Sep 18 16:13:47.924974 ip-10-0-142-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 16:13:47.924987 ip-10-0-142-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 16:13:47.924997 ip-10-0-142-62 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 16:13:47.925202 ip-10-0-142-62 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 16:13:58.043474 ip-10-0-142-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 16:13:58.043495 ip-10-0-142-62 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 187ad3b72dc64e768efd124c4be1cfca --
Sep 18 16:15:28.391741 ip-10-0-142-62 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:15:28.830984 ip-10-0-142-62 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:28.830984 ip-10-0-142-62 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:15:28.830984 ip-10-0-142-62 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:28.830984 ip-10-0-142-62 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:15:28.830984 ip-10-0-142-62 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:28.833027 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.832933    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:15:28.837132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837117    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.837132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837133    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837136    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837139    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837143    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837146    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837149    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837152    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837154    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837157    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837159    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837163    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837167    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837175    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837178    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837180    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837183    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837186    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837189    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837192    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837194    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.837205 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837204    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837209    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837213    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837217    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837219    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837222    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837224    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837227    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837230    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837232    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837236    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837239    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837242    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837245    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837247    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837250    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837252    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837255    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837258    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.837686 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837261    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837263    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837266    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837268    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837272    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837274    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837277    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837279    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837282    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837284    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837286    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837289    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837291    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837294    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837296    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837300    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837303    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837305    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837308    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837311    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.838143 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837313    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837316    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837319    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837322    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837325    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837329    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837332    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837335    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837337    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837340    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837343    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837346    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837348    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837350    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837353    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837355    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837358    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837361    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837363    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.838679 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837366    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837369    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837372    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837375    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837377    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837380    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837383    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837385    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837388    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837391    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837394    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837396    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837399    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837401    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837404    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837407    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837409    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837425    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837428    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.837431    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.839139 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838131    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838137    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838140    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838144    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838148    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838151    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838153    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838156    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838159    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838161    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838165    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838169    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838173    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838176    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838179    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838182    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838185    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838188    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838191    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.839641 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838194    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838197    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838200    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838203    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838205    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838208    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838212    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838216    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838219    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838221    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838224    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838227    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838230    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838232    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838235    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838237    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838240    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838242    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838246    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838248    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.840132 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838251    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838254    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838258    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838261    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838264    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838267    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838270    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838273    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838276    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838278    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838281    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838285    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838287    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838291    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838294    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838298    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838301    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838303    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838306    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.840643 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838308    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838311    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838314    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838316    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838319    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838322    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838324    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838327    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838329    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838332    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838334    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838337    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838340    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838342    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838345    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838347    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838350    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838352    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838355    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838359    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.841106 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838361    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838364    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838367    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838369    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838372    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838375    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838378    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838380    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838383    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838385    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838388    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838390    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838393    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838395    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838398    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838400    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838403    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838405    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838408    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.841625 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838411    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838427    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838801    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838806    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838809    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838811    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838814    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838817    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838819    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838822    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838824    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838827    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838830    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838833    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838835    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838838    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838841    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838844    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838846    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838849    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.842082 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838852    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838854    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838857    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838859    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838862    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838864    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838867    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838869    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838872    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838874    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838877    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838879    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838882    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838884    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838888    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838891    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838894    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838897    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838900    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.842662 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838903    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838906    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838910    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838912    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838915    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838919    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838922    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838925    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838928    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838931    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838935    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838938    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838940    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838943    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838946    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838949    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838951    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838954    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838956    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.843171 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838959    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838961    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838964    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838966    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838969    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838971    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838974    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838976    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838979    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838982    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838984    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838987    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838990    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838992    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838994    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.838997    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839000    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839003    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839006    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839011    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.843657 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839014    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839017    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839019    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839022    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839024    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839027    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839030    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839033    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839036    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839038    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839041    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839043    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839046    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839048    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839052    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839055    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839058    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839060    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839063    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.844135 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839066    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839068    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839071    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839074    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839145    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839179    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839185    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839189    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839194    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839199    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839203    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839208    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839212    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839215    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839219    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839222    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839225    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839229    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839232    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839235    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839237    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839240    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839244    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839247    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:15:28.844609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839250    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839253    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839256    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839260    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839263    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839266    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839270    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839272    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839275    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839278    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839281    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839284    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839289    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839291    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839294    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839297    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839300    2554 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839304    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839308    2554 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839311    2554 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839314    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839317    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839320    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839324    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:15:28.845187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839327    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839330    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839333    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839337    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839339    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839342    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839345    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839348    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839351    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839354    2554 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839358    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839361    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839364    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839367    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839370    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839374    2554 flags.go:64] FLAG: --help="false"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839377    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-142-62.ec2.internal"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839380    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839383    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839385    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839389    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839392    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839395    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:15:28.845807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839397    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839400    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839404    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839408    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839425    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839430    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839434    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839439    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839442    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839446    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839449    2554 flags.go:64] FLAG: --lock-file=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839452    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839455    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839458    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839464    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839467    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839470    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839473    2554 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839475    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839479    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839481    2554 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839484    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839489    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839492    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839496    2554 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:15:28.846391 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839499    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839502    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839505    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839508    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839511    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839514    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839517    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839524    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839528    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839532    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839535    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839538    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839544    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839546    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839550    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839552    2554 flags.go:64] FLAG: --port="10250"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839556    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839559    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0623429bd148ee6b3"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839562    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839565    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839568    2554 flags.go:64] FLAG: --register-node="true"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839571    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839575    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839578    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:15:28.847046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839580    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839583    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839587    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839590    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839593    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839596    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839598    2554 flags.go:64] FLAG: --runonce="false"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839601    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839604    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839607    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839610    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839613    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839616    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839620    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839623    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839629    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839632    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839637    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839641    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839644    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839648    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839651    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839657    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839660    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839663    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:15:28.847677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839676    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839679    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839682    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839685    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839688    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839691    2554 flags.go:64] FLAG: --v="2"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839695    2554 flags.go:64] FLAG: --version="false"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839700    2554 flags.go:64] FLAG: --vmodule=""
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839703    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.839707    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839822    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839825    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839828    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839844    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839847    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839850    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839853    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839856    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839859    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839862    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839865    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839868    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.848294 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839872    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839875    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839881    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839885    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839888    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839891    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839893    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839896    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839899    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839901    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839904    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839907    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839911    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839914    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839916    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839919    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839922    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839925    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839927    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.848970 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839930    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839933    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839935    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839938    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839941    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839943    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839946    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839949    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839951    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839954    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839957    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839959    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839962    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839966    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839969    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839973    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839976    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839979    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839981    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839984    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.849766 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839987    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839989    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839992    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839994    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.839997    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840000    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840003    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840005    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840008    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840012    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840016    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840019    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840021    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840024    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840027    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840029    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840032    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840035    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840037    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840040    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.850533 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840043    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840045    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840048    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840050    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840054    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840058    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840061    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840066    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840068    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840071    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840074    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840077    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840080    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840082    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840085    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840088    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840090    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840093    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840096    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.851074 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840099    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840101    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840104    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840106    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840109    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840111    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840114    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840116    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.840119    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.841134    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.850687    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:15:28.851554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.850707    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.851644    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.851656    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851716    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851721    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851724    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851728    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851731    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851734    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851737    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851740    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851743    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851745    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851750    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851754    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851757    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851760    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851763    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851765    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.851837 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851769    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851772    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851774    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851777    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851779    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851782    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851785    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851788    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851790    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851793    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851796    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851798    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851801    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851804    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851806    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851810    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851813    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851816    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851818    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851821    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.852305 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851824    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851826    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851829    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851832    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851834    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851837    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851840    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851842    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851845    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851848    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851851    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851853    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851856    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851859    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851861    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851864    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851866    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851869    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851872    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851875    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.852810 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851877    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851880    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851883    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851886    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851888    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851891    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851893    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851897    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851899    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851902    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851905    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851907    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851910    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851912    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851915    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851917    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851920    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851922    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851925    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851928    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.853300 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851930    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851933    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851935    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851939    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851942    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851944    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851947    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851949    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851953    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851957    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851960    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851964    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851967    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851971    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851974    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851976    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851979    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851982    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851985    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.853807 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851988    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851990    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851993    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.851996    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.852001    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852132    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852137    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852140    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852143    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852146    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852148    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852151    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852154    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852156    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852160    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:28.854277 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852164    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852167    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852170    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852172    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852175    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852178    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852180    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852183    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852185    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852188    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852191    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852194    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852197    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852200    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852202    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852205    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852208    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852210    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852213    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852216    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:28.854752 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852219    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852222    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852224    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852227    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852230    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852233    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852235    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852238    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852240    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852243    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852246    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852250    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852253    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852256    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852259    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852262    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852265    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852268    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852270    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:28.855241 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852274    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852277    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852281    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852284    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852286    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852289    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852291    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852294    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852297    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852299    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852302    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852305    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852308    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852311    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852314    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852316    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852319    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852321    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852324    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:28.855703 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852327    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852329    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852331    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852334    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852337    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852339    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852342    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852345    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852347    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852350    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852353    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852356    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852358    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852361    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852363    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852366    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852369    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852373    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852375    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852378    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:28.856163 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852380    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852383    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852386    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852389    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852391    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852394    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852397    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852399    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852402    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852404    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:28.852407    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.852428    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.852562    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:15:28.856709 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.856507    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:15:28.857603 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.857591    2554 server.go:1019] "Starting client certificate rotation"
Sep 18 16:15:28.857714 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.857695    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:15:28.857748 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.857729    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:15:28.860510 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.860496    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:15:28.864711 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.864697    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:15:28.873378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.873359    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:15:28.876023 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.876009    2554 log.go:25] "Validated CRI v1 image API"
Sep 18 16:15:28.876918 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.876899    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:15:28.880375 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.880355    2554 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 820a875e-9d8a-4a19-92ff-73738b9da07a:/dev/nvme0n1p4 e52b28f4-c49f-4926-a68b-924e4be0c148:/dev/nvme0n1p3]
Sep 18 16:15:28.880468 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.880373    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:15:28.885880 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.885739    2554 manager.go:216] Machine: {Timestamp:2026-09-18 16:15:28.884387045 +0000 UTC m=+0.364360581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096455 MemoryCapacity:33164464128 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e5342cb10899b4bc03e72aa68f384 SystemUUID:ec2e5342-cb10-899b-4bc0-3e72aa68f384 BootID:187ad3b7-2dc6-4e76-8efd-124c4be1cfca Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582230016 Type:vfs Inodes:4048396 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:36:b4:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:36:b4:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:6f:ee:bb:3f:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164464128 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:15:28.885880 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.885876    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:15:28.885989 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.885967    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:15:28.889570 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.889554    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:15:28.889893 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.889867    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:15:28.890072 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.889897    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:15:28.890773 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.890762    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:15:28.890812 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.890776    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:15:28.890812 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.890784    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:15:28.890812 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.890798    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:15:28.890897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.890877    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:15:28.895703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895686    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:15:28.895846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895836    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:15:28.895878 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895868    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:15:28.895906 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895886    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:15:28.895906 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895901    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:15:28.895956 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895915    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:15:28.896007 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.895997    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:15:28.896712 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.896698    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:15:28.896823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.896722    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:15:28.900674 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.900652    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:15:28.903612 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.903597    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:15:28.903666 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.903629    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:15:28.905623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905611    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905628    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905634    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905640    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905648    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905653    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905667    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905672    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905678    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905684    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905692    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:15:28.905693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.905697    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:15:28.907503 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.907492    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:15:28.907503 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.907503    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:15:28.908788 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.908751    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:15:28.908788 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.908752    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:15:28.911359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.911346    2554 server.go:1268] "Started kubelet"
Sep 18 16:15:28.911459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.911435    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:15:28.912094 ip-10-0-142-62 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:15:28.913061 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.912225    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:15:28.913123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.913084    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:15:28.913450 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.913437    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:15:28.914592 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.914579    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:15:28.920098 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.920082    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:15:28.921534 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.921516    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:15:28.922178 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.922165    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:15:28.922232 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.922178    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:15:28.922383 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.922342    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:28.922580 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.921354    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d52de241  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.911319617 +0000 UTC m=+0.391293154,LastTimestamp:2026-09-18 16:15:28.911319617 +0000 UTC m=+0.391293154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.922887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.922863    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:15:28.922887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.922868    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:15:28.923049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.922896    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:15:28.923049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923015    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:15:28.923049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923028    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:15:28.923179 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923082    2554 factory.go:55] Registering systemd factory
Sep 18 16:15:28.923179 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923096    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:15:28.923381 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923354    2554 factory.go:153] Registering CRI-O factory
Sep 18 16:15:28.923381 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923375    2554 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:15:28.923538 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923496    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:15:28.923538 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923516    2554 factory.go:103] Registering Raw factory
Sep 18 16:15:28.923538 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923534    2554 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:15:28.923939 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.923928    2554 manager.go:349] Starting recovery of all containers
Sep 18 16:15:28.925433 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.925396    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:15:28.925548 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.925531    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:15:28.935480 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.935462    2554 manager.go:354] Recovery completed
Sep 18 16:15:28.940338 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.940299    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:28.942237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942221    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:28.942299 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942249    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:28.942299 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942260    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:28.942829 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942814    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:15:28.942829 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942826    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:15:28.942958 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942843    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:28.942958 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.942845    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:15:28.943549 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.943533    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:28.943626 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.943559    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:28.943626 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.943569    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:28.944658 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.944587    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d705a517  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-62.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.942236951 +0000 UTC m=+0.422210487,LastTimestamp:2026-09-18 16:15:28.942236951 +0000 UTC m=+0.422210487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.945053 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.945043    2554 policy_none.go:49] "None policy: Start"
Sep 18 16:15:28.945101 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.945057    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:15:28.945101 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.945067    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:15:28.947019 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.947006    2554 policy_none.go:47] "Start"
Sep 18 16:15:28.954866 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.954797    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d705e7aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-62.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.942253994 +0000 UTC m=+0.422227531,LastTimestamp:2026-09-18 16:15:28.942253994 +0000 UTC m=+0.422227531,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.960029 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.959959    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d7060fbd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-62.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.942264253 +0000 UTC m=+0.422237790,LastTimestamp:2026-09-18 16:15:28.942264253 +0000 UTC m=+0.422237790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.963439 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.963358    2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-62.ec2.internal.18d67670d705a517\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d705a517  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-62.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.942236951 +0000 UTC m=+0.422210487,LastTimestamp:2026-09-18 16:15:28.943547489 +0000 UTC m=+0.423521026,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.966973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.966951    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pr4c5"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.972506    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pr4c5"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.971665    2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-62.ec2.internal.18d67670d705e7aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-62.ec2.internal.18d67670d705e7aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-62.ec2.internal,UID:ip-10-0-142-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-62.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-62.ec2.internal,},FirstTimestamp:2026-09-18 16:15:28.942253994 +0000 UTC m=+0.422227531,LastTimestamp:2026-09-18 16:15:28.943564224 +0000 UTC m=+0.423537761,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-62.ec2.internal,}"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973352    2554 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.973384    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973397    2554 server.go:87] "Starting device plugin registration server"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973666    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973677    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973761    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973866    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.973874    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.974329    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.974377    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.988741    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.989961    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.990010    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.990033    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:28.990076    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:15:28.998579 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:28.992597    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:29.074841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.074795    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:29.075799 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.075779    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:29.075906 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.075812    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:29.075906 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.075827    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:29.075906 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.075854    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.082226 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.082181    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.082226 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.082203    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-62.ec2.internal\": node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.090894 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.090870    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"]
Sep 18 16:15:29.090953 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.090943    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:29.091742 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.091725    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:29.091811 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.091753    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:29.091811 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.091763    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:29.094113 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094101    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:29.094239 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094224    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.094300 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094262    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:29.094872 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094853    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:29.094955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094880    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:29.094955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094894    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:29.094955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094905    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:29.094955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094925    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:29.095073 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.094956    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:29.097273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.097259    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.097348 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.097284    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:29.097971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.097956    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:29.097971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.097977    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:29.098100 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.097990    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:29.107368 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.107351    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.118089 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.118058    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-62.ec2.internal\" not found" node="ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.121716 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.121700    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-62.ec2.internal\" not found" node="ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.207787 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.207759    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.224199 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.224167    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.224336 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.224202    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.224336 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.224236    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5410219c58050432591751550b16d506-config\") pod \"kube-apiserver-proxy-ip-10-0-142-62.ec2.internal\" (UID: \"5410219c58050432591751550b16d506\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.308330 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.308284    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.324619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324595    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.324708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324623    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5410219c58050432591751550b16d506-config\") pod \"kube-apiserver-proxy-ip-10-0-142-62.ec2.internal\" (UID: \"5410219c58050432591751550b16d506\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.324708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.324708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.324708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324681    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8461611a1118a44b99c586656907a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal\" (UID: \"8461611a1118a44b99c586656907a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.324708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.324703    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5410219c58050432591751550b16d506-config\") pod \"kube-apiserver-proxy-ip-10-0-142-62.ec2.internal\" (UID: \"5410219c58050432591751550b16d506\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.409107 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.409019    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.423193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.423169    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.427514 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.427497    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:29.509934 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.509735    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.610259 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.610218    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.710680 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.710640    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.810977 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.810934    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.857222 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.857189    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:15:29.857717 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.857383    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:15:29.911490 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:29.911462    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:29.979612 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.979573    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:10:28 +0000 UTC" deadline="2028-04-23 15:35:06.167967991 +0000 UTC"
Sep 18 16:15:29.979612 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:29.979609    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13991h19m36.188366341s"
Sep 18 16:15:30.011772 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.011743    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:30.035124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.035093    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:30.036332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.036311    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:30.036462 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.036344    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:30.036462 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.036357    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:30.112012 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.111978    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:30.114998 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:30.114970    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8461611a1118a44b99c586656907a0d0.slice/crio-f0f351474fbdbeac2f2a39114fe53dafa0f0f703cd3853519cf94459b3598ab2 WatchSource:0}: Error finding container f0f351474fbdbeac2f2a39114fe53dafa0f0f703cd3853519cf94459b3598ab2: Status 404 returned error can't find the container with id f0f351474fbdbeac2f2a39114fe53dafa0f0f703cd3853519cf94459b3598ab2
Sep 18 16:15:30.115440 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:30.115405    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5410219c58050432591751550b16d506.slice/crio-e6c62afbff56ac86acc327fb02ec07703e412d60c2976a02e3bc0b97772024fd WatchSource:0}: Error finding container e6c62afbff56ac86acc327fb02ec07703e412d60c2976a02e3bc0b97772024fd: Status 404 returned error can't find the container with id e6c62afbff56ac86acc327fb02ec07703e412d60c2976a02e3bc0b97772024fd
Sep 18 16:15:30.120935 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.120923    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:15:30.142635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.142611    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:30.159775 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.159748    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:30.212704 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.212657    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:30.313090 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.313010    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:30.413530 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.413495    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-62.ec2.internal\" not found"
Sep 18 16:15:30.465739 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.465713    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:30.522494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.522463    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:30.533032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.533012    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:15:30.534117 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.534105    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal"
Sep 18 16:15:30.544750 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.544725    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:15:30.897231 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.897199    2554 apiserver.go:52] "Watching apiserver"
Sep 18 16:15:30.907239 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.907009    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:15:30.907531 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.907468    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2684q","openshift-ovn-kubernetes/ovnkube-node-754q4","kube-system/konnectivity-agent-l2ngq","kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j","openshift-image-registry/node-ca-wrts2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-vl7gs","kube-system/global-pull-secret-syncer-vn5kq","openshift-cluster-node-tuning-operator/tuned-tt7q5","openshift-dns/node-resolver-kjtm7","openshift-multus/multus-ppstz","openshift-multus/network-metrics-daemon-fxk2s","openshift-network-diagnostics/network-check-target-gpwss"]
Sep 18 16:15:30.910747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.910452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:30.910747 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.910536    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:30.913681 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.912886    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.915935 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.915467    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:30.916697 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.916580    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdccn\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917029    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917135    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917230    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917291    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917393    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:15:30.917481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.917471    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:15:30.918337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.918035    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:15:30.918337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.918219    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:15:30.918337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.918252    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hf8js\""
Sep 18 16:15:30.920568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.920062    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:30.922450 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.922185    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.922681 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.922660    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.922760 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.922681    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:15:30.922974 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.922957    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.923339 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.923121    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:15:30.923339 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.923160    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zx62l\""
Sep 18 16:15:30.924974 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.924955    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:30.925916 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.925898    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.926134 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.926108    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:15:30.926354 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.926337    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.926554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.926537    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgjg8\""
Sep 18 16:15:30.927307 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.927005    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:30.927307 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.927078    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:30.927785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.927618    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hth4q\""
Sep 18 16:15:30.927785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.927638    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:15:30.927785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.927671    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.927785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.927770    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.929467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.929453    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:30.932315 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.931928    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:30.932315 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:30.931996    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:30.932536 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.932371    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sz7kv\""
Sep 18 16:15:30.933168 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.932620    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.933819    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.933987    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.934011    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.934204    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.934736    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:30.935010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.934815    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-netns\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.934866    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.937785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovn-node-metrics-cert\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.937834    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-script-lib\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.937888    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc241aa0-d286-4995-8661-ea3044ecb836-iptables-alerter-script\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-host\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938171    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-kubelet\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-slash\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938439    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-bin\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvj6\" (UniqueName: \"kubernetes.io/projected/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-kube-api-access-5lvj6\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938647    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc241aa0-d286-4995-8661-ea3044ecb836-host-slash\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938684    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938715    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-device-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-ovn\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.941190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-node-log\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93619efd-a838-4e61-9443-e9e48874784b-agent-certs\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.938980    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-serviceca\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939050    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-systemd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-netd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949tl\" (UniqueName: \"kubernetes.io/projected/bc241aa0-d286-4995-8661-ea3044ecb836-kube-api-access-949tl\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-systemd-units\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939208    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-config\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939294    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmv5\" (UniqueName: \"kubernetes.io/projected/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-kube-api-access-5mmv5\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-etc-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939503    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-env-overrides\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939539    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrp7\" (UniqueName: \"kubernetes.io/projected/023a0f44-e934-492c-a9fd-c88a77e45000-kube-api-access-vbrp7\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-var-lib-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942104 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939796    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-log-socket\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939830    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93619efd-a838-4e61-9443-e9e48874784b-konnectivity-ca\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mtc\" (UniqueName: \"kubernetes.io/projected/4d86a9c9-26fc-461b-b262-879b82d20ae1-kube-api-access-s2mtc\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.939983    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.940041    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.940619    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.940632    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.940674    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b6sfz\""
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.940987    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.942887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.941540    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:30.944735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.944096    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:15:30.944735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.944587    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npkl6\""
Sep 18 16:15:30.944735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.944733    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppstz"
Sep 18 16:15:30.945963 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.945807    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:15:30.947297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.946899    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98cgp\""
Sep 18 16:15:30.947297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.947144    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:15:30.974733 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.974696    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pg7fm"
Sep 18 16:15:30.983844 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.983483    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pg7fm"
Sep 18 16:15:30.993132 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.992819    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:30.997897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:30.997836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal" event={"ID":"5410219c58050432591751550b16d506","Type":"ContainerStarted","Data":"e6c62afbff56ac86acc327fb02ec07703e412d60c2976a02e3bc0b97772024fd"}
Sep 18 16:15:31.000469 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.000442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal" event={"ID":"8461611a1118a44b99c586656907a0d0","Type":"ContainerStarted","Data":"f0f351474fbdbeac2f2a39114fe53dafa0f0f703cd3853519cf94459b3598ab2"}
Sep 18 16:15:31.024194 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.024132    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:15:31.040614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-kubernetes\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.040724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040625    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-system-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.040724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-systemd-units\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.040724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-systemd-units\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-etc-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-kubelet-config\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-etc-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.040917 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-var-lib-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040949    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-log-socket\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.040978    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.040992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-log-socket\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-var-lib-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.041041    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:31.54101981 +0000 UTC m=+3.020993353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041061    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041089    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-var-lib-kubelet\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7djz\" (UniqueName: \"kubernetes.io/projected/c375dd5a-62d2-4986-a638-853da1a48559-kube-api-access-s7djz\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041163    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-run\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041206    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-etc-tuned\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-sys-fs\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.041273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041245    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-cni-binary-copy\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-bin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041308    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-multus\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041332    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-kubelet\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041354    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8w6f\" (UniqueName: \"kubernetes.io/projected/cee9d688-9df4-4669-a39d-cde460288b6b-kube-api-access-r8w6f\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041378    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-netns\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041430    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-netns\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041440    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovn-node-metrics-cert\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-host\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041478    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041487    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041543    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-host\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041545    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2w5r\" (UniqueName: \"kubernetes.io/projected/364f7d51-7a38-43f1-a5b3-73756da9c04c-kube-api-access-r2w5r\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-serviceca\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-bin\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.041882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041716    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-bin\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041756    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvj6\" (UniqueName: \"kubernetes.io/projected/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-kube-api-access-5lvj6\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041783    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041805    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041865    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-device-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-socket-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.041809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-device-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-socket-dir-parent\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042059    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83c7a85-6566-44e2-8725-dd1c6429033d-tmp-dir\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042087    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-ovn\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-node-log\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-serviceca\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042165    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-ovn\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042171    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-node-log\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-systemd\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042212    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-multus-certs\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.042637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042264    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxq4d\" (UniqueName: \"kubernetes.io/projected/d83c7a85-6566-44e2-8725-dd1c6429033d-kube-api-access-pxq4d\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-systemd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042348    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-netd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949tl\" (UniqueName: \"kubernetes.io/projected/bc241aa0-d286-4995-8661-ea3044ecb836-kube-api-access-949tl\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042440    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-systemd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-cnibin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042482    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-etc-kubernetes\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-cni-netd\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042579    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83c7a85-6566-44e2-8725-dd1c6429033d-hosts-file\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042604    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-config\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042648    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmv5\" (UniqueName: \"kubernetes.io/projected/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-kube-api-access-5mmv5\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-run-openvswitch\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-sys\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-tmp\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-netns\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042758    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:31.043291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042779    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-system-cni-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042814    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-env-overrides\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042847    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042877    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-multus-daemon-config\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042949    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-cnibin\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.042985    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrp7\" (UniqueName: \"kubernetes.io/projected/023a0f44-e934-492c-a9fd-c88a77e45000-kube-api-access-vbrp7\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93619efd-a838-4e61-9443-e9e48874784b-konnectivity-ca\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043051    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mtc\" (UniqueName: \"kubernetes.io/projected/4d86a9c9-26fc-461b-b262-879b82d20ae1-kube-api-access-s2mtc\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043076    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043100    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-dbus\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043122    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-conf\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-k8s-cni-cncf-io\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043152    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-config\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043159    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-conf-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043252    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.044084 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-modprobe-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-host\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043346    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-env-overrides\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043373    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-script-lib\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc241aa0-d286-4995-8661-ea3044ecb836-iptables-alerter-script\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043527    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043557    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-kubelet\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-slash\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043628    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-kubelet\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043628    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93619efd-a838-4e61-9443-e9e48874784b-konnectivity-ca\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc241aa0-d286-4995-8661-ea3044ecb836-host-slash\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043682    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043685    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc241aa0-d286-4995-8661-ea3044ecb836-host-slash\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d86a9c9-26fc-461b-b262-879b82d20ae1-registration-dir\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-hostroot\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043714    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-host-slash\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.044830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93619efd-a838-4e61-9443-e9e48874784b-agent-certs\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:31.045608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysconfig\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.045608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-lib-modules\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.045608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043941    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc241aa0-d286-4995-8661-ea3044ecb836-iptables-alerter-script\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.045608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.043959    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovnkube-script-lib\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.046454 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.046368    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-ovn-node-metrics-cert\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.046534 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.046509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93619efd-a838-4e61-9443-e9e48874784b-agent-certs\") pod \"konnectivity-agent-l2ngq\" (UID: \"93619efd-a838-4e61-9443-e9e48874784b\") " pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:31.051442 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.051409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvj6\" (UniqueName: \"kubernetes.io/projected/a5b4cb8d-c838-48fd-8531-708ca2a3ab4b-kube-api-access-5lvj6\") pod \"ovnkube-node-754q4\" (UID: \"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.054087 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.053941    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mtc\" (UniqueName: \"kubernetes.io/projected/4d86a9c9-26fc-461b-b262-879b82d20ae1-kube-api-access-s2mtc\") pod \"aws-ebs-csi-driver-node-mqp6j\" (UID: \"4d86a9c9-26fc-461b-b262-879b82d20ae1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.054482 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.054462    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-949tl\" (UniqueName: \"kubernetes.io/projected/bc241aa0-d286-4995-8661-ea3044ecb836-kube-api-access-949tl\") pod \"iptables-alerter-2684q\" (UID: \"bc241aa0-d286-4995-8661-ea3044ecb836\") " pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.054555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.054522    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmv5\" (UniqueName: \"kubernetes.io/projected/dd1bdd77-ade8-4626-b01e-2b3c6fe51d98-kube-api-access-5mmv5\") pod \"node-ca-wrts2\" (UID: \"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98\") " pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.055359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.055337    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrp7\" (UniqueName: \"kubernetes.io/projected/023a0f44-e934-492c-a9fd-c88a77e45000-kube-api-access-vbrp7\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:31.144130 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-socket-dir-parent\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144130 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83c7a85-6566-44e2-8725-dd1c6429033d-tmp-dir\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144145    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-systemd\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144166    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-multus-certs\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144182    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144199    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxq4d\" (UniqueName: \"kubernetes.io/projected/d83c7a85-6566-44e2-8725-dd1c6429033d-kube-api-access-pxq4d\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-cnibin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144242    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-etc-kubernetes\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144266    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83c7a85-6566-44e2-8725-dd1c6429033d-hosts-file\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144287    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-sys\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144308    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-tmp\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-socket-dir-parent\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-netns\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:31.144362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144367    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-system-cni-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-multus-daemon-config\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144410    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-cnibin\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-dbus\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-conf\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144509    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-k8s-cni-cncf-io\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144538    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-conf-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144551    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144569    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-modprobe-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-host\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144603    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-hostroot\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144627    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysconfig\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144652    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-lib-modules\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-kubernetes\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-system-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-kubelet-config\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.144945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144764    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144781    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-var-lib-kubelet\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7djz\" (UniqueName: \"kubernetes.io/projected/c375dd5a-62d2-4986-a638-853da1a48559-kube-api-access-s7djz\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-run\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-etc-tuned\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144868    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83c7a85-6566-44e2-8725-dd1c6429033d-tmp-dir\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-cni-binary-copy\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144946    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-bin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144972    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-multus\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.144996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-kubelet\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145020    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8w6f\" (UniqueName: \"kubernetes.io/projected/cee9d688-9df4-4669-a39d-cde460288b6b-kube-api-access-r8w6f\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145090    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145124    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2w5r\" (UniqueName: \"kubernetes.io/projected/364f7d51-7a38-43f1-a5b3-73756da9c04c-kube-api-access-r2w5r\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145310    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-cni-binary-copy\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-cnibin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-etc-kubernetes\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-systemd\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.145677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145462    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83c7a85-6566-44e2-8725-dd1c6429033d-hosts-file\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145474    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-multus-certs\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145494    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-sys\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145871    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-kubelet\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145926    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-bin\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145962    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-var-lib-cni-multus\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.145987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146097    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-host\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146384    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-netns\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146391    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146627    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-system-cni-dir\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146810    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-host-run-k8s-cni-cncf-io\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/364f7d51-7a38-43f1-a5b3-73756da9c04c-cnibin\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee9d688-9df4-4669-a39d-cde460288b6b-multus-daemon-config\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146902    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-dbus\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.146952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.146920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-multus-conf-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.147298 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-conf\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147298 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147128    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-modprobe-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147298 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.147204    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:31.147298 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.147269    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:31.647252761 +0000 UTC m=+3.127226299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147305    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/364f7d51-7a38-43f1-a5b3-73756da9c04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147311    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-hostroot\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysconfig\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee9d688-9df4-4669-a39d-cde460288b6b-system-cni-dir\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147471    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-kubernetes\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147481    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-lib-modules\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147500    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-var-lib-kubelet\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147840 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147531    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-run\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.147840 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147558    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25c3b578-0fb3-4d56-bb23-da9ba897582a-kubelet-config\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.147840 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.147652    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c375dd5a-62d2-4986-a638-853da1a48559-etc-sysctl-d\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.148048 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.148006    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-tmp\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.149686 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.149646    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c375dd5a-62d2-4986-a638-853da1a48559-etc-tuned\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.151221 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.151184    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:31.151221 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.151208    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:31.151221 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.151222    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:31.151461 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.151291    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:31.651274971 +0000 UTC m=+3.131248514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:31.152516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.152480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxq4d\" (UniqueName: \"kubernetes.io/projected/d83c7a85-6566-44e2-8725-dd1c6429033d-kube-api-access-pxq4d\") pod \"node-resolver-kjtm7\" (UID: \"d83c7a85-6566-44e2-8725-dd1c6429033d\") " pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.154075 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.154048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2w5r\" (UniqueName: \"kubernetes.io/projected/364f7d51-7a38-43f1-a5b3-73756da9c04c-kube-api-access-r2w5r\") pod \"multus-additional-cni-plugins-vl7gs\" (UID: \"364f7d51-7a38-43f1-a5b3-73756da9c04c\") " pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.154718 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.154270    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8w6f\" (UniqueName: \"kubernetes.io/projected/cee9d688-9df4-4669-a39d-cde460288b6b-kube-api-access-r8w6f\") pod \"multus-ppstz\" (UID: \"cee9d688-9df4-4669-a39d-cde460288b6b\") " pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.158566 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.158539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7djz\" (UniqueName: \"kubernetes.io/projected/c375dd5a-62d2-4986-a638-853da1a48559-kube-api-access-s7djz\") pod \"tuned-tt7q5\" (UID: \"c375dd5a-62d2-4986-a638-853da1a48559\") " pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.227778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.227750    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:31.244535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.244508    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:31.255304 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.255280    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2684q"
Sep 18 16:15:31.262116 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.262091    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppstz"
Sep 18 16:15:31.269959 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.269936    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j"
Sep 18 16:15:31.278597 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.278576    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrts2"
Sep 18 16:15:31.287256 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.287237    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vl7gs"
Sep 18 16:15:31.294918 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.294897    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tt7q5"
Sep 18 16:15:31.300506 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.300484    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjtm7"
Sep 18 16:15:31.547886 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.547842    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:31.548083 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.548009    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:31.548149 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.548106    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:32.548085126 +0000 UTC m=+4.028058659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:31.648907 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.648872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:31.649064 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.649033    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:31.649124 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.649099    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:32.64907721 +0000 UTC m=+4.129050733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:31.750236 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.750198    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:31.750396 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.750340    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:31.750396 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.750359    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:31.750396 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.750372    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:31.750582 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.750456    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:32.750438519 +0000 UTC m=+4.230412072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:31.986861 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.986821    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:30 +0000 UTC" deadline="2028-04-25 06:01:54.198011872 +0000 UTC"
Sep 18 16:15:31.986861 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.986853    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14029h46m22.211161645s"
Sep 18 16:15:31.991154 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:31.991121    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:31.991298 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:31.991264    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:32.191824 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.191377    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364f7d51_7a38_43f1_a5b3_73756da9c04c.slice/crio-fc6a7848ed1d97ab506dd21219664a81e968de149a051090658e0865de67035f WatchSource:0}: Error finding container fc6a7848ed1d97ab506dd21219664a81e968de149a051090658e0865de67035f: Status 404 returned error can't find the container with id fc6a7848ed1d97ab506dd21219664a81e968de149a051090658e0865de67035f
Sep 18 16:15:32.193102 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.193076    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc375dd5a_62d2_4986_a638_853da1a48559.slice/crio-bdbc16ccfe6b110000a901fb60cd66d58423d4bfceac91f134992852e3a79d46 WatchSource:0}: Error finding container bdbc16ccfe6b110000a901fb60cd66d58423d4bfceac91f134992852e3a79d46: Status 404 returned error can't find the container with id bdbc16ccfe6b110000a901fb60cd66d58423d4bfceac91f134992852e3a79d46
Sep 18 16:15:32.194372 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.194255    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b4cb8d_c838_48fd_8531_708ca2a3ab4b.slice/crio-9c8ae2763d8b28e313d646625966b683b88ba313cdf7ea25bbbb3d94daace32d WatchSource:0}: Error finding container 9c8ae2763d8b28e313d646625966b683b88ba313cdf7ea25bbbb3d94daace32d: Status 404 returned error can't find the container with id 9c8ae2763d8b28e313d646625966b683b88ba313cdf7ea25bbbb3d94daace32d
Sep 18 16:15:32.196902 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.196877    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc241aa0_d286_4995_8661_ea3044ecb836.slice/crio-0ae743a673a6b055e106edeb9ca7a399deecda77d0332866158684e34744dd77 WatchSource:0}: Error finding container 0ae743a673a6b055e106edeb9ca7a399deecda77d0332866158684e34744dd77: Status 404 returned error can't find the container with id 0ae743a673a6b055e106edeb9ca7a399deecda77d0332866158684e34744dd77
Sep 18 16:15:32.198842 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.198431    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d86a9c9_26fc_461b_b262_879b82d20ae1.slice/crio-45344661abd5dd6957390f96d91a7c3aac1a4a499a815744aeb15bb6ae2c16ed WatchSource:0}: Error finding container 45344661abd5dd6957390f96d91a7c3aac1a4a499a815744aeb15bb6ae2c16ed: Status 404 returned error can't find the container with id 45344661abd5dd6957390f96d91a7c3aac1a4a499a815744aeb15bb6ae2c16ed
Sep 18 16:15:32.199715 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.199692    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee9d688_9df4_4669_a39d_cde460288b6b.slice/crio-62a862413862d355d97c21af706fc5bf1cb287805b9dba6ab059cc0cc5a55553 WatchSource:0}: Error finding container 62a862413862d355d97c21af706fc5bf1cb287805b9dba6ab059cc0cc5a55553: Status 404 returned error can't find the container with id 62a862413862d355d97c21af706fc5bf1cb287805b9dba6ab059cc0cc5a55553
Sep 18 16:15:32.200881 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.200561    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83c7a85_6566_44e2_8725_dd1c6429033d.slice/crio-5840784c9deada85b1edf05d92110035a5406fec878d5069299db28bc163a21e WatchSource:0}: Error finding container 5840784c9deada85b1edf05d92110035a5406fec878d5069299db28bc163a21e: Status 404 returned error can't find the container with id 5840784c9deada85b1edf05d92110035a5406fec878d5069299db28bc163a21e
Sep 18 16:15:32.202444 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:15:32.202314    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1bdd77_ade8_4626_b01e_2b3c6fe51d98.slice/crio-9b79bd615b8acac297e352851d7580ad173a052682b8a1d7edead17244319521 WatchSource:0}: Error finding container 9b79bd615b8acac297e352851d7580ad173a052682b8a1d7edead17244319521: Status 404 returned error can't find the container with id 9b79bd615b8acac297e352851d7580ad173a052682b8a1d7edead17244319521
Sep 18 16:15:32.556792 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.556701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:32.556935 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.556848    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:32.556935 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.556909    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:34.556894971 +0000 UTC m=+6.036868507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:32.657027 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.656980    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:32.657229 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.657134    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:32.657229 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.657204    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:34.657187732 +0000 UTC m=+6.137161269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:32.757648 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.757608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:32.757834 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.757777    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:32.757834 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.757796    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:32.757834 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.757808    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:32.757993 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.757867    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:34.757848267 +0000 UTC m=+6.237821810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:32.988030 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.987961    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:30 +0000 UTC" deadline="2028-04-30 15:19:31.544917057 +0000 UTC"
Sep 18 16:15:32.988030 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.988031    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14159h3m58.55689074s"
Sep 18 16:15:32.990549 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.990522    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:32.990683 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.990654    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:32.991690 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:32.991671    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:32.991789 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:32.991766    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:33.026965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.026475    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal" event={"ID":"5410219c58050432591751550b16d506","Type":"ContainerStarted","Data":"b046052d683a6e3c83c81c4dcc3c3accbe3aae32969d71b085ba7fd9854fcfec"}
Sep 18 16:15:33.038727 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.038476    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjtm7" event={"ID":"d83c7a85-6566-44e2-8725-dd1c6429033d","Type":"ContainerStarted","Data":"5840784c9deada85b1edf05d92110035a5406fec878d5069299db28bc163a21e"}
Sep 18 16:15:33.049978 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.049938    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j" event={"ID":"4d86a9c9-26fc-461b-b262-879b82d20ae1","Type":"ContainerStarted","Data":"45344661abd5dd6957390f96d91a7c3aac1a4a499a815744aeb15bb6ae2c16ed"}
Sep 18 16:15:33.057868 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.057830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"9c8ae2763d8b28e313d646625966b683b88ba313cdf7ea25bbbb3d94daace32d"}
Sep 18 16:15:33.060854 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.060825    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2684q" event={"ID":"bc241aa0-d286-4995-8661-ea3044ecb836","Type":"ContainerStarted","Data":"0ae743a673a6b055e106edeb9ca7a399deecda77d0332866158684e34744dd77"}
Sep 18 16:15:33.065591 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.065558    2554 generic.go:356] "Generic (PLEG): container finished" podID="8461611a1118a44b99c586656907a0d0" containerID="f5b86456e8fe08d7c5198756537f791110a7905b17f7887893add2a0c46e9bbf" exitCode=0
Sep 18 16:15:33.065728 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.065646    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal" event={"ID":"8461611a1118a44b99c586656907a0d0","Type":"ContainerDied","Data":"f5b86456e8fe08d7c5198756537f791110a7905b17f7887893add2a0c46e9bbf"}
Sep 18 16:15:33.083399 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.082979    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-62.ec2.internal" podStartSLOduration=3.082959127 podStartE2EDuration="3.082959127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:33.043964611 +0000 UTC m=+4.523938158" watchObservedRunningTime="2026-09-18 16:15:33.082959127 +0000 UTC m=+4.562932674"
Sep 18 16:15:33.086330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.086286    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2ngq" event={"ID":"93619efd-a838-4e61-9443-e9e48874784b","Type":"ContainerStarted","Data":"3109c746fdcec63113841e80c49387c27667f185430fa186c64e3af49618e3fd"}
Sep 18 16:15:33.094896 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.094854    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrts2" event={"ID":"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98","Type":"ContainerStarted","Data":"9b79bd615b8acac297e352851d7580ad173a052682b8a1d7edead17244319521"}
Sep 18 16:15:33.105430 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.105386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppstz" event={"ID":"cee9d688-9df4-4669-a39d-cde460288b6b","Type":"ContainerStarted","Data":"62a862413862d355d97c21af706fc5bf1cb287805b9dba6ab059cc0cc5a55553"}
Sep 18 16:15:33.109348 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.109319    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tt7q5" event={"ID":"c375dd5a-62d2-4986-a638-853da1a48559","Type":"ContainerStarted","Data":"bdbc16ccfe6b110000a901fb60cd66d58423d4bfceac91f134992852e3a79d46"}
Sep 18 16:15:33.124362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.121243    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerStarted","Data":"fc6a7848ed1d97ab506dd21219664a81e968de149a051090658e0865de67035f"}
Sep 18 16:15:33.991287 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:33.990810    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:33.991287 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:33.990945    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:34.155898 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.155808    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal" event={"ID":"8461611a1118a44b99c586656907a0d0","Type":"ContainerStarted","Data":"7abc07ee7efdcf287ff149bf1cb515da25e3e10a09eef72677e3f63c202732af"}
Sep 18 16:15:34.573663 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.573621    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:34.573972 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.573801    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:34.573972 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.573861    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.573841138 +0000 UTC m=+10.053814679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:34.675014 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.674977    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:34.675188 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.675172    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:34.675251 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.675240    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.67522072 +0000 UTC m=+10.155194282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:34.776640 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.775975    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:34.776640 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.776193    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:34.776640 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.776213    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:34.776640 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.776225    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:34.776640 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.776284    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.776265408 +0000 UTC m=+10.256238946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:34.993124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.992475    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:34.993124 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.992602    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:34.993124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:34.992918    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:34.993124 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:34.993013    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:35.990690 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:35.990652    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:35.990867 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:35.990793    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:36.990881 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:36.990846    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:36.991313 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:36.990973    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:36.993339 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:36.990805    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:36.993339 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:36.993241    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:37.991026 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:37.990987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:37.991479 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:37.991136    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:38.609974 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:38.609937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:38.610142 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.610091    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:38.610218 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.610171    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:46.610150653 +0000 UTC m=+18.090124179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:38.711014 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:38.710974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:38.711196 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.711102    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:38.711196 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.711169    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:46.711150554 +0000 UTC m=+18.191124101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:38.811972 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:38.811934    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:38.812152 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.812129    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:38.812202 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.812154    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:38.812202 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.812168    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:38.812271 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.812232    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:46.812213812 +0000 UTC m=+18.292187339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:38.992234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:38.992198    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:38.992707 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.992321    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:38.992773 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:38.992707    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:38.992830 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:38.992799    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:39.990517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:39.990213    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:39.990517 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:39.990357    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:40.990887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:40.990852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:40.991332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:40.990852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:40.991332 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:40.990972    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:40.991332 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:40.991053    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:41.990424 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:41.990380    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:41.990584 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:41.990509    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:42.990234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:42.990196    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:42.990234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:42.990242    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:42.990755 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:42.990339    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:42.990755 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:42.990448    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:43.990581 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:43.990551    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:43.991028 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:43.990697    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:44.990362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:44.990323    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:44.990362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:44.990347    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:44.990612 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:44.990455    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:44.990612 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:44.990562    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:45.990530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:45.990496    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:45.990731 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:45.990631    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:46.670197 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:46.670165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:46.670435 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.670334    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:46.670435 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.670427    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:02.670391254 +0000 UTC m=+34.150364800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:46.771324 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:46.771280    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:46.771530 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.771477    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:46.771599 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.771558    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:02.771535075 +0000 UTC m=+34.251508624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:46.872307 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:46.872265    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:46.872485 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.872456    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:46.872485 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.872482    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:46.872582 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.872492    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:46.872582 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.872546    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:02.872530931 +0000 UTC m=+34.352504477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:46.990594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:46.990560    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:46.990778 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.990693    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:46.990778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:46.990731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:46.991117 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:46.990802    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:47.991234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:47.991195    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:47.991736 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:47.991349    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:48.991001 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:48.990966    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:48.991161 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:48.991060    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:48.991161 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:48.991144    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:48.991260 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:48.991239    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:49.991077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:49.991041    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:49.991213 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:49.991155    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:50.990616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:50.990431    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:50.991434 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:50.990439    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:50.991434 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:50.990727    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:50.991434 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:50.990879    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:51.198476 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.198442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2ngq" event={"ID":"93619efd-a838-4e61-9443-e9e48874784b","Type":"ContainerStarted","Data":"11e22c6dc1c39ef2809c7591ec53a2e1c37e834040074dcff5b1e994c16147f2"}
Sep 18 16:15:51.199858 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.199829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrts2" event={"ID":"dd1bdd77-ade8-4626-b01e-2b3c6fe51d98","Type":"ContainerStarted","Data":"5be99d62fd9db6d70a4d6f7ee1c5caf32d8d2f8db0609eab94b280121fa44239"}
Sep 18 16:15:51.201195 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.201174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppstz" event={"ID":"cee9d688-9df4-4669-a39d-cde460288b6b","Type":"ContainerStarted","Data":"f4d4e2a9e175f9b90741192b332120ece1d142251029770c8d719e8e8919d055"}
Sep 18 16:15:51.203960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.203932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tt7q5" event={"ID":"c375dd5a-62d2-4986-a638-853da1a48559","Type":"ContainerStarted","Data":"75550f56ae4b517d4ed928fafa1a1f500ae8e9cf0ff7c14d7f02f06b210b6cdd"}
Sep 18 16:15:51.205400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.205379    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="c4782d4630e897c94369c870362134e0d16c0fc0acd95373201c1e0d49a33ee5" exitCode=0
Sep 18 16:15:51.205512 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.205444    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"c4782d4630e897c94369c870362134e0d16c0fc0acd95373201c1e0d49a33ee5"}
Sep 18 16:15:51.206873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.206840    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjtm7" event={"ID":"d83c7a85-6566-44e2-8725-dd1c6429033d","Type":"ContainerStarted","Data":"f02f6e7ae179064e89efad6263ffed484a8bfc804cbbd9d7686abd0da5d21f49"}
Sep 18 16:15:51.208237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.208217    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j" event={"ID":"4d86a9c9-26fc-461b-b262-879b82d20ae1","Type":"ContainerStarted","Data":"a2f63f14ca38e2e94ec8b920437f89e40e36bdf3320769028350262df4150f5b"}
Sep 18 16:15:51.210982 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.210946    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:15:51.211274 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211254    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5b4cb8d-c838-48fd-8531-708ca2a3ab4b" containerID="458ff610f57386f0a435a0e423cdfea7844f169091fc6b5cd070908be2de3460" exitCode=1
Sep 18 16:15:51.211354 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"09e0e65d507332ed4256764e3b889253537ed2626744b49eb50c623df934dfb2"}
Sep 18 16:15:51.211433 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211352    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"42a2959bc371d6b1d4f457d02f54f05301db73dcbeeb273d0053d1d8192e88b7"}
Sep 18 16:15:51.211433 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211401    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"3cc3b9b5f51c2d68942a6498fa57310081f5d0712286675f91f6fba4bf52c845"}
Sep 18 16:15:51.211545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211467    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"179ca6cb926caa3c10b791b96870d22f14009ae340cbcf0c62d9bfef263a20be"}
Sep 18 16:15:51.211545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerDied","Data":"458ff610f57386f0a435a0e423cdfea7844f169091fc6b5cd070908be2de3460"}
Sep 18 16:15:51.211633 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.211564    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"a479ed1dcc51f996e76d030c6e2f0b8b24db3aac038c1eb132d30e07d3dbf6c7"}
Sep 18 16:15:51.214788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.214754    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l2ngq" podStartSLOduration=11.667314212 podStartE2EDuration="22.214743332s" totalImagesPullingTime="10.54742912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:51.214400757 +0000 UTC m=+22.694374302" watchObservedRunningTime="2026-09-18 16:15:51.214743332 +0000 UTC m=+22.694716877"
Sep 18 16:15:51.214931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.214914    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-62.ec2.internal" podStartSLOduration=21.214909989 podStartE2EDuration="21.214909989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:34.174188727 +0000 UTC m=+5.654162276" watchObservedRunningTime="2026-09-18 16:15:51.214909989 +0000 UTC m=+22.694883534"
Sep 18 16:15:51.249379 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.249336    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wrts2" podStartSLOduration=4.310264572 podStartE2EDuration="22.249319856s" totalImagesPullingTime="17.939055284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:51.22828244 +0000 UTC m=+22.708255986" watchObservedRunningTime="2026-09-18 16:15:51.249319856 +0000 UTC m=+22.729293401"
Sep 18 16:15:51.249550 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.249529    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tt7q5" podStartSLOduration=4.298932129 podStartE2EDuration="22.249523512s" totalImagesPullingTime="17.950591383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:51.249507067 +0000 UTC m=+22.729480613" watchObservedRunningTime="2026-09-18 16:15:51.249523512 +0000 UTC m=+22.729497120"
Sep 18 16:15:51.264860 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.264810    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kjtm7" podStartSLOduration=4.322721376 podStartE2EDuration="22.264792457s" totalImagesPullingTime="17.942071081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:51.264649609 +0000 UTC m=+22.744623157" watchObservedRunningTime="2026-09-18 16:15:51.264792457 +0000 UTC m=+22.744766004"
Sep 18 16:15:51.297043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.296988    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ppstz" podStartSLOduration=4.048119096 podStartE2EDuration="22.296970801s" totalImagesPullingTime="18.248851705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:51.296666633 +0000 UTC m=+22.776640181" watchObservedRunningTime="2026-09-18 16:15:51.296970801 +0000 UTC m=+22.776944353"
Sep 18 16:15:51.768482 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.768455    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:15:51.989916 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.989824    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:15:51.768476696Z","UUID":"f1c5c0b3-f762-4f53-8e41-dde6126f72b7","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:15:51.990223 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.990208    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:51.990317 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:51.990303    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:51.991312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.991289    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:15:51.991312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:51.991317    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:15:52.215766 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:52.215732    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j" event={"ID":"4d86a9c9-26fc-461b-b262-879b82d20ae1","Type":"ContainerStarted","Data":"580b84511c42ac16c7e915ccac027154fa7e471c3512a63937b06cea8d999b06"}
Sep 18 16:15:52.217202 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:52.217174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2684q" event={"ID":"bc241aa0-d286-4995-8661-ea3044ecb836","Type":"ContainerStarted","Data":"a31a70349e83ebc33ddd5d87bc723327cbb97abb587cc4a597da5ebbb2366e2a"}
Sep 18 16:15:52.233838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:52.233786    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2684q" podStartSLOduration=5.287979033 podStartE2EDuration="23.23377143s" totalImagesPullingTime="17.945792397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:52.233609069 +0000 UTC m=+23.713582615" watchObservedRunningTime="2026-09-18 16:15:52.23377143 +0000 UTC m=+23.713744976"
Sep 18 16:15:52.991219 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:52.991192    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:52.991348 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:52.991231    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:52.991348 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:52.991317    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:52.991761 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:52.991411    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:53.223092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:53.223067    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:15:53.223513 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:53.223487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"1fad489fc306364de5bd979bdc40646b73cac21f9fe4ebcba4303c66147b05e8"}
Sep 18 16:15:53.949968 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:53.949935    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:53.950651 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:53.950613    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:53.991299 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:53.991131    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:53.991494 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:53.991399    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:54.228251 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.228216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j" event={"ID":"4d86a9c9-26fc-461b-b262-879b82d20ae1","Type":"ContainerStarted","Data":"c9801befb9744d841fb1c67b03d32a3d207268c5b00ec4f62ba4cc50f6a95891"}
Sep 18 16:15:54.228532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.228514    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:54.228982 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.228965    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l2ngq"
Sep 18 16:15:54.256928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.256870    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqp6j" podStartSLOduration=4.055837093 podStartE2EDuration="25.256847258s" totalImagesPullingTime="21.201010165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:54.25327117 +0000 UTC m=+25.733244715" watchObservedRunningTime="2026-09-18 16:15:54.256847258 +0000 UTC m=+25.736820805"
Sep 18 16:15:54.990497 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.990460    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:54.990680 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:54.990593    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:54.990887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:54.990460    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:54.991058 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:54.991018    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:55.235746 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:55.235705    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:15:55.236604 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:55.236136    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"6ca989e03f44d0f8b0704c7beac28cde60993e07199f4996a08d27393a3e6cad"}
Sep 18 16:15:55.236838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:55.236817    2554 scope.go:117] "RemoveContainer" containerID="458ff610f57386f0a435a0e423cdfea7844f169091fc6b5cd070908be2de3460"
Sep 18 16:15:55.991009 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:55.990973    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:55.991192 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:55.991092    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:56.246141 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.246065    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:15:56.246594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.246569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" event={"ID":"a5b4cb8d-c838-48fd-8531-708ca2a3ab4b","Type":"ContainerStarted","Data":"276cef085ef72ab14718609794603f74bf357bb0449d89ff3e2dc69436fc7969"}
Sep 18 16:15:56.246931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.246910    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:56.247043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.246993    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:56.247043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.247042    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:56.263598 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.263573    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:56.263805 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.263790    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:15:56.288749 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.288701    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-754q4" podStartSLOduration=9.326908343 podStartE2EDuration="27.288685832s" totalImagesPullingTime="17.961777489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:56.28804992 +0000 UTC m=+27.768023468" watchObservedRunningTime="2026-09-18 16:15:56.288685832 +0000 UTC m=+27.768659377"
Sep 18 16:15:56.993758 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.993588    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:56.993898 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:56.993650    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:56.993898 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:56.993835    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:56.993999 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:56.993907    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:57.109096 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.109062    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vn5kq"]
Sep 18 16:15:57.111939 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.111910    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fxk2s"]
Sep 18 16:15:57.112098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.112037    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:57.112173 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:57.112154    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:57.112699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.112672    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpwss"]
Sep 18 16:15:57.248488 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.248434    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:57.248785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:57.248442    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:57.248785 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:57.248552    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:57.248785 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:57.248608    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:58.252837 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:58.252801    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="fee06dae570fb8cad247b1530cc262eb430bec17e1393268365ad6281360f1d6" exitCode=0
Sep 18 16:15:58.253213 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:58.252884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"fee06dae570fb8cad247b1530cc262eb430bec17e1393268365ad6281360f1d6"}
Sep 18 16:15:58.991721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:58.991699    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:15:58.991849 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:58.991827    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:15:58.992242 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:58.992217    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:15:58.992428 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:58.992377    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:15:58.992428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:58.992426    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:15:58.992552 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:15:58.992516    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:15:59.257211 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:59.257109    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="a18947693d7dd8d95f9cd6bfd642c1c88b8ac2619efc538042d30c66650a43a9" exitCode=0
Sep 18 16:15:59.257211 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:15:59.257192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"a18947693d7dd8d95f9cd6bfd642c1c88b8ac2619efc538042d30c66650a43a9"}
Sep 18 16:16:00.261931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:00.261755    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="7b1f224798104103127f23afa4ebf47bac81a628ada807027bdb6613dbe3f967" exitCode=0
Sep 18 16:16:00.261931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:00.261845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"7b1f224798104103127f23afa4ebf47bac81a628ada807027bdb6613dbe3f967"}
Sep 18 16:16:00.994588 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:00.994557    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:00.994765 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:00.994650    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:00.994765 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:00.994657    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:16:00.994765 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:00.994666    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:16:00.994902 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:00.994765    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:16:00.994902 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:00.994882    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:16:02.683713 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.683678    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:16:02.684177 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.683819    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:02.684177 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.683883    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:34.683864166 +0000 UTC m=+66.163837705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:02.784590 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.784549    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:02.784776 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.784687    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:02.784776 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.784750    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret podName:25c3b578-0fb3-4d56-bb23-da9ba897582a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:34.784733956 +0000 UTC m=+66.264707487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret") pod "global-pull-secret-syncer-vn5kq" (UID: "25c3b578-0fb3-4d56-bb23-da9ba897582a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:02.885664 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.885627    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:02.885845 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.885781    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:02.885845 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.885799    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:02.885845 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.885808    2554 projected.go:196] Error preparing data for projected volume kube-api-access-qpbst for pod openshift-network-diagnostics/network-check-target-gpwss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:02.885981 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.885858    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst podName:2163f10f-d538-46c4-8f93-9e637a8574c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:34.885842218 +0000 UTC m=+66.365815747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qpbst" (UniqueName: "kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst") pod "network-check-target-gpwss" (UID: "2163f10f-d538-46c4-8f93-9e637a8574c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:02.994063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.994025    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:16:02.994063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.994052    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:02.994266 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:02.994052    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:02.994266 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.994151    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:16:02.994266 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.994245    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpwss" podUID="2163f10f-d538-46c4-8f93-9e637a8574c2"
Sep 18 16:16:02.994402 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:02.994338    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn5kq" podUID="25c3b578-0fb3-4d56-bb23-da9ba897582a"
Sep 18 16:16:03.284283 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.284205    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-62.ec2.internal" event="NodeReady"
Sep 18 16:16:03.284467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.284348    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:16:03.326649 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.326615    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:16:03.368955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.368913    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"]
Sep 18 16:16:03.369148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.369100    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.372515 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.372483    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hlk9j\""
Sep 18 16:16:03.372690 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.372610    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:16:03.373071 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.373051    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:16:03.373192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.373119    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:16:03.380167 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.380140    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 16:16:03.383193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.383163    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pgftx"]
Sep 18 16:16:03.383330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.383283    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.396730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.396370    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 16:16:03.396730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.396433    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 16:16:03.396730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.396458    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zv89s\""
Sep 18 16:16:03.413636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.413593    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"]
Sep 18 16:16:03.413636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.413632    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:16:03.413832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.413650    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6wbnf"]
Sep 18 16:16:03.413832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.413737    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.416483 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.416460    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:16:03.416596 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.416460    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:16:03.416596 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.416460    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:16:03.435661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.435625    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgftx"]
Sep 18 16:16:03.435838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.435679    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6wbnf"]
Sep 18 16:16:03.435838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.435766    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.438096 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.438072    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:16:03.438096 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.438091    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:16:03.438096 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.438097    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6h8pt\""
Sep 18 16:16:03.438337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.438097    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:16:03.489650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-config-volume\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dknr\" (UniqueName: \"kubernetes.io/projected/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-kube-api-access-7dknr\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489764    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm9t\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-tmp-dir\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.489832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489883    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca2d477-1869-49a3-adc2-7f8da9249342-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.489912    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.490007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.490042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.490192 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.490069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.588855 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.588762    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:03.590679 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.590832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590686    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.590832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590716    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.590832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-config-volume\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.590832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590791    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwjv\" (UniqueName: \"kubernetes.io/projected/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-kube-api-access-jgwjv\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.590832 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590824    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dknr\" (UniqueName: \"kubernetes.io/projected/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-kube-api-access-7dknr\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm9t\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.590960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-tmp-dir\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.591059 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591056    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591120    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.091100688 +0000 UTC m=+35.571074212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591155    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591175    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591226    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.091209032 +0000 UTC m=+35.571182557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591251    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591341 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-cert\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591347    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca2d477-1869-49a3-adc2-7f8da9249342-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591375    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591286    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-tmp-dir\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591521    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:03.591564    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.091550386 +0000 UTC m=+35.571523911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:03.591703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-config-volume\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.591981 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591812    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591981 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.591981 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.591859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.592234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.592205    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca2d477-1869-49a3-adc2-7f8da9249342-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:03.595932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.595908    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.596047 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.595935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.600125 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.600100    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.600349 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.600326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm9t\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:03.600449 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.600336    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dknr\" (UniqueName: \"kubernetes.io/projected/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-kube-api-access-7dknr\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:03.691441 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.691392    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:03.692058 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.691767    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwjv\" (UniqueName: \"kubernetes.io/projected/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-kube-api-access-jgwjv\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.692058 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.691838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-cert\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.694617 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.694592    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-cert\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.700614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.700572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwjv\" (UniqueName: \"kubernetes.io/projected/56cf29d2-ddb5-4c98-a9d8-693d27a505d7-kube-api-access-jgwjv\") pod \"ingress-canary-6wbnf\" (UID: \"56cf29d2-ddb5-4c98-a9d8-693d27a505d7\") " pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:03.744521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:03.744484    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6wbnf"
Sep 18 16:16:04.011558 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.011515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:04.090569 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.090527    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:04.095540 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.095503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:04.095712 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.095548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:04.095712 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.095632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:04.095712 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095674    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:04.095712 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095696    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:04.095712 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095709    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:04.095927 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095760    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.095744004 +0000 UTC m=+36.575717529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:04.095927 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095780    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.095769124 +0000 UTC m=+36.575742651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:04.095927 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095793    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:04.095927 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:04.095858    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.095842721 +0000 UTC m=+36.575816251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:04.993843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.993802    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:16:04.993843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.993844    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:04.994514 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.993903    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.997976    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.997994    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.997978    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.997976    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.998051    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:04.998164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:04.998142    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.104530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:05.104492    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:05.104699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:05.104607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:05.104699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:05.104633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:05.104699 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104636    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:05.104699 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104692    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.104677931 +0000 UTC m=+38.584651455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:05.104912 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104735    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:05.104912 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104752    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:05.104912 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104794    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.104781179 +0000 UTC m=+38.584754704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:05.104912 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104825    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:05.104912 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:05.104875    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.104861646 +0000 UTC m=+38.584835175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:05.281400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:05.281320    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6wbnf"]
Sep 18 16:16:05.285599 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:16:05.285572    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cf29d2_ddb5_4c98_a9d8_693d27a505d7.slice/crio-28bc4ce9670e6184cfdd0f0cf740abe72eae3f3504e038535ce617b1e36d2175 WatchSource:0}: Error finding container 28bc4ce9670e6184cfdd0f0cf740abe72eae3f3504e038535ce617b1e36d2175: Status 404 returned error can't find the container with id 28bc4ce9670e6184cfdd0f0cf740abe72eae3f3504e038535ce617b1e36d2175
Sep 18 16:16:06.281631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:06.281587    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="592cbb60e45b2e1842ca4ffed6167e08f12326cff5de2e8bfdc99e12ce1ffba9" exitCode=0
Sep 18 16:16:06.282202 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:06.281687    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"592cbb60e45b2e1842ca4ffed6167e08f12326cff5de2e8bfdc99e12ce1ffba9"}
Sep 18 16:16:06.283601 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:06.283568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6wbnf" event={"ID":"56cf29d2-ddb5-4c98-a9d8-693d27a505d7","Type":"ContainerStarted","Data":"28bc4ce9670e6184cfdd0f0cf740abe72eae3f3504e038535ce617b1e36d2175"}
Sep 18 16:16:07.120477 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:07.120426    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:07.120477 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:07.120479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:07.120515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120525    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120545    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120601    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.120585006 +0000 UTC m=+42.600558530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120608    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120619    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120647    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.120635946 +0000 UTC m=+42.600609470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:07.120730 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:07.120670    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.120653902 +0000 UTC m=+42.600627431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:07.289201 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:07.289168    2554 generic.go:356] "Generic (PLEG): container finished" podID="364f7d51-7a38-43f1-a5b3-73756da9c04c" containerID="9e0c4c4730eb88daf0a0ccc62cf74b10a8f974fbab03465cda61280dd621fd05" exitCode=0
Sep 18 16:16:07.289671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:07.289247    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerDied","Data":"9e0c4c4730eb88daf0a0ccc62cf74b10a8f974fbab03465cda61280dd621fd05"}
Sep 18 16:16:08.210468 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.210436    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.295166 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.295134    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" event={"ID":"364f7d51-7a38-43f1-a5b3-73756da9c04c","Type":"ContainerStarted","Data":"f39f8c73957a7a28b973736e978b07d8ebf5e555d7ee28c154bb69413605f076"}
Sep 18 16:16:08.313591 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.313564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.331531 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.331483    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vl7gs" podStartSLOduration=8.825793196 podStartE2EDuration="39.331470194s" totalImagesPullingTime="30.505676998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:08.330224536 +0000 UTC m=+39.810198083" watchObservedRunningTime="2026-09-18 16:16:08.331470194 +0000 UTC m=+39.811443740"
Sep 18 16:16:08.432298 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.432252    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.569512 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.566798    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.657430 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.657388    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.758301 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.758265    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.822668 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.822571    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:08.880347 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:08.880313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:09.299426 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:09.299381    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6wbnf" event={"ID":"56cf29d2-ddb5-4c98-a9d8-693d27a505d7","Type":"ContainerStarted","Data":"63e815db61bfd16bc2514586a2eb9858ee9590929395afd825672f8869b29f08"}
Sep 18 16:16:09.315129 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:09.315079    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6wbnf" podStartSLOduration=3.313624851 podStartE2EDuration="6.315063871s" totalImagesPullingTime="3.00143902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:09.314443477 +0000 UTC m=+40.794417024" watchObservedRunningTime="2026-09-18 16:16:09.315063871 +0000 UTC m=+40.795037417"
Sep 18 16:16:11.152112 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:11.152071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:11.152112 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:11.152114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:11.152147    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152235    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152246    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152275    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152308    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.152292721 +0000 UTC m=+50.632266244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152245    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152340    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.1523228 +0000 UTC m=+50.632296329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:11.152637 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:11.152383    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.152372624 +0000 UTC m=+50.632346148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:19.214672 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:19.214632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:19.214672 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:19.214670    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:19.214702    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214815    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214824    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214847    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214871    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.21485907 +0000 UTC m=+66.694832594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214903    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.21488981 +0000 UTC m=+66.694863339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214822    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:19.215148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:19.214943    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.214934568 +0000 UTC m=+66.694908093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:23.863960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:23.863925    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:23.934831 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:23.934802    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:24.010009 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:24.009976    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:24.809967 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:24.809911    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:24.888721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:24.888685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.273185 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:28.273155    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-754q4"
Sep 18 16:16:28.424220 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:28.424185    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.523733 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:28.523646    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.624892 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:28.624857    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.699627 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:29.699590    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.805727 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:29.805697    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.908434 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:29.908386    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.974342 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:29.974303    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.787195 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:30.787158    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.886705 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:30.886668    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:33.303553 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:33.303514    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.734724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.734677    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:16:34.737496 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.737475    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:34.745532 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:34.745513    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:16:34.745619 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:34.745572    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:38.74555627 +0000 UTC m=+130.225529793 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : secret "metrics-daemon-secret" not found
Sep 18 16:16:34.835747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.835709    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:34.838568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.838546    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:34.849799 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.849773    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25c3b578-0fb3-4d56-bb23-da9ba897582a-original-pull-secret\") pod \"global-pull-secret-syncer-vn5kq\" (UID: \"25c3b578-0fb3-4d56-bb23-da9ba897582a\") " pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:34.936723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.936676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:34.939829 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.939803    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:34.949988 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.949964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:34.960184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:34.960160    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbst\" (UniqueName: \"kubernetes.io/projected/2163f10f-d538-46c4-8f93-9e637a8574c2-kube-api-access-qpbst\") pod \"network-check-target-gpwss\" (UID: \"2163f10f-d538-46c4-8f93-9e637a8574c2\") " pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:35.013784 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.013707    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:16:35.014746 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.014731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn5kq"
Sep 18 16:16:35.021736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.021712    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:35.239552 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.239512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:16:35.239552 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.239550    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.239582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239688    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239700    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239725    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239751    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:07.239735526 +0000 UTC m=+98.719709064 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239688    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239785    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:07.239763326 +0000 UTC m=+98.719736868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:16:35.239840 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:16:35.239834    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:07.239822309 +0000 UTC m=+98.719795833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:16:35.302026 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.301943    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vn5kq"]
Sep 18 16:16:35.305121 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:16:35.305094    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c3b578_0fb3_4d56_bb23_da9ba897582a.slice/crio-43b8383687d7865734581ab566d6542ff643cb0c0333006506fd061f7d8c2979 WatchSource:0}: Error finding container 43b8383687d7865734581ab566d6542ff643cb0c0333006506fd061f7d8c2979: Status 404 returned error can't find the container with id 43b8383687d7865734581ab566d6542ff643cb0c0333006506fd061f7d8c2979
Sep 18 16:16:35.325628 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.325469    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpwss"]
Sep 18 16:16:35.327853 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:16:35.327829    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2163f10f_d538_46c4_8f93_9e637a8574c2.slice/crio-971e2cebc8be7628572403529041f9abb9a6e960f022a75cb19a0ccc7503df9b WatchSource:0}: Error finding container 971e2cebc8be7628572403529041f9abb9a6e960f022a75cb19a0ccc7503df9b: Status 404 returned error can't find the container with id 971e2cebc8be7628572403529041f9abb9a6e960f022a75cb19a0ccc7503df9b
Sep 18 16:16:35.359859 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.359824    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpwss" event={"ID":"2163f10f-d538-46c4-8f93-9e637a8574c2","Type":"ContainerStarted","Data":"971e2cebc8be7628572403529041f9abb9a6e960f022a75cb19a0ccc7503df9b"}
Sep 18 16:16:35.360994 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:35.360972    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vn5kq" event={"ID":"25c3b578-0fb3-4d56-bb23-da9ba897582a","Type":"ContainerStarted","Data":"43b8383687d7865734581ab566d6542ff643cb0c0333006506fd061f7d8c2979"}
Sep 18 16:16:36.663108 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:36.663064    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:36.726316 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:36.726279    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:40.375132 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.375032    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vn5kq" event={"ID":"25c3b578-0fb3-4d56-bb23-da9ba897582a","Type":"ContainerStarted","Data":"2b80f97c35b659ff203eaf15fa8fa87e2d498d22e66b438ac164af02ab1af60f"}
Sep 18 16:16:40.376546 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.376521    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpwss" event={"ID":"2163f10f-d538-46c4-8f93-9e637a8574c2","Type":"ContainerStarted","Data":"1baeae4f587a45c5a9f7b5081cf5dd7dd71748197ce599e49c3357994c703b39"}
Sep 18 16:16:40.376672 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.376655    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:16:40.389430 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.389380    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vn5kq" podStartSLOduration=66.624763277 podStartE2EDuration="1m11.389365837s" totalImagesPullingTime="4.76460256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:40.388782084 +0000 UTC m=+71.868755633" watchObservedRunningTime="2026-09-18 16:16:40.389365837 +0000 UTC m=+71.869339383"
Sep 18 16:16:40.404488 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.404438    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gpwss" podStartSLOduration=66.667544943 podStartE2EDuration="1m11.404396644s" totalImagesPullingTime="4.736851701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:40.404113557 +0000 UTC m=+71.884087104" watchObservedRunningTime="2026-09-18 16:16:40.404396644 +0000 UTC m=+71.884370187"
Sep 18 16:16:40.692470 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.692382    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:40.760572 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.760484    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:40.893242 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.893203    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:40.951463 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:40.951373    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.734029 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:42.733991    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.794724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:42.794695    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:45.893431 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:45.893387    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:45.984885 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:45.984854    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:46.581920 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:46.581876    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:46.680980 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:16:46.680949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:01.866247 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:01.866127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:01.969574 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:01.969541    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.273713 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:07.273665    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:07.273742    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:07.273762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273831    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273880    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273898    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273838    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273925    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:11.273903893 +0000 UTC m=+162.753877443 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.273998    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:11.273986516 +0000 UTC m=+162.753960040 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:17:07.274165 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:07.274009    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:11.274002939 +0000 UTC m=+162.753976463 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:07.867887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:07.867850    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.978774 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:07.978742    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.382283 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:11.382249    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gpwss"
Sep 18 16:17:32.979237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:32.979200    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:33.079753 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:33.079720    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:38.800708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:17:38.800654    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:17:38.801104 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:38.800808    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:17:38.801104 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:17:38.800871    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:40.800854639 +0000 UTC m=+252.280828176 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : secret "metrics-daemon-secret" not found
Sep 18 16:18:06.379690 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:06.379654    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3"
Sep 18 16:18:06.392759 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:06.392726    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk" podUID="3ca2d477-1869-49a3-adc2-7f8da9249342"
Sep 18 16:18:06.422208 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:06.422160    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pgftx" podUID="35655ad6-0a9b-41cb-a9ef-031e5880b4c2"
Sep 18 16:18:06.586088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:06.586054    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:18:06.586248 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:06.586054    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:18:06.586248 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:06.586060    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:18:08.004372 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:08.004322    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:18:11.345389 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:11.345353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:11.345394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:11.345445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345526    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345586    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls podName:35655ad6-0a9b-41cb-a9ef-031e5880b4c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.345571443 +0000 UTC m=+284.825544980 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls") pod "dns-default-pgftx" (UID: "35655ad6-0a9b-41cb-a9ef-031e5880b4c2") : secret "dns-default-metrics-tls" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345528    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345676    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert podName:3ca2d477-1869-49a3-adc2-7f8da9249342 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.34566315 +0000 UTC m=+284.825636679 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7m6xk" (UID: "3ca2d477-1869-49a3-adc2-7f8da9249342") : secret "networking-console-plugin-cert" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345531    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345695    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d5c6f56-fxx4q: secret "image-registry-tls" not found
Sep 18 16:18:11.345767 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:11.345728    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls podName:69a8bf97-52a6-499a-9b63-2567dac930e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.345718885 +0000 UTC m=+284.825692422 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls") pod "image-registry-579d5c6f56-fxx4q" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3") : secret "image-registry-tls" not found
Sep 18 16:18:12.612201 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:18:12.612153    2554 controller.go:195] "Failed to update lease" err="Put \"https://a7fcdc15a722d441d84d411110ab9c85-cc4f6123aed674b7.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-62.ec2.internal?timeout=10s\": read tcp 10.0.142.62:46230->184.192.68.255:6443: read: connection reset by peer"
Sep 18 16:18:18.991954 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:18:18.991917    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:19:16.487855 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:16.487824    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:40.883262 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:40.883206    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:19:40.883693 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:40.883333    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:19:40.883693 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:40.883389    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs podName:023a0f44-e934-492c-a9fd-c88a77e45000 nodeName:}" failed. No retries permitted until 2026-09-18 16:21:42.883374925 +0000 UTC m=+374.363348449 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs") pod "network-metrics-daemon-fxk2s" (UID: "023a0f44-e934-492c-a9fd-c88a77e45000") : secret "metrics-daemon-secret" not found
Sep 18 16:19:53.671102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.671018    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"]
Sep 18 16:19:53.673863 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.673843    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.674177 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.674152    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"]
Sep 18 16:19:53.676870 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.676841    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.680476 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680292    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 16:19:53.680476 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680444    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 16:19:53.680609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680584    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.680710 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680688    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t4vsr\""
Sep 18 16:19:53.680878 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680856    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.680878 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.680868    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 16:19:53.681282 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.681264    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.681373 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.681355    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.681449 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.681405    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"]
Sep 18 16:19:53.681515 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.681464    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mrrvl\""
Sep 18 16:19:53.682560 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.682543    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 16:19:53.684073 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.684059    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.690551 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.690532    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 16:19:53.691110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.691093    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.695042 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.695025    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ptt9t\""
Sep 18 16:19:53.695268 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.695256    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.695858 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.695841    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"]
Sep 18 16:19:53.701159 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.701136    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"]
Sep 18 16:19:53.718621 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.718598    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"]
Sep 18 16:19:53.774378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.774378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774378    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjvw\" (UniqueName: \"kubernetes.io/projected/8c182ae2-4b16-467a-8f88-9768ce080373-kube-api-access-bpjvw\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.774589 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.774589 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcpx\" (UniqueName: \"kubernetes.io/projected/0953c73e-e2c8-4e23-9917-c19c5325f849-kube-api-access-qpcpx\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.774656 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prllm\" (UniqueName: \"kubernetes.io/projected/1ecca491-0527-4d26-813f-f401182d7bd4-kube-api-access-prllm\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.774656 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774638    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0953c73e-e2c8-4e23-9917-c19c5325f849-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.774716 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c182ae2-4b16-467a-8f88-9768ce080373-serving-cert\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.774716 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.774672    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c182ae2-4b16-467a-8f88-9768ce080373-config\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.875190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0953c73e-e2c8-4e23-9917-c19c5325f849-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.875190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c182ae2-4b16-467a-8f88-9768ce080373-serving-cert\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875204    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c182ae2-4b16-467a-8f88-9768ce080373-config\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875223    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjvw\" (UniqueName: \"kubernetes.io/projected/8c182ae2-4b16-467a-8f88-9768ce080373-kube-api-access-bpjvw\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875288    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875308    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcpx\" (UniqueName: \"kubernetes.io/projected/0953c73e-e2c8-4e23-9917-c19c5325f849-kube-api-access-qpcpx\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:53.875330    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prllm\" (UniqueName: \"kubernetes.io/projected/1ecca491-0527-4d26-813f-f401182d7bd4-kube-api-access-prllm\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:53.875409    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:54.375385872 +0000 UTC m=+265.855359396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:53.875478 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:53.875439    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:19:53.875929 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:53.875495    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls podName:1ecca491-0527-4d26-813f-f401182d7bd4 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:54.375482305 +0000 UTC m=+265.855455853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dvtfx" (UID: "1ecca491-0527-4d26-813f-f401182d7bd4") : secret "samples-operator-tls" not found
Sep 18 16:19:53.875929 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.875921    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c182ae2-4b16-467a-8f88-9768ce080373-config\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.876127 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.876104    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0953c73e-e2c8-4e23-9917-c19c5325f849-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.877563 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.877547    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c182ae2-4b16-467a-8f88-9768ce080373-serving-cert\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.884060 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.884036    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjvw\" (UniqueName: \"kubernetes.io/projected/8c182ae2-4b16-467a-8f88-9768ce080373-kube-api-access-bpjvw\") pod \"service-ca-operator-57cdc4b768-mrcj6\" (UID: \"8c182ae2-4b16-467a-8f88-9768ce080373\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:53.884150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.884134    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prllm\" (UniqueName: \"kubernetes.io/projected/1ecca491-0527-4d26-813f-f401182d7bd4-kube-api-access-prllm\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:53.884648 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.884630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcpx\" (UniqueName: \"kubernetes.io/projected/0953c73e-e2c8-4e23-9917-c19c5325f849-kube-api-access-qpcpx\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:53.901043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.901009    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:53.989660 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:53.989630    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"
Sep 18 16:19:54.258530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:54.258502    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6"]
Sep 18 16:19:54.259174 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:19:54.259146    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c182ae2_4b16_467a_8f88_9768ce080373.slice/crio-1b8422fab4da0652fc2ee9dc4eca4bbed0a4e8710da6d11dfed0b682f1458356 WatchSource:0}: Error finding container 1b8422fab4da0652fc2ee9dc4eca4bbed0a4e8710da6d11dfed0b682f1458356: Status 404 returned error can't find the container with id 1b8422fab4da0652fc2ee9dc4eca4bbed0a4e8710da6d11dfed0b682f1458356
Sep 18 16:19:54.379783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:54.379748    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:54.379984 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:54.379820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:54.379984 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:54.379905    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:54.379984 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:54.379944    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:19:54.379984 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:54.379971    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.379954584 +0000 UTC m=+266.859928109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:54.380148 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:54.380003    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls podName:1ecca491-0527-4d26-813f-f401182d7bd4 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.379987934 +0000 UTC m=+266.859961476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dvtfx" (UID: "1ecca491-0527-4d26-813f-f401182d7bd4") : secret "samples-operator-tls" not found
Sep 18 16:19:54.840544 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:54.840512    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6" event={"ID":"8c182ae2-4b16-467a-8f88-9768ce080373","Type":"ContainerStarted","Data":"1b8422fab4da0652fc2ee9dc4eca4bbed0a4e8710da6d11dfed0b682f1458356"}
Sep 18 16:19:55.038170 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:55.038138    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:55.387841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:55.387799    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:55.388039 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:55.387980    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:19:55.388106 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:55.388029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:55.388106 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:55.388054    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls podName:1ecca491-0527-4d26-813f-f401182d7bd4 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:57.388033204 +0000 UTC m=+268.868006737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dvtfx" (UID: "1ecca491-0527-4d26-813f-f401182d7bd4") : secret "samples-operator-tls" not found
Sep 18 16:19:55.388219 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:55.388119    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:55.388219 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:55.388171    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:57.388155368 +0000 UTC m=+268.868128898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:56.273796 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:56.273760    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:56.331634 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:56.331586    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:56.846800 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:56.846766    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6" event={"ID":"8c182ae2-4b16-467a-8f88-9768ce080373","Type":"ContainerStarted","Data":"419f54e19c1a585917cf56c57912ff93a641f4fd7a1082b9cb74803d2d7be2d1"}
Sep 18 16:19:56.864109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:56.864004    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6" podStartSLOduration=1.522571439 podStartE2EDuration="3.863988733s" totalImagesPullingTime="2.341417294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:56.863219461 +0000 UTC m=+268.343193006" watchObservedRunningTime="2026-09-18 16:19:56.863988733 +0000 UTC m=+268.343962321"
Sep 18 16:19:57.384643 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.384609    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.405310 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.405280    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:19:57.405467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.405332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:19:57.405467 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.405437    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:19:57.405467 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.405446    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:57.405601 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.405490    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls podName:1ecca491-0527-4d26-813f-f401182d7bd4 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.405476456 +0000 UTC m=+272.885449980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dvtfx" (UID: "1ecca491-0527-4d26-813f-f401182d7bd4") : secret "samples-operator-tls" not found
Sep 18 16:19:57.405601 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.405502    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.405496574 +0000 UTC m=+272.885470097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:19:57.485617 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.485581    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.633088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.633055    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bvdpc"]
Sep 18 16:19:57.636348 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.636274    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.638662 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.638636    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:19:57.638805 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.638684    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:19:57.638805 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.638688    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:19:57.639714 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.639692    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:19:57.639803 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.639753    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpr2n\""
Sep 18 16:19:57.650101 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.650076    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvdpc"]
Sep 18 16:19:57.661634 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.661598    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.708983 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.708950    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-data-volume\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.709150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.709013    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.709150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.709085    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-api-access-w4snc\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.709150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.709115    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.709150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.709132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-crio-socket\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.745645 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.745612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.809843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.809807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-api-access-w4snc\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.809843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.809844    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.809860    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-crio-socket\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.809946    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-crio-socket\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.809943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-data-volume\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.809987    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.810054    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.310033769 +0000 UTC m=+269.790007300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.810095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.810074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.810334 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.810201    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.810334 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:57.810255    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.31024206 +0000 UTC m=+269.790215601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.810334 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.810291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-data-volume\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:57.820915 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:57.820897    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-api-access-w4snc\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:58.095950 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.095915    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.195454 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.195404    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.314382 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.314338    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:58.314547 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.314403    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:58.314547 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:58.314485    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:58.314547 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:58.314518    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.314664 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:58.314565    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.314544881 +0000 UTC m=+270.794518418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:58.314664 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:58.314579    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.314573679 +0000 UTC m=+270.794547202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.582827 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.582794    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.685013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.684974    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.785964 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:58.785934    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:59.321856 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:59.321818    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:59.322117 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:59.321880    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:19:59.322117 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:59.321946    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.322117 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:59.322012    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.321995676 +0000 UTC m=+272.801969200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.322117 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:59.322018    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.322117 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:19:59.322060    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.32204684 +0000 UTC m=+272.802020364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.412795 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:59.412769    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kjtm7_d83c7a85-6566-44e2-8725-dd1c6429033d/dns-node-resolver/0.log"
Sep 18 16:19:59.723902 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:59.723872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:59.784809 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:19:59.784780    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:00.706754 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:00.706725    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:00.813516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:00.813491    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrts2_dd1bdd77-ade8-4626-b01e-2b3c6fe51d98/node-ca/0.log"
Sep 18 16:20:01.212889 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:01.212852    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6wbnf_56cf29d2-ddb5-4c98-a9d8-693d27a505d7/serve-healthcheck-canary/0.log"
Sep 18 16:20:01.336174 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:01.336137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:01.336359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:01.336217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:01.336359 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.336283    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.336359 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.336284    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.336359 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.336335    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.336320706 +0000 UTC m=+276.816294229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.336359 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.336347    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.336341742 +0000 UTC m=+276.816315266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.437432 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:01.437381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:20:01.437583 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:01.437513    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:20:01.437583 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.437537    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:01.437666 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.437612    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:09.437592204 +0000 UTC m=+280.917565738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:01.437666 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.437616    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:01.437666 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:01.437649    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls podName:1ecca491-0527-4d26-813f-f401182d7bd4 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:09.437640271 +0000 UTC m=+280.917613811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dvtfx" (UID: "1ecca491-0527-4d26-813f-f401182d7bd4") : secret "samples-operator-tls" not found
Sep 18 16:20:03.102016 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:03.101979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:03.238356 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:03.238320    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:03.523829 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:03.523794    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:03.623157 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:03.623127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:03.797950 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:03.797874    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:04.494216 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:04.494185    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:04.607967 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:04.607936    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:05.371098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:05.371059    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:05.371272 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:05.371133    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:05.371272 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:05.371188    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:05.371272 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:05.371246    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:05.371272 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:05.371261    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.37124163 +0000 UTC m=+284.851215171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:05.371411 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:05.371281    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls podName:9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.371270876 +0000 UTC m=+284.851244401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvdpc" (UID: "9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:05.481410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:05.481355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:05.561847 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:05.561821    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:05.645970 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:05.645898    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:06.004214 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:06.004181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:06.404005 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:06.403924    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:06.624775 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:06.624742    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:07.017209 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:07.017177    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:07.404726 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:07.404636    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:07.505103 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:07.505069    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:07.606258 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:07.606231    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:08.110565 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:08.110533    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:08.210867 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:08.210835    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:08.598914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:08.598881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:08.708900 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:08.708874    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:09.009850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.009815    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:09.204314 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.204283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:09.506547 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.506511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:20:09.506728 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.506580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:20:09.506728 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:09.506691    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:09.506838 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:09.506759    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls podName:0953c73e-e2c8-4e23-9917-c19c5325f849 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:25.50674382 +0000 UTC m=+296.986717369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-s46vk" (UID: "0953c73e-e2c8-4e23-9917-c19c5325f849") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:09.508997 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.508971    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ecca491-0527-4d26-813f-f401182d7bd4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dvtfx\" (UID: \"1ecca491-0527-4d26-813f-f401182d7bd4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:20:09.586556 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:09.586505    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk" podUID="3ca2d477-1869-49a3-adc2-7f8da9249342"
Sep 18 16:20:09.586556 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:09.586535    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3"
Sep 18 16:20:09.586556 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:09.586517    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pgftx" podUID="35655ad6-0a9b-41cb-a9ef-031e5880b4c2"
Sep 18 16:20:09.594577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.594551    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"
Sep 18 16:20:09.610007 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.609981    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:09.864736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.864658    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx"]
Sep 18 16:20:09.878558 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.878535    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:20:09.878685 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.878535    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:09.878728 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:09.878538    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:10.197940 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:10.197862    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:10.607459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:10.607408    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:10.804616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:10.804586    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:10.882650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:10.882565    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx" event={"ID":"1ecca491-0527-4d26-813f-f401182d7bd4","Type":"ContainerStarted","Data":"5dd14f40c6b4d287724f06b87d170c4823ea1486d6195ff13f230b01d61ffa3e"}
Sep 18 16:20:11.196516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:11.196442    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:11.806315 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:11.806286    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:12.409494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:12.409440    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:12.601867 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:12.601643    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:12.889168 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:12.889130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx" event={"ID":"1ecca491-0527-4d26-813f-f401182d7bd4","Type":"ContainerStarted","Data":"1d1d6173d0bd287245c26ad3c5483e6b4e28c980fc2acd1c41e99331907c4ffb"}
Sep 18 16:20:12.889623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:12.889188    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx" event={"ID":"1ecca491-0527-4d26-813f-f401182d7bd4","Type":"ContainerStarted","Data":"c7718a0c8fbb79d62b17697234da045d22f8a0d79bf2da69a56f5ba42633edc1"}
Sep 18 16:20:12.999957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:12.999925    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.211674 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.211587    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.439241 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439197    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:20:13.439241 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:13.439520 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439297    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:13.439520 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:13.439645 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:13.439849 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.439818    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:13.441667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.441640    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca2d477-1869-49a3-adc2-7f8da9249342-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7m6xk\" (UID: \"3ca2d477-1869-49a3-adc2-7f8da9249342\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:20:13.441762 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.441683    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvdpc\" (UID: \"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3\") " pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:13.441762 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.441701    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35655ad6-0a9b-41cb-a9ef-031e5880b4c2-metrics-tls\") pod \"dns-default-pgftx\" (UID: \"35655ad6-0a9b-41cb-a9ef-031e5880b4c2\") " pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:13.441762 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.441749    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"image-registry-579d5c6f56-fxx4q\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") " pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:13.483151 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.482702    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:20:13.483151 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.482798    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zv89s\""
Sep 18 16:20:13.483409 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.483368    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hlk9j\""
Sep 18 16:20:13.489948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.489925    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:13.490073 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.490000    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:13.490073 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.490053    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"
Sep 18 16:20:13.545758 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.545657    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvdpc"
Sep 18 16:20:13.615056 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.615012    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.768298 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.768188    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dvtfx" podStartSLOduration=18.174461217 podStartE2EDuration="20.768166243s" totalImagesPullingTime="2.593705026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:12.905884226 +0000 UTC m=+284.385857773" watchObservedRunningTime="2026-09-18 16:20:13.768166243 +0000 UTC m=+285.248139793"
Sep 18 16:20:13.768550 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.768535    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgftx"]
Sep 18 16:20:13.771955 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:13.771928    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35655ad6_0a9b_41cb_a9ef_031e5880b4c2.slice/crio-1ab1e4f6a4976b794702102173f4bed5dad62a06ecd961ca569a0ce9e51f1131 WatchSource:0}: Error finding container 1ab1e4f6a4976b794702102173f4bed5dad62a06ecd961ca569a0ce9e51f1131: Status 404 returned error can't find the container with id 1ab1e4f6a4976b794702102173f4bed5dad62a06ecd961ca569a0ce9e51f1131
Sep 18 16:20:13.795221 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.795193    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk"]
Sep 18 16:20:13.798025 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:13.798004    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca2d477_1869_49a3_adc2_7f8da9249342.slice/crio-f6c6b4feb3cdd9ce58090786628ba211987abb48243639d41492583cacea9ead WatchSource:0}: Error finding container f6c6b4feb3cdd9ce58090786628ba211987abb48243639d41492583cacea9ead: Status 404 returned error can't find the container with id f6c6b4feb3cdd9ce58090786628ba211987abb48243639d41492583cacea9ead
Sep 18 16:20:13.800581 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.800508    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:20:13.802742 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:13.802720    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a8bf97_52a6_499a_9b63_2567dac930e3.slice/crio-f214dd5de82a511586b514cf02bc20b42472fea552151da09e65b8102166db24 WatchSource:0}: Error finding container f214dd5de82a511586b514cf02bc20b42472fea552151da09e65b8102166db24: Status 404 returned error can't find the container with id f214dd5de82a511586b514cf02bc20b42472fea552151da09e65b8102166db24
Sep 18 16:20:13.829032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.829011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.847908 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.847878    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvdpc"]
Sep 18 16:20:13.853994 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:13.853969    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f70c7e6_2e72_4f4e_bd49_4760ac36aaa3.slice/crio-1a7bb476465277d7d2f330d6f5c0606a0bb84ec394317ba29263ee10c42e344f WatchSource:0}: Error finding container 1a7bb476465277d7d2f330d6f5c0606a0bb84ec394317ba29263ee10c42e344f: Status 404 returned error can't find the container with id 1a7bb476465277d7d2f330d6f5c0606a0bb84ec394317ba29263ee10c42e344f
Sep 18 16:20:13.893995 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.893960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" event={"ID":"69a8bf97-52a6-499a-9b63-2567dac930e3","Type":"ContainerStarted","Data":"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"}
Sep 18 16:20:13.894377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.894047    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:13.894377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.894073    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" event={"ID":"69a8bf97-52a6-499a-9b63-2567dac930e3","Type":"ContainerStarted","Data":"f214dd5de82a511586b514cf02bc20b42472fea552151da09e65b8102166db24"}
Sep 18 16:20:13.895138 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.895117    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvdpc" event={"ID":"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3","Type":"ContainerStarted","Data":"1a7bb476465277d7d2f330d6f5c0606a0bb84ec394317ba29263ee10c42e344f"}
Sep 18 16:20:13.896337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.896317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk" event={"ID":"3ca2d477-1869-49a3-adc2-7f8da9249342","Type":"ContainerStarted","Data":"f6c6b4feb3cdd9ce58090786628ba211987abb48243639d41492583cacea9ead"}
Sep 18 16:20:13.897571 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.897543    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgftx" event={"ID":"35655ad6-0a9b-41cb-a9ef-031e5880b4c2","Type":"ContainerStarted","Data":"1ab1e4f6a4976b794702102173f4bed5dad62a06ecd961ca569a0ce9e51f1131"}
Sep 18 16:20:13.915931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:13.915887    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" podStartSLOduration=284.915872503 podStartE2EDuration="4m44.915872503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:13.915345228 +0000 UTC m=+285.395318775" watchObservedRunningTime="2026-09-18 16:20:13.915872503 +0000 UTC m=+285.395846049"
Sep 18 16:20:14.903147 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:14.903061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvdpc" event={"ID":"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3","Type":"ContainerStarted","Data":"70d382ffadc3fb2837b475f3ecd647ac18fa0fa8b66e458385f36b1f2e69490e"}
Sep 18 16:20:14.903147 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:14.903121    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvdpc" event={"ID":"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3","Type":"ContainerStarted","Data":"6ea3bb4005ba62253504f29f4fb29b5eaf77d70f2e6a0c7a8614b074d3496d29"}
Sep 18 16:20:15.908246 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:15.908149    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk" event={"ID":"3ca2d477-1869-49a3-adc2-7f8da9249342","Type":"ContainerStarted","Data":"e223a5d11013111ecb40cfb162babdf7b332ca8c01a6f0565ea10712cf880208"}
Sep 18 16:20:15.909871 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:15.909847    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgftx" event={"ID":"35655ad6-0a9b-41cb-a9ef-031e5880b4c2","Type":"ContainerStarted","Data":"6e9fce43040374aaea6ffc3094e1180af65f3f919ba3ff978cf68ef24bb8fdf6"}
Sep 18 16:20:15.926096 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:15.926047    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7m6xk" podStartSLOduration=283.176697882 podStartE2EDuration="4m44.926030306s" totalImagesPullingTime="1.749332424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:15.92474429 +0000 UTC m=+287.404717838" watchObservedRunningTime="2026-09-18 16:20:15.926030306 +0000 UTC m=+287.406003852"
Sep 18 16:20:16.919802 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:16.919763    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvdpc" event={"ID":"9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3","Type":"ContainerStarted","Data":"f6e94cfa18b4375e0b597ffd7e522d2040efdf880ee22c155d8a3e67688f30f4"}
Sep 18 16:20:16.921481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:16.921446    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgftx" event={"ID":"35655ad6-0a9b-41cb-a9ef-031e5880b4c2","Type":"ContainerStarted","Data":"d74faad3508fd1583d8f3b28112d962cca591dcd31c1d0a30bcb6529bab90edd"}
Sep 18 16:20:16.921667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:16.921646    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:16.937085 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:16.937035    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bvdpc" podStartSLOduration=17.866969387 podStartE2EDuration="19.937019975s" totalImagesPullingTime="2.070050588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:19:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:16.937012066 +0000 UTC m=+288.416985613" watchObservedRunningTime="2026-09-18 16:20:16.937019975 +0000 UTC m=+288.416993520"
Sep 18 16:20:16.954507 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:16.954458    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pgftx" podStartSLOduration=252.173023035 podStartE2EDuration="4m13.954441387s" totalImagesPullingTime="1.781418352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:16.953452088 +0000 UTC m=+288.433425637" watchObservedRunningTime="2026-09-18 16:20:16.954441387 +0000 UTC m=+288.434414948"
Sep 18 16:20:19.052661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:19.052624    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:19.144751 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:19.144714    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:20.793566 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:20.793537    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:21.992745 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:20:21.992703    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fxk2s" podUID="023a0f44-e934-492c-a9fd-c88a77e45000"
Sep 18 16:20:23.263051 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:23.263007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.355244 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:23.355211    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.617686 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:23.617605    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.728082 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:23.728052    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.853367 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:23.853336    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:24.038314 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:24.038276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:24.158684 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:24.158653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:25.535139 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:25.535083    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:20:25.537548 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:25.537528    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0953c73e-e2c8-4e23-9917-c19c5325f849-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-s46vk\" (UID: \"0953c73e-e2c8-4e23-9917-c19c5325f849\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:20:25.784174 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:25.784135    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"
Sep 18 16:20:26.059064 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:26.058990    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk"]
Sep 18 16:20:26.062009 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:26.061983    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0953c73e_e2c8_4e23_9917_c19c5325f849.slice/crio-a6a7d8f1ded0f8265594959a98608a02833bfc19ab9fff4c351ea5d4d586fff0 WatchSource:0}: Error finding container a6a7d8f1ded0f8265594959a98608a02833bfc19ab9fff4c351ea5d4d586fff0: Status 404 returned error can't find the container with id a6a7d8f1ded0f8265594959a98608a02833bfc19ab9fff4c351ea5d4d586fff0
Sep 18 16:20:26.926348 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:26.926269    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgftx"
Sep 18 16:20:26.956018 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:26.955971    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk" event={"ID":"0953c73e-e2c8-4e23-9917-c19c5325f849","Type":"ContainerStarted","Data":"a6a7d8f1ded0f8265594959a98608a02833bfc19ab9fff4c351ea5d4d586fff0"}
Sep 18 16:20:27.960457 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:27.960410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk" event={"ID":"0953c73e-e2c8-4e23-9917-c19c5325f849","Type":"ContainerStarted","Data":"6fa526f2f56533a37bd989afa05c63febe659922a730f6ad8795e41d26f57c25"}
Sep 18 16:20:27.979580 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:27.979520    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-s46vk" podStartSLOduration=33.199654221 podStartE2EDuration="34.97949816s" totalImagesPullingTime="1.779843939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:27.977794352 +0000 UTC m=+299.457767899" watchObservedRunningTime="2026-09-18 16:20:27.97949816 +0000 UTC m=+299.459471724"
Sep 18 16:20:28.166962 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.166881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.271875 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.271839    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.351626 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.351592    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.431599 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.431507    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.944263 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.944227    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:20:28.945064 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.945035    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:20:28.951996 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:28.951969    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:20:31.363747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.363714    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.385737 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.385694    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"]
Sep 18 16:20:31.388726 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.388707    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.392667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.392636    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 16:20:31.392807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.392789    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mr4lv\""
Sep 18 16:20:31.393145 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.393129    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 16:20:31.393213 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.393147    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:20:31.399772 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.399749    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"]
Sep 18 16:20:31.513321 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.513283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.586444 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.586393    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.586593 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.586456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.586593 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.586542    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st86z\" (UniqueName: \"kubernetes.io/projected/42838fd5-f64e-4be8-8665-8969f844f13f-kube-api-access-st86z\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.586702 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.586641    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42838fd5-f64e-4be8-8665-8969f844f13f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.590062 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.590039    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.687721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.687640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42838fd5-f64e-4be8-8665-8969f844f13f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.687721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.687691    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.687928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.687842    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.687928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.687900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st86z\" (UniqueName: \"kubernetes.io/projected/42838fd5-f64e-4be8-8665-8969f844f13f-kube-api-access-st86z\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.688330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.688311    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42838fd5-f64e-4be8-8665-8969f844f13f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.690042 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.690020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.690135 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.690119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42838fd5-f64e-4be8-8665-8969f844f13f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.696502 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.696480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st86z\" (UniqueName: \"kubernetes.io/projected/42838fd5-f64e-4be8-8665-8969f844f13f-kube-api-access-st86z\") pod \"prometheus-operator-7bd5d56d6-w9pfz\" (UID: \"42838fd5-f64e-4be8-8665-8969f844f13f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.698095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.698081    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"
Sep 18 16:20:31.969337 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.969313    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz"]
Sep 18 16:20:31.971512 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:31.971474    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42838fd5_f64e_4be8_8665_8969f844f13f.slice/crio-d88537ac4d8e06182fe5dd35230e2df0c6b4d8fc8b43d193ed49735fd5fd9e3a WatchSource:0}: Error finding container d88537ac4d8e06182fe5dd35230e2df0c6b4d8fc8b43d193ed49735fd5fd9e3a: Status 404 returned error can't find the container with id d88537ac4d8e06182fe5dd35230e2df0c6b4d8fc8b43d193ed49735fd5fd9e3a
Sep 18 16:20:31.973398 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:31.973381    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:20:32.979268 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:32.979229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz" event={"ID":"42838fd5-f64e-4be8-8665-8969f844f13f","Type":"ContainerStarted","Data":"d88537ac4d8e06182fe5dd35230e2df0c6b4d8fc8b43d193ed49735fd5fd9e3a"}
Sep 18 16:20:32.990580 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:32.990548    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:20:33.423753 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:33.423710    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:33.495021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:33.494940    2554 patch_prober.go:28] interesting pod/image-registry-579d5c6f56-fxx4q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:20:33.495021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:33.494993    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:20:33.984781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:33.984744    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz" event={"ID":"42838fd5-f64e-4be8-8665-8969f844f13f","Type":"ContainerStarted","Data":"4de069ba8ddf40d8166aa1a831edccff7f118d414661d195f54e531dccbd04cb"}
Sep 18 16:20:33.985159 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:33.984794    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz" event={"ID":"42838fd5-f64e-4be8-8665-8969f844f13f","Type":"ContainerStarted","Data":"83ec998924d381999e0d8d2b522dfdc1037fc59293755ec803b450c25a0fe4f8"}
Sep 18 16:20:34.003837 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:34.003791    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w9pfz" podStartSLOduration=1.814369825 podStartE2EDuration="3.003777371s" totalImagesPullingTime="1.189407546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:34.002625528 +0000 UTC m=+305.482599074" watchObservedRunningTime="2026-09-18 16:20:34.003777371 +0000 UTC m=+305.483750917"
Sep 18 16:20:34.907526 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:34.907494    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:20:35.190879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.190792    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.259868 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.259838    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.489717 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.489684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.720120 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.720085    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.750077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.749955    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l5bbs"]
Sep 18 16:20:35.753307 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.753286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.755548 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.755529    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:20:35.757512 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.757496    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:20:35.757700 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.757681    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57lgg\""
Sep 18 16:20:35.758383 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.758364    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:20:35.813437 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-wtmp\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813437 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813441    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lx8\" (UniqueName: \"kubernetes.io/projected/63849575-d8c1-4f76-8e0f-46d5268ea942-kube-api-access-c4lx8\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813521    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-root\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813570    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-textfile\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-sys\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-tls\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813630    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.813661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.813657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-metrics-client-ca\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.827287 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.827260    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.914061 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914026    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-root\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-textfile\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-sys\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914148    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-sys\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-root\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-tls\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914228    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914271    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-metrics-client-ca\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914297    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-wtmp\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914323    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lx8\" (UniqueName: \"kubernetes.io/projected/63849575-d8c1-4f76-8e0f-46d5268ea942-kube-api-access-c4lx8\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914407    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914486    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-wtmp\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914494    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-textfile\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.914919 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.914895    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.915040 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.915020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63849575-d8c1-4f76-8e0f-46d5268ea942-metrics-client-ca\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.916538 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.916509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.916646 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.916631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63849575-d8c1-4f76-8e0f-46d5268ea942-node-exporter-tls\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:35.923851 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.923830    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:35.925997 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:35.925975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lx8\" (UniqueName: \"kubernetes.io/projected/63849575-d8c1-4f76-8e0f-46d5268ea942-kube-api-access-c4lx8\") pod \"node-exporter-l5bbs\" (UID: \"63849575-d8c1-4f76-8e0f-46d5268ea942\") " pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:36.021398 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.021312    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.062830 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.062801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5bbs"
Sep 18 16:20:36.077759 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:36.077730    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63849575_d8c1_4f76_8e0f_46d5268ea942.slice/crio-495a9504e7885d6a2ac1515d84e8e7d21121673220ef2240bc8db9c0177133ba WatchSource:0}: Error finding container 495a9504e7885d6a2ac1515d84e8e7d21121673220ef2240bc8db9c0177133ba: Status 404 returned error can't find the container with id 495a9504e7885d6a2ac1515d84e8e7d21121673220ef2240bc8db9c0177133ba
Sep 18 16:20:36.122740 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.122700    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.224323 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.224286    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.322330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.322250    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.423462 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.423429    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.997754 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:36.997715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5bbs" event={"ID":"63849575-d8c1-4f76-8e0f-46d5268ea942","Type":"ContainerStarted","Data":"495a9504e7885d6a2ac1515d84e8e7d21121673220ef2240bc8db9c0177133ba"}
Sep 18 16:20:38.003480 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:38.003398    2554 generic.go:356] "Generic (PLEG): container finished" podID="63849575-d8c1-4f76-8e0f-46d5268ea942" containerID="1e8bc942022223dab0f445e11c02e6fefcc06901f41d6fa142a12ed61862ea58" exitCode=0
Sep 18 16:20:38.003958 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:38.003495    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5bbs" event={"ID":"63849575-d8c1-4f76-8e0f-46d5268ea942","Type":"ContainerDied","Data":"1e8bc942022223dab0f445e11c02e6fefcc06901f41d6fa142a12ed61862ea58"}
Sep 18 16:20:39.008610 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:39.008570    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5bbs" event={"ID":"63849575-d8c1-4f76-8e0f-46d5268ea942","Type":"ContainerStarted","Data":"8284e7629f9268284528a585d4e1cb93dada95629d19e58424a7d9d4534b08f8"}
Sep 18 16:20:39.009097 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:39.008630    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5bbs" event={"ID":"63849575-d8c1-4f76-8e0f-46d5268ea942","Type":"ContainerStarted","Data":"9766941dd350446233b3d7a03daa94c0e0fbe9d754d9a3b1099feb82d6dd82c7"}
Sep 18 16:20:39.032541 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:39.032484    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l5bbs" podStartSLOduration=2.788483094 podStartE2EDuration="4.032464162s" totalImagesPullingTime="1.182159527s" totalInitContainerRuntime="61.821541ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:39.030785115 +0000 UTC m=+310.510758664" watchObservedRunningTime="2026-09-18 16:20:39.032464162 +0000 UTC m=+310.512437721"
Sep 18 16:20:40.143815 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.143783    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-654dcd44d-nl4r8"]
Sep 18 16:20:40.146842 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.146816    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.149329 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.149309    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:20:40.149456 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.149365    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 16:20:40.150293 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.150277    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:20:40.150460 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.150408    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9ngbw\""
Sep 18 16:20:40.150460 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.150433    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3iqo9c3hmdrq4\""
Sep 18 16:20:40.150590 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.150489    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:20:40.157185 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.157159    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654dcd44d-nl4r8"]
Sep 18 16:20:40.243336 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243293    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-tls\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243336 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-client-certs\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243546 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243383    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-client-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243546 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243452    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqm4\" (UniqueName: \"kubernetes.io/projected/ef681765-8424-45d8-ba0d-b3797d7fbfe1-kube-api-access-8vqm4\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243546 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243546 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef681765-8424-45d8-ba0d-b3797d7fbfe1-audit-log\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.243688 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.243565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-metrics-server-audit-profiles\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344595    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-tls\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-client-certs\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-client-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344725    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqm4\" (UniqueName: \"kubernetes.io/projected/ef681765-8424-45d8-ba0d-b3797d7fbfe1-kube-api-access-8vqm4\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.344879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344773    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef681765-8424-45d8-ba0d-b3797d7fbfe1-audit-log\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.345064 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.344908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-metrics-server-audit-profiles\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.345263 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.345226    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef681765-8424-45d8-ba0d-b3797d7fbfe1-audit-log\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.345606 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.345582    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.345963 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.345942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef681765-8424-45d8-ba0d-b3797d7fbfe1-metrics-server-audit-profiles\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.347223 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.347197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-client-ca-bundle\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.347312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.347225    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-client-certs\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.347312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.347254    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef681765-8424-45d8-ba0d-b3797d7fbfe1-secret-metrics-server-tls\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.353641 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.353621    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqm4\" (UniqueName: \"kubernetes.io/projected/ef681765-8424-45d8-ba0d-b3797d7fbfe1-kube-api-access-8vqm4\") pod \"metrics-server-654dcd44d-nl4r8\" (UID: \"ef681765-8424-45d8-ba0d-b3797d7fbfe1\") " pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.456301 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.456207    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:20:40.503325 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.503283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:40.679946 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.679913    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:40.730472 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.730442    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654dcd44d-nl4r8"]
Sep 18 16:20:40.733369 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:40.733343    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef681765_8424_45d8_ba0d_b3797d7fbfe1.slice/crio-44668014fecc7b35307f90e1a1f8f7dd3708ee8427677233b2a8a8973ad9ecea WatchSource:0}: Error finding container 44668014fecc7b35307f90e1a1f8f7dd3708ee8427677233b2a8a8973ad9ecea: Status 404 returned error can't find the container with id 44668014fecc7b35307f90e1a1f8f7dd3708ee8427677233b2a8a8973ad9ecea
Sep 18 16:20:40.978405 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.978372    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bcd745757-h2jkt"]
Sep 18 16:20:40.982750 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.982699    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:40.986371 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.986349    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 16:20:40.986535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.986520    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q5dhr\""
Sep 18 16:20:40.986629 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.986613    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 16:20:40.986865 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.986847    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 16:20:40.986964 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:40.986883    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 16:20:41.000107 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.000083    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 16:20:41.000694 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.000679    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 16:20:41.009130 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.008959    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bcd745757-h2jkt"]
Sep 18 16:20:41.016732 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.016702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8" event={"ID":"ef681765-8424-45d8-ba0d-b3797d7fbfe1","Type":"ContainerStarted","Data":"44668014fecc7b35307f90e1a1f8f7dd3708ee8427677233b2a8a8973ad9ecea"}
Sep 18 16:20:41.053881 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.053851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-serving-certs-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054002 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.053892    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-metrics-client-ca\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054002 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.053916    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054002 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.053946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054119 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.054012    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-federate-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054119 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.054040    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69c8\" (UniqueName: \"kubernetes.io/projected/39b86d54-e098-415f-b586-6e72eb9eb34e-kube-api-access-f69c8\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054119 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.054059    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.054119 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.054108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155034 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.154998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155048    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-serving-certs-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155080    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-metrics-client-ca\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155104    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155124    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-federate-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155160    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69c8\" (UniqueName: \"kubernetes.io/projected/39b86d54-e098-415f-b586-6e72eb9eb34e-kube-api-access-f69c8\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155185    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.155960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155936    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-serving-certs-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.156020 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.155934    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-metrics-client-ca\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.156650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.156622    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.157675 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.157641    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-telemeter-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.157839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.157819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.157990 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.157975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-federate-client-tls\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.158028 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.157976    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39b86d54-e098-415f-b586-6e72eb9eb34e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.166779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.166761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69c8\" (UniqueName: \"kubernetes.io/projected/39b86d54-e098-415f-b586-6e72eb9eb34e-kube-api-access-f69c8\") pod \"telemeter-client-7bcd745757-h2jkt\" (UID: \"39b86d54-e098-415f-b586-6e72eb9eb34e\") " pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.247128 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.247044    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:41.291606 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.291570    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt"
Sep 18 16:20:41.328560 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.328488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:41.579609 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:20:41.579463    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b86d54_e098_415f_b586_6e72eb9eb34e.slice/crio-146f0242b10a116ab5afc483d2529ef9106478b05432ba7329749a33b692bbef WatchSource:0}: Error finding container 146f0242b10a116ab5afc483d2529ef9106478b05432ba7329749a33b692bbef: Status 404 returned error can't find the container with id 146f0242b10a116ab5afc483d2529ef9106478b05432ba7329749a33b692bbef
Sep 18 16:20:41.581007 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:41.580982    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bcd745757-h2jkt"]
Sep 18 16:20:42.022500 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:42.022458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt" event={"ID":"39b86d54-e098-415f-b586-6e72eb9eb34e","Type":"ContainerStarted","Data":"146f0242b10a116ab5afc483d2529ef9106478b05432ba7329749a33b692bbef"}
Sep 18 16:20:42.244588 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:42.244564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:42.347593 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:42.347558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:43.029478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:43.029433    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8" event={"ID":"ef681765-8424-45d8-ba0d-b3797d7fbfe1","Type":"ContainerStarted","Data":"6512f07cf27f21283e229decb2a005601e591a1b3059d44e9836fa990aa87cc0"}
Sep 18 16:20:43.054042 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:43.053990    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8" podStartSLOduration=1.56641702 podStartE2EDuration="3.053971507s" totalImagesPullingTime="1.487554487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:43.053524048 +0000 UTC m=+314.533497595" watchObservedRunningTime="2026-09-18 16:20:43.053971507 +0000 UTC m=+314.533945052"
Sep 18 16:20:44.034576 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:44.034548    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt" event={"ID":"39b86d54-e098-415f-b586-6e72eb9eb34e","Type":"ContainerStarted","Data":"64f77b249fb04353c3e1c33be5ab7d951a4bfdd4d5f48158e8be6188b4c6de1e"}
Sep 18 16:20:45.039969 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:45.039931    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt" event={"ID":"39b86d54-e098-415f-b586-6e72eb9eb34e","Type":"ContainerStarted","Data":"35bb590b33b927a639cf9d8dff97c1a316dd9bd54d74e920bcdcaad9971511f7"}
Sep 18 16:20:45.040323 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:45.039989    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt" event={"ID":"39b86d54-e098-415f-b586-6e72eb9eb34e","Type":"ContainerStarted","Data":"b249578dd4e878ead0ae2305aae8aacd018ea9ff5a492d550df303a0d598da50"}
Sep 18 16:20:45.066680 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:45.066633    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bcd745757-h2jkt" podStartSLOduration=1.888634857 podStartE2EDuration="5.066618331s" totalImagesPullingTime="3.177983474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:45.065943151 +0000 UTC m=+316.545916696" watchObservedRunningTime="2026-09-18 16:20:45.066618331 +0000 UTC m=+316.546591876"
Sep 18 16:20:46.390590 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:46.390555    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:20:46.484064 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:46.484030    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:46.585608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:46.585571    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:46.685875 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:46.685792    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:49.086931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:49.086897    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:49.155068 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:49.155037    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:51.949311 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:51.949280    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.045522 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.045490    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.105215 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.105184    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.253779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.253751    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.438058 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.438022    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.539275 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.539190    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.636620 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.636592    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:52.737441 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:52.737398    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:53.055561 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:53.055528    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:53.142772 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:53.142735    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:53.350955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:53.350887    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:53.457961 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:53.457933    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:53.841772 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:53.841742    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:54.048094 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:54.048067    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:54.148632 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:54.148552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:54.862461 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:54.862411    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:58.327393 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:58.327358    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:58.440254 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:20:58.440220    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:00.456607 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:00.456568    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:21:00.456995 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:00.456629    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:21:01.251006 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:01.250972    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:02.100727 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:02.100691    2554 generic.go:356] "Generic (PLEG): container finished" podID="8c182ae2-4b16-467a-8f88-9768ce080373" containerID="419f54e19c1a585917cf56c57912ff93a641f4fd7a1082b9cb74803d2d7be2d1" exitCode=0
Sep 18 16:21:02.101080 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:02.100764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6" event={"ID":"8c182ae2-4b16-467a-8f88-9768ce080373","Type":"ContainerDied","Data":"419f54e19c1a585917cf56c57912ff93a641f4fd7a1082b9cb74803d2d7be2d1"}
Sep 18 16:21:02.101126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:02.101102    2554 scope.go:117] "RemoveContainer" containerID="419f54e19c1a585917cf56c57912ff93a641f4fd7a1082b9cb74803d2d7be2d1"
Sep 18 16:21:03.105697 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:03.105661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mrcj6" event={"ID":"8c182ae2-4b16-467a-8f88-9768ce080373","Type":"ContainerStarted","Data":"d22960ce4cb9a8ff862162e8d6eb8422df56008123c70c5d65f73105eb809da8"}
Sep 18 16:21:03.849193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:03.849161    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:03.910530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:03.910500    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:06.653402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:06.653371    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:06.716718 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:06.716686    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.232564 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:08.232533    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.290555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:08.290525    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.561048 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:08.560979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.662144 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:08.662114    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:11.409729 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.409690    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerName="registry" containerID="cri-o://60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8" gracePeriod=30
Sep 18 16:21:11.651498 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.651472    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:21:11.809952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.809913    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.809962    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810025    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810051    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810118    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810389 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810152    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsm9t\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810389 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810187    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token\") pod \"69a8bf97-52a6-499a-9b63-2567dac930e3\" (UID: \"69a8bf97-52a6-499a-9b63-2567dac930e3\") "
Sep 18 16:21:11.810521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810395    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:11.810521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.810495    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:11.812730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.812678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t" (OuterVolumeSpecName: "kube-api-access-hsm9t") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "kube-api-access-hsm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:11.812850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.812730    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:11.812850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.812776    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:11.812850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.812801    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:11.812949 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.812911    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:11.818907 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.818880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "69a8bf97-52a6-499a-9b63-2567dac930e3" (UID: "69a8bf97-52a6-499a-9b63-2567dac930e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:11.910761 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910723    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-tls\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910761 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910754    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69a8bf97-52a6-499a-9b63-2567dac930e3-ca-trust-extracted\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910761 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910767    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm9t\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-kube-api-access-hsm9t\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910777    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a8bf97-52a6-499a-9b63-2567dac930e3-bound-sa-token\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910786    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-registry-certificates\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910794    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69a8bf97-52a6-499a-9b63-2567dac930e3-trusted-ca\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910803    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-image-registry-private-configuration\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:11.910965 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:11.910811    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69a8bf97-52a6-499a-9b63-2567dac930e3-installation-pull-secrets\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:21:12.137351 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.137254    2554 generic.go:356] "Generic (PLEG): container finished" podID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerID="60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8" exitCode=0
Sep 18 16:21:12.137351 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.137297    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q"
Sep 18 16:21:12.137351 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.137338    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" event={"ID":"69a8bf97-52a6-499a-9b63-2567dac930e3","Type":"ContainerDied","Data":"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"}
Sep 18 16:21:12.137635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.137384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579d5c6f56-fxx4q" event={"ID":"69a8bf97-52a6-499a-9b63-2567dac930e3","Type":"ContainerDied","Data":"f214dd5de82a511586b514cf02bc20b42472fea552151da09e65b8102166db24"}
Sep 18 16:21:12.137635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.137406    2554 scope.go:117] "RemoveContainer" containerID="60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"
Sep 18 16:21:12.146365 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.146345    2554 scope.go:117] "RemoveContainer" containerID="60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"
Sep 18 16:21:12.146673 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:21:12.146654    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8\": container with ID starting with 60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8 not found: ID does not exist" containerID="60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"
Sep 18 16:21:12.146730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.146680    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8"} err="failed to get container status \"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8\": rpc error: code = NotFound desc = could not find container \"60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8\": container with ID starting with 60cabde3025825ad33090f0b6011b66c1bedbc1bb20be5405d641ad2036106e8 not found: ID does not exist"
Sep 18 16:21:12.159685 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.159657    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:21:12.165589 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.165567    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579d5c6f56-fxx4q"]
Sep 18 16:21:12.842150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.842119    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:12.906260 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.906219    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:12.995341 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:12.995304    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" path="/var/lib/kubelet/pods/69a8bf97-52a6-499a-9b63-2567dac930e3/volumes"
Sep 18 16:21:13.444261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.442362    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:21:13.444261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.442935    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerName="registry"
Sep 18 16:21:13.444261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.442954    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerName="registry"
Sep 18 16:21:13.444261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.443069    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a8bf97-52a6-499a-9b63-2567dac930e3" containerName="registry"
Sep 18 16:21:13.448625 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.448598    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.452338 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452317    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:21:13.452489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452360    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:21:13.452489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452455    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:21:13.452601 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452580    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdwv8\""
Sep 18 16:21:13.452655 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452600    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:21:13.452655 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452646    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:21:13.452752 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452671    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:21:13.452752 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.452700    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:21:13.454991 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.454966    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:21:13.457702 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.457680    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:21:13.523712 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523671    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523712 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523716    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523734    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zq8\" (UniqueName: \"kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.523933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.523884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625276 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625276 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625279    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zq8\" (UniqueName: \"kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625303    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.625568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.625439    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.626093 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.626053    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.626208 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.626145    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.626260 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.626234    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.626599 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.626574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.627918 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.627890    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.627997 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.627929    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.633816 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.633793    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zq8\" (UniqueName: \"kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8\") pod \"console-bf7bdc5b9-57zvl\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") " pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:13.758715 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:13.758669    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:14.030568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:14.030492    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:21:14.034259 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:21:14.034233    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a02151_e8d5_4b28_9539_b238e899b72b.slice/crio-53317c1986dc37192e73cccb11c9c1d535ceb321c3427ef8c0805b907c14521d WatchSource:0}: Error finding container 53317c1986dc37192e73cccb11c9c1d535ceb321c3427ef8c0805b907c14521d: Status 404 returned error can't find the container with id 53317c1986dc37192e73cccb11c9c1d535ceb321c3427ef8c0805b907c14521d
Sep 18 16:21:14.147139 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:14.147098    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf7bdc5b9-57zvl" event={"ID":"c7a02151-e8d5-4b28-9539-b238e899b72b","Type":"ContainerStarted","Data":"53317c1986dc37192e73cccb11c9c1d535ceb321c3427ef8c0805b907c14521d"}
Sep 18 16:21:17.703127 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:17.703088    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:17.779498 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:17.779472    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:18.164695 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:18.164657    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf7bdc5b9-57zvl" event={"ID":"c7a02151-e8d5-4b28-9539-b238e899b72b","Type":"ContainerStarted","Data":"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"}
Sep 18 16:21:18.181699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:18.181641    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bf7bdc5b9-57zvl" podStartSLOduration=1.383857425 podStartE2EDuration="5.181622027s" totalImagesPullingTime="3.797764602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:18.181175199 +0000 UTC m=+349.661148746" watchObservedRunningTime="2026-09-18 16:21:18.181622027 +0000 UTC m=+349.661595574"
Sep 18 16:21:19.266693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:19.266649    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:19.331519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:19.331489    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:20.461995 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:20.461967    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:21:20.465841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:20.465814    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-654dcd44d-nl4r8"
Sep 18 16:21:20.904676 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:20.904646    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:21.003683 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:21.003639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:23.231219 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:23.231144    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:23.307918 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:23.307890    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:23.759549 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:23.759520    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:23.759703 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:23.759599    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:23.764609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:23.764585    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:24.188821 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:24.188744    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:21:24.348726 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:24.348697    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-s46vk_0953c73e-e2c8-4e23-9917-c19c5325f849/cluster-monitoring-operator/0.log"
Sep 18 16:21:25.147638 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:25.147606    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-654dcd44d-nl4r8_ef681765-8424-45d8-ba0d-b3797d7fbfe1/metrics-server/0.log"
Sep 18 16:21:25.659689 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:25.659656    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:25.723331 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:25.723307    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:26.667992 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:26.667958    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:26.736048 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:26.735961    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:26.749735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:26.749715    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/init-textfile/0.log"
Sep 18 16:21:26.953035 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:26.953008    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/node-exporter/0.log"
Sep 18 16:21:27.148291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:27.148224    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/kube-rbac-proxy/0.log"
Sep 18 16:21:27.948884 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:27.948857    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w9pfz_42838fd5-f64e-4be8-8665-8969f844f13f/prometheus-operator/0.log"
Sep 18 16:21:28.147805 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:28.147779    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w9pfz_42838fd5-f64e-4be8-8665-8969f844f13f/kube-rbac-proxy/0.log"
Sep 18 16:21:28.548069 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:28.548034    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/telemeter-client/0.log"
Sep 18 16:21:28.747720 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:28.747687    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/reload/0.log"
Sep 18 16:21:28.948010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:28.947986    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/kube-rbac-proxy/0.log"
Sep 18 16:21:29.147871 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:29.147843    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7m6xk_3ca2d477-1869-49a3-adc2-7f8da9249342/networking-console-plugin/0.log"
Sep 18 16:21:29.948033 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:29.948007    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bf7bdc5b9-57zvl_c7a02151-e8d5-4b28-9539-b238e899b72b/console/0.log"
Sep 18 16:21:30.268183 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:30.268155    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:30.320988 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:30.320955    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:30.548046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:30.547968    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6wbnf_56cf29d2-ddb5-4c98-a9d8-693d27a505d7/serve-healthcheck-canary/0.log"
Sep 18 16:21:32.034813 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:32.034786    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:32.136671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:32.136641    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:32.208548 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:32.208523    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:42.964143 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:42.964091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:21:42.966519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:42.966497    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a0f44-e934-492c-a9fd-c88a77e45000-metrics-certs\") pod \"network-metrics-daemon-fxk2s\" (UID: \"023a0f44-e934-492c-a9fd-c88a77e45000\") " pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:21:43.194309 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:43.194270    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:21:43.202793 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:43.202770    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxk2s"
Sep 18 16:21:43.471885 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:43.471852    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fxk2s"]
Sep 18 16:21:43.474937 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:21:43.474907    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023a0f44_e934_492c_a9fd_c88a77e45000.slice/crio-a4faabe7319869ea9612aee54daf067e1bb6f928cd78635469730782798762c6 WatchSource:0}: Error finding container a4faabe7319869ea9612aee54daf067e1bb6f928cd78635469730782798762c6: Status 404 returned error can't find the container with id a4faabe7319869ea9612aee54daf067e1bb6f928cd78635469730782798762c6
Sep 18 16:21:44.266359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:44.266318    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxk2s" event={"ID":"023a0f44-e934-492c-a9fd-c88a77e45000","Type":"ContainerStarted","Data":"a4faabe7319869ea9612aee54daf067e1bb6f928cd78635469730782798762c6"}
Sep 18 16:21:45.272150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:45.272115    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxk2s" event={"ID":"023a0f44-e934-492c-a9fd-c88a77e45000","Type":"ContainerStarted","Data":"43a6189055ea23064d3e2f607fd9e42c812d8773fce06a9e9011891a7b4de610"}
Sep 18 16:21:45.272534 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:45.272165    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxk2s" event={"ID":"023a0f44-e934-492c-a9fd-c88a77e45000","Type":"ContainerStarted","Data":"ea97b0434934426e3da24f65c9bfc9cf6b53c01a8c3eaf7f34f33900acc5bac1"}
Sep 18 16:21:45.290567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:45.290516    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fxk2s" podStartSLOduration=375.247263233 podStartE2EDuration="6m16.290499735s" totalImagesPullingTime="1.043236502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:45.289475608 +0000 UTC m=+376.769449154" watchObservedRunningTime="2026-09-18 16:21:45.290499735 +0000 UTC m=+376.770473281"
Sep 18 16:21:45.403976 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:45.403929    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:45.505429 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:45.505393    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:53.224794 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:53.224751    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:53.285739 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:21:53.285696    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:53.570717 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:22:53.570632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:02.370555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:24:02.370524    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:15.778444 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:24:15.778397    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:28.969616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:28.969586    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:25:28.970167 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:28.969628    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:25:39.836250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.836217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:39.940167 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.940131    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:25:39.944369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.943724    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:39.946578 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946544    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:25:39.946578 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946562    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:25:39.946754 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946614    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:25:39.947013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946873    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:25:39.947013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946900    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:25:39.947013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946914    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:25:39.947013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946902    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dtswb\""
Sep 18 16:25:39.947013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.946879    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:25:39.947295 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.947281    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:25:39.953987 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.953971    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:25:39.966654 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:39.966626    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:25:40.115406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115360    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115657 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115657 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115506    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7qh\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-kube-api-access-wf7qh\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115657 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115657 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115562    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115657 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115629    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-web-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-config-out\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115689    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115721    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.115823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.115745    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.216788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.216788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7qh\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-kube-api-access-wf7qh\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216909    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216946    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-web-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216975    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-config-out\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.216995    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.217015    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217469 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:25:40.217020    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 16:25:40.217469 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.217036    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217469 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.217070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217469 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.217092    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.217469 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:25:40.217123    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls podName:145151be-5eab-4898-b428-dc9cac9250ce nodeName:}" failed. No retries permitted until 2026-09-18 16:25:40.717095342 +0000 UTC m=+612.197068887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "145151be-5eab-4898-b428-dc9cac9250ce") : secret "alertmanager-main-tls" not found
Sep 18 16:25:40.217845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.217820    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.219095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.219064    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.219694 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.219668    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/145151be-5eab-4898-b428-dc9cac9250ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220183 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220281 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220220    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220500    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-web-config\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220592 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220520    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220629 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220595    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220629 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220602    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.220975 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.220956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.221041 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.221022    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/145151be-5eab-4898-b428-dc9cac9250ce-config-out\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.227149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.227126    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7qh\" (UniqueName: \"kubernetes.io/projected/145151be-5eab-4898-b428-dc9cac9250ce-kube-api-access-wf7qh\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.722062 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.721991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.724620 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.724586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/145151be-5eab-4898-b428-dc9cac9250ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"145151be-5eab-4898-b428-dc9cac9250ce\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:40.853365 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:40.853323    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:25:41.111479 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:41.111385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:41.137006 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:41.136977    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:25:41.138133 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:25:41.138111    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145151be_5eab_4898_b428_dc9cac9250ce.slice/crio-f70b94b31a71f2e31ee80e54d6f4754683de17f8b85dd13567ac303934a6b993 WatchSource:0}: Error finding container f70b94b31a71f2e31ee80e54d6f4754683de17f8b85dd13567ac303934a6b993: Status 404 returned error can't find the container with id f70b94b31a71f2e31ee80e54d6f4754683de17f8b85dd13567ac303934a6b993
Sep 18 16:25:41.836624 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:41.836584    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:41.937409 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:41.937365    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:42.064451 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.064396    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:42.094767 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.094683    2554 generic.go:356] "Generic (PLEG): container finished" podID="145151be-5eab-4898-b428-dc9cac9250ce" containerID="a0452fd3285111c6a30d8fdf02c7c36a6603185caf8bdf5ae451e15bfdaab7d6" exitCode=0
Sep 18 16:25:42.094767 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.094726    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerDied","Data":"a0452fd3285111c6a30d8fdf02c7c36a6603185caf8bdf5ae451e15bfdaab7d6"}
Sep 18 16:25:42.094993 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.094792    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"f70b94b31a71f2e31ee80e54d6f4754683de17f8b85dd13567ac303934a6b993"}
Sep 18 16:25:42.095978 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.095962    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:25:42.391082 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:42.390992    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:44.105296 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:44.105264    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"88c63724b93e2d3d855146a47b10a24a190b5d651f5b575d5447aa78f95c4072"}
Sep 18 16:25:44.105671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:44.105328    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"079da050822970004fb35e9763aa344e93624730b6ad5d511c4b18a6f1612240"}
Sep 18 16:25:44.105671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:44.105355    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"abe7c780301503eaa82c689e442b88c69ac461e9242fa982d01413a2f9e1d3c1"}
Sep 18 16:25:44.105671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:44.105379    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"f441372033690db6ced26d93b0d12ce0690887a48207fcb538dbb40c5b08c6f6"}
Sep 18 16:25:44.105671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:44.105402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"8cdf3fbb57d0c27ee12ec0fd38ce15ecacf9f779db169768ff849aac5029a1d3"}
Sep 18 16:25:45.093825 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.093785    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:45.101890 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.101854    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:25:45.110506 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.109232    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.113079 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113052    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:25:45.113236 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113119    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:25:45.113236 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113119    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a605rlf7d46fh\""
Sep 18 16:25:45.113363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113296    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:25:45.113363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113330    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:25:45.114063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113880    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sdx86\""
Sep 18 16:25:45.114063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113942    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:25:45.114063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.113996    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:25:45.114063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114010    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:25:45.114312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114154    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:25:45.114312 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114172    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:25:45.114558 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114538    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:25:45.114632 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114609    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:25:45.114800 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.114776    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:25:45.117841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.117818    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:25:45.119928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.119904    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"145151be-5eab-4898-b428-dc9cac9250ce","Type":"ContainerStarted","Data":"c0e24660fcc2ae5db29e6295dd14f1fc9dcea8f612a805478021f4b7d303452c"}
Sep 18 16:25:45.120996 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.120973    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:25:45.127374 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.127348    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:25:45.162327 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162287    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162327 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162357    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162448    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162513    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162714    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.162972 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162954    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.162985    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163152 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163064    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcstb\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-kube-api-access-fcstb\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163152 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163159    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163207    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163234    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.163394 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.163297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.186876 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.186805    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.15654498 podStartE2EDuration="6.186780668s" totalImagesPullingTime="2.015611979s" totalInitContainerRuntime="14.623709ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:45.184576798 +0000 UTC m=+616.664550344" watchObservedRunningTime="2026-09-18 16:25:45.186780668 +0000 UTC m=+616.666754229"
Sep 18 16:25:45.264779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264924    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.264971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264972    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.264998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265023    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265098    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265690 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265662    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265769 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265769 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265743    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265811    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcstb\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-kube-api-access-fcstb\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265852    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.265873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265866    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.266114 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265925    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.266114 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265962    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.266114 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.265992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.266114 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.266028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.266339 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.266315    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.267443 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.266947    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.267443 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.267081    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.270557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.267775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.270557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.268997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.270557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.269445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.271105 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.271085    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.271198 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.271154    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.271356 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.271292    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a23802-3f52-44bf-98c1-36add00a9c41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.271699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.271678    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a23802-3f52-44bf-98c1-36add00a9c41-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.271851 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.271809    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.272144 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.272124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.272238 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.272216    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.272289 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.272271    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.273015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.272983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.273679 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.273658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a23802-3f52-44bf-98c1-36add00a9c41-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.279743 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.279712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcstb\" (UniqueName: \"kubernetes.io/projected/07a23802-3f52-44bf-98c1-36add00a9c41-kube-api-access-fcstb\") pod \"prometheus-k8s-0\" (UID: \"07a23802-3f52-44bf-98c1-36add00a9c41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.422293 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.422195    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:25:45.715399 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:45.715321    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:25:45.718761 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:25:45.718733    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a23802_3f52_44bf_98c1_36add00a9c41.slice/crio-45709b28eb9a587e5572aa3aa2c0b6b726f08bcf956e5d690e715ccb520515db WatchSource:0}: Error finding container 45709b28eb9a587e5572aa3aa2c0b6b726f08bcf956e5d690e715ccb520515db: Status 404 returned error can't find the container with id 45709b28eb9a587e5572aa3aa2c0b6b726f08bcf956e5d690e715ccb520515db
Sep 18 16:25:46.125831 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:46.125797    2554 generic.go:356] "Generic (PLEG): container finished" podID="07a23802-3f52-44bf-98c1-36add00a9c41" containerID="be9510b6e525dd9e2e511cd6bbb392f859700cb3dac63fede2e245312afd20cf" exitCode=0
Sep 18 16:25:46.126292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:46.125888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerDied","Data":"be9510b6e525dd9e2e511cd6bbb392f859700cb3dac63fede2e245312afd20cf"}
Sep 18 16:25:46.126292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:46.125983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"45709b28eb9a587e5572aa3aa2c0b6b726f08bcf956e5d690e715ccb520515db"}
Sep 18 16:25:50.148116 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:50.148025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"34e0540062a2530892b7962e59ed7cefb9152828e6355e3f75556e1b402bb99f"}
Sep 18 16:25:50.148530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:50.148121    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"027ca1999d33e2f76c9d15a7cae7d1af39e18f4a16fe290d70bc6220388cf79b"}
Sep 18 16:25:51.157124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:51.157077    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"afca67eec7cef95432e0663bfb819c6c67fc5ea8d8c4278b1bf38f0208304958"}
Sep 18 16:25:52.165481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:52.165447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"ca095a0cd68f69fc7c81577ca6263ec5fb8bd6e5475f35fe436fe63b192eca12"}
Sep 18 16:25:52.165853 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:52.165515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"1929fd57fa23fea2467926cf2e45abc5badca1312bb16f5c13f5ed2f1e41e237"}
Sep 18 16:25:52.165853 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:52.165541    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a23802-3f52-44bf-98c1-36add00a9c41","Type":"ContainerStarted","Data":"35b634501ef2b742aade992acb4b36e84e6156a489b447e1d542fb0755de5c52"}
Sep 18 16:25:52.197095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:52.197024    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.373486654 podStartE2EDuration="7.196995149s" totalImagesPullingTime="4.802459913s" totalInitContainerRuntime="21.048582ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:52.193685106 +0000 UTC m=+623.673658653" watchObservedRunningTime="2026-09-18 16:25:52.196995149 +0000 UTC m=+623.676968696"
Sep 18 16:25:55.422688 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:25:55.422653    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:26:45.422661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:45.422618    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:26:45.441143 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:45.441105    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:26:45.538870 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:45.538827    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:46.389810 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:46.389782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:26:54.989627 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:54.989591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:55.090130 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:55.090097    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:57.069241 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:57.069206    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:57.176006 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:26:57.175973    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:06.994024 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:06.993995    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:07.095229 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:07.095198    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:07.740571 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:07.740531    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:27:07.859515 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:07.859485    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:07.960166 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:07.960132    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:08.019407 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:08.019331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:23.243309 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:23.243217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:23.343659 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:23.343628    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:30.103309 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:30.103272    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:30.199381 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:30.199349    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:32.761457 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:32.761387    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bf7bdc5b9-57zvl" podUID="c7a02151-e8d5-4b28-9539-b238e899b72b" containerName="console" containerID="cri-o://0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032" gracePeriod=15
Sep 18 16:27:32.998759 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:32.998735    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:27:33.083567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083475    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083532    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5zq8\" (UniqueName: \"kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083645    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083689    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.083783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.083753    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca\") pod \"c7a02151-e8d5-4b28-9539-b238e899b72b\" (UID: \"c7a02151-e8d5-4b28-9539-b238e899b72b\") "
Sep 18 16:27:33.084056 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.084023    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config" (OuterVolumeSpecName: "console-config") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:33.084122 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.084063    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:33.084122 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.084074    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:33.084309 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.084285    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca" (OuterVolumeSpecName: "service-ca") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:33.085874 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.085839    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:27:33.085991 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.085891    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:27:33.085991 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.085936    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8" (OuterVolumeSpecName: "kube-api-access-r5zq8") pod "c7a02151-e8d5-4b28-9539-b238e899b72b" (UID: "c7a02151-e8d5-4b28-9539-b238e899b72b"). InnerVolumeSpecName "kube-api-access-r5zq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:27:33.184577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184535    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-oauth-serving-cert\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184571    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zq8\" (UniqueName: \"kubernetes.io/projected/c7a02151-e8d5-4b28-9539-b238e899b72b-kube-api-access-r5zq8\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184585    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-console-config\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184596    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-trusted-ca-bundle\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184608    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-serving-cert\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184619    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a02151-e8d5-4b28-9539-b238e899b72b-service-ca\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.184814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.184629    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a02151-e8d5-4b28-9539-b238e899b72b-console-oauth-config\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:27:33.565715 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.565681    2554 generic.go:356] "Generic (PLEG): container finished" podID="c7a02151-e8d5-4b28-9539-b238e899b72b" containerID="0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032" exitCode=0
Sep 18 16:27:33.565887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.565724    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf7bdc5b9-57zvl"
Sep 18 16:27:33.565887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.565731    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf7bdc5b9-57zvl" event={"ID":"c7a02151-e8d5-4b28-9539-b238e899b72b","Type":"ContainerDied","Data":"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"}
Sep 18 16:27:33.565887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.565769    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf7bdc5b9-57zvl" event={"ID":"c7a02151-e8d5-4b28-9539-b238e899b72b","Type":"ContainerDied","Data":"53317c1986dc37192e73cccb11c9c1d535ceb321c3427ef8c0805b907c14521d"}
Sep 18 16:27:33.565887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.565788    2554 scope.go:117] "RemoveContainer" containerID="0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"
Sep 18 16:27:33.574345 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.574325    2554 scope.go:117] "RemoveContainer" containerID="0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"
Sep 18 16:27:33.574659 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:27:33.574639    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032\": container with ID starting with 0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032 not found: ID does not exist" containerID="0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"
Sep 18 16:27:33.574731 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.574667    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032"} err="failed to get container status \"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032\": rpc error: code = NotFound desc = could not find container \"0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032\": container with ID starting with 0c155a778b394349f5a44ed8167724e2d1de4fbc12d52b97dfa02fa687140032 not found: ID does not exist"
Sep 18 16:27:33.586648 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.586619    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:27:33.590613 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:33.590587    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bf7bdc5b9-57zvl"]
Sep 18 16:27:34.995126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:34.995082    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a02151-e8d5-4b28-9539-b238e899b72b" path="/var/lib/kubelet/pods/c7a02151-e8d5-4b28-9539-b238e899b72b/volumes"
Sep 18 16:27:47.401572 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:47.401530    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:47.508439 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:47.508393    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:47.582265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:27:47.582228    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:50.264702 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:28:50.264670    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:50.367043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:28:50.367017    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:50.465860 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:28:50.465822    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:24.730664 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:29:24.730577    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:24.799380 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:29:24.799352    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:59.563869 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:29:59.563835    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:05.011060 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:05.011023    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:05.116776 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:05.116740    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:11.368655 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:11.368616    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:11.715986 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:11.715903    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.022633 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.022601    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.121315 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.121276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.303741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.303656    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.403435 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.403388    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.508148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.508116    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.639009 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.638933    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.737121 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.737094    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:12.983381 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:12.983339    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:13.189594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:13.189562    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:13.288369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:13.288294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:28.995949 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:28.995920    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:30:28.996466 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:30:28.996045    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:31:03.128229 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:03.128169    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:04.470734 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:04.470698    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:14.550914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:14.550879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:14.880267 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:14.880184    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:15.845315 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:15.845282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:17.622990 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.622954    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bfd6579b6-bqphh"]
Sep 18 16:31:17.623385 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.623255    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a02151-e8d5-4b28-9539-b238e899b72b" containerName="console"
Sep 18 16:31:17.623385 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.623265    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a02151-e8d5-4b28-9539-b238e899b72b" containerName="console"
Sep 18 16:31:17.623385 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.623318    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a02151-e8d5-4b28-9539-b238e899b72b" containerName="console"
Sep 18 16:31:17.626330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.626313    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.630009 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.629986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:31:17.630160 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.629986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdwv8\""
Sep 18 16:31:17.631059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.631037    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:31:17.631179 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.631140    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:31:17.631179 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.631175    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:31:17.631321 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.631246    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:31:17.631388 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.631334    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:31:17.633278 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.633259    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:31:17.644297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.644275    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:31:17.651162 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.651132    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bfd6579b6-bqphh"]
Sep 18 16:31:17.674688 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.674688 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674687    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-oauth-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.674897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674728    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-trusted-ca-bundle\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.674897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlh7\" (UniqueName: \"kubernetes.io/projected/c6e0698a-60f5-4159-9d44-a3f5a9537461-kube-api-access-phlh7\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.674897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674799    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-oauth-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.674897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674858    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.675058 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.674904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-service-ca\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.754291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.754253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:17.775465 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-trusted-ca-bundle\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775559    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlh7\" (UniqueName: \"kubernetes.io/projected/c6e0698a-60f5-4159-9d44-a3f5a9537461-kube-api-access-phlh7\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775680 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775623    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-oauth-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775680 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775696    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-service-ca\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775729    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.775788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.775755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-oauth-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.776393 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.776362    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.776532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.776362    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-trusted-ca-bundle\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.776532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.776374    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-service-ca\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.776532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.776454    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e0698a-60f5-4159-9d44-a3f5a9537461-oauth-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.778728 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.778704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-serving-cert\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.778845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.778820    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e0698a-60f5-4159-9d44-a3f5a9537461-console-oauth-config\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.789960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.789939    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlh7\" (UniqueName: \"kubernetes.io/projected/c6e0698a-60f5-4159-9d44-a3f5a9537461-kube-api-access-phlh7\") pod \"console-5bfd6579b6-bqphh\" (UID: \"c6e0698a-60f5-4159-9d44-a3f5a9537461\") " pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:17.868362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.868326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:17.935238 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:17.935160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:18.027944 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.027913    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:18.240321 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:31:18.240295    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e0698a_60f5_4159_9d44_a3f5a9537461.slice/crio-69d76563d8ed74b6fa5628571585b943659b2efeb5217c659f4bd3fd7c0a7fa1 WatchSource:0}: Error finding container 69d76563d8ed74b6fa5628571585b943659b2efeb5217c659f4bd3fd7c0a7fa1: Status 404 returned error can't find the container with id 69d76563d8ed74b6fa5628571585b943659b2efeb5217c659f4bd3fd7c0a7fa1
Sep 18 16:31:18.242943 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.242923    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bfd6579b6-bqphh"]
Sep 18 16:31:18.363403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.363371    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:18.429227 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.429194    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfd6579b6-bqphh" event={"ID":"c6e0698a-60f5-4159-9d44-a3f5a9537461","Type":"ContainerStarted","Data":"2d380f40ee0d150fba97dd6cc7255ed5501d25393996e7379b6c3fb288d0aee6"}
Sep 18 16:31:18.429394 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.429240    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfd6579b6-bqphh" event={"ID":"c6e0698a-60f5-4159-9d44-a3f5a9537461","Type":"ContainerStarted","Data":"69d76563d8ed74b6fa5628571585b943659b2efeb5217c659f4bd3fd7c0a7fa1"}
Sep 18 16:31:18.456235 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.456185    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bfd6579b6-bqphh" podStartSLOduration=1.456168697 podStartE2EDuration="1.456168697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:18.454484917 +0000 UTC m=+949.934458464" watchObservedRunningTime="2026-09-18 16:31:18.456168697 +0000 UTC m=+949.936142243"
Sep 18 16:31:18.472622 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:18.472569    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:20.567167 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:20.567124    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:20.668072 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:20.668043    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:22.904824 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:22.904791    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:23.139636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.139578    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:23.224977 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.224943    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"]
Sep 18 16:31:23.228350 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.228334    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.231957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.231935    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fb52m\""
Sep 18 16:31:23.232088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.231939    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 16:31:23.232088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.232016    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 16:31:23.239070 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.239050    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"]
Sep 18 16:31:23.325735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.325694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.325916 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.325768    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p775l\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-kube-api-access-p775l\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.384616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.384574    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:23.426839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.426797    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.427042 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.426870    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p775l\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-kube-api-access-p775l\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.435031 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.435000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.435683 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.435661    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p775l\" (UniqueName: \"kubernetes.io/projected/7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad-kube-api-access-p775l\") pod \"cert-manager-cainjector-559cdd8c87-8zbnr\" (UID: \"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.484972 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.484885    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:23.544872 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.544837    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"
Sep 18 16:31:23.838406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.838379    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr"]
Sep 18 16:31:23.841190 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:31:23.841162    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff176c9_0c7b_4e44_bd2f_2d4ab3ee0bad.slice/crio-06a4bb2db83bce40025159e77d01ddb480eb09812e321b75d21900bebe7161fe WatchSource:0}: Error finding container 06a4bb2db83bce40025159e77d01ddb480eb09812e321b75d21900bebe7161fe: Status 404 returned error can't find the container with id 06a4bb2db83bce40025159e77d01ddb480eb09812e321b75d21900bebe7161fe
Sep 18 16:31:23.843675 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:23.843654    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:31:24.457298 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:24.457236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr" event={"ID":"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad","Type":"ContainerStarted","Data":"06a4bb2db83bce40025159e77d01ddb480eb09812e321b75d21900bebe7161fe"}
Sep 18 16:31:24.615843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:24.615761    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:24.728356 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:24.728278    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:24.855147 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:24.855107    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:27.472573 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:27.472539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr" event={"ID":"7ff176c9-0c7b-4e44-bd2f-2d4ab3ee0bad","Type":"ContainerStarted","Data":"65c4c3272723221d690548b2451b7fb3bb455ae3b6c1d50312b1702c88d7d451"}
Sep 18 16:31:27.489899 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:27.489851    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8zbnr" podStartSLOduration=1.62630199 podStartE2EDuration="4.489835616s" totalImagesPullingTime="2.863533626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:27.489517334 +0000 UTC m=+958.969490881" watchObservedRunningTime="2026-09-18 16:31:27.489835616 +0000 UTC m=+958.969809162"
Sep 18 16:31:27.935750 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:27.935663    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:27.935750 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:27.935714    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:27.940490 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:27.940465    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:28.481428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:28.481387    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bfd6579b6-bqphh"
Sep 18 16:31:28.660124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:28.660090    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:28.760995 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:28.760905    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:28.822086 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:28.822054    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:30.506384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:30.506346    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:30.628375 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:30.628337    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:31.398919 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.398886    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rznx"]
Sep 18 16:31:31.402217 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.402197    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.404646 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.404627    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hp4k6\""
Sep 18 16:31:31.410231 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.410206    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rznx"]
Sep 18 16:31:31.499839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.499808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hq7\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-kube-api-access-c6hq7\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.500021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.499875    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.600722 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.600672    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hq7\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-kube-api-access-c6hq7\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.601057 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.600762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.609725 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.609695    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.609824 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.609697    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hq7\" (UniqueName: \"kubernetes.io/projected/6c107074-eee9-4903-9682-ea1d0b89a834-kube-api-access-c6hq7\") pod \"cert-manager-55b4d954d5-4rznx\" (UID: \"6c107074-eee9-4903-9682-ea1d0b89a834\") " pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.712063 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.711978    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4rznx"
Sep 18 16:31:31.982748 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:31.982722    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rznx"]
Sep 18 16:31:31.985264 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:31:31.985234    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c107074_eee9_4903_9682_ea1d0b89a834.slice/crio-ff4a43b8ed6865c98c68dd3e9874a41227a3dd6a644855eb7cc4d65c04272a29 WatchSource:0}: Error finding container ff4a43b8ed6865c98c68dd3e9874a41227a3dd6a644855eb7cc4d65c04272a29: Status 404 returned error can't find the container with id ff4a43b8ed6865c98c68dd3e9874a41227a3dd6a644855eb7cc4d65c04272a29
Sep 18 16:31:32.496903 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:32.496868    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4rznx" event={"ID":"6c107074-eee9-4903-9682-ea1d0b89a834","Type":"ContainerStarted","Data":"8a9e6377ad9e4ab7e715515bcace11f1169331dfe0e44994d0a450dca22b7ba2"}
Sep 18 16:31:32.497082 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:32.496915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4rznx" event={"ID":"6c107074-eee9-4903-9682-ea1d0b89a834","Type":"ContainerStarted","Data":"ff4a43b8ed6865c98c68dd3e9874a41227a3dd6a644855eb7cc4d65c04272a29"}
Sep 18 16:31:32.513124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:32.513075    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4rznx" podStartSLOduration=1.513051615 podStartE2EDuration="1.513051615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:32.511961769 +0000 UTC m=+963.991935315" watchObservedRunningTime="2026-09-18 16:31:32.513051615 +0000 UTC m=+963.993025170"
Sep 18 16:31:35.569543 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:35.569509    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:35.669707 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:35.669670    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:41.936954 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:41.936916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:43.061194 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:43.061165    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:47.747748 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:47.747708    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:47.821596 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:47.821547    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:48.393482 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:48.393445    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:48.455741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:48.455704    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:48.758790 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:48.758735    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:51.875567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:51.875458    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:51.978135 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:51.978103    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:55.593015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:55.592982    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:55.652499 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:55.652459    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:58.978916 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:31:58.978877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:01.940390 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:01.940352    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:02.046987 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:02.046950    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:07.314920 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.314884    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:07.443246 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.443212    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"]
Sep 18 16:32:07.446995 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.446964    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:07.449779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.449761    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.454193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.454161    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:32:07.454193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.454180    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 16:32:07.454402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.454182    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 16:32:07.454402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.454161    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 16:32:07.461782 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.461756    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n5fqj\""
Sep 18 16:32:07.462215 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.462200    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 16:32:07.469744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.469714    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"]
Sep 18 16:32:07.495562 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.495525    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.495744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.495618    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qxn\" (UniqueName: \"kubernetes.io/projected/f4236bc3-d243-4e84-be74-d4ef205ebbec-kube-api-access-85qxn\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.495744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.495651    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4236bc3-d243-4e84-be74-d4ef205ebbec-manager-config\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.495744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.495689    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.596346 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.596252    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.596346 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.596327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qxn\" (UniqueName: \"kubernetes.io/projected/f4236bc3-d243-4e84-be74-d4ef205ebbec-kube-api-access-85qxn\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.596561 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.596351    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4236bc3-d243-4e84-be74-d4ef205ebbec-manager-config\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.596561 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.596378    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.597090 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.597058    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4236bc3-d243-4e84-be74-d4ef205ebbec-manager-config\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.598875 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.598850    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.598966 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.598907    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4236bc3-d243-4e84-be74-d4ef205ebbec-cert\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.613747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.613708    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qxn\" (UniqueName: \"kubernetes.io/projected/f4236bc3-d243-4e84-be74-d4ef205ebbec-kube-api-access-85qxn\") pod \"lws-controller-manager-6fd4df8954-4q8l9\" (UID: \"f4236bc3-d243-4e84-be74-d4ef205ebbec\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:07.760545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:07.760505    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:08.033228 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:08.033200    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"]
Sep 18 16:32:08.035764 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:32:08.035739    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4236bc3_d243_4e84_be74_d4ef205ebbec.slice/crio-a760c206ac604ab32a1bb626f0d8b25cc1dd9c3be2358a87eb415cf3894ea9f3 WatchSource:0}: Error finding container a760c206ac604ab32a1bb626f0d8b25cc1dd9c3be2358a87eb415cf3894ea9f3: Status 404 returned error can't find the container with id a760c206ac604ab32a1bb626f0d8b25cc1dd9c3be2358a87eb415cf3894ea9f3
Sep 18 16:32:08.639156 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:08.639120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9" event={"ID":"f4236bc3-d243-4e84-be74-d4ef205ebbec","Type":"ContainerStarted","Data":"a760c206ac604ab32a1bb626f0d8b25cc1dd9c3be2358a87eb415cf3894ea9f3"}
Sep 18 16:32:14.666764 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:14.666663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9" event={"ID":"f4236bc3-d243-4e84-be74-d4ef205ebbec","Type":"ContainerStarted","Data":"e686db32bebf8c5e9d5173ef7bca95c8d3bc6853e9f97fe6a2dbdc17d6ce2704"}
Sep 18 16:32:14.666764 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:14.666725    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:14.692453 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:14.692233    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9" podStartSLOduration=1.38861187 podStartE2EDuration="7.692212243s" totalImagesPullingTime="6.303600373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:14.691896518 +0000 UTC m=+1006.171870064" watchObservedRunningTime="2026-09-18 16:32:14.692212243 +0000 UTC m=+1006.172185800"
Sep 18 16:32:16.319477 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:16.319444    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:18.348800 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.348757    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"]
Sep 18 16:32:18.351756 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.351736    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:18.354919 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.354895    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 16:32:18.355092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.354895    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 16:32:18.355092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.355000    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zqf58\""
Sep 18 16:32:18.355384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.355362    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 16:32:18.368757 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.368715    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"]
Sep 18 16:32:18.493706 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.493662    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5w4j\" (UniqueName: \"kubernetes.io/projected/8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c-kube-api-access-c5w4j\") pod \"dns-operator-controller-manager-65dc8f76f5-lzmkz\" (UID: \"8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:18.594456 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.594391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5w4j\" (UniqueName: \"kubernetes.io/projected/8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c-kube-api-access-c5w4j\") pod \"dns-operator-controller-manager-65dc8f76f5-lzmkz\" (UID: \"8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:18.604567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.604503    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5w4j\" (UniqueName: \"kubernetes.io/projected/8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c-kube-api-access-c5w4j\") pod \"dns-operator-controller-manager-65dc8f76f5-lzmkz\" (UID: \"8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:18.662570 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.662537    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:18.935494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:18.935466    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"]
Sep 18 16:32:18.937905 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:32:18.937873    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb9d37a_fe35_41d1_8fb5_80ea9fd65e7c.slice/crio-c14eac2fad5ac0511c9434224714b3ac512c77f4ada5f5d9e743be299340dc61 WatchSource:0}: Error finding container c14eac2fad5ac0511c9434224714b3ac512c77f4ada5f5d9e743be299340dc61: Status 404 returned error can't find the container with id c14eac2fad5ac0511c9434224714b3ac512c77f4ada5f5d9e743be299340dc61
Sep 18 16:32:19.118074 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:19.118040    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:19.689048 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:19.689017    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz" event={"ID":"8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c","Type":"ContainerStarted","Data":"c14eac2fad5ac0511c9434224714b3ac512c77f4ada5f5d9e743be299340dc61"}
Sep 18 16:32:19.747087 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:19.747055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:20.914067 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:20.914034    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:21.169838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:21.169756    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:21.299446 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:21.299399    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:22.704933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:22.704887    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz" event={"ID":"8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c","Type":"ContainerStarted","Data":"ea76a1854c27aa26d5da058617d0805d84ba5a5a6a47355f82eb519de9811e77"}
Sep 18 16:32:22.705375 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:22.704980    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:22.726532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:22.726478    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz" podStartSLOduration=1.969547998 podStartE2EDuration="4.726460986s" totalImagesPullingTime="2.756912988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:22.725341508 +0000 UTC m=+1014.205315055" watchObservedRunningTime="2026-09-18 16:32:22.726460986 +0000 UTC m=+1014.206434532"
Sep 18 16:32:23.341109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.341077    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"]
Sep 18 16:32:23.344390 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.344374    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:23.347843 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.347814    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kscg2\""
Sep 18 16:32:23.379705 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.379668    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"]
Sep 18 16:32:23.387120 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.387093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:23.440097 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.440062    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7zb\" (UniqueName: \"kubernetes.io/projected/fbe1c95a-cf4f-4988-ae84-361190df0401-kube-api-access-4q7zb\") pod \"authorino-operator-6fc8f864f4-hvwsj\" (UID: \"fbe1c95a-cf4f-4988-ae84-361190df0401\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:23.535900 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.535867    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:23.541048 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.541024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7zb\" (UniqueName: \"kubernetes.io/projected/fbe1c95a-cf4f-4988-ae84-361190df0401-kube-api-access-4q7zb\") pod \"authorino-operator-6fc8f864f4-hvwsj\" (UID: \"fbe1c95a-cf4f-4988-ae84-361190df0401\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:23.563997 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.563968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7zb\" (UniqueName: \"kubernetes.io/projected/fbe1c95a-cf4f-4988-ae84-361190df0401-kube-api-access-4q7zb\") pod \"authorino-operator-6fc8f864f4-hvwsj\" (UID: \"fbe1c95a-cf4f-4988-ae84-361190df0401\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:23.655291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.655190    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:23.918884 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.918811    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:23.931157 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:23.931134    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"]
Sep 18 16:32:23.933795 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:32:23.933770    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe1c95a_cf4f_4988_ae84_361190df0401.slice/crio-5a7f8fbd4505f74d4a0f1cdfecf9325600cffe97ed5a006f08899a57a76b054f WatchSource:0}: Error finding container 5a7f8fbd4505f74d4a0f1cdfecf9325600cffe97ed5a006f08899a57a76b054f: Status 404 returned error can't find the container with id 5a7f8fbd4505f74d4a0f1cdfecf9325600cffe97ed5a006f08899a57a76b054f
Sep 18 16:32:24.326222 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:24.326190    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:24.716611 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:24.716525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj" event={"ID":"fbe1c95a-cf4f-4988-ae84-361190df0401","Type":"ContainerStarted","Data":"5a7f8fbd4505f74d4a0f1cdfecf9325600cffe97ed5a006f08899a57a76b054f"}
Sep 18 16:32:24.719264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:24.719245    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:25.175631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.175598    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:25.293290 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.293255    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:25.674076 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.674040    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-4q8l9"
Sep 18 16:32:25.722247 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.722208    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj" event={"ID":"fbe1c95a-cf4f-4988-ae84-361190df0401","Type":"ContainerStarted","Data":"3218e0ad2d626a52113a707cedc6e8d324761a067ac438a86c1e96922e076987"}
Sep 18 16:32:25.722406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.722349    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:25.752815 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:25.752760    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj" podStartSLOduration=1.056173267 podStartE2EDuration="2.752742906s" totalImagesPullingTime="1.696569639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:25.752408343 +0000 UTC m=+1017.232381890" watchObservedRunningTime="2026-09-18 16:32:25.752742906 +0000 UTC m=+1017.232716450"
Sep 18 16:32:29.709239 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:29.709207    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:29.816163 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:29.816128    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:33.505285 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:33.505247    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:33.618197 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:33.618166    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:33.712671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:33.712641    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-lzmkz"
Sep 18 16:32:34.724536 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:34.724501    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:34.850574 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:34.850544    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:36.729294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:36.729267    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-hvwsj"
Sep 18 16:32:37.356456 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:37.356403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:37.455972 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:37.455938    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:42.082592 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:42.082563    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:42.180489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:42.180455    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:42.282164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:42.282125    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:48.722608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:48.722574    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:48.826044 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:48.826012    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:51.918545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:51.918512    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:52.018989 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:52.018952    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:56.373718 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:56.373683    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:56.435831 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:56.435792    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:56.534196 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:56.534164    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:56.614520 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:32:56.614488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:01.490631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:01.490587    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:01.591368 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:01.591335    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:01.690162 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:01.690131    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:11.745565 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:11.745515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:11.807833 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:11.807801    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:19.680919 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.680887    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:19.686876 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.686845    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:19.691086 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.691069    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.693758 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.693731    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 16:33:19.693873 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.693780    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-24526\""
Sep 18 16:33:19.699737 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.699711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:19.783472 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.783440    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:19.794270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.794241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:19.834189 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.834149    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5wq\" (UniqueName: \"kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.834371 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.834222    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.853860 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.853831    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:19.935685 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.935597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.935685 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.935675    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5wq\" (UniqueName: \"kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.936232 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.936211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:19.948499 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:19.946093    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5wq\" (UniqueName: \"kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq\") pod \"limitador-limitador-8588749977-pwbl7\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") " pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:20.002005 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.001970    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:20.274088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.274058    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:20.276729 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:33:20.276700    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0278ea82_3609_4efd_8339_37df1a2987e8.slice/crio-06ed4a793266501f8528b8c01ccb417f536302072c034a99d9b22cb61061e884 WatchSource:0}: Error finding container 06ed4a793266501f8528b8c01ccb417f536302072c034a99d9b22cb61061e884: Status 404 returned error can't find the container with id 06ed4a793266501f8528b8c01ccb417f536302072c034a99d9b22cb61061e884
Sep 18 16:33:20.486284 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.486249    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:20.491782 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.491751    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:20.494524 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.494492    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cxxmj\""
Sep 18 16:33:20.497588 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.497564    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:20.514106 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.514082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:20.591694 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.591617    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:20.640727 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.640691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8j5\" (UniqueName: \"kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5\") pod \"authorino-7bcfd8f9cd-ds9br\" (UID: \"736b21aa-5a67-4b4e-9425-d2ede79bff1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:20.741472 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.741437    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8j5\" (UniqueName: \"kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5\") pod \"authorino-7bcfd8f9cd-ds9br\" (UID: \"736b21aa-5a67-4b4e-9425-d2ede79bff1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:20.751498 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.751462    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8j5\" (UniqueName: \"kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5\") pod \"authorino-7bcfd8f9cd-ds9br\" (UID: \"736b21aa-5a67-4b4e-9425-d2ede79bff1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:20.803075 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.803039    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:20.970121 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:20.970086    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" event={"ID":"0278ea82-3609-4efd-8339-37df1a2987e8","Type":"ContainerStarted","Data":"06ed4a793266501f8528b8c01ccb417f536302072c034a99d9b22cb61061e884"}
Sep 18 16:33:21.070436 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:21.070376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:21.073556 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:33:21.073430    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod736b21aa_5a67_4b4e_9425_d2ede79bff1b.slice/crio-2c398c8472ea74ffbea47b4b87534d972641089e3048fcde7bb1308ea3ae5611 WatchSource:0}: Error finding container 2c398c8472ea74ffbea47b4b87534d972641089e3048fcde7bb1308ea3ae5611: Status 404 returned error can't find the container with id 2c398c8472ea74ffbea47b4b87534d972641089e3048fcde7bb1308ea3ae5611
Sep 18 16:33:21.976941 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:21.976899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" event={"ID":"736b21aa-5a67-4b4e-9425-d2ede79bff1b","Type":"ContainerStarted","Data":"2c398c8472ea74ffbea47b4b87534d972641089e3048fcde7bb1308ea3ae5611"}
Sep 18 16:33:21.978648 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:21.978624    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" event={"ID":"0278ea82-3609-4efd-8339-37df1a2987e8","Type":"ContainerStarted","Data":"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"}
Sep 18 16:33:21.978789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:21.978754    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:21.995197 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:21.995150    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" podStartSLOduration=1.8136268119999999 podStartE2EDuration="2.995133048s" totalImagesPullingTime="1.181506236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:33:21.993599338 +0000 UTC m=+1073.473572908" watchObservedRunningTime="2026-09-18 16:33:21.995133048 +0000 UTC m=+1073.475106593"
Sep 18 16:33:22.986458 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:22.986388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" event={"ID":"736b21aa-5a67-4b4e-9425-d2ede79bff1b","Type":"ContainerStarted","Data":"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"}
Sep 18 16:33:23.009603 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:23.009539    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" podStartSLOduration=1.198891597 podStartE2EDuration="3.009513152s" totalImagesPullingTime="1.810621555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:33:23.008644556 +0000 UTC m=+1074.488618174" watchObservedRunningTime="2026-09-18 16:33:23.009513152 +0000 UTC m=+1074.489486699"
Sep 18 16:33:29.118270 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.118232    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:29.118690 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.118430    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" podUID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" containerName="authorino" containerID="cri-o://c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44" gracePeriod=30
Sep 18 16:33:29.366525 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.365857    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:29.416213 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.416131    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv8j5\" (UniqueName: \"kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5\") pod \"736b21aa-5a67-4b4e-9425-d2ede79bff1b\" (UID: \"736b21aa-5a67-4b4e-9425-d2ede79bff1b\") "
Sep 18 16:33:29.418255 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.418227    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5" (OuterVolumeSpecName: "kube-api-access-dv8j5") pod "736b21aa-5a67-4b4e-9425-d2ede79bff1b" (UID: "736b21aa-5a67-4b4e-9425-d2ede79bff1b"). InnerVolumeSpecName "kube-api-access-dv8j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:29.516963 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:29.516925    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8j5\" (UniqueName: \"kubernetes.io/projected/736b21aa-5a67-4b4e-9425-d2ede79bff1b-kube-api-access-dv8j5\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:33:30.024590 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.024556    2554 generic.go:356] "Generic (PLEG): container finished" podID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" containerID="c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44" exitCode=0
Sep 18 16:33:30.024736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.024611    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br"
Sep 18 16:33:30.024736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.024642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" event={"ID":"736b21aa-5a67-4b4e-9425-d2ede79bff1b","Type":"ContainerDied","Data":"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"}
Sep 18 16:33:30.024736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.024687    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ds9br" event={"ID":"736b21aa-5a67-4b4e-9425-d2ede79bff1b","Type":"ContainerDied","Data":"2c398c8472ea74ffbea47b4b87534d972641089e3048fcde7bb1308ea3ae5611"}
Sep 18 16:33:30.024736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.024702    2554 scope.go:117] "RemoveContainer" containerID="c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"
Sep 18 16:33:30.033267 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.033251    2554 scope.go:117] "RemoveContainer" containerID="c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"
Sep 18 16:33:30.033552 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:33:30.033535    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44\": container with ID starting with c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44 not found: ID does not exist" containerID="c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"
Sep 18 16:33:30.033605 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.033562    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44"} err="failed to get container status \"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44\": rpc error: code = NotFound desc = could not find container \"c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44\": container with ID starting with c38bf841ae76a68ea41b59441ae3db72fe446c566e9f083f2f52dae91c43cd44 not found: ID does not exist"
Sep 18 16:33:30.046288 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.046262    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:30.050131 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.050095    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ds9br"]
Sep 18 16:33:30.996471 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:30.996436    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" path="/var/lib/kubelet/pods/736b21aa-5a67-4b4e-9425-d2ede79bff1b/volumes"
Sep 18 16:33:32.987244 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:32.987207    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:33.677443 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:33.677384    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:33.677641 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:33.677614    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" podUID="0278ea82-3609-4efd-8339-37df1a2987e8" containerName="limitador" containerID="cri-o://dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711" gracePeriod=30
Sep 18 16:33:34.225432 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.225386    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:34.257795 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.257698    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file\") pod \"0278ea82-3609-4efd-8339-37df1a2987e8\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") "
Sep 18 16:33:34.257947 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.257821    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr5wq\" (UniqueName: \"kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq\") pod \"0278ea82-3609-4efd-8339-37df1a2987e8\" (UID: \"0278ea82-3609-4efd-8339-37df1a2987e8\") "
Sep 18 16:33:34.258164 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.258136    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file" (OuterVolumeSpecName: "config-file") pod "0278ea82-3609-4efd-8339-37df1a2987e8" (UID: "0278ea82-3609-4efd-8339-37df1a2987e8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:34.259879 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.259851    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq" (OuterVolumeSpecName: "kube-api-access-hr5wq") pod "0278ea82-3609-4efd-8339-37df1a2987e8" (UID: "0278ea82-3609-4efd-8339-37df1a2987e8"). InnerVolumeSpecName "kube-api-access-hr5wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:34.358857 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.358819    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5wq\" (UniqueName: \"kubernetes.io/projected/0278ea82-3609-4efd-8339-37df1a2987e8-kube-api-access-hr5wq\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:33:34.358857 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:34.358849    2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0278ea82-3609-4efd-8339-37df1a2987e8-config-file\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:33:35.048619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.048587    2554 generic.go:356] "Generic (PLEG): container finished" podID="0278ea82-3609-4efd-8339-37df1a2987e8" containerID="dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711" exitCode=0
Sep 18 16:33:35.048814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.048633    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7"
Sep 18 16:33:35.048814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.048679    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" event={"ID":"0278ea82-3609-4efd-8339-37df1a2987e8","Type":"ContainerDied","Data":"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"}
Sep 18 16:33:35.048814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.048738    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pwbl7" event={"ID":"0278ea82-3609-4efd-8339-37df1a2987e8","Type":"ContainerDied","Data":"06ed4a793266501f8528b8c01ccb417f536302072c034a99d9b22cb61061e884"}
Sep 18 16:33:35.048814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.048771    2554 scope.go:117] "RemoveContainer" containerID="dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"
Sep 18 16:33:35.057505 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.057486    2554 scope.go:117] "RemoveContainer" containerID="dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"
Sep 18 16:33:35.057781 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:33:35.057762    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711\": container with ID starting with dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711 not found: ID does not exist" containerID="dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"
Sep 18 16:33:35.057836 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.057794    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711"} err="failed to get container status \"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711\": rpc error: code = NotFound desc = could not find container \"dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711\": container with ID starting with dc7aae409a647e3a6e97f539c805a146e78fe2cc32d31222ea810143b9852711 not found: ID does not exist"
Sep 18 16:33:35.065468 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.065441    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:35.067295 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:35.067272    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pwbl7"]
Sep 18 16:33:36.996280 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:36.996248    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0278ea82-3609-4efd-8339-37df1a2987e8" path="/var/lib/kubelet/pods/0278ea82-3609-4efd-8339-37df1a2987e8/volumes"
Sep 18 16:33:39.628311 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:39.628265    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:40.195299 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:40.195267    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:42.511326 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511297    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-x62qz"]
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511640    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" containerName="authorino"
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511651    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" containerName="authorino"
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511668    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0278ea82-3609-4efd-8339-37df1a2987e8" containerName="limitador"
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511674    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0278ea82-3609-4efd-8339-37df1a2987e8" containerName="limitador"
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511722    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="736b21aa-5a67-4b4e-9425-d2ede79bff1b" containerName="authorino"
Sep 18 16:33:42.511744 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.511734    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0278ea82-3609-4efd-8339-37df1a2987e8" containerName="limitador"
Sep 18 16:33:42.516024 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.516008    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.519469 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.519444    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cxxmj\""
Sep 18 16:33:42.519629 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.519446    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 16:33:42.521372 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.521345    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-x62qz"]
Sep 18 16:33:42.630035 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.630003    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlqp\" (UniqueName: \"kubernetes.io/projected/6bc7580a-5ed1-4e06-95de-7715075b6b5a-kube-api-access-qjlqp\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.630231 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.630114    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bc7580a-5ed1-4e06-95de-7715075b6b5a-tls-cert\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.731551 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.731510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bc7580a-5ed1-4e06-95de-7715075b6b5a-tls-cert\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.731715 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.731578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlqp\" (UniqueName: \"kubernetes.io/projected/6bc7580a-5ed1-4e06-95de-7715075b6b5a-kube-api-access-qjlqp\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.734000 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.733973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bc7580a-5ed1-4e06-95de-7715075b6b5a-tls-cert\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.740032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.740004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlqp\" (UniqueName: \"kubernetes.io/projected/6bc7580a-5ed1-4e06-95de-7715075b6b5a-kube-api-access-qjlqp\") pod \"authorino-6d9898bd99-x62qz\" (UID: \"6bc7580a-5ed1-4e06-95de-7715075b6b5a\") " pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:42.825654 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:42.825569    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-x62qz"
Sep 18 16:33:43.092445 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:43.092354    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-x62qz"]
Sep 18 16:33:43.095118 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:33:43.095090    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc7580a_5ed1_4e06_95de_7715075b6b5a.slice/crio-b3d7493fa5f48d47d16efb2898725e9b90201cfe2f666f20cb0a940a888083bf WatchSource:0}: Error finding container b3d7493fa5f48d47d16efb2898725e9b90201cfe2f666f20cb0a940a888083bf: Status 404 returned error can't find the container with id b3d7493fa5f48d47d16efb2898725e9b90201cfe2f666f20cb0a940a888083bf
Sep 18 16:33:44.092723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:44.092688    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-x62qz" event={"ID":"6bc7580a-5ed1-4e06-95de-7715075b6b5a","Type":"ContainerStarted","Data":"0542cf92558aef373d93c59653fac00212d234ed64ae4e5bc6a16780521dc36f"}
Sep 18 16:33:44.092723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:44.092730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-x62qz" event={"ID":"6bc7580a-5ed1-4e06-95de-7715075b6b5a","Type":"ContainerStarted","Data":"b3d7493fa5f48d47d16efb2898725e9b90201cfe2f666f20cb0a940a888083bf"}
Sep 18 16:33:44.110022 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:44.109972    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-x62qz" podStartSLOduration=1.709790312 podStartE2EDuration="2.1099544s" totalImagesPullingTime="400.164088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:33:44.109001204 +0000 UTC m=+1095.588974751" watchObservedRunningTime="2026-09-18 16:33:44.1099544 +0000 UTC m=+1095.589927946"
Sep 18 16:33:47.979034 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:47.978995    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:48.072405 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:33:48.072368    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:13.912149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.912106    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:13.993239 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.993205    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:13.994973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.994945    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:13.996639 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.996618    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:13.999245 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.999226    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-kp8r9\""
Sep 18 16:34:13.999346 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:13.999258    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 16:34:14.000258 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.000233    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 16:34:14.000365 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.000292    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 16:34:14.006644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.006622    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:14.016397 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.016363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.016547 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.016409    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ngz\" (UniqueName: \"kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.038372 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.038341    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-f2sqm"]
Sep 18 16:34:14.041683 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.041661    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.044802 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.044783    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-28qmm\""
Sep 18 16:34:14.044901 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.044815    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 16:34:14.051344 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.051320    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f2sqm"]
Sep 18 16:34:14.111644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.111596    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:14.116934 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.116913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.117015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.116946    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ngz\" (UniqueName: \"kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.117015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.116970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23e7a231-5e71-4f02-a799-5a8a3a01883b-data\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.117099 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:34:14.117037    2554 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 16:34:14.117099 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.117046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/23e7a231-5e71-4f02-a799-5a8a3a01883b-kube-api-access-8jhbm\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.117099 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:34:14.117097    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert podName:66aeacc4-5b47-4e83-a24f-6a3bad1e78c5 nodeName:}" failed. No retries permitted until 2026-09-18 16:34:14.617081052 +0000 UTC m=+1126.097054577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert") pod "kserve-controller-manager-7dcf4bff6-xxbzj" (UID: "66aeacc4-5b47-4e83-a24f-6a3bad1e78c5") : secret "kserve-webhook-server-cert" not found
Sep 18 16:34:14.126789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.126754    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ngz\" (UniqueName: \"kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.187687 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.187606    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:14.217456 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.217404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23e7a231-5e71-4f02-a799-5a8a3a01883b-data\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.217609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.217501    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/23e7a231-5e71-4f02-a799-5a8a3a01883b-kube-api-access-8jhbm\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.217801 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.217782    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23e7a231-5e71-4f02-a799-5a8a3a01883b-data\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.240503 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.240473    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/23e7a231-5e71-4f02-a799-5a8a3a01883b-kube-api-access-8jhbm\") pod \"seaweedfs-64568bcd49-f2sqm\" (UID: \"23e7a231-5e71-4f02-a799-5a8a3a01883b\") " pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.351892 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.351849    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:14.438913 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.438830    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:14.620522 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.620480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.623922 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.623806    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f2sqm"]
Sep 18 16:34:14.624269 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.624245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") pod \"kserve-controller-manager-7dcf4bff6-xxbzj\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") " pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:14.626217 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:34:14.626187    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e7a231_5e71_4f02_a799_5a8a3a01883b.slice/crio-773a201bf66601fd64a77f6d63a01866788c46805c6299bb4bf8fa1560f968be WatchSource:0}: Error finding container 773a201bf66601fd64a77f6d63a01866788c46805c6299bb4bf8fa1560f968be: Status 404 returned error can't find the container with id 773a201bf66601fd64a77f6d63a01866788c46805c6299bb4bf8fa1560f968be
Sep 18 16:34:14.908305 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:14.908261    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:15.031008 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:15.030980    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:15.130436 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:15.130385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:15.183557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:15.183531    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:15.200855 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:15.200804    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:15.237980 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:15.237945    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f2sqm" event={"ID":"23e7a231-5e71-4f02-a799-5a8a3a01883b","Type":"ContainerStarted","Data":"773a201bf66601fd64a77f6d63a01866788c46805c6299bb4bf8fa1560f968be"}
Sep 18 16:34:15.251137 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:34:15.251099    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66aeacc4_5b47_4e83_a24f_6a3bad1e78c5.slice/crio-0d80dafe6ae1790cf7a5bef6080b3e60a56f9de120c8fa34ff6a39f96da0250e WatchSource:0}: Error finding container 0d80dafe6ae1790cf7a5bef6080b3e60a56f9de120c8fa34ff6a39f96da0250e: Status 404 returned error can't find the container with id 0d80dafe6ae1790cf7a5bef6080b3e60a56f9de120c8fa34ff6a39f96da0250e
Sep 18 16:34:16.246568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:16.246530    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" event={"ID":"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5","Type":"ContainerStarted","Data":"0d80dafe6ae1790cf7a5bef6080b3e60a56f9de120c8fa34ff6a39f96da0250e"}
Sep 18 16:34:19.265736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.265657    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f2sqm" event={"ID":"23e7a231-5e71-4f02-a799-5a8a3a01883b","Type":"ContainerStarted","Data":"78666e7259e1ee356e7b6c6a09da2e606624bcf3a523f373a04a3ba133082040"}
Sep 18 16:34:19.266125 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.265744    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:19.267207 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.267185    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" event={"ID":"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5","Type":"ContainerStarted","Data":"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"}
Sep 18 16:34:19.267313 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.267236    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:19.282948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.282897    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-f2sqm" podStartSLOduration=0.968556722 podStartE2EDuration="5.282883083s" totalImagesPullingTime="4.314326361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:19.281297039 +0000 UTC m=+1130.761270586" watchObservedRunningTime="2026-09-18 16:34:19.282883083 +0000 UTC m=+1130.762856628"
Sep 18 16:34:19.298082 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:19.298025    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" podStartSLOduration=2.661388933 podStartE2EDuration="6.298004509s" totalImagesPullingTime="3.636615576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:19.297663918 +0000 UTC m=+1130.777637465" watchObservedRunningTime="2026-09-18 16:34:19.298004509 +0000 UTC m=+1130.777978050"
Sep 18 16:34:25.273227 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:25.273193    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-f2sqm"
Sep 18 16:34:50.276530 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:50.276498    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:55.076783 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:55.076701    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:57.109824 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.109788    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:57.110281 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.109985    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" podUID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" containerName="manager" containerID="cri-o://d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268" gracePeriod=10
Sep 18 16:34:57.134010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.133977    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-btz59"]
Sep 18 16:34:57.137261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.137236    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.145785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.145759    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-btz59"]
Sep 18 16:34:57.304756 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.304724    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ps8\" (UniqueName: \"kubernetes.io/projected/fa6da241-6405-42c6-bf14-a629c6f95e06-kube-api-access-q7ps8\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.304905 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.304855    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa6da241-6405-42c6-bf14-a629c6f95e06-cert\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.352292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.352265    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:57.406316 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.406236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ps8\" (UniqueName: \"kubernetes.io/projected/fa6da241-6405-42c6-bf14-a629c6f95e06-kube-api-access-q7ps8\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.406487 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.406317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa6da241-6405-42c6-bf14-a629c6f95e06-cert\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.408686 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.408658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa6da241-6405-42c6-bf14-a629c6f95e06-cert\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.415718 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.415676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ps8\" (UniqueName: \"kubernetes.io/projected/fa6da241-6405-42c6-bf14-a629c6f95e06-kube-api-access-q7ps8\") pod \"kserve-controller-manager-7dcf4bff6-btz59\" (UID: \"fa6da241-6405-42c6-bf14-a629c6f95e06\") " pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.446009 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.445979    2554 generic.go:356] "Generic (PLEG): container finished" podID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" containerID="d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268" exitCode=0
Sep 18 16:34:57.446184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.446023    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj"
Sep 18 16:34:57.446184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.446059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" event={"ID":"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5","Type":"ContainerDied","Data":"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"}
Sep 18 16:34:57.446184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.446116    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-xxbzj" event={"ID":"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5","Type":"ContainerDied","Data":"0d80dafe6ae1790cf7a5bef6080b3e60a56f9de120c8fa34ff6a39f96da0250e"}
Sep 18 16:34:57.446184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.446144    2554 scope.go:117] "RemoveContainer" containerID="d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"
Sep 18 16:34:57.454294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.454277    2554 scope.go:117] "RemoveContainer" containerID="d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"
Sep 18 16:34:57.454697 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:34:57.454675    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268\": container with ID starting with d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268 not found: ID does not exist" containerID="d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"
Sep 18 16:34:57.454789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.454704    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268"} err="failed to get container status \"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268\": rpc error: code = NotFound desc = could not find container \"d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268\": container with ID starting with d444d85a3d69e9bc2cc1535934c25f358e573a9d5ba080f93b1c05022fa9d268 not found: ID does not exist"
Sep 18 16:34:57.499935 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.499899    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:57.507478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.507450    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") pod \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") "
Sep 18 16:34:57.507606 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.507500    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4ngz\" (UniqueName: \"kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz\") pod \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\" (UID: \"66aeacc4-5b47-4e83-a24f-6a3bad1e78c5\") "
Sep 18 16:34:57.509636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.509605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz" (OuterVolumeSpecName: "kube-api-access-t4ngz") pod "66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" (UID: "66aeacc4-5b47-4e83-a24f-6a3bad1e78c5"). InnerVolumeSpecName "kube-api-access-t4ngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:57.509779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.509756    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert" (OuterVolumeSpecName: "cert") pod "66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" (UID: "66aeacc4-5b47-4e83-a24f-6a3bad1e78c5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:57.609077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.609040    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-cert\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:34:57.609077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.609073    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ngz\" (UniqueName: \"kubernetes.io/projected/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5-kube-api-access-t4ngz\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:34:57.768055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.768023    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:57.772862 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.772825    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-xxbzj"]
Sep 18 16:34:57.774433 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:57.774397    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-btz59"]
Sep 18 16:34:57.775530 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:34:57.775505    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6da241_6405_42c6_bf14_a629c6f95e06.slice/crio-1d277dadf924a30426e66049f535dc95b413435523b6c9f27202c58ccd723869 WatchSource:0}: Error finding container 1d277dadf924a30426e66049f535dc95b413435523b6c9f27202c58ccd723869: Status 404 returned error can't find the container with id 1d277dadf924a30426e66049f535dc95b413435523b6c9f27202c58ccd723869
Sep 18 16:34:58.453109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:58.453012    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59" event={"ID":"fa6da241-6405-42c6-bf14-a629c6f95e06","Type":"ContainerStarted","Data":"4ad7188047a39b00c2e1aef2cf2e595f2cef42fd2671c12bd3645a5922fab744"}
Sep 18 16:34:58.453109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:58.453070    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59" event={"ID":"fa6da241-6405-42c6-bf14-a629c6f95e06","Type":"ContainerStarted","Data":"1d277dadf924a30426e66049f535dc95b413435523b6c9f27202c58ccd723869"}
Sep 18 16:34:58.453109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:58.453096    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:34:58.470953 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:58.470896    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59" podStartSLOduration=1.095059594 podStartE2EDuration="1.470873008s" totalImagesPullingTime="375.813414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:58.47052388 +0000 UTC m=+1169.950497427" watchObservedRunningTime="2026-09-18 16:34:58.470873008 +0000 UTC m=+1169.950846571"
Sep 18 16:34:58.996482 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:34:58.996441    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" path="/var/lib/kubelet/pods/66aeacc4-5b47-4e83-a24f-6a3bad1e78c5/volumes"
Sep 18 16:35:29.024790 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:29.024757    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:35:29.025846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:29.025829    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:35:29.463331 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:29.463258    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-btz59"
Sep 18 16:35:30.697043 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.697010    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:30.751426 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.751378    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-8rthl"]
Sep 18 16:35:30.751780 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.751766    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" containerName="manager"
Sep 18 16:35:30.751823 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.751781    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" containerName="manager"
Sep 18 16:35:30.751857 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.751839    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="66aeacc4-5b47-4e83-a24f-6a3bad1e78c5" containerName="manager"
Sep 18 16:35:30.755241 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.755218    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-bcvg2"]
Sep 18 16:35:30.755363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.755330    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:30.757869 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.757845    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 16:35:30.757970 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.757893    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vfknb\""
Sep 18 16:35:30.758297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.758283    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:30.760635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.760617    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 16:35:30.760841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.760827    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 16:35:30.761245 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.761226    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-m8f55\""
Sep 18 16:35:30.768197 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.768179    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bcvg2"]
Sep 18 16:35:30.769939 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.769916    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-8rthl"]
Sep 18 16:35:30.799018 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.798989    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:30.907532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.907492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqjk\" (UniqueName: \"kubernetes.io/projected/698b4837-4bb2-4158-afc4-7090f0509ca2-kube-api-access-kbqjk\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:30.907736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.907540    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:30.907736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.907635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b80e01-1408-4b29-ad42-13765cff2024-tls-certs\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:30.907736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.907713    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-metrics-certs\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:30.907877 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.907752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbnd\" (UniqueName: \"kubernetes.io/projected/d3b80e01-1408-4b29-ad42-13765cff2024-kube-api-access-6pbnd\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:30.939550 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:30.939509    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.008325 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.008290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.008535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.008340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b80e01-1408-4b29-ad42-13765cff2024-tls-certs\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:31.008535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.008379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-metrics-certs\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.008535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.008402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbnd\" (UniqueName: \"kubernetes.io/projected/d3b80e01-1408-4b29-ad42-13765cff2024-kube-api-access-6pbnd\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:31.008535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.008471    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqjk\" (UniqueName: \"kubernetes.io/projected/698b4837-4bb2-4158-afc4-7090f0509ca2-kube-api-access-kbqjk\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.008535 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:35:31.008470    2554 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 16:35:31.008733 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:35:31.008563    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert podName:698b4837-4bb2-4158-afc4-7090f0509ca2 nodeName:}" failed. No retries permitted until 2026-09-18 16:35:31.508546876 +0000 UTC m=+1202.988520400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert") pod "odh-model-controller-7b966d7984-bcvg2" (UID: "698b4837-4bb2-4158-afc4-7090f0509ca2") : secret "odh-model-controller-webhook-cert" not found
Sep 18 16:35:31.010862 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.010833    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b80e01-1408-4b29-ad42-13765cff2024-tls-certs\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:31.010992 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.010973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-metrics-certs\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.019088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.019048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbnd\" (UniqueName: \"kubernetes.io/projected/d3b80e01-1408-4b29-ad42-13765cff2024-kube-api-access-6pbnd\") pod \"model-serving-api-7b954ffc95-8rthl\" (UID: \"d3b80e01-1408-4b29-ad42-13765cff2024\") " pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:31.019254 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.019231    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqjk\" (UniqueName: \"kubernetes.io/projected/698b4837-4bb2-4158-afc4-7090f0509ca2-kube-api-access-kbqjk\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.038850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.038817    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.068578 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.068540    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:31.345404 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.345378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-8rthl"]
Sep 18 16:35:31.347737 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:35:31.347696    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b80e01_1408_4b29_ad42_13765cff2024.slice/crio-6568f79e7d1e3ef70a3c3668b08be95a641d1aa2b75dc17a3fcfcc9a8ff19446 WatchSource:0}: Error finding container 6568f79e7d1e3ef70a3c3668b08be95a641d1aa2b75dc17a3fcfcc9a8ff19446: Status 404 returned error can't find the container with id 6568f79e7d1e3ef70a3c3668b08be95a641d1aa2b75dc17a3fcfcc9a8ff19446
Sep 18 16:35:31.368507 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.368478    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.513064 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.513024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.515430 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.515387    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/698b4837-4bb2-4158-afc4-7090f0509ca2-cert\") pod \"odh-model-controller-7b966d7984-bcvg2\" (UID: \"698b4837-4bb2-4158-afc4-7090f0509ca2\") " pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.608604 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.608523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-8rthl" event={"ID":"d3b80e01-1408-4b29-ad42-13765cff2024","Type":"ContainerStarted","Data":"6568f79e7d1e3ef70a3c3668b08be95a641d1aa2b75dc17a3fcfcc9a8ff19446"}
Sep 18 16:35:31.675369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.675339    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:31.910692 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.910575    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.947005 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:31.946973    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bcvg2"]
Sep 18 16:35:32.012280 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:32.012234    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:32.616319 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:32.616282    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bcvg2" event={"ID":"698b4837-4bb2-4158-afc4-7090f0509ca2","Type":"ContainerStarted","Data":"a9955351bb74ad26411e04082467e087243a55878e35a280b16b5cfc9129c7ae"}
Sep 18 16:35:35.635714 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.635650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bcvg2" event={"ID":"698b4837-4bb2-4158-afc4-7090f0509ca2","Type":"ContainerStarted","Data":"7a3f1711934d9d535d9b7396c4ab69601eebf33a3d0c4b17cad076d426e8004c"}
Sep 18 16:35:35.636202 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.635725    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:35.637564 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.637534    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-8rthl" event={"ID":"d3b80e01-1408-4b29-ad42-13765cff2024","Type":"ContainerStarted","Data":"5c57a6e2eafcf08b3fa2ecce63bfa8abffc6714a5444db13fc2f2cfcb5b672f7"}
Sep 18 16:35:35.637698 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.637601    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:35:35.661445 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.659259    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-bcvg2" podStartSLOduration=2.392848996 podStartE2EDuration="5.659237608s" totalImagesPullingTime="3.266388612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:35.655786877 +0000 UTC m=+1207.135760423" watchObservedRunningTime="2026-09-18 16:35:35.659237608 +0000 UTC m=+1207.139211154"
Sep 18 16:35:35.676005 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:35.675953    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-8rthl" podStartSLOduration=1.8544007059999998 podStartE2EDuration="5.675936441s" totalImagesPullingTime="3.821535735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:35.674142309 +0000 UTC m=+1207.154115868" watchObservedRunningTime="2026-09-18 16:35:35.675936441 +0000 UTC m=+1207.155909987"
Sep 18 16:35:46.644991 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:46.644960    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-bcvg2"
Sep 18 16:35:46.646915 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:35:46.646894    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-8rthl"
Sep 18 16:36:08.475031 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:36:08.475000    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:09.844493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:36:09.844462    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:23.727242 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:36:23.727157    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:42.030914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:36:42.030880    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:01.530025 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:01.529987    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:01.627577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:01.627546    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:07.268477 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:07.268444    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:07.368806 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:07.368777    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:20.393093 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:20.393060    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:20.584563 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:20.584527    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:27.129366 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:27.129327    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:49.887623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.884989    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:37:49.890237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.890209    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:49.893931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.893899    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 16:37:49.893931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.893922    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 16:37:49.894126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.893953    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 16:37:49.894126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.894017    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-zcrtd\""
Sep 18 16:37:49.900052 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.900029    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:37:49.923622 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:49.923590    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:50.067914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.067870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.067914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.067916    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.068126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.067936    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.068126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.067994    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bn9\" (UniqueName: \"kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.068126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.068052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.068225 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.068131    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169099 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169099 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169099 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169120    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bn9\" (UniqueName: \"kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169493    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169658 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169580    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.169658 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.169633    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.171403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.171380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.171568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.171550    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.177976 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.177952    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bn9\" (UniqueName: \"kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.201987 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.201955    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:37:50.290020 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.289940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:50.478824 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.478797    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:37:50.481673 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:37:50.481641    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8800a819_965c_47f3_ba9e_3298e49ec10c.slice/crio-50a37e3a8d709eb4de98ed89cff3bbce0bc12f304b79e61b9df09716a73668e5 WatchSource:0}: Error finding container 50a37e3a8d709eb4de98ed89cff3bbce0bc12f304b79e61b9df09716a73668e5: Status 404 returned error can't find the container with id 50a37e3a8d709eb4de98ed89cff3bbce0bc12f304b79e61b9df09716a73668e5
Sep 18 16:37:50.483531 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:50.483512    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:37:51.280547 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:51.280484    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" event={"ID":"8800a819-965c-47f3-ba9e-3298e49ec10c","Type":"ContainerStarted","Data":"50a37e3a8d709eb4de98ed89cff3bbce0bc12f304b79e61b9df09716a73668e5"}
Sep 18 16:37:52.287763 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:52.287722    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" event={"ID":"8800a819-965c-47f3-ba9e-3298e49ec10c","Type":"ContainerStarted","Data":"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"}
Sep 18 16:37:52.305892 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:52.305839    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" podStartSLOduration=1.691724459 podStartE2EDuration="3.305822621s" totalImagesPullingTime="1.614098162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:52.303899016 +0000 UTC m=+1343.783872562" watchObservedRunningTime="2026-09-18 16:37:52.305822621 +0000 UTC m=+1343.785796167"
Sep 18 16:37:53.142818 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:53.142789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:53.202863 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:37:53.202826    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:00.202386 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:00.202344    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:00.215024 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:00.214995    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:00.337877 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:00.337847    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:05.186715 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:05.186682    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:30.466603 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:30.466572    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:37.003741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.003711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:37.007321 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.007292    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:38:37.018839 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:38:37.018805    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:38:37.018839 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:38:37.018839 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:38:37.018839 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d" cmd=["/bin/sleep","15"]
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.018859    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" message="" err=<
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:38:37.018871    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main" containerID="cri-o://fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"
Sep 18 16:38:37.019160 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.018917    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main" containerID="cri-o://fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d" gracePeriod=60
Sep 18 16:38:37.092373 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.092343    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:37.392370 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.392334    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:37.502710 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502679    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.502710 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502712    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.502973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502746    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.502973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502792    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6bn9\" (UniqueName: \"kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.502973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502812    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.502973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.502871    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir\") pod \"8800a819-965c-47f3-ba9e-3298e49ec10c\" (UID: \"8800a819-965c-47f3-ba9e-3298e49ec10c\") "
Sep 18 16:38:37.503188 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.503016    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home" (OuterVolumeSpecName: "home") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:37.503188 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.503145    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:37.503291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.503253    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:37.503343 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.503281    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:37.505126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.505094    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm" (OuterVolumeSpecName: "dshm") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:37.505410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.505387    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:38:37.505527 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.505461    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9" (OuterVolumeSpecName: "kube-api-access-q6bn9") pod "8800a819-965c-47f3-ba9e-3298e49ec10c" (UID: "8800a819-965c-47f3-ba9e-3298e49ec10c"). InnerVolumeSpecName "kube-api-access-q6bn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:38:37.510539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.510514    2554 generic.go:356] "Generic (PLEG): container finished" podID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerID="fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d" exitCode=0
Sep 18 16:38:37.510650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.510566    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"
Sep 18 16:38:37.510650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.510601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" event={"ID":"8800a819-965c-47f3-ba9e-3298e49ec10c","Type":"ContainerDied","Data":"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"}
Sep 18 16:38:37.510766 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.510672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" event={"ID":"8800a819-965c-47f3-ba9e-3298e49ec10c","Type":"ContainerDied","Data":"50a37e3a8d709eb4de98ed89cff3bbce0bc12f304b79e61b9df09716a73668e5"}
Sep 18 16:38:37.510766 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.510709    2554 scope.go:117] "RemoveContainer" containerID="fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"
Sep 18 16:38:37.525842 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.525818    2554 scope.go:117] "RemoveContainer" containerID="fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"
Sep 18 16:38:37.526173 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:38:37.526154    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d\": container with ID starting with fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d not found: ID does not exist" containerID="fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"
Sep 18 16:38:37.526250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.526181    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d"} err="failed to get container status \"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d\": rpc error: code = NotFound desc = could not find container \"fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d\": container with ID starting with fad1357915fff2f8566bda81ee469f98ef96685fe8b00b6b9298c6db21dfc08d not found: ID does not exist"
Sep 18 16:38:37.541150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.541116    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:38:37.560314 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.560235    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k"]
Sep 18 16:38:37.604795 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.604755    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8800a819-965c-47f3-ba9e-3298e49ec10c-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:37.604795 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.604796    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bn9\" (UniqueName: \"kubernetes.io/projected/8800a819-965c-47f3-ba9e-3298e49ec10c-kube-api-access-q6bn9\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:37.604989 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.604811    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:37.604989 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.604825    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:37.604989 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:37.604838    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8800a819-965c-47f3-ba9e-3298e49ec10c-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:38:38.327475 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:38.327396    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjq75k" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:38.996888 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:38.996852    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" path="/var/lib/kubelet/pods/8800a819-965c-47f3-ba9e-3298e49ec10c/volumes"
Sep 18 16:38:45.371137 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:38:45.371097    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:26.972653 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:26.972608    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:27.063325 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:27.063283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:31.117085 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:31.117043    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:31.215731 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:31.215696    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:34.072363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:34.072333    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:36.111384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:36.111352    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:39.506662 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.506629    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:39:39.507038 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.506979    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main"
Sep 18 16:39:39.507038 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.506989    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main"
Sep 18 16:39:39.507126 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.507061    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8800a819-965c-47f3-ba9e-3298e49ec10c" containerName="main"
Sep 18 16:39:39.510027 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.509999    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.513455 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.513404    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4k22c\""
Sep 18 16:39:39.514508 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.514486    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 16:39:39.514508 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.514506    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 16:39:39.514664 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.514590    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 16:39:39.522557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.522531    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.531903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.531983    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.532027    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.532119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.532162    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.539032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.532249    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smds2\" (UniqueName: \"kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632672 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632622    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632672 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632672    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632736    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smds2\" (UniqueName: \"kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632765    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.632932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.632828    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.633254 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.633226    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.633332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.633252    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.633332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.633263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.635132 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.635108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.635250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.635227    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.642426 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.642390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smds2\" (UniqueName: \"kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:39.821081 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:39.820984    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:40.092209 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.092181    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:39:40.094876 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:39:40.094846    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82388c76_084d_41eb_bca0_9a518d9dd0ad.slice/crio-4587e74e9ac1ee1d4ef2f82232cc7aa5bed15cbc132f6d904856cd7b08d1bc04 WatchSource:0}: Error finding container 4587e74e9ac1ee1d4ef2f82232cc7aa5bed15cbc132f6d904856cd7b08d1bc04: Status 404 returned error can't find the container with id 4587e74e9ac1ee1d4ef2f82232cc7aa5bed15cbc132f6d904856cd7b08d1bc04
Sep 18 16:39:40.179175 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.179146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:40.283760 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.283718    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:40.381396 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.381319    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:40.814614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.814581    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" event={"ID":"82388c76-084d-41eb-bca0-9a518d9dd0ad","Type":"ContainerStarted","Data":"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"}
Sep 18 16:39:40.814993 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.814629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" event={"ID":"82388c76-084d-41eb-bca0-9a518d9dd0ad","Type":"ContainerStarted","Data":"4587e74e9ac1ee1d4ef2f82232cc7aa5bed15cbc132f6d904856cd7b08d1bc04"}
Sep 18 16:39:40.821333 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.821299    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:40.838033 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:40.837980    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" podStartSLOduration=1.8379592059999998 podStartE2EDuration="1.837959206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:40.836302216 +0000 UTC m=+1452.316275763" watchObservedRunningTime="2026-09-18 16:39:40.837959206 +0000 UTC m=+1452.317932752"
Sep 18 16:39:49.821343 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:49.821299    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:49.833964 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:49.833935    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:39:49.874888 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:39:49.874858    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:40:04.358826 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:04.358795    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:13.235137 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:13.235103    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:29.058088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:29.058058    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:40:29.058485 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:29.058475    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:40:30.427704 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:30.427662    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:35.107678 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.107635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:35.112577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.112549    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:40:35.121784 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:40:35.121745    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:40:35.121784 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:40:35.121784 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:40:35.121784 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f" cmd=["/bin/sleep","15"]
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.121798    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" message="" err=<
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:40:35.121812    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main" containerID="cri-o://b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"
Sep 18 16:40:35.122027 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.121863    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main" containerID="cri-o://b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f" gracePeriod=60
Sep 18 16:40:35.204177 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.204143    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:35.477175 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.477144    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:35.869842 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.869813    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:40:35.933113 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933076    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933122    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933166    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smds2\" (UniqueName: \"kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933227    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933261    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933561 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933346    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir\") pod \"82388c76-084d-41eb-bca0-9a518d9dd0ad\" (UID: \"82388c76-084d-41eb-bca0-9a518d9dd0ad\") "
Sep 18 16:40:35.933625 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home" (OuterVolumeSpecName: "home") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:35.933625 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:35.933730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933690    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:35.933730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933711    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:35.933730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.933725    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:35.935494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.935466    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2" (OuterVolumeSpecName: "kube-api-access-smds2") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "kube-api-access-smds2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:40:35.935494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.935477    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm" (OuterVolumeSpecName: "dshm") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:35.935494 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:35.935482    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82388c76-084d-41eb-bca0-9a518d9dd0ad" (UID: "82388c76-084d-41eb-bca0-9a518d9dd0ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:40:36.034555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.034445    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:36.034555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.034490    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82388c76-084d-41eb-bca0-9a518d9dd0ad-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:36.034555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.034503    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smds2\" (UniqueName: \"kubernetes.io/projected/82388c76-084d-41eb-bca0-9a518d9dd0ad-kube-api-access-smds2\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:36.034555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.034514    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82388c76-084d-41eb-bca0-9a518d9dd0ad-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:40:36.084655 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.084620    2554 generic.go:356] "Generic (PLEG): container finished" podID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerID="b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f" exitCode=0
Sep 18 16:40:36.084813 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.084667    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"
Sep 18 16:40:36.084813 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.084709    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" event={"ID":"82388c76-084d-41eb-bca0-9a518d9dd0ad","Type":"ContainerDied","Data":"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"}
Sep 18 16:40:36.084813 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.084765    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" event={"ID":"82388c76-084d-41eb-bca0-9a518d9dd0ad","Type":"ContainerDied","Data":"4587e74e9ac1ee1d4ef2f82232cc7aa5bed15cbc132f6d904856cd7b08d1bc04"}
Sep 18 16:40:36.084813 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.084790    2554 scope.go:117] "RemoveContainer" containerID="b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"
Sep 18 16:40:36.094542 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.094522    2554 scope.go:117] "RemoveContainer" containerID="b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"
Sep 18 16:40:36.094781 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:40:36.094753    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f\": container with ID starting with b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f not found: ID does not exist" containerID="b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"
Sep 18 16:40:36.094845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.094783    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f"} err="failed to get container status \"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f\": rpc error: code = NotFound desc = could not find container \"b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f\": container with ID starting with b0874f0ba68b99c501825b8cf45991940b3f46c632ca6b602e0cfa0b0ab0441f not found: ID does not exist"
Sep 18 16:40:36.106619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.106593    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:40:36.108957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.108933    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r"]
Sep 18 16:40:36.864373 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.864324    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-m9q8r" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded"
Sep 18 16:40:36.996519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:36.996486    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" path="/var/lib/kubelet/pods/82388c76-084d-41eb-bca0-9a518d9dd0ad/volumes"
Sep 18 16:40:41.572356 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:41.572314    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:41.673963 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:41.673930    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:47.634914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:47.634837    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:47.708382 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:47.708335    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:50.517846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.517818    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:50.745864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.745827    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:40:50.746369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.746333    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main"
Sep 18 16:40:50.746369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.746359    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main"
Sep 18 16:40:50.746602 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.746486    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="82388c76-084d-41eb-bca0-9a518d9dd0ad" containerName="main"
Sep 18 16:40:50.749706 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.749683    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.753796 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.753768    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-f4qgz\""
Sep 18 16:40:50.753987 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.753768    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 18 16:40:50.754089 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.753845    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cpu-accelerator-preset-ser-afc820a4-epp-sa-dockercfg-crk4j\""
Sep 18 16:40:50.754089 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.753874    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 18 16:40:50.754202 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.753902    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 18 16:40:50.766061 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.766026    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:40:50.804363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.804275    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:50.873650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.873610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.873650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.873655    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6jj\" (UniqueName: \"kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.974596 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.974561    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.974762 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.974603    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6jj\" (UniqueName: \"kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.977071 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.977051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:50.984093 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:50.984067    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6jj\" (UniqueName: \"kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:51.060574 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:51.060485    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:51.337410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:51.337287    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:40:51.339788 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:40:51.339755    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8ee040_4e41_41c5_89b8_2cae569e297e.slice/crio-ab0f08b0251f111f41a8c9f4b476f065e1b833199b5c7ef4735f90a8468b34b0 WatchSource:0}: Error finding container ab0f08b0251f111f41a8c9f4b476f065e1b833199b5c7ef4735f90a8468b34b0: Status 404 returned error can't find the container with id ab0f08b0251f111f41a8c9f4b476f065e1b833199b5c7ef4735f90a8468b34b0
Sep 18 16:40:52.166208 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:52.166171    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" event={"ID":"db8ee040-4e41-41c5-89b8-2cae569e297e","Type":"ContainerStarted","Data":"ab0f08b0251f111f41a8c9f4b476f065e1b833199b5c7ef4735f90a8468b34b0"}
Sep 18 16:40:54.178208 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:54.178174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" event={"ID":"db8ee040-4e41-41c5-89b8-2cae569e297e","Type":"ContainerStarted","Data":"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"}
Sep 18 16:40:54.178686 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:54.178255    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:40:54.198411 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:40:54.198360    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podStartSLOduration=1.880146178 podStartE2EDuration="4.198343191s" totalImagesPullingTime="2.318197013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:40:54.196681565 +0000 UTC m=+1525.676655114" watchObservedRunningTime="2026-09-18 16:40:54.198343191 +0000 UTC m=+1525.678316754"
Sep 18 16:41:11.915411 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:11.915373    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:21.933784 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:21.933750    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:22.051237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:22.051209    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:22.153614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:22.153585    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:25.185219 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:25.185189    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:41:28.909310 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:28.909277    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:29.110671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.110645    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:29.337534 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.337503    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:41:29.346017 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.345986    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.356591 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.356563    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 16:41:29.357895 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.357868    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-t9h9x\""
Sep 18 16:41:29.358040 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.357894    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 16:41:29.358040 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.357902    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 16:41:29.358040 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.357925    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-lxdjv\""
Sep 18 16:41:29.361123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.361090    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:41:29.379005 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.378973    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:29.431603 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.431562    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8g6x\" (UniqueName: \"kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.431754 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.431685    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.532567 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.532524    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.532755 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.532596    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8g6x\" (UniqueName: \"kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.535124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.535090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.540895 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.540867    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8g6x\" (UniqueName: \"kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.658102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.658018    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:29.930331 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:29.930305    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:41:29.932537 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:41:29.932508    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377accc5_815b_4b13_96a1_bda2c665c1a0.slice/crio-4c0ad678808dfae7b8165a58d0fb6562f19e9e5f6b7ae56ba4deb176c7c609ef WatchSource:0}: Error finding container 4c0ad678808dfae7b8165a58d0fb6562f19e9e5f6b7ae56ba4deb176c7c609ef: Status 404 returned error can't find the container with id 4c0ad678808dfae7b8165a58d0fb6562f19e9e5f6b7ae56ba4deb176c7c609ef
Sep 18 16:41:30.360328 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:30.360285    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" event={"ID":"377accc5-815b-4b13-96a1-bda2c665c1a0","Type":"ContainerStarted","Data":"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"}
Sep 18 16:41:30.360532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:30.360368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" event={"ID":"377accc5-815b-4b13-96a1-bda2c665c1a0","Type":"ContainerStarted","Data":"4c0ad678808dfae7b8165a58d0fb6562f19e9e5f6b7ae56ba4deb176c7c609ef"}
Sep 18 16:41:30.360532 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:30.360477    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:41:30.379689 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:30.379631    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" podStartSLOduration=1.379613459 podStartE2EDuration="1.379613459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:41:30.378777045 +0000 UTC m=+1561.858750592" watchObservedRunningTime="2026-09-18 16:41:30.379613459 +0000 UTC m=+1561.859587006"
Sep 18 16:41:34.847003 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:34.846961    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:34.911052 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:34.911002    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:41.292649 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:41:41.292612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:01.367407 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:01.367375    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:42:02.985533 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:02.985495    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:42:02.992964 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:02.992935    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:03.080586 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:03.080553    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:11.367865 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:11.367822    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:14.215950 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.215914    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 18 16:42:14.216329 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:42:14.215951    2554 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main" containerID="cri-o://d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"
Sep 18 16:42:14.216329 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.216002    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main" containerID="cri-o://d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd" gracePeriod=49
Sep 18 16:42:14.352264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.352238    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh_377accc5-815b-4b13-96a1-bda2c665c1a0/main/0.log"
Sep 18 16:42:14.354636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.354614    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:42:14.446343 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.446306    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8g6x\" (UniqueName: \"kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x\") pod \"377accc5-815b-4b13-96a1-bda2c665c1a0\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") "
Sep 18 16:42:14.446343 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.446350    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs\") pod \"377accc5-815b-4b13-96a1-bda2c665c1a0\" (UID: \"377accc5-815b-4b13-96a1-bda2c665c1a0\") "
Sep 18 16:42:14.448543 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.448512    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "377accc5-815b-4b13-96a1-bda2c665c1a0" (UID: "377accc5-815b-4b13-96a1-bda2c665c1a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:42:14.448635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.448576    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x" (OuterVolumeSpecName: "kube-api-access-j8g6x") pod "377accc5-815b-4b13-96a1-bda2c665c1a0" (UID: "377accc5-815b-4b13-96a1-bda2c665c1a0"). InnerVolumeSpecName "kube-api-access-j8g6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:42:14.547539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.547503    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8g6x\" (UniqueName: \"kubernetes.io/projected/377accc5-815b-4b13-96a1-bda2c665c1a0-kube-api-access-j8g6x\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:42:14.547539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.547533    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377accc5-815b-4b13-96a1-bda2c665c1a0-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:42:14.584611 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584583    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh_377accc5-815b-4b13-96a1-bda2c665c1a0/main/0.log"
Sep 18 16:42:14.584771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584623    2554 generic.go:356] "Generic (PLEG): container finished" podID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerID="d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd" exitCode=1
Sep 18 16:42:14.584771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584675    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"
Sep 18 16:42:14.584771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584706    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" event={"ID":"377accc5-815b-4b13-96a1-bda2c665c1a0","Type":"ContainerDied","Data":"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"}
Sep 18 16:42:14.584771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584749    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh" event={"ID":"377accc5-815b-4b13-96a1-bda2c665c1a0","Type":"ContainerDied","Data":"4c0ad678808dfae7b8165a58d0fb6562f19e9e5f6b7ae56ba4deb176c7c609ef"}
Sep 18 16:42:14.584771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.584772    2554 scope.go:117] "RemoveContainer" containerID="d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"
Sep 18 16:42:14.594631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.594614    2554 scope.go:117] "RemoveContainer" containerID="d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"
Sep 18 16:42:14.594911 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:42:14.594892    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd\": container with ID starting with d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd not found: ID does not exist" containerID="d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"
Sep 18 16:42:14.594951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.594920    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd"} err="failed to get container status \"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd\": rpc error: code = NotFound desc = could not find container \"d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd\": container with ID starting with d5ebb3cbbf893905573b611afc43cd1a36a21742ae043c24f18156e884b62efd not found: ID does not exist"
Sep 18 16:42:14.607707 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.607672    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:42:14.614036 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.614009    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f57zh"]
Sep 18 16:42:14.997295 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:14.997261    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" path="/var/lib/kubelet/pods/377accc5-815b-4b13-96a1-bda2c665c1a0/volumes"
Sep 18 16:42:15.185285 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:15.185238    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:15.225100 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:15.225068    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:42:15.225699 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:15.225669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:15.289922 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:15.289834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:25.185117 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:25.185071    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:30.249857 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:30.249792    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" containerID="cri-o://6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d" gracePeriod=45
Sep 18 16:42:30.260899 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:30.260872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:35.185090 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:35.185051    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:35.185521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:35.185170    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:42:45.185625 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:45.185583    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:53.025519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:53.025484    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:53.115572 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:53.115536    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:55.186095 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:55.186052    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:59.231140 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:42:59.231105    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:00.493487 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.493461    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:43:00.571152 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.571060    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c6jj\" (UniqueName: \"kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj\") pod \"db8ee040-4e41-41c5-89b8-2cae569e297e\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") "
Sep 18 16:43:00.571313 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.571161    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs\") pod \"db8ee040-4e41-41c5-89b8-2cae569e297e\" (UID: \"db8ee040-4e41-41c5-89b8-2cae569e297e\") "
Sep 18 16:43:00.573321 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.573289    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db8ee040-4e41-41c5-89b8-2cae569e297e" (UID: "db8ee040-4e41-41c5-89b8-2cae569e297e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:43:00.573467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.573315    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj" (OuterVolumeSpecName: "kube-api-access-8c6jj") pod "db8ee040-4e41-41c5-89b8-2cae569e297e" (UID: "db8ee040-4e41-41c5-89b8-2cae569e297e"). InnerVolumeSpecName "kube-api-access-8c6jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:43:00.672636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.672599    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8ee040-4e41-41c5-89b8-2cae569e297e-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:43:00.672636 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.672635    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6jj\" (UniqueName: \"kubernetes.io/projected/db8ee040-4e41-41c5-89b8-2cae569e297e-kube-api-access-8c6jj\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:43:00.801240 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.801206    2554 generic.go:356] "Generic (PLEG): container finished" podID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerID="6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d" exitCode=0
Sep 18 16:43:00.801465 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.801258    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"
Sep 18 16:43:00.801465 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.801291    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" event={"ID":"db8ee040-4e41-41c5-89b8-2cae569e297e","Type":"ContainerDied","Data":"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"}
Sep 18 16:43:00.801465 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.801345    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb" event={"ID":"db8ee040-4e41-41c5-89b8-2cae569e297e","Type":"ContainerDied","Data":"ab0f08b0251f111f41a8c9f4b476f065e1b833199b5c7ef4735f90a8468b34b0"}
Sep 18 16:43:00.801465 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.801366    2554 scope.go:117] "RemoveContainer" containerID="6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"
Sep 18 16:43:00.810201 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.810183    2554 scope.go:117] "RemoveContainer" containerID="6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"
Sep 18 16:43:00.810535 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:43:00.810518    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d\": container with ID starting with 6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d not found: ID does not exist" containerID="6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"
Sep 18 16:43:00.810594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.810544    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d"} err="failed to get container status \"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d\": rpc error: code = NotFound desc = could not find container \"6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d\": container with ID starting with 6935dd8e68fdbe00baec01ffbfe680b2892a12aa2b588c36473db2320496fb2d not found: ID does not exist"
Sep 18 16:43:00.823364 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.823295    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:43:00.832620 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.832594    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche4khmb"]
Sep 18 16:43:00.996459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:00.996406    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" path="/var/lib/kubelet/pods/db8ee040-4e41-41c5-89b8-2cae569e297e/volumes"
Sep 18 16:43:03.268185 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.268148    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:03.503781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.503744    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:43:03.504344 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504324    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main"
Sep 18 16:43:03.504344 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504344    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main"
Sep 18 16:43:03.504516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504374    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main"
Sep 18 16:43:03.504516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504383    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main"
Sep 18 16:43:03.504516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504495    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8ee040-4e41-41c5-89b8-2cae569e297e" containerName="main"
Sep 18 16:43:03.504516 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.504509    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="377accc5-815b-4b13-96a1-bda2c665c1a0" containerName="main"
Sep 18 16:43:03.509243 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.509222    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.512880 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.512856    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 16:43:03.513016 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.512975    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-bjl2l\""
Sep 18 16:43:03.513302 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.513269    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 16:43:03.513302 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.513278    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 16:43:03.513468 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.513306    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rqldl\""
Sep 18 16:43:03.518633 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.518580    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:43:03.529939 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.529904    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:03.600992 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.600945    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.601172 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.601009    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.601172 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.601096    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfrp\" (UniqueName: \"kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.601172 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.601140    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.701970 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.701926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.702181 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.701984    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfrp\" (UniqueName: \"kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.702181 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.702101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.702341 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.702187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.702459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.702403    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.702522 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.702475    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.704623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.704604    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.711346 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.711321    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfrp\" (UniqueName: \"kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:03.824493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:03.824368    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:04.099035 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:04.098948    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:43:04.102086 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:43:04.102058    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6258a99e_7376_47c6_bb94_3335cace4cad.slice/crio-b77e7ac26b77a34fe24fb120cdbd67366f3dd7fef24b1250c9ad2c1d033864c0 WatchSource:0}: Error finding container b77e7ac26b77a34fe24fb120cdbd67366f3dd7fef24b1250c9ad2c1d033864c0: Status 404 returned error can't find the container with id b77e7ac26b77a34fe24fb120cdbd67366f3dd7fef24b1250c9ad2c1d033864c0
Sep 18 16:43:04.822786 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:04.822743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" event={"ID":"6258a99e-7376-47c6-bb94-3335cace4cad","Type":"ContainerStarted","Data":"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"}
Sep 18 16:43:04.822786 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:04.822795    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" event={"ID":"6258a99e-7376-47c6-bb94-3335cace4cad","Type":"ContainerStarted","Data":"b77e7ac26b77a34fe24fb120cdbd67366f3dd7fef24b1250c9ad2c1d033864c0"}
Sep 18 16:43:04.823330 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:04.822878    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:04.843378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:04.843328    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podStartSLOduration=1.8433130979999999 podStartE2EDuration="1.843313098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:04.842398662 +0000 UTC m=+1656.322372211" watchObservedRunningTime="2026-09-18 16:43:04.843313098 +0000 UTC m=+1656.323286655"
Sep 18 16:43:05.256671 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:05.256644    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:05.348467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:05.348429    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:07.085665 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:07.085626    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:07.186927 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:07.186886    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:11.772475 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:11.772439    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:12.006987 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:12.006951    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:18.571247 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:18.571208    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:18.673232 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:18.673192    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:23.279860 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:23.279832    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:23.480614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:23.480581    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:29.815389 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:29.815354    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:35.832345 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:35.832287    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:43:37.344258 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:37.344215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:37.352123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:37.352091    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:43:37.440734 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:37.440702    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:45.831999 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:45.831961    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:43:52.371730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:52.371600    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main" containerID="cri-o://ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24" gracePeriod=45
Sep 18 16:43:55.831702 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:43:55.831660    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:02.761921 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:02.761881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:02.866072 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:02.865932    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:05.831918 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:05.831871    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:05.832356 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:05.832054    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:44:15.832044 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:15.832007    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:22.620786 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.620758    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:44:22.724817 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.724783    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache\") pod \"6258a99e-7376-47c6-bb94-3335cace4cad\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") "
Sep 18 16:44:22.725001 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.724831    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache\") pod \"6258a99e-7376-47c6-bb94-3335cace4cad\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") "
Sep 18 16:44:22.725001 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.724894    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssfrp\" (UniqueName: \"kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp\") pod \"6258a99e-7376-47c6-bb94-3335cace4cad\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") "
Sep 18 16:44:22.725001 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.724916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs\") pod \"6258a99e-7376-47c6-bb94-3335cace4cad\" (UID: \"6258a99e-7376-47c6-bb94-3335cace4cad\") "
Sep 18 16:44:22.725184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.725124    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "6258a99e-7376-47c6-bb94-3335cace4cad" (UID: "6258a99e-7376-47c6-bb94-3335cace4cad"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:22.725184 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.725142    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6258a99e-7376-47c6-bb94-3335cace4cad" (UID: "6258a99e-7376-47c6-bb94-3335cace4cad"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:22.727123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.727089    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6258a99e-7376-47c6-bb94-3335cace4cad" (UID: "6258a99e-7376-47c6-bb94-3335cace4cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:22.727123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.727114    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp" (OuterVolumeSpecName: "kube-api-access-ssfrp") pod "6258a99e-7376-47c6-bb94-3335cace4cad" (UID: "6258a99e-7376-47c6-bb94-3335cace4cad"). InnerVolumeSpecName "kube-api-access-ssfrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:22.826049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.826013    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfrp\" (UniqueName: \"kubernetes.io/projected/6258a99e-7376-47c6-bb94-3335cace4cad-kube-api-access-ssfrp\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:44:22.826049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.826042    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6258a99e-7376-47c6-bb94-3335cace4cad-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:44:22.826049 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.826053    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-tokenizer-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:44:22.826286 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:22.826061    2554 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/6258a99e-7376-47c6-bb94-3335cace4cad-cachi2-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:44:23.216471 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.216405    2554 generic.go:356] "Generic (PLEG): container finished" podID="6258a99e-7376-47c6-bb94-3335cace4cad" containerID="ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24" exitCode=0
Sep 18 16:44:23.216652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.216478    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"
Sep 18 16:44:23.216652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.216487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" event={"ID":"6258a99e-7376-47c6-bb94-3335cace4cad","Type":"ContainerDied","Data":"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"}
Sep 18 16:44:23.216652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.216551    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7" event={"ID":"6258a99e-7376-47c6-bb94-3335cace4cad","Type":"ContainerDied","Data":"b77e7ac26b77a34fe24fb120cdbd67366f3dd7fef24b1250c9ad2c1d033864c0"}
Sep 18 16:44:23.216652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.216580    2554 scope.go:117] "RemoveContainer" containerID="ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"
Sep 18 16:44:23.225742 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.225720    2554 scope.go:117] "RemoveContainer" containerID="ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"
Sep 18 16:44:23.226026 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:44:23.226006    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24\": container with ID starting with ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24 not found: ID does not exist" containerID="ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"
Sep 18 16:44:23.226100 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.226039    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24"} err="failed to get container status \"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24\": rpc error: code = NotFound desc = could not find container \"ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24\": container with ID starting with ba57dcc748c963b30c07a14dc0d38e614f6964303d1f7a32bb27f0aa654dbd24 not found: ID does not exist"
Sep 18 16:44:23.234900 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.234873    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:44:23.241291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:23.241266    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbzqr7"]
Sep 18 16:44:24.996214 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:24.996181    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" path="/var/lib/kubelet/pods/6258a99e-7376-47c6-bb94-3335cace4cad/volumes"
Sep 18 16:44:26.958408 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:26.958325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:27.057557 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:27.057522    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:30.031371 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.031333    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:44:30.031781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.031697    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main"
Sep 18 16:44:30.031781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.031709    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main"
Sep 18 16:44:30.031781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.031779    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6258a99e-7376-47c6-bb94-3335cace4cad" containerName="main"
Sep 18 16:44:30.036276 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.036255    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.039385 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.039364    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 16:44:30.040543 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.040522    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 16:44:30.040760 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.040742    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 16:44:30.040824 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.040778    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gnnmb\""
Sep 18 16:44:30.046604 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.046581    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:44:30.068668 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.068636    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:30.192852 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.192811    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.193032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.192864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.193032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.192953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.193032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.192987    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjc6m\" (UniqueName: \"kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.193032 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.193008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.193178 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.193042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjc6m\" (UniqueName: \"kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294402 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294764 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294764 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294513    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294882 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294932 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.294970 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.294930    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.296752 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.296728    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.297031 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.297011    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.306993 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.306964    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjc6m\" (UniqueName: \"kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.308059 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.308035    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:44:30.312210 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.312188    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.314861 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.314838    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-fk5lp\""
Sep 18 16:44:30.321077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.321049    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:44:30.348489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.348462    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:30.351771 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.351396    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:30.397490 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.395828    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.397490 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.395907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh8l\" (UniqueName: \"kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.496948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.496914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.497098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.496968    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh8l\" (UniqueName: \"kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.499359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.499333    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.505208 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.505180    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh8l\" (UniqueName: \"kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.622825 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.622797    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:44:30.624821 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:44:30.624787    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa98f6f_54a5_4226_9db6_10d8c7638696.slice/crio-07539e3ebacd735639429a6f0a1ed0fe0350c3be1f935709a060400fff5f6218 WatchSource:0}: Error finding container 07539e3ebacd735639429a6f0a1ed0fe0350c3be1f935709a060400fff5f6218: Status 404 returned error can't find the container with id 07539e3ebacd735639429a6f0a1ed0fe0350c3be1f935709a060400fff5f6218
Sep 18 16:44:30.629871 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.629845    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:30.912124 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:30.911943    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:44:30.914763 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:44:30.914735    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65d261a_bddb_4106_b2e5_d5f1702d60ff.slice/crio-ce7fe33fa78dfc56a135ba4f4a9b71ac176d42697d0080b64d84c508a0dd640c WatchSource:0}: Error finding container ce7fe33fa78dfc56a135ba4f4a9b71ac176d42697d0080b64d84c508a0dd640c: Status 404 returned error can't find the container with id ce7fe33fa78dfc56a135ba4f4a9b71ac176d42697d0080b64d84c508a0dd640c
Sep 18 16:44:31.258180 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.258142    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" event={"ID":"4aa98f6f-54a5-4226-9db6-10d8c7638696","Type":"ContainerStarted","Data":"d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453"}
Sep 18 16:44:31.258693 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.258210    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" event={"ID":"4aa98f6f-54a5-4226-9db6-10d8c7638696","Type":"ContainerStarted","Data":"07539e3ebacd735639429a6f0a1ed0fe0350c3be1f935709a060400fff5f6218"}
Sep 18 16:44:31.259826 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.259799    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" event={"ID":"c65d261a-bddb-4106-b2e5-d5f1702d60ff","Type":"ContainerStarted","Data":"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"}
Sep 18 16:44:31.259938 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.259846    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:44:31.259938 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.259862    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" event={"ID":"c65d261a-bddb-4106-b2e5-d5f1702d60ff","Type":"ContainerStarted","Data":"ce7fe33fa78dfc56a135ba4f4a9b71ac176d42697d0080b64d84c508a0dd640c"}
Sep 18 16:44:31.276322 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.275874    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" podStartSLOduration=1.2758558629999999 podStartE2EDuration="1.275855863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:31.274867872 +0000 UTC m=+1742.754841420" watchObservedRunningTime="2026-09-18 16:44:31.275855863 +0000 UTC m=+1742.755829410"
Sep 18 16:44:31.292540 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.292489    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podStartSLOduration=1.292474195 podStartE2EDuration="1.292474195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:31.291658781 +0000 UTC m=+1742.771632330" watchObservedRunningTime="2026-09-18 16:44:31.292474195 +0000 UTC m=+1742.772447760"
Sep 18 16:44:31.349149 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:31.349105    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:33.881462 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:33.881407    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:39.552328 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:39.552298    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:39.643302 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:39.643270    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:40.349127 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:40.349080    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:40.361793 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:40.361765    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:41.322283 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:41.322256    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:44:50.402007 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:50.401972    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:52.725554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:52.725516    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:52.816631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:52.816595    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:59.395919 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:44:59.395877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:02.266378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:02.266349    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:45:05.398696 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:05.398659    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:05.498894 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:05.498857    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:09.501938 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:09.501902    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:09.839946 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:09.839862    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:17.309526 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:17.309492    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:25.683655 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:25.683625    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:25.784549 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:25.784519    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:29.087820 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:29.087791    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:45:29.090301 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:29.090282    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:45:48.703202 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:48.703173    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:48.765790 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:45:48.765757    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:19.514634 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.514587    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:19.530990 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.530959    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:46:19.535454 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.535409    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:46:19.544239 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:46:19.544164    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:46:19.544239 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.544239 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.544239 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453" cmd=["/bin/sleep","15"]
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.544287    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" message="" err=<
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:46:19.544321    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerName="main" containerID="cri-o://d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453"
Sep 18 16:46:19.544609 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.544410    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerName="main" containerID="cri-o://d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453" gracePeriod=60
Sep 18 16:46:19.815508 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.815425    2554 generic.go:356] "Generic (PLEG): container finished" podID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerID="d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453" exitCode=0
Sep 18 16:46:19.815508 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.815490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" event={"ID":"4aa98f6f-54a5-4226-9db6-10d8c7638696","Type":"ContainerDied","Data":"d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453"}
Sep 18 16:46:19.901869 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:19.901839    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:46:20.016761 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016725    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.016951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016779    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.016951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016831    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.016951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016860    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.016951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016903    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjc6m\" (UniqueName: \"kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.016951 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.016934    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm\") pod \"4aa98f6f-54a5-4226-9db6-10d8c7638696\" (UID: \"4aa98f6f-54a5-4226-9db6-10d8c7638696\") "
Sep 18 16:46:20.017201 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.017071    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home" (OuterVolumeSpecName: "home") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.017262 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.017191    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache" (OuterVolumeSpecName: "model-cache") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.017262 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.017199    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.017355 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.017263    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.019116 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.019084    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m" (OuterVolumeSpecName: "kube-api-access-qjc6m") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "kube-api-access-qjc6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:46:20.019116 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.019106    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:46:20.019264 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.019082    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm" (OuterVolumeSpecName: "dshm") pod "4aa98f6f-54a5-4226-9db6-10d8c7638696" (UID: "4aa98f6f-54a5-4226-9db6-10d8c7638696"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.118265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.118175    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjc6m\" (UniqueName: \"kubernetes.io/projected/4aa98f6f-54a5-4226-9db6-10d8c7638696-kube-api-access-qjc6m\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.118265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.118206    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.118265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.118218    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa98f6f-54a5-4226-9db6-10d8c7638696-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.118265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.118226    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.118265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.118234    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa98f6f-54a5-4226-9db6-10d8c7638696-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.821300 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.821272    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"
Sep 18 16:46:20.821720 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.821311    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2" event={"ID":"4aa98f6f-54a5-4226-9db6-10d8c7638696","Type":"ContainerDied","Data":"07539e3ebacd735639429a6f0a1ed0fe0350c3be1f935709a060400fff5f6218"}
Sep 18 16:46:20.821720 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.821369    2554 scope.go:117] "RemoveContainer" containerID="d98bd4c0dba49b50bbcc29136b383144d4ec381f4d479f925048839dff6e4453"
Sep 18 16:46:20.843554 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.843526    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:46:20.850343 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.850315    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t8sq2"]
Sep 18 16:46:20.996649 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:20.996616    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" path="/var/lib/kubelet/pods/4aa98f6f-54a5-4226-9db6-10d8c7638696/volumes"
Sep 18 16:46:21.174352 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:21.174275    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:21.890698 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:21.890654    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:21.990720 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:21.990685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:22.266566 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:22.266524    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:32.266635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:32.266596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:34.552112 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:34.552063    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" containerID="cri-o://04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47" gracePeriod=45
Sep 18 16:46:38.756100 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:38.756061    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:38.858459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:38.858403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:38.958335 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:38.958296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:39.056107 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:39.056076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:39.131732 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:39.131696    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:42.266352 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:42.266310    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:42.266807 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:42.266444    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:46:42.539207 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:42.539179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:45.271675 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:45.271636    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:51.095493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:51.095453    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:51.157294 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:51.157258    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:52.266632 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:52.266593    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:56.008820 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:56.008785    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:58.146207 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:58.146157    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:58.247846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:58.247812    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:58.359072 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:58.359043    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:58.559664 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:58.559630    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:58.658723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:46:58.658688    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:01.409024 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:01.408977    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:01.824291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:01.824263    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:02.266360 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:02.266321    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:47:03.174881 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:03.174852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:04.803447 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:04.803402    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:47:04.911489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:04.911369    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs\") pod \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") "
Sep 18 16:47:04.911489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:04.911486    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxh8l\" (UniqueName: \"kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l\") pod \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\" (UID: \"c65d261a-bddb-4106-b2e5-d5f1702d60ff\") "
Sep 18 16:47:04.913577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:04.913552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c65d261a-bddb-4106-b2e5-d5f1702d60ff" (UID: "c65d261a-bddb-4106-b2e5-d5f1702d60ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:47:04.913687 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:04.913605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l" (OuterVolumeSpecName: "kube-api-access-bxh8l") pod "c65d261a-bddb-4106-b2e5-d5f1702d60ff" (UID: "c65d261a-bddb-4106-b2e5-d5f1702d60ff"). InnerVolumeSpecName "kube-api-access-bxh8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:05.012506 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.012477    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d261a-bddb-4106-b2e5-d5f1702d60ff-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.012506 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.012506    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh8l\" (UniqueName: \"kubernetes.io/projected/c65d261a-bddb-4106-b2e5-d5f1702d60ff-kube-api-access-bxh8l\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.033814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.033777    2554 generic.go:356] "Generic (PLEG): container finished" podID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerID="04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47" exitCode=0
Sep 18 16:47:05.033958 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.033839    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"
Sep 18 16:47:05.033958 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.033872    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" event={"ID":"c65d261a-bddb-4106-b2e5-d5f1702d60ff","Type":"ContainerDied","Data":"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"}
Sep 18 16:47:05.034094 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.033968    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw" event={"ID":"c65d261a-bddb-4106-b2e5-d5f1702d60ff","Type":"ContainerDied","Data":"ce7fe33fa78dfc56a135ba4f4a9b71ac176d42697d0080b64d84c508a0dd640c"}
Sep 18 16:47:05.034094 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.034000    2554 scope.go:117] "RemoveContainer" containerID="04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"
Sep 18 16:47:05.042967 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.042948    2554 scope.go:117] "RemoveContainer" containerID="04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"
Sep 18 16:47:05.043257 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:47:05.043237    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47\": container with ID starting with 04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47 not found: ID does not exist" containerID="04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"
Sep 18 16:47:05.043322 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.043271    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47"} err="failed to get container status \"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47\": rpc error: code = NotFound desc = could not find container \"04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47\": container with ID starting with 04a5e8dd7ca5b7e30b94aeee321663a7f4243d4f34b22417edd6db668a26bb47 not found: ID does not exist"
Sep 18 16:47:05.050921 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.050895    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:47:05.055090 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:05.055066    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bkrpw"]
Sep 18 16:47:06.996908 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:06.996874    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" path="/var/lib/kubelet/pods/c65d261a-bddb-4106-b2e5-d5f1702d60ff/volumes"
Sep 18 16:47:08.714966 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:08.714932    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:09.586802 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:09.586728    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:09.682072 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:09.682041    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:10.019677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:10.019648    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:12.295640 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:12.295599    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:15.099284 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:15.099254    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:15.407522 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:15.407443    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:15.507130 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:15.507097    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:16.656362 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:16.656326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:16.757226 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:16.757194    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:17.033299 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:17.033267    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:20.445656 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:20.445627    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:22.972601 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:22.972562    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:23.073013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:23.072967    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:26.095817 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:26.095783    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:26.194353 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:26.194321    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:26.932369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:26.932328    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:32.981148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:32.981120    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:34.915969 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.915932    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:47:34.916349 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916333    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerName="main"
Sep 18 16:47:34.916410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916353    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerName="main"
Sep 18 16:47:34.916410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916374    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main"
Sep 18 16:47:34.916410 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916382    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main"
Sep 18 16:47:34.916521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916482    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa98f6f-54a5-4226-9db6-10d8c7638696" containerName="main"
Sep 18 16:47:34.916521 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.916498    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65d261a-bddb-4106-b2e5-d5f1702d60ff" containerName="main"
Sep 18 16:47:34.919878 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.919856    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:34.923985 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.923955    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-bd5ws\""
Sep 18 16:47:34.924159 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.923985    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 16:47:34.924159 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.923970    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-rbt9f\""
Sep 18 16:47:34.924159 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.923971    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 16:47:34.924470 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.924452    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 16:47:34.934167 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.934143    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:47:34.976174 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:34.976142    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:35.097775 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.097737    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.097957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.097788    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.097957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.097868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.097957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.097898    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jljm\" (UniqueName: \"kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.097957 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.097931    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.098110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.098016    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.192228 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.192146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:35.199359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199329    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199377    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jljm\" (UniqueName: \"kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199539 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199713 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199569    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199833 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199811    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199914 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199897    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.199969 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.199914    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.201729 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.201710    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.202046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.202029    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.208176 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.208148    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jljm\" (UniqueName: \"kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.230015 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.229977    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:35.503368 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.503341    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:47:35.505910 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:47:35.505886    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b0d7da_edeb_4078_8253_4d15405f37cb.slice/crio-1603df320af9a95e69e198764b975a6cfe9cf1150bae6eec03fdb654d4b2f681 WatchSource:0}: Error finding container 1603df320af9a95e69e198764b975a6cfe9cf1150bae6eec03fdb654d4b2f681: Status 404 returned error can't find the container with id 1603df320af9a95e69e198764b975a6cfe9cf1150bae6eec03fdb654d4b2f681
Sep 18 16:47:35.508403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:35.508384    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:47:36.185003 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:36.184960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerStarted","Data":"1603df320af9a95e69e198764b975a6cfe9cf1150bae6eec03fdb654d4b2f681"}
Sep 18 16:47:37.191505 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:37.191466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerStarted","Data":"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"}
Sep 18 16:47:37.191945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:37.191554    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:38.198708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:38.198668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerStarted","Data":"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"}
Sep 18 16:47:38.217787 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:38.217725    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" podStartSLOduration=3.105820668 podStartE2EDuration="4.217703122s" totalImagesPullingTime="1.111882454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:38.217032904 +0000 UTC m=+1929.697006446" watchObservedRunningTime="2026-09-18 16:47:38.217703122 +0000 UTC m=+1929.697676672"
Sep 18 16:47:38.230265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:38.230234    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:41.781756 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:41.781724    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:41.882350 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:41.882312    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:43.918286 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:43.918255    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:44.188400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.188319    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:47:44.192113 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.192085    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.194955 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.194926    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 16:47:44.195081 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.194973    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-lwjsc\""
Sep 18 16:47:44.195081 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.195006    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 16:47:44.196333 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.196313    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-kp6x9\""
Sep 18 16:47:44.196333 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.196354    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 16:47:44.199220 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.199197    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:47:44.212881 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.212853    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:44.275555 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.275517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzs2\" (UniqueName: \"kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.275733 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.275607    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.376069 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.376030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzs2\" (UniqueName: \"kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.376237 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.376086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.378653 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.378631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.396741 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.396713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzs2\" (UniqueName: \"kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.503372 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.503331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:44.781213 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:44.781106    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:47:44.783823 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:47:44.783782    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910249fe_1259_43e5_8337_992523b3544e.slice/crio-3c7c133b6156c3f2fd1723b3dd6d641e5cd3a435a0a95c6d70c148fdde106cba WatchSource:0}: Error finding container 3c7c133b6156c3f2fd1723b3dd6d641e5cd3a435a0a95c6d70c148fdde106cba: Status 404 returned error can't find the container with id 3c7c133b6156c3f2fd1723b3dd6d641e5cd3a435a0a95c6d70c148fdde106cba
Sep 18 16:47:45.230801 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.230767    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:45.233997 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.233970    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" event={"ID":"910249fe-1259-43e5-8337-992523b3544e","Type":"ContainerStarted","Data":"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"}
Sep 18 16:47:45.234138 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.234017    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" event={"ID":"910249fe-1259-43e5-8337-992523b3544e","Type":"ContainerStarted","Data":"3c7c133b6156c3f2fd1723b3dd6d641e5cd3a435a0a95c6d70c148fdde106cba"}
Sep 18 16:47:45.234138 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.234063    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:47:45.246922 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.246898    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:45.254175 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:45.254131    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podStartSLOduration=1.254117748 podStartE2EDuration="1.254117748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:45.251833936 +0000 UTC m=+1936.731807482" watchObservedRunningTime="2026-09-18 16:47:45.254117748 +0000 UTC m=+1936.734091294"
Sep 18 16:47:46.243512 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:46.243483    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:46.252659 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:46.252633    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:47:59.785361 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:47:59.785326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:11.829722 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:11.829682    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:16.241948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:16.241917    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:48:17.005576 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.005531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:17.077003 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.076965    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:48:17.086096 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:48:17.086064    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:48:17.086096 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.086096 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.086096 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e" cmd=["/bin/sleep","15"]
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.086109    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" message="" err=<
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:48:17.086121    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main" containerID="cri-o://648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"
Sep 18 16:48:17.086345 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.086167    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main" containerID="cri-o://648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e" gracePeriod=60
Sep 18 16:48:17.138730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.138695    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:17.244089 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.244045    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8001/health\": dial tcp 10.132.0.41:8001: connect: connection refused"
Sep 18 16:48:17.399584 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.399502    2554 generic.go:356] "Generic (PLEG): container finished" podID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerID="648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e" exitCode=0
Sep 18 16:48:17.399584 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.399566    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerDied","Data":"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"}
Sep 18 16:48:17.699946 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.699911    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="llm-d-routing-sidecar" containerID="cri-o://5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1" gracePeriod=60
Sep 18 16:48:17.962912 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:17.962850    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:48:18.088897 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.088863    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jljm\" (UniqueName: \"kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089080 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.088965    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089080 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.088987    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089080 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089006    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089080 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089071    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089279 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089098    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm\") pod \"02b0d7da-edeb-4078-8253-4d15405f37cb\" (UID: \"02b0d7da-edeb-4078-8253-4d15405f37cb\") "
Sep 18 16:48:18.089279 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089205    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home" (OuterVolumeSpecName: "home") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:18.089279 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:18.089279 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089259    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:18.089778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089734    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.089778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089768    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.089971 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.089783    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.091382 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.091360    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:48:18.091688 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.091673    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm" (OuterVolumeSpecName: "dshm") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:18.091759 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.091705    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm" (OuterVolumeSpecName: "kube-api-access-8jljm") pod "02b0d7da-edeb-4078-8253-4d15405f37cb" (UID: "02b0d7da-edeb-4078-8253-4d15405f37cb"). InnerVolumeSpecName "kube-api-access-8jljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:48:18.190669 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.190629    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b0d7da-edeb-4078-8253-4d15405f37cb-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.190669 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.190660    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b0d7da-edeb-4078-8253-4d15405f37cb-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.190669 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.190669    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jljm\" (UniqueName: \"kubernetes.io/projected/02b0d7da-edeb-4078-8253-4d15405f37cb-kube-api-access-8jljm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:48:18.405905 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.405861    2554 generic.go:356] "Generic (PLEG): container finished" podID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerID="5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1" exitCode=0
Sep 18 16:48:18.406428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.405917    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"
Sep 18 16:48:18.406428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.405941    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerDied","Data":"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"}
Sep 18 16:48:18.406428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.406009    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg" event={"ID":"02b0d7da-edeb-4078-8253-4d15405f37cb","Type":"ContainerDied","Data":"1603df320af9a95e69e198764b975a6cfe9cf1150bae6eec03fdb654d4b2f681"}
Sep 18 16:48:18.406428 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.406046    2554 scope.go:117] "RemoveContainer" containerID="648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"
Sep 18 16:48:18.415388 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.415371    2554 scope.go:117] "RemoveContainer" containerID="5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"
Sep 18 16:48:18.423221 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.423193    2554 scope.go:117] "RemoveContainer" containerID="648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"
Sep 18 16:48:18.423493 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:48:18.423477    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e\": container with ID starting with 648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e not found: ID does not exist" containerID="648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"
Sep 18 16:48:18.423552 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.423502    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e"} err="failed to get container status \"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e\": rpc error: code = NotFound desc = could not find container \"648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e\": container with ID starting with 648f13c2486bf5615082516ca7513af3627eed4333ef5d3e83140bad74fe687e not found: ID does not exist"
Sep 18 16:48:18.423552 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.423520    2554 scope.go:117] "RemoveContainer" containerID="5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"
Sep 18 16:48:18.423801 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:48:18.423785    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1\": container with ID starting with 5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1 not found: ID does not exist" containerID="5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"
Sep 18 16:48:18.423850 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.423804    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1"} err="failed to get container status \"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1\": rpc error: code = NotFound desc = could not find container \"5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1\": container with ID starting with 5744e02ef8cd7d07042a889f33e3a9e3ce6efd3e5ac69782a19237c12a0074c1 not found: ID does not exist"
Sep 18 16:48:18.426475 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.426400    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:48:18.429818 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.429796    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7mgjg"]
Sep 18 16:48:18.998250 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:48:18.998215    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" path="/var/lib/kubelet/pods/02b0d7da-edeb-4078-8253-4d15405f37cb/volumes"
Sep 18 16:49:05.668978 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:05.668940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:05.765248 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:05.765212    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:12.116174 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:12.116140    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:15.813762 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:15.813727    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:16.069405 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069312    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:49:16.069713 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069698    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main"
Sep 18 16:49:16.069713 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069712    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main"
Sep 18 16:49:16.069864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069732    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="llm-d-routing-sidecar"
Sep 18 16:49:16.069864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069738    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="llm-d-routing-sidecar"
Sep 18 16:49:16.069864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069807    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="main"
Sep 18 16:49:16.069864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.069815    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b0d7da-edeb-4078-8253-4d15405f37cb" containerName="llm-d-routing-sidecar"
Sep 18 16:49:16.072890 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.072862    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.076743 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.076704    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-bptcr\""
Sep 18 16:49:16.076743 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.076724    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-gkb8r\""
Sep 18 16:49:16.076945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.076779    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 16:49:16.076945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.076729    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 16:49:16.076945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.076720    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 16:49:16.084519 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.084485    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:16.085127 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.085106    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:49:16.155644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.155610    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:16.215760 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.215726    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzl7\" (UniqueName: \"kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.215908 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.215793    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.317233 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.317183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzl7\" (UniqueName: \"kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.317459 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.317281    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.319885 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.319823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.326186 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.326156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzl7\" (UniqueName: \"kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.386701 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.386665    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:16.659568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.659544    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:49:16.662223 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:49:16.662200    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8a68d8_527a_43fd_aaf9_c8d599a47826.slice/crio-5944bc22308323cc1ff004beb45044a8c06e2b82e66b79d347f94f1f335d1cc5 WatchSource:0}: Error finding container 5944bc22308323cc1ff004beb45044a8c06e2b82e66b79d347f94f1f335d1cc5: Status 404 returned error can't find the container with id 5944bc22308323cc1ff004beb45044a8c06e2b82e66b79d347f94f1f335d1cc5
Sep 18 16:49:16.695004 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:16.694971    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" event={"ID":"fe8a68d8-527a-43fd-aaf9-c8d599a47826","Type":"ContainerStarted","Data":"5944bc22308323cc1ff004beb45044a8c06e2b82e66b79d347f94f1f335d1cc5"}
Sep 18 16:49:17.702209 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:17.702170    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" event={"ID":"fe8a68d8-527a-43fd-aaf9-c8d599a47826","Type":"ContainerStarted","Data":"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"}
Sep 18 16:49:17.702653 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:17.702320    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:18.795952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:18.795920    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:18.896451 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:18.896407    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:23.310631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:23.310598    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:48.710273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:48.710240    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:49:48.733916 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:48.733851    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podStartSLOduration=32.733828564 podStartE2EDuration="32.733828564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:17.754300578 +0000 UTC m=+2029.234274123" watchObservedRunningTime="2026-09-18 16:49:48.733828564 +0000 UTC m=+2060.213802112"
Sep 18 16:49:53.651608 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:53.651519    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:53.666023 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:53.665986    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:49:53.726743 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:53.726711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:58.710644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:49:58.710607    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:08.686466 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:08.686380    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" containerID="cri-o://0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0" gracePeriod=45
Sep 18 16:50:08.710570 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:08.710536    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:18.710578 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:18.710537    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:18.711075 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:18.710658    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:50:27.443594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:27.443559    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:28.710490 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:28.710451    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:29.118717 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:29.118685    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:50:29.122378 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:29.122356    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:50:29.982031 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:29.981994    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:50:29.984768 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:29.984738    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:30.050757 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:30.050726    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:36.241888 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:36.241844    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:38.709637 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:50:38.709607    2554 logging.go:55] [core] [Channel #296 SubChannel #297]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.43:9003", ServerName: "10.132.0.43:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.43:9003: connect: connection refused"
Sep 18 16:50:38.946284 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:38.946223    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:50:39.071051 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.071010    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgzl7\" (UniqueName: \"kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7\") pod \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") "
Sep 18 16:50:39.071224 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.071186    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs\") pod \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\" (UID: \"fe8a68d8-527a-43fd-aaf9-c8d599a47826\") "
Sep 18 16:50:39.073306 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.073272    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe8a68d8-527a-43fd-aaf9-c8d599a47826" (UID: "fe8a68d8-527a-43fd-aaf9-c8d599a47826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:50:39.073446 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.073275    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7" (OuterVolumeSpecName: "kube-api-access-bgzl7") pod "fe8a68d8-527a-43fd-aaf9-c8d599a47826" (UID: "fe8a68d8-527a-43fd-aaf9-c8d599a47826"). InnerVolumeSpecName "kube-api-access-bgzl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:50:39.096482 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.096448    2554 generic.go:356] "Generic (PLEG): container finished" podID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerID="0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0" exitCode=0
Sep 18 16:50:39.096667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.096499    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"
Sep 18 16:50:39.096667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.096532    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" event={"ID":"fe8a68d8-527a-43fd-aaf9-c8d599a47826","Type":"ContainerDied","Data":"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"}
Sep 18 16:50:39.096667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.096586    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" event={"ID":"fe8a68d8-527a-43fd-aaf9-c8d599a47826","Type":"ContainerDied","Data":"5944bc22308323cc1ff004beb45044a8c06e2b82e66b79d347f94f1f335d1cc5"}
Sep 18 16:50:39.096667 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.096608    2554 scope.go:117] "RemoveContainer" containerID="0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"
Sep 18 16:50:39.105467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.105448    2554 scope.go:117] "RemoveContainer" containerID="0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"
Sep 18 16:50:39.105752 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:50:39.105731    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0\": container with ID starting with 0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0 not found: ID does not exist" containerID="0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"
Sep 18 16:50:39.105809 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.105762    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0"} err="failed to get container status \"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0\": rpc error: code = NotFound desc = could not find container \"0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0\": container with ID starting with 0d7c24573526a1100251cbd8abe1b3490d15c49f29c28184617b3560519352c0 not found: ID does not exist"
Sep 18 16:50:39.119081 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.119053    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:50:39.123815 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.123790    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj"]
Sep 18 16:50:39.172556 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.172518    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8a68d8-527a-43fd-aaf9-c8d599a47826-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:50:39.172556 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.172548    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgzl7\" (UniqueName: \"kubernetes.io/projected/fe8a68d8-527a-43fd-aaf9-c8d599a47826-kube-api-access-bgzl7\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:50:39.709660 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:39.709617    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69nmlkj" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.43:9003\" within 1s: context deadline exceeded"
Sep 18 16:50:40.996802 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:40.996764    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" path="/var/lib/kubelet/pods/fe8a68d8-527a-43fd-aaf9-c8d599a47826/volumes"
Sep 18 16:50:43.723562 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:43.723526    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:43.822087 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:43.822055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:45.004288 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:45.004232    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main" containerID="cri-o://dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530" gracePeriod=45
Sep 18 16:50:46.241750 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:46.241709    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:50.311207 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.311172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:50.562785 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.562670    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:50.569808 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.569776    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:50:50.574438 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.570771    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main"
Sep 18 16:50:50.574438 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.570797    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main"
Sep 18 16:50:50.574438 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.570999    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8a68d8-527a-43fd-aaf9-c8d599a47826" containerName="main"
Sep 18 16:50:50.577952 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.577921    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.581659 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.581635    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 16:50:50.581871 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.581651    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 16:50:50.582098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.582066    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 16:50:50.582170 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.582107    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xcr42\""
Sep 18 16:50:50.582408 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.582390    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-zjqss\""
Sep 18 16:50:50.584478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.584457    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:50:50.663674 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.663634    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z8j\" (UniqueName: \"kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.663848 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.663704    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.764928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.764896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z8j\" (UniqueName: \"kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.765100 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.764958    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.767303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.767284    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.773828 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.773798    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z8j\" (UniqueName: \"kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:50.812586 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.812553    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:50.890150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:50.890066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:51.020306 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:51.020274    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:51.168287 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:51.168262    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:50:51.170908 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:50:51.170867    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7975e8e6_cfaa_4e54_b2cb_823c1c559553.slice/crio-2e7f5be3deccbcbf418a58e96fb3de91a69f9feb16880d59fa93992985925126 WatchSource:0}: Error finding container 2e7f5be3deccbcbf418a58e96fb3de91a69f9feb16880d59fa93992985925126: Status 404 returned error can't find the container with id 2e7f5be3deccbcbf418a58e96fb3de91a69f9feb16880d59fa93992985925126
Sep 18 16:50:52.161846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:52.161812    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" event={"ID":"7975e8e6-cfaa-4e54-b2cb-823c1c559553","Type":"ContainerStarted","Data":"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"}
Sep 18 16:50:52.162323 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:52.161872    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" event={"ID":"7975e8e6-cfaa-4e54-b2cb-823c1c559553","Type":"ContainerStarted","Data":"2e7f5be3deccbcbf418a58e96fb3de91a69f9feb16880d59fa93992985925126"}
Sep 18 16:50:52.162323 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:52.161908    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:50:52.181165 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:52.181102    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podStartSLOduration=2.181081374 podStartE2EDuration="2.181081374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:52.180634596 +0000 UTC m=+2123.660608149" watchObservedRunningTime="2026-09-18 16:50:52.181081374 +0000 UTC m=+2123.661054923"
Sep 18 16:50:56.241788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:56.241739    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:50:56.242246 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:56.241870    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:50:56.778992 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:56.778947    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:56.878467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:50:56.878403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:00.528195 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:00.528160    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:06.241113 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:06.241066    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:51:08.870876 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:08.870834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:08.971915 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:08.971877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:15.264544 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.264479    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:51:15.277936 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.277907    2554 generic.go:356] "Generic (PLEG): container finished" podID="910249fe-1259-43e5-8337-992523b3544e" containerID="dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530" exitCode=0
Sep 18 16:51:15.278092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.277945    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"
Sep 18 16:51:15.278092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.277983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" event={"ID":"910249fe-1259-43e5-8337-992523b3544e","Type":"ContainerDied","Data":"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"}
Sep 18 16:51:15.278092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.278041    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87" event={"ID":"910249fe-1259-43e5-8337-992523b3544e","Type":"ContainerDied","Data":"3c7c133b6156c3f2fd1723b3dd6d641e5cd3a435a0a95c6d70c148fdde106cba"}
Sep 18 16:51:15.278092 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.278064    2554 scope.go:117] "RemoveContainer" containerID="dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"
Sep 18 16:51:15.287944 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.287920    2554 scope.go:117] "RemoveContainer" containerID="dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"
Sep 18 16:51:15.288258 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:51:15.288237    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530\": container with ID starting with dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530 not found: ID does not exist" containerID="dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"
Sep 18 16:51:15.288319 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.288268    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530"} err="failed to get container status \"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530\": rpc error: code = NotFound desc = could not find container \"dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530\": container with ID starting with dac225c32d0c20c38d3662ae8e813d2c27f8c3cf29d7ae31464f1ba722ed2530 not found: ID does not exist"
Sep 18 16:51:15.383892 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.383859    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs\") pod \"910249fe-1259-43e5-8337-992523b3544e\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") "
Sep 18 16:51:15.384069 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.383956    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zzs2\" (UniqueName: \"kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2\") pod \"910249fe-1259-43e5-8337-992523b3544e\" (UID: \"910249fe-1259-43e5-8337-992523b3544e\") "
Sep 18 16:51:15.386169 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.386139    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "910249fe-1259-43e5-8337-992523b3544e" (UID: "910249fe-1259-43e5-8337-992523b3544e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:51:15.386283 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.386148    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2" (OuterVolumeSpecName: "kube-api-access-4zzs2") pod "910249fe-1259-43e5-8337-992523b3544e" (UID: "910249fe-1259-43e5-8337-992523b3544e"). InnerVolumeSpecName "kube-api-access-4zzs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:51:15.484841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.484810    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zzs2\" (UniqueName: \"kubernetes.io/projected/910249fe-1259-43e5-8337-992523b3544e-kube-api-access-4zzs2\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:51:15.484841 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.484838    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910249fe-1259-43e5-8337-992523b3544e-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:51:15.600993 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.600955    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:51:15.605168 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:15.605134    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2lk87"]
Sep 18 16:51:16.997556 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:16.997525    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910249fe-1259-43e5-8337-992523b3544e" path="/var/lib/kubelet/pods/910249fe-1259-43e5-8337-992523b3544e/volumes"
Sep 18 16:51:20.040054 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:20.040018    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:20.140851 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:20.140812    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:23.171702 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:23.171623    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:51:26.441733 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:26.441693    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:28.387592 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:28.387560    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:32.354524 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:32.354489    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:32.420658 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:32.420628    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:34.321234 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.321199    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:34.539990 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.539955    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:51:34.540358 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.540343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main"
Sep 18 16:51:34.540403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.540360    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main"
Sep 18 16:51:34.540476 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.540463    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="910249fe-1259-43e5-8337-992523b3544e" containerName="main"
Sep 18 16:51:34.543426 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.543391    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.547400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.547375    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 16:51:34.547400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.547391    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-5bkq2\""
Sep 18 16:51:34.547623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.547406    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-prdmf\""
Sep 18 16:51:34.547623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.547448    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 16:51:34.547623 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.547449    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 16:51:34.553204 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.553179    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:51:34.573291 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.573219    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:34.667258 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.667216    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmvh\" (UniqueName: \"kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.667439 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.667299    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.768199 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.768156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmvh\" (UniqueName: \"kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.768396 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.768249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.770791 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.770770    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.776925 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.776905    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmvh\" (UniqueName: \"kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:34.854602 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:34.854512    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:35.128925 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:35.128889    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:51:35.130646 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:51:35.130600    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bff433c_ddac_4dc9_9b35_e37a640d92a7.slice/crio-832d38b8d7adcda1a60f6dbd8aef631213fb731181581ad1c558ac1dffea5fbd WatchSource:0}: Error finding container 832d38b8d7adcda1a60f6dbd8aef631213fb731181581ad1c558ac1dffea5fbd: Status 404 returned error can't find the container with id 832d38b8d7adcda1a60f6dbd8aef631213fb731181581ad1c558ac1dffea5fbd
Sep 18 16:51:35.375439 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:35.375380    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" event={"ID":"3bff433c-ddac-4dc9-9b35-e37a640d92a7","Type":"ContainerStarted","Data":"ff8ab44da1373fb4cd0dbf7ac6deca5455bf19ff71acb338a701e527086e54eb"}
Sep 18 16:51:35.375804 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:35.375463    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:51:35.375804 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:35.375490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" event={"ID":"3bff433c-ddac-4dc9-9b35-e37a640d92a7","Type":"ContainerStarted","Data":"832d38b8d7adcda1a60f6dbd8aef631213fb731181581ad1c558ac1dffea5fbd"}
Sep 18 16:51:35.396538 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:35.396428    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podStartSLOduration=1.396395746 podStartE2EDuration="1.396395746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:51:35.395690997 +0000 UTC m=+2166.875664545" watchObservedRunningTime="2026-09-18 16:51:35.396395746 +0000 UTC m=+2166.876369292"
Sep 18 16:51:39.095673 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:39.095635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:43.828602 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:43.828558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:43.850787 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:43.850752    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:51:43.914517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:43.914477    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:53.171403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:53.171358    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:51:58.875173 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:51:58.875131    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main" containerID="cri-o://58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff" gracePeriod=45
Sep 18 16:52:03.171594 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:03.171555    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:52:06.381784 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:06.381752    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:52:10.343254 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:10.343216    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:13.170640 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:13.170596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:52:13.171044 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:13.170700    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:52:23.171265 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:23.171228    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:52:31.533652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.533622    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:52:31.567046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.566958    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs\") pod \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") "
Sep 18 16:52:31.567190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.567167    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44z8j\" (UniqueName: \"kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j\") pod \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\" (UID: \"7975e8e6-cfaa-4e54-b2cb-823c1c559553\") "
Sep 18 16:52:31.569196 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.569159    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7975e8e6-cfaa-4e54-b2cb-823c1c559553" (UID: "7975e8e6-cfaa-4e54-b2cb-823c1c559553"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:52:31.569315 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.569232    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j" (OuterVolumeSpecName: "kube-api-access-44z8j") pod "7975e8e6-cfaa-4e54-b2cb-823c1c559553" (UID: "7975e8e6-cfaa-4e54-b2cb-823c1c559553"). InnerVolumeSpecName "kube-api-access-44z8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:52:31.642506 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.642472    2554 generic.go:356] "Generic (PLEG): container finished" podID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerID="58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff" exitCode=0
Sep 18 16:52:31.642682 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.642519    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"
Sep 18 16:52:31.642682 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.642541    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" event={"ID":"7975e8e6-cfaa-4e54-b2cb-823c1c559553","Type":"ContainerDied","Data":"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"}
Sep 18 16:52:31.642682 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.642587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx" event={"ID":"7975e8e6-cfaa-4e54-b2cb-823c1c559553","Type":"ContainerDied","Data":"2e7f5be3deccbcbf418a58e96fb3de91a69f9feb16880d59fa93992985925126"}
Sep 18 16:52:31.642682 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.642608    2554 scope.go:117] "RemoveContainer" containerID="58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"
Sep 18 16:52:31.652068 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.652050    2554 scope.go:117] "RemoveContainer" containerID="58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"
Sep 18 16:52:31.652369 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:52:31.652349    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff\": container with ID starting with 58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff not found: ID does not exist" containerID="58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"
Sep 18 16:52:31.652455 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.652378    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff"} err="failed to get container status \"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff\": rpc error: code = NotFound desc = could not find container \"58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff\": container with ID starting with 58744b4c904cf5be19bc36e5b2396af7b1b7f962681d1b65832189c0d22152ff not found: ID does not exist"
Sep 18 16:52:31.664864 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.664836    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:52:31.668151 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.668130    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44z8j\" (UniqueName: \"kubernetes.io/projected/7975e8e6-cfaa-4e54-b2cb-823c1c559553-kube-api-access-44z8j\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:52:31.668241 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.668157    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7975e8e6-cfaa-4e54-b2cb-823c1c559553-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:52:31.671180 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:31.671150    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57855gbcx"]
Sep 18 16:52:32.532010 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:32.531977    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:32.625334 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:32.625302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:32.997256 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:32.997221    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" path="/var/lib/kubelet/pods/7975e8e6-cfaa-4e54-b2cb-823c1c559553/volumes"
Sep 18 16:52:39.366867 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:39.366833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:43.306082 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.306051    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:43.428728 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.428691    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:43.528724 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.528689    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:52:43.529088 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.529070    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main"
Sep 18 16:52:43.529200 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.529092    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main"
Sep 18 16:52:43.529200 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.529184    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7975e8e6-cfaa-4e54-b2cb-823c1c559553" containerName="main"
Sep 18 16:52:43.530933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.530905    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:43.532611 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.532592    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.536399 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.536360    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-fz92k\""
Sep 18 16:52:43.536399 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.536365    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 16:52:43.536577 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.536366    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 16:52:43.536717 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.536701    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-h9l5s\""
Sep 18 16:52:43.536768 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.536700    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 16:52:43.544041 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.544017    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:52:43.576698 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.576605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.576838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.576736    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrsx\" (UniqueName: \"kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.592474 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.589440    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:43.678150 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.678114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrsx\" (UniqueName: \"kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.678328 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.678245    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.680734 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.680709    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.685838 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.685815    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrsx\" (UniqueName: \"kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:43.844148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:43.844044    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:44.118261 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:44.118191    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:52:44.120940 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:52:44.120899    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf25a97_d6d9_4478_91ec_b5333ff7e651.slice/crio-e143f01ff8daff2777b7738e75befa1c70f320488088ecc134ed4e30ce7757e4 WatchSource:0}: Error finding container e143f01ff8daff2777b7738e75befa1c70f320488088ecc134ed4e30ce7757e4: Status 404 returned error can't find the container with id e143f01ff8daff2777b7738e75befa1c70f320488088ecc134ed4e30ce7757e4
Sep 18 16:52:44.705503 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:44.705461    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" event={"ID":"3bf25a97-d6d9-4478-91ec-b5333ff7e651","Type":"ContainerStarted","Data":"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"}
Sep 18 16:52:44.705887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:44.705519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" event={"ID":"3bf25a97-d6d9-4478-91ec-b5333ff7e651","Type":"ContainerStarted","Data":"e143f01ff8daff2777b7738e75befa1c70f320488088ecc134ed4e30ce7757e4"}
Sep 18 16:52:44.705887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:44.705561    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:52:44.726570 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:44.726503    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podStartSLOduration=1.72648088 podStartE2EDuration="1.72648088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:44.726091228 +0000 UTC m=+2236.206064777" watchObservedRunningTime="2026-09-18 16:52:44.72648088 +0000 UTC m=+2236.206454448"
Sep 18 16:52:45.909616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:45.909573    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:46.010696 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:46.010660    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:55.829834 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:52:55.829752    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:53:15.712595 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:15.712560    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:53:20.722839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:20.722787    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:53:20.729757 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:20.729718    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:53:20.824436 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:20.821861    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:53:25.712400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:25.712357    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:53:35.711661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:35.711617    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:53:35.751475 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:35.751438    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" containerID="cri-o://f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69" gracePeriod=45
Sep 18 16:53:45.711661 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:45.711622    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:53:45.712187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:45.711742    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:53:46.048735 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:46.048697    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:53:46.054537 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:46.054507    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:53:46.113650 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:46.113616    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:53:46.381795 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:46.381710    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:53:55.711845 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:55.711809    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:53:56.381852 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:53:56.381814    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:54:01.076928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:01.076857    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" containerID="cri-o://ff8ab44da1373fb4cd0dbf7ac6deca5455bf19ff71acb338a701e527086e54eb" gracePeriod=45
Sep 18 16:54:05.711679 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:05.711639    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:54:06.025720 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.025685    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:54:06.109874 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.109841    2554 generic.go:356] "Generic (PLEG): container finished" podID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerID="f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69" exitCode=0
Sep 18 16:54:06.110055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.109888    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"
Sep 18 16:54:06.110055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.109925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" event={"ID":"3bf25a97-d6d9-4478-91ec-b5333ff7e651","Type":"ContainerDied","Data":"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"}
Sep 18 16:54:06.110055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.110008    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk" event={"ID":"3bf25a97-d6d9-4478-91ec-b5333ff7e651","Type":"ContainerDied","Data":"e143f01ff8daff2777b7738e75befa1c70f320488088ecc134ed4e30ce7757e4"}
Sep 18 16:54:06.110055 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.110036    2554 scope.go:117] "RemoveContainer" containerID="f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"
Sep 18 16:54:06.118740 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.118722    2554 scope.go:117] "RemoveContainer" containerID="f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"
Sep 18 16:54:06.119029 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:54:06.119014    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69\": container with ID starting with f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69 not found: ID does not exist" containerID="f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"
Sep 18 16:54:06.119083 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.119037    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69"} err="failed to get container status \"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69\": rpc error: code = NotFound desc = could not find container \"f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69\": container with ID starting with f2c75a6d7e1e1d81ac1634c4b7b8a65e9f5a21d6f16f2d4aa41e141d642bfe69 not found: ID does not exist"
Sep 18 16:54:06.131909 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.131881    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfrsx\" (UniqueName: \"kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx\") pod \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") "
Sep 18 16:54:06.132040 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.131958    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs\") pod \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\" (UID: \"3bf25a97-d6d9-4478-91ec-b5333ff7e651\") "
Sep 18 16:54:06.134178 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.134150    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bf25a97-d6d9-4478-91ec-b5333ff7e651" (UID: "3bf25a97-d6d9-4478-91ec-b5333ff7e651"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:54:06.134277 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.134187    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx" (OuterVolumeSpecName: "kube-api-access-bfrsx") pod "3bf25a97-d6d9-4478-91ec-b5333ff7e651" (UID: "3bf25a97-d6d9-4478-91ec-b5333ff7e651"). InnerVolumeSpecName "kube-api-access-bfrsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:54:06.233037 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.232996    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfrsx\" (UniqueName: \"kubernetes.io/projected/3bf25a97-d6d9-4478-91ec-b5333ff7e651-kube-api-access-bfrsx\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:54:06.233037 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.233032    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf25a97-d6d9-4478-91ec-b5333ff7e651-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:54:06.381677 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.381593    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:54:06.381819 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.381729    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:54:06.431109 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.431075    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:54:06.437142 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.437116    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dn56rk"]
Sep 18 16:54:06.997340 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:06.997308    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" path="/var/lib/kubelet/pods/3bf25a97-d6d9-4478-91ec-b5333ff7e651/volumes"
Sep 18 16:54:10.422565 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:10.422531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:10.516094 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:10.516062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:16.381453 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:16.381406    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:54:16.887333 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:16.887301    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:21.620730 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.620648    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:21.840508 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.840476    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:54:21.840846 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.840834    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main"
Sep 18 16:54:21.840889 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.840848    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main"
Sep 18 16:54:21.840929 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.840920    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf25a97-d6d9-4478-91ec-b5333ff7e651" containerName="main"
Sep 18 16:54:21.852509 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.852478    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:21.856308 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856279    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:54:21.856646 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856625    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-ncdgp\""
Sep 18 16:54:21.856768 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856746    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-rr6vv\""
Sep 18 16:54:21.856835 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856625    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 16:54:21.856891 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856625    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 16:54:21.856891 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.856726    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 16:54:21.970723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.970693    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:21.979833 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.979808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:21.979969 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:21.979846    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk248\" (UniqueName: \"kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.081168 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.081130    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk248\" (UniqueName: \"kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.081396 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.081364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.083922 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.083892    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.089359 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.089335    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk248\" (UniqueName: \"kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.167377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.167337    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:22.443267 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:22.443238    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:54:22.445404 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:54:22.445376    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4320e0c_cef6_4bac_bec7_32829d7c9ee9.slice/crio-c02cee3df88d1d2f3cfbc6417bc9902f05873625a5557a0c9ada655348b7a1b5 WatchSource:0}: Error finding container c02cee3df88d1d2f3cfbc6417bc9902f05873625a5557a0c9ada655348b7a1b5: Status 404 returned error can't find the container with id c02cee3df88d1d2f3cfbc6417bc9902f05873625a5557a0c9ada655348b7a1b5
Sep 18 16:54:23.195831 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.195792    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" event={"ID":"e4320e0c-cef6-4bac-bec7-32829d7c9ee9","Type":"ContainerStarted","Data":"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"}
Sep 18 16:54:23.196224 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.195848    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" event={"ID":"e4320e0c-cef6-4bac-bec7-32829d7c9ee9","Type":"ContainerStarted","Data":"c02cee3df88d1d2f3cfbc6417bc9902f05873625a5557a0c9ada655348b7a1b5"}
Sep 18 16:54:23.196224 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.195914    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:23.212193 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.212132    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podStartSLOduration=2.212111871 podStartE2EDuration="2.212111871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:23.211869286 +0000 UTC m=+2334.691842836" watchObservedRunningTime="2026-09-18 16:54:23.212111871 +0000 UTC m=+2334.692085439"
Sep 18 16:54:23.425135 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.425080    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:23.521736 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:23.521704    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:26.381255 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:26.381218    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:54:27.048260 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:27.048230    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:31.241003 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.240972    2554 generic.go:356] "Generic (PLEG): container finished" podID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerID="ff8ab44da1373fb4cd0dbf7ac6deca5455bf19ff71acb338a701e527086e54eb" exitCode=0
Sep 18 16:54:31.241377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.241020    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" event={"ID":"3bff433c-ddac-4dc9-9b35-e37a640d92a7","Type":"ContainerDied","Data":"ff8ab44da1373fb4cd0dbf7ac6deca5455bf19ff71acb338a701e527086e54eb"}
Sep 18 16:54:31.334681 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.334654    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:54:31.468948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.468917    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs\") pod \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") "
Sep 18 16:54:31.469114 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.468962    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjmvh\" (UniqueName: \"kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh\") pod \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\" (UID: \"3bff433c-ddac-4dc9-9b35-e37a640d92a7\") "
Sep 18 16:54:31.471123 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.471088    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bff433c-ddac-4dc9-9b35-e37a640d92a7" (UID: "3bff433c-ddac-4dc9-9b35-e37a640d92a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:54:31.471235 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.471137    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh" (OuterVolumeSpecName: "kube-api-access-jjmvh") pod "3bff433c-ddac-4dc9-9b35-e37a640d92a7" (UID: "3bff433c-ddac-4dc9-9b35-e37a640d92a7"). InnerVolumeSpecName "kube-api-access-jjmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:54:31.570461 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.570406    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff433c-ddac-4dc9-9b35-e37a640d92a7-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:54:31.570461 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:31.570455    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjmvh\" (UniqueName: \"kubernetes.io/projected/3bff433c-ddac-4dc9-9b35-e37a640d92a7-kube-api-access-jjmvh\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:54:32.247718 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.247683    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"
Sep 18 16:54:32.248148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.247718    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5" event={"ID":"3bff433c-ddac-4dc9-9b35-e37a640d92a7","Type":"ContainerDied","Data":"832d38b8d7adcda1a60f6dbd8aef631213fb731181581ad1c558ac1dffea5fbd"}
Sep 18 16:54:32.248148 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.247785    2554 scope.go:117] "RemoveContainer" containerID="ff8ab44da1373fb4cd0dbf7ac6deca5455bf19ff71acb338a701e527086e54eb"
Sep 18 16:54:32.271896 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.271861    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:54:32.273499 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.273476    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmr2l5"]
Sep 18 16:54:32.999484 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:32.999446    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" path="/var/lib/kubelet/pods/3bff433c-ddac-4dc9-9b35-e37a640d92a7/volumes"
Sep 18 16:54:36.110190 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:36.110155    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:36.210429 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:36.210392    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:42.410307 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:42.410271    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:45.226134 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:45.226100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:45.447321 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:45.447286    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:49.318307 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:49.318271    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:49.422108 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:49.422078    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:52.496833 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:52.496798    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:54.202457 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:54.202398    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:54:56.019021 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:56.018990    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:54:56.026960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:56.026931    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:54:56.125405 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:54:56.125365    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:04.202689 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:04.202646    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:55:11.040224 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:11.040158    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main" containerID="cri-o://19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2" gracePeriod=45
Sep 18 16:55:14.202175 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:14.202134    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:55:24.202493 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:24.202442    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:55:24.202931 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:24.202569    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:55:25.856302 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:25.856266    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:25.911629 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:25.911598    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:29.148518 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:29.148490    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:55:29.152734 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:29.152712    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 16:55:34.202393 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:34.202354    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:55:41.291890 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.291859    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:55:41.395660 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.395576    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk248\" (UniqueName: \"kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248\") pod \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") "
Sep 18 16:55:41.395660 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.395625    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs\") pod \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\" (UID: \"e4320e0c-cef6-4bac-bec7-32829d7c9ee9\") "
Sep 18 16:55:41.397777 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.397744    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4320e0c-cef6-4bac-bec7-32829d7c9ee9" (UID: "e4320e0c-cef6-4bac-bec7-32829d7c9ee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:55:41.397904 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.397854    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248" (OuterVolumeSpecName: "kube-api-access-rk248") pod "e4320e0c-cef6-4bac-bec7-32829d7c9ee9" (UID: "e4320e0c-cef6-4bac-bec7-32829d7c9ee9"). InnerVolumeSpecName "kube-api-access-rk248". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:55:41.496859 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.496820    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk248\" (UniqueName: \"kubernetes.io/projected/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-kube-api-access-rk248\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:55:41.496859 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.496853    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4320e0c-cef6-4bac-bec7-32829d7c9ee9-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:55:41.580411 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.580369    2554 generic.go:356] "Generic (PLEG): container finished" podID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerID="19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2" exitCode=0
Sep 18 16:55:41.580411 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.580423    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"
Sep 18 16:55:41.580615 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.580459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" event={"ID":"e4320e0c-cef6-4bac-bec7-32829d7c9ee9","Type":"ContainerDied","Data":"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"}
Sep 18 16:55:41.580615 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.580511    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45" event={"ID":"e4320e0c-cef6-4bac-bec7-32829d7c9ee9","Type":"ContainerDied","Data":"c02cee3df88d1d2f3cfbc6417bc9902f05873625a5557a0c9ada655348b7a1b5"}
Sep 18 16:55:41.580615 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.580535    2554 scope.go:117] "RemoveContainer" containerID="19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"
Sep 18 16:55:41.593564 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.593550    2554 scope.go:117] "RemoveContainer" containerID="19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"
Sep 18 16:55:41.593862 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:55:41.593841    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2\": container with ID starting with 19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2 not found: ID does not exist" containerID="19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"
Sep 18 16:55:41.593902 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.593872    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2"} err="failed to get container status \"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2\": rpc error: code = NotFound desc = could not find container \"19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2\": container with ID starting with 19e69894a3f27ca92a125d6ff70ac4caa220011773cf77553d4f3f9014d797e2 not found: ID does not exist"
Sep 18 16:55:41.606748 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.606725    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:55:41.612896 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:41.612873    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheztg45"]
Sep 18 16:55:42.997187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:42.997150    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" path="/var/lib/kubelet/pods/e4320e0c-cef6-4bac-bec7-32829d7c9ee9/volumes"
Sep 18 16:55:45.738852 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:45.738820    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:45.839023 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:45.838993    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:52.235363 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:52.235284    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:58.429640 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:58.429599    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:55:58.530743 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:55:58.530710    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:11.579945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.579908    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:56:11.580522 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580502    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main"
Sep 18 16:56:11.580619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580524    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main"
Sep 18 16:56:11.580619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580541    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main"
Sep 18 16:56:11.580619 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580549    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main"
Sep 18 16:56:11.580766 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580654    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4320e0c-cef6-4bac-bec7-32829d7c9ee9" containerName="main"
Sep 18 16:56:11.580766 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.580672    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bff433c-ddac-4dc9-9b35-e37a640d92a7" containerName="main"
Sep 18 16:56:11.583721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.583698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.586297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.586274    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 16:56:11.587400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.587379    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-khwgj\""
Sep 18 16:56:11.587545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.587451    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 16:56:11.587545 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.587476    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 16:56:11.595297 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.595271    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:56:11.611529 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.611487    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:11.676874 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.676840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.676874 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.676879    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtqnl\" (UniqueName: \"kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.677110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.676909    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.677110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.677000    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.677110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.677048    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.677110 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.677097    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778187 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778492 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778229    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778492 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778492 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778295    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtqnl\" (UniqueName: \"kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778492 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.778933 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.778905    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.779029 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.779005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.779220 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.779194    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.785377 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.785349    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.786069 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.786047    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.787737 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.787716    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtqnl\" (UniqueName: \"kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.850905 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.850815    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:56:11.854499 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.854475    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.857244 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.857213    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qctbw\""
Sep 18 16:56:11.863292 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.863262    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:56:11.874683 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.874655    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:11.878877 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.878854    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmwp\" (UniqueName: \"kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.878973 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.878904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.894820 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.894791    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:11.980107 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.980073    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmwp\" (UniqueName: \"kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.980273 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.980153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.982791 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.982759    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:11.989179 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:11.989150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmwp\" (UniqueName: \"kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:12.165517 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.165365    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:12.165950 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.165917    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:56:12.168232 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:56:12.168209    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d70f32_6841_441e_897a_b2ecb186b0b9.slice/crio-8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74 WatchSource:0}: Error finding container 8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74: Status 404 returned error can't find the container with id 8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74
Sep 18 16:56:12.439046 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.439018    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:56:12.441596 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:56:12.441563    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436c9e88_e268_4cb2_83b0_036eb789f019.slice/crio-ced401f93ab0c6028fc9bb587b445b4b45303405ce3e45867b1f64e1d6a44c6a WatchSource:0}: Error finding container ced401f93ab0c6028fc9bb587b445b4b45303405ce3e45867b1f64e1d6a44c6a: Status 404 returned error can't find the container with id ced401f93ab0c6028fc9bb587b445b4b45303405ce3e45867b1f64e1d6a44c6a
Sep 18 16:56:12.725858 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.725816    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" event={"ID":"436c9e88-e268-4cb2-83b0-036eb789f019","Type":"ContainerStarted","Data":"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"}
Sep 18 16:56:12.726332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.725879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" event={"ID":"436c9e88-e268-4cb2-83b0-036eb789f019","Type":"ContainerStarted","Data":"ced401f93ab0c6028fc9bb587b445b4b45303405ce3e45867b1f64e1d6a44c6a"}
Sep 18 16:56:12.726332 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.726011    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:12.727715 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.727691    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" event={"ID":"f5d70f32-6841-441e-897a-b2ecb186b0b9","Type":"ContainerStarted","Data":"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"}
Sep 18 16:56:12.727868 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.727735    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" event={"ID":"f5d70f32-6841-441e-897a-b2ecb186b0b9","Type":"ContainerStarted","Data":"8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74"}
Sep 18 16:56:12.745669 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.745613    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podStartSLOduration=1.745596427 podStartE2EDuration="1.745596427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:12.74371751 +0000 UTC m=+2444.223691057" watchObservedRunningTime="2026-09-18 16:56:12.745596427 +0000 UTC m=+2444.225569973"
Sep 18 16:56:12.764117 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.764057    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" podStartSLOduration=1.764038928 podStartE2EDuration="1.764038928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:12.762088293 +0000 UTC m=+2444.242061839" watchObservedRunningTime="2026-09-18 16:56:12.764038928 +0000 UTC m=+2444.244012491"
Sep 18 16:56:12.895631 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:12.895587    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:17.714528 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:17.714487    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:21.895614 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:21.895562    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:21.907961 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:21.907931    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:22.788591 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:22.788559    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:56:43.734087 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:43.734058    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:56:48.715723 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.715685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:48.908698 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.908668    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:56:48.912488 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.912461    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:48.916217 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.916190    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 16:56:48.916400 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.916238    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-znpdf\""
Sep 18 16:56:48.921805 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:48.921779    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:56:49.020940 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.020911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.021121 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.020987    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7r78\" (UniqueName: \"kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.122320 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.122277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.122548 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.122399    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7r78\" (UniqueName: \"kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.125065 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.125034    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.131253 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.131228    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7r78\" (UniqueName: \"kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.224137 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.224093    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.494635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.494607    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:56:49.497075 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:56:49.497039    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15116b72_079b_4810_b429_23b18d5edc86.slice/crio-54fdf5cad3c9d11a8ddde8d73df7cc6cdc2d11e4f97ddecc84f15ef36a1309ef WatchSource:0}: Error finding container 54fdf5cad3c9d11a8ddde8d73df7cc6cdc2d11e4f97ddecc84f15ef36a1309ef: Status 404 returned error can't find the container with id 54fdf5cad3c9d11a8ddde8d73df7cc6cdc2d11e4f97ddecc84f15ef36a1309ef
Sep 18 16:56:49.741949 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.741916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:49.915218 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.915135    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" event={"ID":"15116b72-079b-4810-b429-23b18d5edc86","Type":"ContainerStarted","Data":"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"}
Sep 18 16:56:49.915218 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.915190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" event={"ID":"15116b72-079b-4810-b429-23b18d5edc86","Type":"ContainerStarted","Data":"54fdf5cad3c9d11a8ddde8d73df7cc6cdc2d11e4f97ddecc84f15ef36a1309ef"}
Sep 18 16:56:49.915406 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.915276    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:56:49.932994 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:56:49.932939    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podStartSLOduration=1.932918837 podStartE2EDuration="1.932918837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:49.932915734 +0000 UTC m=+2481.412889281" watchObservedRunningTime="2026-09-18 16:56:49.932918837 +0000 UTC m=+2481.412892397"
Sep 18 16:57:13.799922 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:13.799882    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:57:20.923107 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:20.923068    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:57:22.399511 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:22.399368    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:57:22.496369 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:22.496332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:57:22.805790 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:22.805752    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:57:22.817398 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:22.817362    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:57:22.903285 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:22.903244    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:57:30.922928 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:30.922890    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:37.848734 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:37.848662    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main" containerID="cri-o://be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098" gracePeriod=45
Sep 18 16:57:40.922640 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:40.922601    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:50.922535 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:50.922491    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:50.922944 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:57:50.922596    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:58:00.922313 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:00.922271    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:08.096621 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.096592    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:58:08.247245 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.247210    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7r78\" (UniqueName: \"kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78\") pod \"15116b72-079b-4810-b429-23b18d5edc86\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") "
Sep 18 16:58:08.247486 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.247465    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs\") pod \"15116b72-079b-4810-b429-23b18d5edc86\" (UID: \"15116b72-079b-4810-b429-23b18d5edc86\") "
Sep 18 16:58:08.249616 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.249582    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15116b72-079b-4810-b429-23b18d5edc86" (UID: "15116b72-079b-4810-b429-23b18d5edc86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:58:08.249737 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.249630    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78" (OuterVolumeSpecName: "kube-api-access-b7r78") pod "15116b72-079b-4810-b429-23b18d5edc86" (UID: "15116b72-079b-4810-b429-23b18d5edc86"). InnerVolumeSpecName "kube-api-access-b7r78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:58:08.309599 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.309555    2554 generic.go:356] "Generic (PLEG): container finished" podID="15116b72-079b-4810-b429-23b18d5edc86" containerID="be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098" exitCode=0
Sep 18 16:58:08.309788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.309604    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"
Sep 18 16:58:08.309788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.309642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" event={"ID":"15116b72-079b-4810-b429-23b18d5edc86","Type":"ContainerDied","Data":"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"}
Sep 18 16:58:08.309788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.309701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq" event={"ID":"15116b72-079b-4810-b429-23b18d5edc86","Type":"ContainerDied","Data":"54fdf5cad3c9d11a8ddde8d73df7cc6cdc2d11e4f97ddecc84f15ef36a1309ef"}
Sep 18 16:58:08.309788 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.309724    2554 scope.go:117] "RemoveContainer" containerID="be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"
Sep 18 16:58:08.319248 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.319228    2554 scope.go:117] "RemoveContainer" containerID="be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"
Sep 18 16:58:08.319597 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:08.319572    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098\": container with ID starting with be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098 not found: ID does not exist" containerID="be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"
Sep 18 16:58:08.319666 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.319611    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098"} err="failed to get container status \"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098\": rpc error: code = NotFound desc = could not find container \"be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098\": container with ID starting with be77b396de0d838113c266b0f70f544497badb209004270e7cfaeaa1cc88f098 not found: ID does not exist"
Sep 18 16:58:08.331872 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.331832    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:58:08.337458 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.337426    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schejchsq"]
Sep 18 16:58:08.348985 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.348956    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15116b72-079b-4810-b429-23b18d5edc86-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:08.348985 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.348986    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7r78\" (UniqueName: \"kubernetes.io/projected/15116b72-079b-4810-b429-23b18d5edc86-kube-api-access-b7r78\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:08.996996 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:08.996962    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15116b72-079b-4810-b429-23b18d5edc86" path="/var/lib/kubelet/pods/15116b72-079b-4810-b429-23b18d5edc86/volumes"
Sep 18 16:58:11.947816 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:11.947784    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:12.048323 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.048294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:12.328658 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.328621    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:12.412844 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.412804    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:12.688403 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.688300    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:12.708489 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.708456    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:58:12.711540 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.711509    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:58:12.720752 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:12.720715    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:58:12.720752 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:58:12.720752 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:58:12.720752 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239" cmd=["/bin/sleep","15"]
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.720768    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" message="" err=<
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:12.720781    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main" containerID="cri-o://493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"
Sep 18 16:58:12.721090 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.720821    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main" containerID="cri-o://493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239" gracePeriod=60
Sep 18 16:58:12.777912 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.777860    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 16:58:12.788135 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:12.788106    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:13.084803 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.084533    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:58:13.197708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197629    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.197708 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197670    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.197904 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197783    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.197904 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197853    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.197904 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197897    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.198036 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197899    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:58:13.198036 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197908    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:58:13.198036 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197940    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtqnl\" (UniqueName: \"kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl\") pod \"f5d70f32-6841-441e-897a-b2ecb186b0b9\" (UID: \"f5d70f32-6841-441e-897a-b2ecb186b0b9\") "
Sep 18 16:58:13.198036 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.197994    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home" (OuterVolumeSpecName: "home") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:58:13.198285 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.198262    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.198364 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.198288    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.198364 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.198299    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.200062 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.200036    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:58:13.200401 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.200380    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm" (OuterVolumeSpecName: "dshm") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:58:13.200467 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.200407    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl" (OuterVolumeSpecName: "kube-api-access-mtqnl") pod "f5d70f32-6841-441e-897a-b2ecb186b0b9" (UID: "f5d70f32-6841-441e-897a-b2ecb186b0b9"). InnerVolumeSpecName "kube-api-access-mtqnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:58:13.298778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.298741    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d70f32-6841-441e-897a-b2ecb186b0b9-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.298778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.298774    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d70f32-6841-441e-897a-b2ecb186b0b9-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.298778 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.298785    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtqnl\" (UniqueName: \"kubernetes.io/projected/f5d70f32-6841-441e-897a-b2ecb186b0b9-kube-api-access-mtqnl\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:13.336927 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.336888    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerID="493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239" exitCode=0
Sep 18 16:58:13.337102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.336933    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"
Sep 18 16:58:13.337102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.336996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" event={"ID":"f5d70f32-6841-441e-897a-b2ecb186b0b9","Type":"ContainerDied","Data":"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"}
Sep 18 16:58:13.337102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.337051    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx" event={"ID":"f5d70f32-6841-441e-897a-b2ecb186b0b9","Type":"ContainerDied","Data":"8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74"}
Sep 18 16:58:13.337102 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.337085    2554 scope.go:117] "RemoveContainer" containerID="493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"
Sep 18 16:58:13.347481 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.347461    2554 scope.go:117] "RemoveContainer" containerID="493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"
Sep 18 16:58:13.347811 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:13.347789    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239\": container with ID starting with 493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239 not found: ID does not exist" containerID="493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"
Sep 18 16:58:13.347887 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.347817    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239"} err="failed to get container status \"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239\": rpc error: code = NotFound desc = could not find container \"493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239\": container with ID starting with 493ad092badebc086e5c47b9eaa165f518c18e52be4b03c841b58297ce62c239 not found: ID does not exist"
Sep 18 16:58:13.361518 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.361480    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:58:13.366087 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.366058    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-h2sqx"]
Sep 18 16:58:13.366596 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:13.366567    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d70f32_6841_441e_897a_b2ecb186b0b9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d70f32_6841_441e_897a_b2ecb186b0b9.slice/crio-8b216c08726500ba47519e1a075bd4207d60417f41dac4131fc67ca18408ec74\": RecentStats: unable to find data in memory cache]"
Sep 18 16:58:13.734797 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:13.734761    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:14.996812 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:14.996777    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" path="/var/lib/kubelet/pods/f5d70f32-6841-441e-897a-b2ecb186b0b9/volumes"
Sep 18 16:58:15.912721 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:15.912692    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:18.361786 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:18.361749    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:19.917305 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:19.917264    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:23.734013 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:23.733973    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:24.066012 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:24.065933    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:24.130221 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:24.130157    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:58:27.732801 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:27.732736    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" containerID="cri-o://572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88" gracePeriod=45
Sep 18 16:58:33.734255 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:33.734213    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:33.734814 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:33.734323    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:58:43.734589 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:43.734550    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:53.734568 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:53.734482    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:57.992707 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:57.992680    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:58:58.116666 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.116566    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs\") pod \"436c9e88-e268-4cb2-83b0-036eb789f019\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") "
Sep 18 16:58:58.116666 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.116613    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqmwp\" (UniqueName: \"kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp\") pod \"436c9e88-e268-4cb2-83b0-036eb789f019\" (UID: \"436c9e88-e268-4cb2-83b0-036eb789f019\") "
Sep 18 16:58:58.118839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.118801    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp" (OuterVolumeSpecName: "kube-api-access-cqmwp") pod "436c9e88-e268-4cb2-83b0-036eb789f019" (UID: "436c9e88-e268-4cb2-83b0-036eb789f019"). InnerVolumeSpecName "kube-api-access-cqmwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:58:58.118839 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.118826    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "436c9e88-e268-4cb2-83b0-036eb789f019" (UID: "436c9e88-e268-4cb2-83b0-036eb789f019"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:58:58.218288 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.218250    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436c9e88-e268-4cb2-83b0-036eb789f019-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:58.218288 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.218283    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmwp\" (UniqueName: \"kubernetes.io/projected/436c9e88-e268-4cb2-83b0-036eb789f019-kube-api-access-cqmwp\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 16:58:58.552210 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.552174    2554 generic.go:356] "Generic (PLEG): container finished" podID="436c9e88-e268-4cb2-83b0-036eb789f019" containerID="572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88" exitCode=0
Sep 18 16:58:58.552384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.552217    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"
Sep 18 16:58:58.552384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.552236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" event={"ID":"436c9e88-e268-4cb2-83b0-036eb789f019","Type":"ContainerDied","Data":"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"}
Sep 18 16:58:58.552384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.552280    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj" event={"ID":"436c9e88-e268-4cb2-83b0-036eb789f019","Type":"ContainerDied","Data":"ced401f93ab0c6028fc9bb587b445b4b45303405ce3e45867b1f64e1d6a44c6a"}
Sep 18 16:58:58.552384 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.552301    2554 scope.go:117] "RemoveContainer" containerID="572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"
Sep 18 16:58:58.561496 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.561477    2554 scope.go:117] "RemoveContainer" containerID="572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"
Sep 18 16:58:58.561783 ip-10-0-142-62 kubenswrapper[2554]: E0918 16:58:58.561762    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88\": container with ID starting with 572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88 not found: ID does not exist" containerID="572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"
Sep 18 16:58:58.561861 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.561790    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88"} err="failed to get container status \"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88\": rpc error: code = NotFound desc = could not find container \"572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88\": container with ID starting with 572df74113609e896144b6bd9973d97721f4159164eb5f737f2e3216cb358a88 not found: ID does not exist"
Sep 18 16:58:58.573930 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.573901    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:58:58.577948 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.577922    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schef8lxj"]
Sep 18 16:58:58.997106 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:58:58.997076    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" path="/var/lib/kubelet/pods/436c9e88-e268-4cb2-83b0-036eb789f019/volumes"
Sep 18 16:59:02.216228 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:02.216184    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:02.313945 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:02.313884    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:08.901484 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:08.901449    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:11.080627 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.080584    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081010    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main"
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081029    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main"
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081056    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main"
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081065    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main"
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081091    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main"
Sep 18 16:59:11.081098 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081098    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main"
Sep 18 16:59:11.081441 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081199    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="436c9e88-e268-4cb2-83b0-036eb789f019" containerName="main"
Sep 18 16:59:11.081441 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081211    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15116b72-079b-4810-b429-23b18d5edc86" containerName="main"
Sep 18 16:59:11.081441 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.081221    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d70f32-6841-441e-897a-b2ecb186b0b9" containerName="main"
Sep 18 16:59:11.083627 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.083610    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.086779 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.086760    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 16:59:11.087644 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.087624    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 16:59:11.087747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.087676    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-v27sr\""
Sep 18 16:59:11.087747 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.087704    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 16:59:11.102208 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.102177    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 16:59:11.133105 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.133303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxmt\" (UniqueName: \"kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.133303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.133303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.133303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133234    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.133303 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.133260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.134478 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.134456    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:11.234504 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxmt\" (UniqueName: \"kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.234504 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.234789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.234789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.234789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.234789 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.234661    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.235031 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.235008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.235077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.235031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.235077 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.235059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.236960 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.236939    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.237138 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.237119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.243652 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.243626    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxmt\" (UniqueName: \"kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.324796 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.324764    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:11.393691 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.393615    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:11.665635 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:11.665564    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 16:59:11.668071 ip-10-0-142-62 kubenswrapper[2554]: W0918 16:59:11.668041    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b71f6b_1c8a_41c7_811f_78207494dd4c.slice/crio-79e1a10c9de8b6e187673215c3de51af17bfb1f7f5ce557e918813700c30de51 WatchSource:0}: Error finding container 79e1a10c9de8b6e187673215c3de51af17bfb1f7f5ce557e918813700c30de51: Status 404 returned error can't find the container with id 79e1a10c9de8b6e187673215c3de51af17bfb1f7f5ce557e918813700c30de51
Sep 18 16:59:12.403334 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:12.403298    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:12.621275 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:12.621240    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" event={"ID":"a5b71f6b-1c8a-41c7-811f-78207494dd4c","Type":"ContainerStarted","Data":"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"}
Sep 18 16:59:12.621461 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:12.621290    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" event={"ID":"a5b71f6b-1c8a-41c7-811f-78207494dd4c","Type":"ContainerStarted","Data":"79e1a10c9de8b6e187673215c3de51af17bfb1f7f5ce557e918813700c30de51"}
Sep 18 16:59:12.639196 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:12.639143    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" podStartSLOduration=1.6391277990000002 podStartE2EDuration="1.639127799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:59:12.637564777 +0000 UTC m=+2624.117538324" watchObservedRunningTime="2026-09-18 16:59:12.639127799 +0000 UTC m=+2624.119101389"
Sep 18 16:59:13.394600 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:13.394565    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:14.897654 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:14.897617    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:14.968781 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:14.968747    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:21.394586 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:21.394549    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:21.407251 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:21.407218    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:21.676224 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:21.676143    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 16:59:27.979146 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:27.979110    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:29.601204 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:29.601115    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:29.666280 ip-10-0-142-62 kubenswrapper[2554]: I0918 16:59:29.666247    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:01.234953 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.234914    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:01.428470 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.428435    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:00:01.431770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.431746    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.437783 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.437753    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-2hqvh\""
Sep 18 17:00:01.437986 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.437812    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 17:00:01.450636 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.450605    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:00:01.485492 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.485376    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.485492 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.485430    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvsp\" (UniqueName: \"kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.551322 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.551281    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:01.586826 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.586787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.587022 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.586831    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvsp\" (UniqueName: \"kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.589395 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.589361    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.598607 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.598578    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvsp\" (UniqueName: \"kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:01.743220 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:01.743141    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:02.016948 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:02.016918    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:00:02.019248 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:00:02.019219    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51f51d7_9bcc_473e_a044_0ecc65fe8dfe.slice/crio-bbd7af7cdccb4b816f89736af991e97b026bbea1b8116e2c5dd6d5a2d64cb9a9 WatchSource:0}: Error finding container bbd7af7cdccb4b816f89736af991e97b026bbea1b8116e2c5dd6d5a2d64cb9a9: Status 404 returned error can't find the container with id bbd7af7cdccb4b816f89736af991e97b026bbea1b8116e2c5dd6d5a2d64cb9a9
Sep 18 17:00:02.860102 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:02.860061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" event={"ID":"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe","Type":"ContainerStarted","Data":"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"}
Sep 18 17:00:02.860619 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:02.860122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" event={"ID":"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe","Type":"ContainerStarted","Data":"bbd7af7cdccb4b816f89736af991e97b026bbea1b8116e2c5dd6d5a2d64cb9a9"}
Sep 18 17:00:02.860619 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:02.860151    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:02.878628 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:02.878571    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podStartSLOduration=1.8785532919999999 podStartE2EDuration="1.878553292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:00:02.876737906 +0000 UTC m=+2674.356711453" watchObservedRunningTime="2026-09-18 17:00:02.878553292 +0000 UTC m=+2674.358526837"
Sep 18 17:00:19.572784 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:19.572748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:29.178553 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:29.178522    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:00:29.184606 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:29.184582    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:00:33.866460 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:33.866401    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:00:35.748255 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:35.748213    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:00:35.748911 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:35.748872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:35.815125 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:35.815089    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:37.491490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:37.491456    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:43.426277 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:43.426240    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:43.866737 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:43.866702    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:00:48.161868 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:48.161834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:48.261570 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:48.261539    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:50.768063 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:50.768022    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main" containerID="cri-o://1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1" gracePeriod=45
Sep 18 17:00:53.179627 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:53.179596    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:00:53.866404 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:53.866368    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:00:55.844571 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:00:55.844530    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:03.866837 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:03.866794    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:01:03.867334 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:03.866911    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:01:04.604515 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:04.604481    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:04.899847 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:04.899761    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:12.945089 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:12.945045    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:13.866956 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:13.866910    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:01:15.220480 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:15.220443    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:15.694853 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:15.694768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:21.016609 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.016576    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:01:21.137244 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.137150    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs\") pod \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") "
Sep 18 17:01:21.137463 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.137286    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slvsp\" (UniqueName: \"kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp\") pod \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\" (UID: \"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe\") "
Sep 18 17:01:21.139518 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.139481    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" (UID: "c51f51d7-9bcc-473e-a044-0ecc65fe8dfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:01:21.139624 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.139500    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp" (OuterVolumeSpecName: "kube-api-access-slvsp") pod "c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" (UID: "c51f51d7-9bcc-473e-a044-0ecc65fe8dfe"). InnerVolumeSpecName "kube-api-access-slvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:01:21.238211 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.238179    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvsp\" (UniqueName: \"kubernetes.io/projected/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-kube-api-access-slvsp\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:21.238211 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.238203    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:21.240442 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.240399    2554 generic.go:356] "Generic (PLEG): container finished" podID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerID="1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1" exitCode=0
Sep 18 17:01:21.240559 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.240451    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" event={"ID":"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe","Type":"ContainerDied","Data":"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"}
Sep 18 17:01:21.240559 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.240468    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"
Sep 18 17:01:21.240559 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.240508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm" event={"ID":"c51f51d7-9bcc-473e-a044-0ecc65fe8dfe","Type":"ContainerDied","Data":"bbd7af7cdccb4b816f89736af991e97b026bbea1b8116e2c5dd6d5a2d64cb9a9"}
Sep 18 17:01:21.240559 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.240529    2554 scope.go:117] "RemoveContainer" containerID="1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"
Sep 18 17:01:21.249688 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.249663    2554 scope.go:117] "RemoveContainer" containerID="1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"
Sep 18 17:01:21.249992 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:01:21.249963    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1\": container with ID starting with 1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1 not found: ID does not exist" containerID="1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"
Sep 18 17:01:21.250072 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.250009    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1"} err="failed to get container status \"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1\": rpc error: code = NotFound desc = could not find container \"1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1\": container with ID starting with 1b4c7dd53faafbb1ff5f402b2d8c866ad8cad486ba1e1049425f8a0d4c8966b1 not found: ID does not exist"
Sep 18 17:01:21.261920 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.261890    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:01:21.266407 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.266378    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4fffm"]
Sep 18 17:01:21.796596 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:21.796562    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:22.996806 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:22.996769    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" path="/var/lib/kubelet/pods/c51f51d7-9bcc-473e-a044-0ecc65fe8dfe/volumes"
Sep 18 17:01:25.815241 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:25.815206    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 17:01:25.826117 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:01:25.826079    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:01:25.826117 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:01:25.826117 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:01:25.826117 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173" cmd=["/bin/sleep","15"]
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:25.826138    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" message="" err=<
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:01:25.826152    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerName="main" containerID="cri-o://16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"
Sep 18 17:01:25.826406 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:25.826204    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerName="main" containerID="cri-o://16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173" gracePeriod=60
Sep 18 17:01:25.829678 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:25.829647    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:25.890371 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:25.890331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:26.178729 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.178701    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 17:01:26.183004 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.182984    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183091 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183013    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183091 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183065    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsxmt\" (UniqueName: \"kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183172 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183133    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183172 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183164    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183253 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183196    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home\") pod \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\" (UID: \"a5b71f6b-1c8a-41c7-811f-78207494dd4c\") "
Sep 18 17:01:26.183296 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183266    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:01:26.183490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183471    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.183490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183456    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:01:26.183644 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.183530    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home" (OuterVolumeSpecName: "home") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:01:26.185334 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.185307    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:01:26.185334 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.185310    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt" (OuterVolumeSpecName: "kube-api-access-hsxmt") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "kube-api-access-hsxmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:01:26.185483 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.185332    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm" (OuterVolumeSpecName: "dshm") pod "a5b71f6b-1c8a-41c7-811f-78207494dd4c" (UID: "a5b71f6b-1c8a-41c7-811f-78207494dd4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:01:26.266128 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.266090    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerID="16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173" exitCode=0
Sep 18 17:01:26.266290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.266137    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"
Sep 18 17:01:26.266290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.266153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" event={"ID":"a5b71f6b-1c8a-41c7-811f-78207494dd4c","Type":"ContainerDied","Data":"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"}
Sep 18 17:01:26.266290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.266205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9" event={"ID":"a5b71f6b-1c8a-41c7-811f-78207494dd4c","Type":"ContainerDied","Data":"79e1a10c9de8b6e187673215c3de51af17bfb1f7f5ce557e918813700c30de51"}
Sep 18 17:01:26.266290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.266235    2554 scope.go:117] "RemoveContainer" containerID="16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"
Sep 18 17:01:26.275642 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.275613    2554 scope.go:117] "RemoveContainer" containerID="16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"
Sep 18 17:01:26.275976 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:01:26.275956    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173\": container with ID starting with 16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173 not found: ID does not exist" containerID="16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"
Sep 18 17:01:26.276036 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.275987    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173"} err="failed to get container status \"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173\": rpc error: code = NotFound desc = could not find container \"16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173\": container with ID starting with 16e75cf66c065d4a1c1763205fb598ae700f3bbec7ada80cf648a6053dd04173 not found: ID does not exist"
Sep 18 17:01:26.284423 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.284383    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b71f6b-1c8a-41c7-811f-78207494dd4c-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.284423 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.284407    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.284659 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.284431    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.284659 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.284443    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b71f6b-1c8a-41c7-811f-78207494dd4c-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.284659 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.284456    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxmt\" (UniqueName: \"kubernetes.io/projected/a5b71f6b-1c8a-41c7-811f-78207494dd4c-kube-api-access-hsxmt\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:01:26.288143 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.288115    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 17:01:26.292123 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.292096    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kh9b9"]
Sep 18 17:01:26.997426 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:26.997389    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" path="/var/lib/kubelet/pods/a5b71f6b-1c8a-41c7-811f-78207494dd4c/volumes"
Sep 18 17:01:41.911969 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:41.911936    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:45.469917 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:45.469881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:01:45.573761 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:01:45.573726    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:15.834898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:15.834770    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:17.163815 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:17.163782    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:22.794314 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:22.794274    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:27.200796 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.200757    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:27.428897 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.428861    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:02:27.429307 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429286    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main"
Sep 18 17:02:27.429307 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429302    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main"
Sep 18 17:02:27.429390 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429322    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerName="main"
Sep 18 17:02:27.429390 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429328    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerName="main"
Sep 18 17:02:27.429472 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429391    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b71f6b-1c8a-41c7-811f-78207494dd4c" containerName="main"
Sep 18 17:02:27.429472 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.429399    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51f51d7-9bcc-473e-a044-0ecc65fe8dfe" containerName="main"
Sep 18 17:02:27.432476 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.432455    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.436163 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.436136    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 17:02:27.436359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.436134    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 17:02:27.436359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.436140    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-h4cw4\""
Sep 18 17:02:27.436359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.436139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 17:02:27.436359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.436238    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-pvnzc\""
Sep 18 17:02:27.443101 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.443077    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:02:27.454833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.454733    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:27.526959 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.526924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.526959 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.526959    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrtm\" (UniqueName: \"kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.627454 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.627393    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.627454 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.627459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrtm\" (UniqueName: \"kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.629962 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.629941    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.640290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.640259    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrtm\" (UniqueName: \"kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:27.743301 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:27.743263    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:28.016683 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:28.016554    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:02:28.020138 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:02:28.020104    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc28a54_c0c0_4914_a389_4dd32373aba6.slice/crio-b1722e0e9e98e1c0556f9f17886745e51a9b6cbaa1033660e42b80958f4e5d6e WatchSource:0}: Error finding container b1722e0e9e98e1c0556f9f17886745e51a9b6cbaa1033660e42b80958f4e5d6e: Status 404 returned error can't find the container with id b1722e0e9e98e1c0556f9f17886745e51a9b6cbaa1033660e42b80958f4e5d6e
Sep 18 17:02:28.567134 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:28.567090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" event={"ID":"2fc28a54-c0c0-4914-a389-4dd32373aba6","Type":"ContainerStarted","Data":"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"}
Sep 18 17:02:28.567568 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:28.567154    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" event={"ID":"2fc28a54-c0c0-4914-a389-4dd32373aba6","Type":"ContainerStarted","Data":"b1722e0e9e98e1c0556f9f17886745e51a9b6cbaa1033660e42b80958f4e5d6e"}
Sep 18 17:02:28.567568 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:28.567212    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:02:28.584533 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:28.584473    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podStartSLOduration=1.584404339 podStartE2EDuration="1.584404339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:28.583498997 +0000 UTC m=+2820.063472543" watchObservedRunningTime="2026-09-18 17:02:28.584404339 +0000 UTC m=+2820.064377905"
Sep 18 17:02:29.523646 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:29.523579    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:29.666611 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:29.666576    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:35.198732 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:35.198691    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:59.575296 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:02:59.575267    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:03:19.736786 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:03:19.736751    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:03:19.837758 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:03:19.837723    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:09.740780 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:09.740723    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:09.837808 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:09.837774    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:16.098275 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:16.098241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:22.032264 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:22.032215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:22.133647 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:22.133614    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:31.618894 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.618852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:31.823837 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.823802    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:04:31.827279 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.827261    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:31.831267 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.831227    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 17:04:31.831267 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.831242    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 17:04:31.831267 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.831257    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 17:04:31.831548 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.831282    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-dwrb8\""
Sep 18 17:04:31.831548 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.831227    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-2dpcf\""
Sep 18 17:04:31.837938 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.837917    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:04:31.855017 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.854988    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:31.912186 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.912104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:31.912342 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:31.912187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fjf\" (UniqueName: \"kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.013112 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.013074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.013319 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.013135    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fjf\" (UniqueName: \"kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.015715 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.015689    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.021534 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.021508    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fjf\" (UniqueName: \"kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.138518 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.138488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:32.412966 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:32.412938    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:04:32.415410 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:04:32.415375    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dd4194_aa33_4099_b6d7_a613f922f5e2.slice/crio-50e6128d9e8ffe74df04a5cbe2aa81d1874b44a9fb437e77929970912392a99a WatchSource:0}: Error finding container 50e6128d9e8ffe74df04a5cbe2aa81d1874b44a9fb437e77929970912392a99a: Status 404 returned error can't find the container with id 50e6128d9e8ffe74df04a5cbe2aa81d1874b44a9fb437e77929970912392a99a
Sep 18 17:04:33.183049 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:33.183014    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" event={"ID":"45dd4194-aa33-4099-b6d7-a613f922f5e2","Type":"ContainerStarted","Data":"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"}
Sep 18 17:04:33.183458 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:33.183066    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" event={"ID":"45dd4194-aa33-4099-b6d7-a613f922f5e2","Type":"ContainerStarted","Data":"50e6128d9e8ffe74df04a5cbe2aa81d1874b44a9fb437e77929970912392a99a"}
Sep 18 17:04:33.183458 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:33.183151    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:04:33.231367 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:33.231291    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podStartSLOduration=2.231272188 podStartE2EDuration="2.231272188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:04:33.230382108 +0000 UTC m=+2944.710355654" watchObservedRunningTime="2026-09-18 17:04:33.231272188 +0000 UTC m=+2944.711245733"
Sep 18 17:04:38.870684 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:38.870643    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:04:50.406066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:04:50.406026    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:01.827379 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:01.827301    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:01.883364 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:01.883325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:04.189839 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:04.189808    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:05:04.256575 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:04.256545    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:04.270242 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:04.270211    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:05:04.357290 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:04.357253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:09.575049 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:09.575004    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:05:12.367059 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:12.367018    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:12.424182 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:12.424142    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:19.290689 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:19.290637    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main" containerID="cri-o://31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac" gracePeriod=45
Sep 18 17:05:19.575176 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:19.575096    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:05:22.726680 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:22.726648    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:29.208303 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:29.208271    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:05:29.216686 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:29.216664    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:05:29.574615 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:29.574573    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:05:29.574813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:29.574693    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:05:39.574901 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:39.574858    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:05:49.546301 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.546272    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:05:49.553439 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.553361    2554 generic.go:356] "Generic (PLEG): container finished" podID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerID="31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac" exitCode=0
Sep 18 17:05:49.553439 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.553395    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"
Sep 18 17:05:49.553594 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.553395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" event={"ID":"2fc28a54-c0c0-4914-a389-4dd32373aba6","Type":"ContainerDied","Data":"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"}
Sep 18 17:05:49.553594 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.553489    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr" event={"ID":"2fc28a54-c0c0-4914-a389-4dd32373aba6","Type":"ContainerDied","Data":"b1722e0e9e98e1c0556f9f17886745e51a9b6cbaa1033660e42b80958f4e5d6e"}
Sep 18 17:05:49.553594 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.553515    2554 scope.go:117] "RemoveContainer" containerID="31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"
Sep 18 17:05:49.561746 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.561726    2554 scope.go:117] "RemoveContainer" containerID="31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"
Sep 18 17:05:49.562034 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:05:49.562013    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac\": container with ID starting with 31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac not found: ID does not exist" containerID="31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"
Sep 18 17:05:49.562083 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.562043    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac"} err="failed to get container status \"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac\": rpc error: code = NotFound desc = could not find container \"31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac\": container with ID starting with 31665bcc059eec20fe015b32fd2ef21291860b9119ca2cc81c5f415814920cac not found: ID does not exist"
Sep 18 17:05:49.634132 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.634097    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs\") pod \"2fc28a54-c0c0-4914-a389-4dd32373aba6\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") "
Sep 18 17:05:49.634315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.634161    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsrtm\" (UniqueName: \"kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm\") pod \"2fc28a54-c0c0-4914-a389-4dd32373aba6\" (UID: \"2fc28a54-c0c0-4914-a389-4dd32373aba6\") "
Sep 18 17:05:49.636325 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.636299    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm" (OuterVolumeSpecName: "kube-api-access-xsrtm") pod "2fc28a54-c0c0-4914-a389-4dd32373aba6" (UID: "2fc28a54-c0c0-4914-a389-4dd32373aba6"). InnerVolumeSpecName "kube-api-access-xsrtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:05:49.636399 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.636312    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fc28a54-c0c0-4914-a389-4dd32373aba6" (UID: "2fc28a54-c0c0-4914-a389-4dd32373aba6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:05:49.735078 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.735041    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsrtm\" (UniqueName: \"kubernetes.io/projected/2fc28a54-c0c0-4914-a389-4dd32373aba6-kube-api-access-xsrtm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:05:49.735078 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.735073    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc28a54-c0c0-4914-a389-4dd32373aba6-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:05:49.881484 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.881433    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:05:49.881670 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:49.881634    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schetlwfr"]
Sep 18 17:05:50.997356 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:50.997319    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" path="/var/lib/kubelet/pods/2fc28a54-c0c0-4914-a389-4dd32373aba6/volumes"
Sep 18 17:05:54.314549 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:54.314510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:05:54.419888 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:05:54.419852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:00.663608 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:00.663570    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:00.832713 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:00.832680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:01.042189 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.042152    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:06:01.042557 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.042543    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main"
Sep 18 17:06:01.042557 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.042558    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main"
Sep 18 17:06:01.042649 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.042620    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc28a54-c0c0-4914-a389-4dd32373aba6" containerName="main"
Sep 18 17:06:01.047136 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.047106    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.050954 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.050925    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-q6d7q\""
Sep 18 17:06:01.051100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.050978    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 17:06:01.051100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.051018    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 17:06:01.051100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.051029    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 17:06:01.051100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.051023    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-hfvrz\""
Sep 18 17:06:01.054040 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.054017    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:06:01.069670 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.069619    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:01.141380 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.141344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.141587 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.141451    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd27w\" (UniqueName: \"kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.242948 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.242913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.243099 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.243002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd27w\" (UniqueName: \"kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.245582 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.245563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.250877 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.250853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd27w\" (UniqueName: \"kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.358268 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.358180    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:01.638899 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:01.638782    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:06:01.642801 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:06:01.642771    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc2f0dd_04b9_42b5_833c_f931c9585341.slice/crio-a0a814c5289aba68fd53740ab0be25b9b2bd28af95f60bc27e2235065ae67eaa WatchSource:0}: Error finding container a0a814c5289aba68fd53740ab0be25b9b2bd28af95f60bc27e2235065ae67eaa: Status 404 returned error can't find the container with id a0a814c5289aba68fd53740ab0be25b9b2bd28af95f60bc27e2235065ae67eaa
Sep 18 17:06:02.617771 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:02.617731    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" event={"ID":"ecc2f0dd-04b9-42b5-833c-f931c9585341","Type":"ContainerStarted","Data":"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"}
Sep 18 17:06:02.618157 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:02.617794    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" event={"ID":"ecc2f0dd-04b9-42b5-833c-f931c9585341","Type":"ContainerStarted","Data":"a0a814c5289aba68fd53740ab0be25b9b2bd28af95f60bc27e2235065ae67eaa"}
Sep 18 17:06:02.618157 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:02.617841    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:02.637387 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:02.637329    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podStartSLOduration=1.637306766 podStartE2EDuration="1.637306766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:06:02.635898459 +0000 UTC m=+3034.115871998" watchObservedRunningTime="2026-09-18 17:06:02.637306766 +0000 UTC m=+3034.117280313"
Sep 18 17:06:06.385083 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:06.385037    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:06.465941 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:06.465904    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:18.189552 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:18.189512    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:20.857553 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:20.857504    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:33.624590 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:33.624557    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:06:47.469268 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:47.469230    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:47.482565 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:47.482529    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:06:47.543218 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:47.543181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:54.190375 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:06:54.190336    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:02.510663 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:02.510624    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main" containerID="cri-o://dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a" gracePeriod=45
Sep 18 17:07:04.190113 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:04.190072    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:14.190477 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:14.190409    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:14.190951 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:14.190547    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:07:22.790495 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:22.790463    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:07:22.794863 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:22.794833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:23.624134 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:23.624087    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:24.190046 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:24.190011    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:24.338976 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:24.338937    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:32.754894 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.754866    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:07:32.895950 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.895858    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5fjf\" (UniqueName: \"kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf\") pod \"45dd4194-aa33-4099-b6d7-a613f922f5e2\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") "
Sep 18 17:07:32.895950 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.895923    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs\") pod \"45dd4194-aa33-4099-b6d7-a613f922f5e2\" (UID: \"45dd4194-aa33-4099-b6d7-a613f922f5e2\") "
Sep 18 17:07:32.898114 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.898085    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45dd4194-aa33-4099-b6d7-a613f922f5e2" (UID: "45dd4194-aa33-4099-b6d7-a613f922f5e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:07:32.898222 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.898142    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf" (OuterVolumeSpecName: "kube-api-access-t5fjf") pod "45dd4194-aa33-4099-b6d7-a613f922f5e2" (UID: "45dd4194-aa33-4099-b6d7-a613f922f5e2"). InnerVolumeSpecName "kube-api-access-t5fjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:32.996523 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.996491    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fjf\" (UniqueName: \"kubernetes.io/projected/45dd4194-aa33-4099-b6d7-a613f922f5e2-kube-api-access-t5fjf\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:07:32.996523 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:32.996518    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dd4194-aa33-4099-b6d7-a613f922f5e2-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:07:33.067929 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.067897    2554 generic.go:356] "Generic (PLEG): container finished" podID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerID="dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a" exitCode=0
Sep 18 17:07:33.068108 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.067944    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"
Sep 18 17:07:33.068108 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.067949    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" event={"ID":"45dd4194-aa33-4099-b6d7-a613f922f5e2","Type":"ContainerDied","Data":"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"}
Sep 18 17:07:33.068108 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.068048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r" event={"ID":"45dd4194-aa33-4099-b6d7-a613f922f5e2","Type":"ContainerDied","Data":"50e6128d9e8ffe74df04a5cbe2aa81d1874b44a9fb437e77929970912392a99a"}
Sep 18 17:07:33.068108 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.068072    2554 scope.go:117] "RemoveContainer" containerID="dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"
Sep 18 17:07:33.076839 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.076821    2554 scope.go:117] "RemoveContainer" containerID="dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"
Sep 18 17:07:33.077102 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:07:33.077079    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a\": container with ID starting with dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a not found: ID does not exist" containerID="dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"
Sep 18 17:07:33.077152 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.077111    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a"} err="failed to get container status \"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a\": rpc error: code = NotFound desc = could not find container \"dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a\": container with ID starting with dcc9d155d1248e699f5e4abd2ec8d975abda6e0c71929dfaa28603c71454879a not found: ID does not exist"
Sep 18 17:07:33.084707 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.084679    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:07:33.089002 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.088980    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2rt4r"]
Sep 18 17:07:33.624360 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:33.624318    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:34.996597 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:34.996560    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" path="/var/lib/kubelet/pods/45dd4194-aa33-4099-b6d7-a613f922f5e2/volumes"
Sep 18 17:07:37.519133 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:37.519099    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:37.649776 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:37.649743    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:37.810949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:37.810865    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" containerID="cri-o://c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db" gracePeriod=45
Sep 18 17:07:41.035516 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.035473    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:41.214287 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.214253    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:07:41.214649 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.214635    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main"
Sep 18 17:07:41.214723 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.214652    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main"
Sep 18 17:07:41.214763 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.214736    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="45dd4194-aa33-4099-b6d7-a613f922f5e2" containerName="main"
Sep 18 17:07:41.217796 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.217772    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.221149 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.221116    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 17:07:41.222524 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.222494    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-x5nt8\""
Sep 18 17:07:41.222642 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.222524    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 17:07:41.222642 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.222523    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 17:07:41.222642 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.222554    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-pg7hq\""
Sep 18 17:07:41.229257 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.229232    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:07:41.252358 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.252326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:41.372405 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.372304    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclq6\" (UniqueName: \"kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.372405 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.372365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.472847 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.472809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclq6\" (UniqueName: \"kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.473035 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.472871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.475327 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.475307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.481116 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.481090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclq6\" (UniqueName: \"kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.529300 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.529260    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:41.801603 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:41.801566    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:07:41.805021 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:07:41.804992    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe25145_eee7_467e_832f_cb1a0a70adc5.slice/crio-50f33b36b63bd4a8b469a362f35b2f7eb2d728f745b06614c0e9a3c274493afd WatchSource:0}: Error finding container 50f33b36b63bd4a8b469a362f35b2f7eb2d728f745b06614c0e9a3c274493afd: Status 404 returned error can't find the container with id 50f33b36b63bd4a8b469a362f35b2f7eb2d728f745b06614c0e9a3c274493afd
Sep 18 17:07:42.113812 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:42.113716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" event={"ID":"afe25145-eee7-467e-832f-cb1a0a70adc5","Type":"ContainerStarted","Data":"00f16a7ee1b311f71cba1565a3b2e307331beace48827634ce8a8db19a16df7b"}
Sep 18 17:07:42.113812 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:42.113774    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" event={"ID":"afe25145-eee7-467e-832f-cb1a0a70adc5","Type":"ContainerStarted","Data":"50f33b36b63bd4a8b469a362f35b2f7eb2d728f745b06614c0e9a3c274493afd"}
Sep 18 17:07:42.113812 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:42.113803    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:07:42.130163 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:42.130114    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podStartSLOduration=1.130097861 podStartE2EDuration="1.130097861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:42.129007935 +0000 UTC m=+3133.608981481" watchObservedRunningTime="2026-09-18 17:07:42.130097861 +0000 UTC m=+3133.610071407"
Sep 18 17:07:43.624381 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:43.624311    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:43.624827 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:43.624485    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:07:44.209641 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:44.209600    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:49.875356 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:49.875319    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:49.975200 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:49.975164    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:53.624285 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:53.624202    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:07:58.906620 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:07:58.906580    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:03.624784 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:03.624747    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:08:08.063429 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.063388    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:08:08.109614 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.109536    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd27w\" (UniqueName: \"kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w\") pod \"ecc2f0dd-04b9-42b5-833c-f931c9585341\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") "
Sep 18 17:08:08.109757 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.109664    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs\") pod \"ecc2f0dd-04b9-42b5-833c-f931c9585341\" (UID: \"ecc2f0dd-04b9-42b5-833c-f931c9585341\") "
Sep 18 17:08:08.111825 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.111789    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w" (OuterVolumeSpecName: "kube-api-access-rd27w") pod "ecc2f0dd-04b9-42b5-833c-f931c9585341" (UID: "ecc2f0dd-04b9-42b5-833c-f931c9585341"). InnerVolumeSpecName "kube-api-access-rd27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:08:08.111825 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.111806    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecc2f0dd-04b9-42b5-833c-f931c9585341" (UID: "ecc2f0dd-04b9-42b5-833c-f931c9585341"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:08:08.210917 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.210882    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2f0dd-04b9-42b5-833c-f931c9585341-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:08:08.210917 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.210914    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd27w\" (UniqueName: \"kubernetes.io/projected/ecc2f0dd-04b9-42b5-833c-f931c9585341-kube-api-access-rd27w\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:08:08.243673 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.243640    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerID="c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db" exitCode=0
Sep 18 17:08:08.243813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.243687    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"
Sep 18 17:08:08.243813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.243689    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" event={"ID":"ecc2f0dd-04b9-42b5-833c-f931c9585341","Type":"ContainerDied","Data":"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"}
Sep 18 17:08:08.243890 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.243851    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx" event={"ID":"ecc2f0dd-04b9-42b5-833c-f931c9585341","Type":"ContainerDied","Data":"a0a814c5289aba68fd53740ab0be25b9b2bd28af95f60bc27e2235065ae67eaa"}
Sep 18 17:08:08.243926 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.243887    2554 scope.go:117] "RemoveContainer" containerID="c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"
Sep 18 17:08:08.253563 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.253539    2554 scope.go:117] "RemoveContainer" containerID="c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"
Sep 18 17:08:08.253842 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:08:08.253823    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db\": container with ID starting with c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db not found: ID does not exist" containerID="c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"
Sep 18 17:08:08.253903 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.253851    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db"} err="failed to get container status \"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db\": rpc error: code = NotFound desc = could not find container \"c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db\": container with ID starting with c1f7678142012fa38e69e74e7d4137f2575129a11bb1c454aba57569769446db not found: ID does not exist"
Sep 18 17:08:08.264900 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.264877    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:08:08.271173 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.271148    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sches5wwx"]
Sep 18 17:08:08.996873 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:08.996840    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" path="/var/lib/kubelet/pods/ecc2f0dd-04b9-42b5-833c-f931c9585341/volumes"
Sep 18 17:08:12.900711 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:12.900656    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:12.993744 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:12.993700    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:13.125292 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:13.125261    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:08:19.482336 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:19.482303    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:25.276364 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.276332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:25.432578 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.432541    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:08:25.432949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.432934    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main"
Sep 18 17:08:25.432949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.432949    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main"
Sep 18 17:08:25.433105 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.433022    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc2f0dd-04b9-42b5-833c-f931c9585341" containerName="main"
Sep 18 17:08:25.439655 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.439621    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.443916 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.443765    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 17:08:25.443916 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.443907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 17:08:25.444128 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.444068    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-dsrhg\""
Sep 18 17:08:25.444190 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.444157    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 17:08:25.444342 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.444302    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qc6zq\""
Sep 18 17:08:25.444779 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.444759    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:08:25.462986 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.462947    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:25.544678 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.544597    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:25.569762 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.569731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.569886 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.569787    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxpg\" (UniqueName: \"kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.606570 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.606535    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:25.670363 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.670328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.670546 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.670380    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxpg\" (UniqueName: \"kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.673001 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.672971    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.680179 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.680151    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxpg\" (UniqueName: \"kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:25.751599 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:25.751561    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:26.029776 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:26.029746    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:08:26.032738 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:08:26.032710    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a15fe8e_08f9_4b83_be60_cd62781983d6.slice/crio-2a0e88ef60f55e34cc56f50399d96cca440a9060fc54419b729e6370cd823999 WatchSource:0}: Error finding container 2a0e88ef60f55e34cc56f50399d96cca440a9060fc54419b729e6370cd823999: Status 404 returned error can't find the container with id 2a0e88ef60f55e34cc56f50399d96cca440a9060fc54419b729e6370cd823999
Sep 18 17:08:26.333021 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:26.332924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" event={"ID":"5a15fe8e-08f9-4b83-be60-cd62781983d6","Type":"ContainerStarted","Data":"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"}
Sep 18 17:08:26.333523 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:26.333033    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" event={"ID":"5a15fe8e-08f9-4b83-be60-cd62781983d6","Type":"ContainerStarted","Data":"2a0e88ef60f55e34cc56f50399d96cca440a9060fc54419b729e6370cd823999"}
Sep 18 17:08:26.333523 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:26.333196    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:08:26.353480 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:26.353394    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podStartSLOduration=1.353376361 podStartE2EDuration="1.353376361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:08:26.352529466 +0000 UTC m=+3177.832503008" watchObservedRunningTime="2026-09-18 17:08:26.353376361 +0000 UTC m=+3177.833349909"
Sep 18 17:08:41.624292 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:41.624260    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:08:57.338778 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:08:57.338745    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:09:34.930653 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:34.930618    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:09:46.715120 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:46.715085    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:09:47.837358 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:47.837324    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:09:47.946518 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:47.946484    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:09:53.125242 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:53.125204    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:09:53.877547 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:09:53.877510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:01.735187 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:01.735115    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main" containerID="cri-o://00f16a7ee1b311f71cba1565a3b2e307331beace48827634ce8a8db19a16df7b" gracePeriod=45
Sep 18 17:10:03.125885 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:03.125846    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:10:05.369400 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:05.369365    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:13.125201 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:13.125160    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:10:13.125626 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:13.125278    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:10:23.125898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:23.125850    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:10:29.239123 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:29.239093    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:10:29.256364 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:29.256337    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:10:31.958326 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:31.958290    2554 generic.go:356] "Generic (PLEG): container finished" podID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerID="00f16a7ee1b311f71cba1565a3b2e307331beace48827634ce8a8db19a16df7b" exitCode=0
Sep 18 17:10:31.958716 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:31.958368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" event={"ID":"afe25145-eee7-467e-832f-cb1a0a70adc5","Type":"ContainerDied","Data":"00f16a7ee1b311f71cba1565a3b2e307331beace48827634ce8a8db19a16df7b"}
Sep 18 17:10:31.984112 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:31.984082    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:10:32.122315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.122218    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rclq6\" (UniqueName: \"kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6\") pod \"afe25145-eee7-467e-832f-cb1a0a70adc5\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") "
Sep 18 17:10:32.122315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.122303    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs\") pod \"afe25145-eee7-467e-832f-cb1a0a70adc5\" (UID: \"afe25145-eee7-467e-832f-cb1a0a70adc5\") "
Sep 18 17:10:32.124558 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.124524    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afe25145-eee7-467e-832f-cb1a0a70adc5" (UID: "afe25145-eee7-467e-832f-cb1a0a70adc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:10:32.124558 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.124544    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6" (OuterVolumeSpecName: "kube-api-access-rclq6") pod "afe25145-eee7-467e-832f-cb1a0a70adc5" (UID: "afe25145-eee7-467e-832f-cb1a0a70adc5"). InnerVolumeSpecName "kube-api-access-rclq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:10:32.223309 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.223268    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclq6\" (UniqueName: \"kubernetes.io/projected/afe25145-eee7-467e-832f-cb1a0a70adc5-kube-api-access-rclq6\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:10:32.223309 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.223300    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe25145-eee7-467e-832f-cb1a0a70adc5-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:10:32.964174 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.964141    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"
Sep 18 17:10:32.964624 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.964142    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb" event={"ID":"afe25145-eee7-467e-832f-cb1a0a70adc5","Type":"ContainerDied","Data":"50f33b36b63bd4a8b469a362f35b2f7eb2d728f745b06614c0e9a3c274493afd"}
Sep 18 17:10:32.964624 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.964275    2554 scope.go:117] "RemoveContainer" containerID="00f16a7ee1b311f71cba1565a3b2e307331beace48827634ce8a8db19a16df7b"
Sep 18 17:10:32.985707 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.985680    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:10:32.996273 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:32.996240    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58tzrsb"]
Sep 18 17:10:34.997862 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:34.997828    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" path="/var/lib/kubelet/pods/afe25145-eee7-467e-832f-cb1a0a70adc5/volumes"
Sep 18 17:10:36.687468 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:36.687402    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:36.779978 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:36.779950    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:36.999813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:36.999772    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:37.096520 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:37.096486    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:38.167509 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:38.167457    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:38.539819 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:38.539778    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:10:38.769591 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:38.769558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:38.869893 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:38.869807    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:40.280256 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:40.280221    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:43.638387 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:43.638355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:47.338813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:47.338774    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:10:49.680342 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:49.680311    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:49.780556 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:49.780515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:10:53.559729 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:53.559641    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main" containerID="cri-o://56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810" gracePeriod=45
Sep 18 17:10:57.339582 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:10:57.339536    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:11:00.233996 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.233960    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:00.460837 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.460800    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:11:00.461428 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.461382    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main"
Sep 18 17:11:00.461428 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.461405    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main"
Sep 18 17:11:00.461640 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.461505    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe25145-eee7-467e-832f-cb1a0a70adc5" containerName="main"
Sep 18 17:11:00.464597 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.464574    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.467428 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.467388    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-jjz24\""
Sep 18 17:11:00.475465 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.475432    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:11:00.492310 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.492229    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:00.578209 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.578170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.578380 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.578223    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xpj\" (UniqueName: \"kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.679114 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.679076    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.679275 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.679131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xpj\" (UniqueName: \"kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.681701 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.681679    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.687269 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.687242    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xpj\" (UniqueName: \"kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:00.776817 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:00.776718    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:01.050031 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:01.049951    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:11:01.053887 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:11:01.053857    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc11d0a5_c773_4719_8497_593c1e1c35f6.slice/crio-fd6af413a8f39856b268bd557f7808463c62c04563746174eb2f1ff7bee4be7f WatchSource:0}: Error finding container fd6af413a8f39856b268bd557f7808463c62c04563746174eb2f1ff7bee4be7f: Status 404 returned error can't find the container with id fd6af413a8f39856b268bd557f7808463c62c04563746174eb2f1ff7bee4be7f
Sep 18 17:11:01.102179 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:01.102149    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" event={"ID":"bc11d0a5-c773-4719-8497-593c1e1c35f6","Type":"ContainerStarted","Data":"fd6af413a8f39856b268bd557f7808463c62c04563746174eb2f1ff7bee4be7f"}
Sep 18 17:11:02.108642 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:02.108599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" event={"ID":"bc11d0a5-c773-4719-8497-593c1e1c35f6","Type":"ContainerStarted","Data":"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"}
Sep 18 17:11:02.109114 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:02.108664    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:11:02.126555 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:02.126471    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podStartSLOduration=2.126453854 podStartE2EDuration="2.126453854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:02.124952764 +0000 UTC m=+3333.604926313" watchObservedRunningTime="2026-09-18 17:11:02.126453854 +0000 UTC m=+3333.606427416"
Sep 18 17:11:07.339504 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:07.339465    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:11:07.339900 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:07.339576    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:11:10.720450 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:10.720403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:11.091181 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:11.091092    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:11.190599 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:11.190563    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:17.338866 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:17.338827    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:11:23.804146 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.804118    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:11:23.897533 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.897403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs\") pod \"5a15fe8e-08f9-4b83-be60-cd62781983d6\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") "
Sep 18 17:11:23.897533 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.897501    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvxpg\" (UniqueName: \"kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg\") pod \"5a15fe8e-08f9-4b83-be60-cd62781983d6\" (UID: \"5a15fe8e-08f9-4b83-be60-cd62781983d6\") "
Sep 18 17:11:23.899751 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.899716    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a15fe8e-08f9-4b83-be60-cd62781983d6" (UID: "5a15fe8e-08f9-4b83-be60-cd62781983d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:11:23.899898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.899831    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg" (OuterVolumeSpecName: "kube-api-access-jvxpg") pod "5a15fe8e-08f9-4b83-be60-cd62781983d6" (UID: "5a15fe8e-08f9-4b83-be60-cd62781983d6"). InnerVolumeSpecName "kube-api-access-jvxpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:11:23.998225 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.998180    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxpg\" (UniqueName: \"kubernetes.io/projected/5a15fe8e-08f9-4b83-be60-cd62781983d6-kube-api-access-jvxpg\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:11:23.998225 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:23.998218    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15fe8e-08f9-4b83-be60-cd62781983d6-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:11:24.218400 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.218372    2554 generic.go:356] "Generic (PLEG): container finished" podID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerID="56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810" exitCode=0
Sep 18 17:11:24.218574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.218438    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"
Sep 18 17:11:24.218574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.218460    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" event={"ID":"5a15fe8e-08f9-4b83-be60-cd62781983d6","Type":"ContainerDied","Data":"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"}
Sep 18 17:11:24.218574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.218518    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5" event={"ID":"5a15fe8e-08f9-4b83-be60-cd62781983d6","Type":"ContainerDied","Data":"2a0e88ef60f55e34cc56f50399d96cca440a9060fc54419b729e6370cd823999"}
Sep 18 17:11:24.218574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.218547    2554 scope.go:117] "RemoveContainer" containerID="56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"
Sep 18 17:11:24.227630 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.227609    2554 scope.go:117] "RemoveContainer" containerID="56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"
Sep 18 17:11:24.227907 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:11:24.227884    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810\": container with ID starting with 56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810 not found: ID does not exist" containerID="56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"
Sep 18 17:11:24.227957 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.227915    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810"} err="failed to get container status \"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810\": rpc error: code = NotFound desc = could not find container \"56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810\": container with ID starting with 56fe5efc1de70f546a582c7abf0e31344bbd2e8c52df0db5e9efb47f0721a810 not found: ID does not exist"
Sep 18 17:11:24.240817 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.240793    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:11:24.244489 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.244467    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-kssd5"]
Sep 18 17:11:24.996511 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:24.996472    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" path="/var/lib/kubelet/pods/5a15fe8e-08f9-4b83-be60-cd62781983d6/volumes"
Sep 18 17:11:27.118715 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:27.118684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:11:33.115508 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:11:33.115473    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:12:11.740340 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:12:11.740305    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:01.971607 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:01.971558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:01.977827 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:01.977795    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:13:03.115292 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:03.115251    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:03.731833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:03.731797    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:13.115633 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:13.115587    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:16.997445 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:16.997385    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" containerID="cri-o://083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8" gracePeriod=45
Sep 18 17:13:23.114933 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:23.114892    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:23.115321 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:23.115001    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:13:26.139406 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:26.139364    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:26.238321 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:26.238282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:33.114983 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:33.114944    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:43.115651 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:43.115607    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:47.247607 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.247578    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:13:47.312661 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.312568    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs\") pod \"bc11d0a5-c773-4719-8497-593c1e1c35f6\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") "
Sep 18 17:13:47.312661 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.312621    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4xpj\" (UniqueName: \"kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj\") pod \"bc11d0a5-c773-4719-8497-593c1e1c35f6\" (UID: \"bc11d0a5-c773-4719-8497-593c1e1c35f6\") "
Sep 18 17:13:47.314716 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.314687    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj" (OuterVolumeSpecName: "kube-api-access-f4xpj") pod "bc11d0a5-c773-4719-8497-593c1e1c35f6" (UID: "bc11d0a5-c773-4719-8497-593c1e1c35f6"). InnerVolumeSpecName "kube-api-access-f4xpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:13:47.314837 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.314775    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc11d0a5-c773-4719-8497-593c1e1c35f6" (UID: "bc11d0a5-c773-4719-8497-593c1e1c35f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:13:47.414024 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.413973    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc11d0a5-c773-4719-8497-593c1e1c35f6-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:13:47.414024 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.414010    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xpj\" (UniqueName: \"kubernetes.io/projected/bc11d0a5-c773-4719-8497-593c1e1c35f6-kube-api-access-f4xpj\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:13:47.905125 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.905087    2554 generic.go:356] "Generic (PLEG): container finished" podID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerID="083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8" exitCode=0
Sep 18 17:13:47.905324 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.905139    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"
Sep 18 17:13:47.905324 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.905176    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" event={"ID":"bc11d0a5-c773-4719-8497-593c1e1c35f6","Type":"ContainerDied","Data":"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"}
Sep 18 17:13:47.905324 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.905234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g" event={"ID":"bc11d0a5-c773-4719-8497-593c1e1c35f6","Type":"ContainerDied","Data":"fd6af413a8f39856b268bd557f7808463c62c04563746174eb2f1ff7bee4be7f"}
Sep 18 17:13:47.905324 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.905256    2554 scope.go:117] "RemoveContainer" containerID="083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"
Sep 18 17:13:47.915072 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.915052    2554 scope.go:117] "RemoveContainer" containerID="083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"
Sep 18 17:13:47.915311 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:13:47.915285    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8\": container with ID starting with 083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8 not found: ID does not exist" containerID="083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"
Sep 18 17:13:47.915371 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.915312    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8"} err="failed to get container status \"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8\": rpc error: code = NotFound desc = could not find container \"083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8\": container with ID starting with 083e7d99fdd2c5c7ba29c237ef06957fbb2460e0bb5fc39cbfac530b090924f8 not found: ID does not exist"
Sep 18 17:13:47.927090 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.927063    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:13:47.931003 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:47.930982    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-scf7g"]
Sep 18 17:13:48.996661 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:48.996626    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" path="/var/lib/kubelet/pods/bc11d0a5-c773-4719-8497-593c1e1c35f6/volumes"
Sep 18 17:13:52.038836 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:52.038754    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:52.158168 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:52.158121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:52.301994 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:52.301898    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:56.376049 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:56.376008    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:13:58.963510 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:13:58.963474    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:05.048946 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:05.048901    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:05.112307 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:05.112258    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:16.208747 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:16.208708    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:16.308287 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:16.308253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:21.272610 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:21.272557    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:23.981342 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:23.981301    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:24.209499 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209439    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:14:24.209814 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209791    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main"
Sep 18 17:14:24.209814 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209806    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main"
Sep 18 17:14:24.209934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209843    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main"
Sep 18 17:14:24.209934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209849    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main"
Sep 18 17:14:24.209934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209907    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc11d0a5-c773-4719-8497-593c1e1c35f6" containerName="main"
Sep 18 17:14:24.209934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.209915    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a15fe8e-08f9-4b83-be60-cd62781983d6" containerName="main"
Sep 18 17:14:24.212657 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.212636    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.216760 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.216730    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 17:14:24.216909 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.216764    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-rgcf4\""
Sep 18 17:14:24.216909 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.216740    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 17:14:24.216909 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.216842    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-j4h4t\""
Sep 18 17:14:24.216909 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.216872    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 17:14:24.222741 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.222718    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:14:24.238299 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.238202    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:24.351893 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.351854    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.352093 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.351972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkgs\" (UniqueName: \"kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.452496 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.452459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.452666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.452540    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkgs\" (UniqueName: \"kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.455068 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.455042    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.460698 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.460668    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkgs\" (UniqueName: \"kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.524384 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.524287    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:24.797335 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:24.797309    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:14:24.799460 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:14:24.799435    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9afa9e8_84d6_44e2_8a2c_2ae9b9ddd456.slice/crio-0da9ea03012b91e2ff4347c1c5c57fa905c493609d232d7a5fc90213c4ceb9c8 WatchSource:0}: Error finding container 0da9ea03012b91e2ff4347c1c5c57fa905c493609d232d7a5fc90213c4ceb9c8: Status 404 returned error can't find the container with id 0da9ea03012b91e2ff4347c1c5c57fa905c493609d232d7a5fc90213c4ceb9c8
Sep 18 17:14:25.087561 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:25.087474    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" event={"ID":"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456","Type":"ContainerStarted","Data":"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"}
Sep 18 17:14:25.087561 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:25.087528    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" event={"ID":"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456","Type":"ContainerStarted","Data":"0da9ea03012b91e2ff4347c1c5c57fa905c493609d232d7a5fc90213c4ceb9c8"}
Sep 18 17:14:25.087561 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:25.087553    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:14:25.107028 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:25.106976    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podStartSLOduration=1.106957171 podStartE2EDuration="1.106957171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:25.104213102 +0000 UTC m=+3536.584186648" watchObservedRunningTime="2026-09-18 17:14:25.106957171 +0000 UTC m=+3536.586930732"
Sep 18 17:14:38.918117 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:38.918082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:43.045264 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:43.045231    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:43.591319 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:43.591289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:49.493488 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:49.493452    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:53.932796 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:53.932760    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:14:56.093854 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:14:56.093816    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:15:04.332555 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:04.332504    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:05.345598 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:05.345562    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:05.445980 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:05.445944    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:12.571738 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:12.571696    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:12.673863 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:12.673812    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:15.831373 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:15.831332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:15.930966 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:15.930916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:15:29.269196 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:29.269161    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:15:29.287653 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:15:29.287622    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:16:05.547315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:05.547282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:16.063670 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:16.063634    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:25.782199 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:25.782163    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:25.879353 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:25.879310    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:31.970915 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:31.970881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:32.071655 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:32.071618    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:34.699787 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:34.699745    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:54.217208 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:54.217120    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:16:54.251388 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:54.251336    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:54.306803 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:54.306754    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:16:56.094157 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:16:56.094117    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:17:06.094096 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:06.094056    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:17:09.245078 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:09.245015    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" containerID="cri-o://1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b" gracePeriod=45
Sep 18 17:17:16.094961 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:16.094918    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:17:16.095453 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:16.095046    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:17:26.094554 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:26.094511    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:17:36.094277 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:36.094239    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:17:39.494697 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.494665    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:17:39.508355 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.508284    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnkgs\" (UniqueName: \"kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs\") pod \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") "
Sep 18 17:17:39.508355 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.508350    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs\") pod \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\" (UID: \"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456\") "
Sep 18 17:17:39.510924 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.510880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" (UID: "c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:39.511181 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.511149    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs" (OuterVolumeSpecName: "kube-api-access-jnkgs") pod "c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" (UID: "c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456"). InnerVolumeSpecName "kube-api-access-jnkgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:17:39.609612 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.609577    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkgs\" (UniqueName: \"kubernetes.io/projected/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-kube-api-access-jnkgs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:17:39.609612 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:39.609608    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:17:40.025441 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.025386    2554 generic.go:356] "Generic (PLEG): container finished" podID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerID="1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b" exitCode=0
Sep 18 17:17:40.025441 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.025442    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"
Sep 18 17:17:40.025662 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.025469    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" event={"ID":"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456","Type":"ContainerDied","Data":"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"}
Sep 18 17:17:40.025662 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.025530    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc" event={"ID":"c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456","Type":"ContainerDied","Data":"0da9ea03012b91e2ff4347c1c5c57fa905c493609d232d7a5fc90213c4ceb9c8"}
Sep 18 17:17:40.025662 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.025559    2554 scope.go:117] "RemoveContainer" containerID="1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"
Sep 18 17:17:40.034685 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.034662    2554 scope.go:117] "RemoveContainer" containerID="1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"
Sep 18 17:17:40.035000 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:17:40.034978    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b\": container with ID starting with 1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b not found: ID does not exist" containerID="1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"
Sep 18 17:17:40.035064 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.035007    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b"} err="failed to get container status \"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b\": rpc error: code = NotFound desc = could not find container \"1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b\": container with ID starting with 1f637cb239b6d79f33dd2ddfe9a4e5f606961db37f5eee4e2c12c91769438d7b not found: ID does not exist"
Sep 18 17:17:40.047980 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.047949    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:17:40.051435 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:40.051391    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemg4cc"]
Sep 18 17:17:41.002283 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:41.002235    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" path="/var/lib/kubelet/pods/c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456/volumes"
Sep 18 17:17:44.302918 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:44.302877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:17:44.398852 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:44.398816    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:17:50.729072 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:50.729033    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:17:56.406359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:56.406326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:17:56.506353 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:17:56.506321    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:18:05.408097 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.408058    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:18:05.639656 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.639620    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:18:05.640001 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.639988    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main"
Sep 18 17:18:05.640051 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.640002    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main"
Sep 18 17:18:05.640092 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.640058    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9afa9e8-84d6-44e2-8a2c-2ae9b9ddd456" containerName="main"
Sep 18 17:18:05.643315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.643293    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.647317 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.647291    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 17:18:05.647478 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.647294    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-5tnzc\""
Sep 18 17:18:05.647478 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.647355    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 17:18:05.647478 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.647294    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-jlp6s\""
Sep 18 17:18:05.647478 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.647397    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 17:18:05.655541 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.655518    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:18:05.741953 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.741923    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.742123 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.741991    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcpg\" (UniqueName: \"kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.842488 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.842445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.842667 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.842541    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcpg\" (UniqueName: \"kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.845113 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.845087    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.851715 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.851684    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcpg\" (UniqueName: \"kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:05.953955 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:05.953920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:06.233731 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:06.233692    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:18:06.237459 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:18:06.237390    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2c31be_0a84_4aba_bf56_0e51283186ee.slice/crio-c94aa3edc2d3f016fa531a1d7330ba31606f12d2fe69b326e570a4b4c9702139 WatchSource:0}: Error finding container c94aa3edc2d3f016fa531a1d7330ba31606f12d2fe69b326e570a4b4c9702139: Status 404 returned error can't find the container with id c94aa3edc2d3f016fa531a1d7330ba31606f12d2fe69b326e570a4b4c9702139
Sep 18 17:18:06.803061 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:06.803025    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:18:07.159925 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:07.159830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" event={"ID":"0c2c31be-0a84-4aba-bf56-0e51283186ee","Type":"ContainerStarted","Data":"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"}
Sep 18 17:18:07.159925 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:07.159886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" event={"ID":"0c2c31be-0a84-4aba-bf56-0e51283186ee","Type":"ContainerStarted","Data":"c94aa3edc2d3f016fa531a1d7330ba31606f12d2fe69b326e570a4b4c9702139"}
Sep 18 17:18:07.160151 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:07.160050    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:07.178401 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:07.178345    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podStartSLOduration=2.178328322 podStartE2EDuration="2.178328322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:18:07.17696299 +0000 UTC m=+3758.656936537" watchObservedRunningTime="2026-09-18 17:18:07.178328322 +0000 UTC m=+3758.658301877"
Sep 18 17:18:18.165399 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:18.165358    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:18:38.168644 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:38.168613    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:18:47.446805 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:18:47.446768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:07.515451 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:07.515398    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:07.616188 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:07.616147    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:07.772159 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:07.772078    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:07.868330 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:07.868296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:07.972071 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:07.972032    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:10.252295 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:10.252260    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:10.355075 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:10.355046    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:10.925429 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:10.925378    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:13.498186 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:13.498151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:13.560881 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:13.560852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:19:34.277646 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:19:34.277611    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:20:19.433970 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:19.433869    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:20:19.496549 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:19.496514    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:20:25.129123 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:25.129081    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:20:25.129500 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:25.129400    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:20:25.191782 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:25.191748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:20:28.168429 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:28.168378    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:20:29.298332 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:29.298306    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:20:29.318930 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:29.318905    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:20:38.168189 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:38.168149    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:20:40.149822 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:40.149764    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" containerID="cri-o://3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e" gracePeriod=45
Sep 18 17:20:48.168512 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:48.168471    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:20:48.168916 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:48.168606    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:20:58.168566 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:20:58.168522    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:21:08.168722 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:08.168679    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:21:10.408320 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.408286    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:21:10.540613 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.540573    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs\") pod \"0c2c31be-0a84-4aba-bf56-0e51283186ee\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") "
Sep 18 17:21:10.540818 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.540669    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgcpg\" (UniqueName: \"kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg\") pod \"0c2c31be-0a84-4aba-bf56-0e51283186ee\" (UID: \"0c2c31be-0a84-4aba-bf56-0e51283186ee\") "
Sep 18 17:21:10.542863 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.542837    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c2c31be-0a84-4aba-bf56-0e51283186ee" (UID: "0c2c31be-0a84-4aba-bf56-0e51283186ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:21:10.542982 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.542956    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg" (OuterVolumeSpecName: "kube-api-access-dgcpg") pod "0c2c31be-0a84-4aba-bf56-0e51283186ee" (UID: "0c2c31be-0a84-4aba-bf56-0e51283186ee"). InnerVolumeSpecName "kube-api-access-dgcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:21:10.641672 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.641633    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2c31be-0a84-4aba-bf56-0e51283186ee-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:21:10.641672 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:10.641665    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcpg\" (UniqueName: \"kubernetes.io/projected/0c2c31be-0a84-4aba-bf56-0e51283186ee-kube-api-access-dgcpg\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:21:11.054620 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.054582    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerID="3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e" exitCode=0
Sep 18 17:21:11.054806 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.054629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" event={"ID":"0c2c31be-0a84-4aba-bf56-0e51283186ee","Type":"ContainerDied","Data":"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"}
Sep 18 17:21:11.054806 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.054655    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"
Sep 18 17:21:11.054806 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.054714    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5" event={"ID":"0c2c31be-0a84-4aba-bf56-0e51283186ee","Type":"ContainerDied","Data":"c94aa3edc2d3f016fa531a1d7330ba31606f12d2fe69b326e570a4b4c9702139"}
Sep 18 17:21:11.054806 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.054738    2554 scope.go:117] "RemoveContainer" containerID="3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"
Sep 18 17:21:11.063606 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.063587    2554 scope.go:117] "RemoveContainer" containerID="3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"
Sep 18 17:21:11.063934 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:21:11.063912    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e\": container with ID starting with 3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e not found: ID does not exist" containerID="3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"
Sep 18 17:21:11.063981 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.063943    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e"} err="failed to get container status \"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e\": rpc error: code = NotFound desc = could not find container \"3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e\": container with ID starting with 3f4a15ae2c3881ef988fe4b3a4819408443776c86aca843185acaeb252c8968e not found: ID does not exist"
Sep 18 17:21:11.072013 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.071987    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:21:11.075329 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:11.075303    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48vg6r5"]
Sep 18 17:21:12.996570 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:12.996535    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" path="/var/lib/kubelet/pods/0c2c31be-0a84-4aba-bf56-0e51283186ee/volumes"
Sep 18 17:21:15.244910 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:15.244872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:15.349753 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:15.349719    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:15.488907 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:15.488852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:15.587058 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:15.586968    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:18.834771 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:18.834735    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:20.070947 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:20.070890    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:20.127457 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:20.127406    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:21.749473 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:21.749393    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:27.583905 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:27.583870    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:27.683425 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:27.683372    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:30.799197 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:30.799161    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:31.055028 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.054946    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:21:31.055329 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.055304    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main"
Sep 18 17:21:31.055329 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.055329    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main"
Sep 18 17:21:31.055529 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.055429    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2c31be-0a84-4aba-bf56-0e51283186ee" containerName="main"
Sep 18 17:21:31.059511 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.059492    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.062160 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.062131    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 17:21:31.063293 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.063240    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-vksnc\""
Sep 18 17:21:31.063293 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.063244    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 17:21:31.063293 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.063249    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 17:21:31.063533 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.063370    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-v5kh6\""
Sep 18 17:21:31.066335 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.066312    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:21:31.123949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.123911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k888s\" (UniqueName: \"kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.124137 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.123978    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.224990 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.224948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.225183 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.225034    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k888s\" (UniqueName: \"kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.227504 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.227484    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.233220 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.233190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k888s\" (UniqueName: \"kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.364755 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.364663    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:31.370553 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.370530    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:31.486774 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.486685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:31.642584 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:31.642378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:21:31.645401 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:21:31.645374    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591c4527_a91e_4ff7_a61a_f4b02c244e0f.slice/crio-2f65a87d2a320b4d27ce9512d03fca3148baba79a24a756c1f082a5b60d28732 WatchSource:0}: Error finding container 2f65a87d2a320b4d27ce9512d03fca3148baba79a24a756c1f082a5b60d28732: Status 404 returned error can't find the container with id 2f65a87d2a320b4d27ce9512d03fca3148baba79a24a756c1f082a5b60d28732
Sep 18 17:21:32.157776 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:32.157650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" event={"ID":"591c4527-a91e-4ff7-a61a-f4b02c244e0f","Type":"ContainerStarted","Data":"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"}
Sep 18 17:21:32.157776 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:32.157710    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" event={"ID":"591c4527-a91e-4ff7-a61a-f4b02c244e0f","Type":"ContainerStarted","Data":"2f65a87d2a320b4d27ce9512d03fca3148baba79a24a756c1f082a5b60d28732"}
Sep 18 17:21:32.157776 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:32.157767    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:21:32.176349 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:32.174858    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podStartSLOduration=1.174841553 podStartE2EDuration="1.174841553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:21:32.17420705 +0000 UTC m=+3963.654180593" watchObservedRunningTime="2026-09-18 17:21:32.174841553 +0000 UTC m=+3963.654815098"
Sep 18 17:21:39.535750 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:39.535707    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:21:59.942631 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:21:59.942594    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:00.044680 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:00.044635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:00.143585 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:00.143536    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:00.234315 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:00.234266    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:00.299319 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:00.299266    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:02.533469 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:02.533437    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:02.592225 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:02.592191    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:03.165205 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:03.165174    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:22:03.357536 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:03.357469    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:05.701143 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:05.701080    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:05.802814 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:05.802777    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:06.289646 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:06.289612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:08.855756 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:08.855720    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:08.913272 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:08.913236    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:12.165155 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:12.165110    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:12.265467 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:12.265411    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:36.988170 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.988132    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:22:36.991527 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.991498    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:36.994025 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.993992    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 18 17:22:36.994301 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.994271    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 17:22:36.994410 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.994328    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-m4skn\""
Sep 18 17:22:36.994495 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:36.994456    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 17:22:37.004783 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.004754    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:22:37.021486 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.021447    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:22:37.092007 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.091970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.092174 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.092019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.092174 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.092085    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.092174 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.092147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.092275 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.092181    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.092275 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.092203    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpkx\" (UniqueName: \"kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.192783 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.192783 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192830    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192861    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpkx\" (UniqueName: \"kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.192945    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193282 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.193252    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193331 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.193283    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.193453 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.193338    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.195231 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.195209    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.195441 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.195408    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:37.208071 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:37.208049    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpkx\" (UniqueName: \"kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.053285 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.053247    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.058773 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.058743    2554 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.153749 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.153715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.153929 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.153830    2554 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.157103 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.157077    2554 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") DevicePath \"csi-d2673ac9af269a93a59f0dd77e5c07728c9833c55b3de2e8a9130bdc892c5614\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.254773 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.254726    2554 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9d615f203ca9bc2eb4423728807638f6b015c6defc17b8fe44837f745c8a01b6/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.275147 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.275113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.504036 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.503950    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:22:44.784849 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.784822    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:22:44.787283 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:22:44.787253    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3f2981_066b_404e_89c4_b2af6cde5527.slice/crio-0f9050f3142f052e83de1630f4726b9aceb8b94130a446955fdfb0da8f6ac52e WatchSource:0}: Error finding container 0f9050f3142f052e83de1630f4726b9aceb8b94130a446955fdfb0da8f6ac52e: Status 404 returned error can't find the container with id 0f9050f3142f052e83de1630f4726b9aceb8b94130a446955fdfb0da8f6ac52e
Sep 18 17:22:44.789525 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:44.789500    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:22:45.511789 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:22:45.511703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" event={"ID":"bc3f2981-066b-404e-89c4-b2af6cde5527","Type":"ContainerStarted","Data":"0f9050f3142f052e83de1630f4726b9aceb8b94130a446955fdfb0da8f6ac52e"}
Sep 18 17:23:13.684019 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:13.683924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" event={"ID":"bc3f2981-066b-404e-89c4-b2af6cde5527","Type":"ContainerStarted","Data":"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"}
Sep 18 17:23:13.704549 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:13.704487    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podStartSLOduration=9.221024069 podStartE2EDuration="37.704465218s" totalImagesPullingTime="28.483441149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:13.70369918 +0000 UTC m=+4065.183672751" watchObservedRunningTime="2026-09-18 17:23:13.704465218 +0000 UTC m=+4065.184438765"
Sep 18 17:23:14.504321 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:14.504272    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:23:14.504525 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:14.504356    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:23:14.505947 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:14.505912    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:23:24.504783 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:24.504739    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:23:34.504623 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:34.504577    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:23:44.504813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:44.504759    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:23:50.456307 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:50.456269    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:23:50.457792 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:50.457757    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:23:51.850317 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:51.850282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:23:53.164969 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:53.164923    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:23:54.504793 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:54.504739    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:23:55.119660 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:23:55.119612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:03.164983 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:03.164941    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:24:04.505332 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:04.505287    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:24:05.478181 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:05.478137    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" containerID="cri-o://2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df" gracePeriod=45
Sep 18 17:24:13.164577 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:13.164534    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:24:13.165070 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:13.164668    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:24:14.504520 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:14.504479    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:24:23.164833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:23.164733    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:24:24.504724 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:24.504668    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 18 17:24:33.165019 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:33.164979    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:24:34.518894 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:34.518860    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:24:34.527050 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:34.527017    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:24:35.738341 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.738254    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:24:35.836237 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.836201    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs\") pod \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") "
Sep 18 17:24:35.836465 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.836290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k888s\" (UniqueName: \"kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s\") pod \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\" (UID: \"591c4527-a91e-4ff7-a61a-f4b02c244e0f\") "
Sep 18 17:24:35.838720 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.838682    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "591c4527-a91e-4ff7-a61a-f4b02c244e0f" (UID: "591c4527-a91e-4ff7-a61a-f4b02c244e0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:24:35.838720 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.838707    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s" (OuterVolumeSpecName: "kube-api-access-k888s") pod "591c4527-a91e-4ff7-a61a-f4b02c244e0f" (UID: "591c4527-a91e-4ff7-a61a-f4b02c244e0f"). InnerVolumeSpecName "kube-api-access-k888s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:24:35.937858 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.937818    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c4527-a91e-4ff7-a61a-f4b02c244e0f-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:24:35.937858 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:35.937852    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k888s\" (UniqueName: \"kubernetes.io/projected/591c4527-a91e-4ff7-a61a-f4b02c244e0f-kube-api-access-k888s\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:24:36.119928 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.119830    2554 generic.go:356] "Generic (PLEG): container finished" podID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerID="2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df" exitCode=0
Sep 18 17:24:36.119928 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.119887    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"
Sep 18 17:24:36.119928 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.119917    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" event={"ID":"591c4527-a91e-4ff7-a61a-f4b02c244e0f","Type":"ContainerDied","Data":"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"}
Sep 18 17:24:36.120203 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.119972    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz" event={"ID":"591c4527-a91e-4ff7-a61a-f4b02c244e0f","Type":"ContainerDied","Data":"2f65a87d2a320b4d27ce9512d03fca3148baba79a24a756c1f082a5b60d28732"}
Sep 18 17:24:36.120203 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.119994    2554 scope.go:117] "RemoveContainer" containerID="2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"
Sep 18 17:24:36.130079 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.130058    2554 scope.go:117] "RemoveContainer" containerID="2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"
Sep 18 17:24:36.130486 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:24:36.130464    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df\": container with ID starting with 2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df not found: ID does not exist" containerID="2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"
Sep 18 17:24:36.130570 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.130496    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df"} err="failed to get container status \"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df\": rpc error: code = NotFound desc = could not find container \"2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df\": container with ID starting with 2180bdc327f6a6f57647b33fa23805c301ed8fba2f937b5d4b16b702f06026df not found: ID does not exist"
Sep 18 17:24:36.143656 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.143620    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:24:36.148758 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.148725    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4582qmz"]
Sep 18 17:24:36.997841 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:36.997804    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" path="/var/lib/kubelet/pods/591c4527-a91e-4ff7-a61a-f4b02c244e0f/volumes"
Sep 18 17:24:40.537597 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:40.537564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:40.629073 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:40.629025    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:45.524090 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:45.524049    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:45.609434 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:45.609380    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:49.715845 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:49.715795    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:24:49.774562 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:24:49.774528    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:02.426722 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:02.426691    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh_bc3f2981-066b-404e-89c4-b2af6cde5527/main/0.log"
Sep 18 17:25:03.492404 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:03.492370    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:25:03.501212 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:03.501177    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:07.315040 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:07.315007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:07.743628 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:07.743594    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:13.511784 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:13.511736    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:18.515966 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:18.515890    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main" containerID="cri-o://8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7" gracePeriod=45
Sep 18 17:25:19.183597 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.183569    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:25:19.250101 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250002    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250101 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250069    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250348 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250115    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250348 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250141    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250348 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250173    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwpkx\" (UniqueName: \"kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250348 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250294    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250586 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250375    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs\") pod \"bc3f2981-066b-404e-89c4-b2af6cde5527\" (UID: \"bc3f2981-066b-404e-89c4-b2af6cde5527\") "
Sep 18 17:25:19.250586 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250453    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:19.250586 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250514    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home" (OuterVolumeSpecName: "home") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:19.250754 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250714    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.250754 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.250732    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.252826 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.252788    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm" (OuterVolumeSpecName: "dshm") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:19.252971 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.252799    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:25:19.252971 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.252802    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx" (OuterVolumeSpecName: "kube-api-access-gwpkx") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "kube-api-access-gwpkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:25:19.259804 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.259778    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65" (OuterVolumeSpecName: "kserve-pvc-source") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 17:25:19.261276 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.261248    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc3f2981-066b-404e-89c4-b2af6cde5527" (UID: "bc3f2981-066b-404e-89c4-b2af6cde5527"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:19.350373 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.350333    2554 generic.go:356] "Generic (PLEG): container finished" podID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerID="8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7" exitCode=0
Sep 18 17:25:19.350604 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.350393    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"
Sep 18 17:25:19.350604 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.350442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" event={"ID":"bc3f2981-066b-404e-89c4-b2af6cde5527","Type":"ContainerDied","Data":"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"}
Sep 18 17:25:19.350604 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.350509    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh" event={"ID":"bc3f2981-066b-404e-89c4-b2af6cde5527","Type":"ContainerDied","Data":"0f9050f3142f052e83de1630f4726b9aceb8b94130a446955fdfb0da8f6ac52e"}
Sep 18 17:25:19.350604 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.350540    2554 scope.go:117] "RemoveContainer" containerID="8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"
Sep 18 17:25:19.351182 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.351155    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.351182 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.351180    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpkx\" (UniqueName: \"kubernetes.io/projected/bc3f2981-066b-404e-89c4-b2af6cde5527-kube-api-access-gwpkx\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.351335 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.351212    2554 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") on node \"ip-10-0-142-62.ec2.internal\" "
Sep 18 17:25:19.351335 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.351226    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3f2981-066b-404e-89c4-b2af6cde5527-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.351335 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.351240    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3f2981-066b-404e-89c4-b2af6cde5527-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:25:19.360350 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.360329    2554 scope.go:117] "RemoveContainer" containerID="8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"
Sep 18 17:25:19.360713 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:25:19.360693    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7\": container with ID starting with 8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7 not found: ID does not exist" containerID="8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"
Sep 18 17:25:19.360790 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.360724    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7"} err="failed to get container status \"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7\": rpc error: code = NotFound desc = could not find container \"8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7\": container with ID starting with 8b948ef5bc38aec3642bcc887f87f66dda369ab1ef2d60a29cf1d864d341bed7 not found: ID does not exist"
Sep 18 17:25:19.374406 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.374363    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:25:19.379118 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.379091    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-mw4xh"]
Sep 18 17:25:19.382059 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.382032    2554 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65") on node "ip-10-0-142-62.ec2.internal"
Sep 18 17:25:19.452505 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:19.452462    2554 reconciler_common.go:299] "Volume detached for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"csi-d2673ac9af269a93a59f0dd77e5c07728c9833c55b3de2e8a9130bdc892c5614\""
Sep 18 17:25:20.997193 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:20.997155    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" path="/var/lib/kubelet/pods/bc3f2981-066b-404e-89c4-b2af6cde5527/volumes"
Sep 18 17:25:23.609698 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:23.609653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:23.710574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:23.710538    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:28.682569 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:28.682506    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:28.782917 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:28.782881    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:29.330362 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:29.330331    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:25:29.356061 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:29.356037    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:25:29.629222 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:29.629135    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:31.966700 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:31.966654    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:32.040393 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:32.040354    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:34.704928 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.704884    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:34.797383 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.797349    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:25:34.797925 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.797890    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main"
Sep 18 17:25:34.797925 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.797913    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main"
Sep 18 17:25:34.798096 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.797946    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main"
Sep 18 17:25:34.798096 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.797955    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main"
Sep 18 17:25:34.798096 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.798043    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3f2981-066b-404e-89c4-b2af6cde5527" containerName="main"
Sep 18 17:25:34.798096 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.798053    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="591c4527-a91e-4ff7-a61a-f4b02c244e0f" containerName="main"
Sep 18 17:25:34.800589 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.800572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:34.803195 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.803174    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 17:25:34.804212 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.804188    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-w58bg\""
Sep 18 17:25:34.804212 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.804206    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 17:25:34.804411 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.804332    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 17:25:34.804411 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.804337    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-v5tx9\""
Sep 18 17:25:34.809903 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.809881    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:25:34.832674 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.832635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:34.896277 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.896233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:34.896490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.896294    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7d7l\" (UniqueName: \"kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:34.997126 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.997089    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:34.997316 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.997149    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7d7l\" (UniqueName: \"kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:34.999660 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:34.999642    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:35.004936 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:35.004913    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7d7l\" (UniqueName: \"kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:35.111183 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:35.111141    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:35.391174 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:35.391147    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:25:35.393752 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:25:35.393721    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e5e231_2046_4dd7_8530_65cb7b051884.slice/crio-32ae9c4e7b0ff4f6567ae33c1003888eda25622e6571a3200bcb0f9bb30a3626 WatchSource:0}: Error finding container 32ae9c4e7b0ff4f6567ae33c1003888eda25622e6571a3200bcb0f9bb30a3626: Status 404 returned error can't find the container with id 32ae9c4e7b0ff4f6567ae33c1003888eda25622e6571a3200bcb0f9bb30a3626
Sep 18 17:25:35.430198 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:35.430158    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" event={"ID":"03e5e231-2046-4dd7-8530-65cb7b051884","Type":"ContainerStarted","Data":"32ae9c4e7b0ff4f6567ae33c1003888eda25622e6571a3200bcb0f9bb30a3626"}
Sep 18 17:25:36.109627 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:36.109591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:36.209727 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:36.209692    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:36.436535 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:36.436446    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" event={"ID":"03e5e231-2046-4dd7-8530-65cb7b051884","Type":"ContainerStarted","Data":"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"}
Sep 18 17:25:36.436693 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:36.436573    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:25:36.454407 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:36.454357    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podStartSLOduration=2.4543413530000002 podStartE2EDuration="2.454341353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:36.453826847 +0000 UTC m=+4207.933800394" watchObservedRunningTime="2026-09-18 17:25:36.454341353 +0000 UTC m=+4207.934314899"
Sep 18 17:25:37.586360 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:37.586326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:37.693351 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:37.693313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:25:43.717466 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:25:43.717409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:26:07.443088 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:26:07.443012    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:26:50.114657 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:26:50.114622    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:00.269528 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:00.269490    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:00.370886 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:00.370846    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:10.142469 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:10.142294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:10.239986 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:10.239940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:16.786737 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:16.786691    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:17.613779 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:17.613742    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:17.733558 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:17.733520    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:17.863896 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:17.863789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:18.194304 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:18.194219    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:22.736286 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:22.736208    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:22.799980 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:22.799940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:24.403513 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:24.403478    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:36.434066 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:36.434025    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:50.008832 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:50.008792    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:27:50.021956 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:50.021923    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:50.083490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:50.083448    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:27:57.442908 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:27:57.442869    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:28:05.028625 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:05.028582    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main" containerID="cri-o://4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6" gracePeriod=45
Sep 18 17:28:07.443408 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:07.443367    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:28:17.443546 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:17.443504    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:28:17.443926 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:17.443624    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:28:27.443353 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:27.443306    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:28:35.269457 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.269429    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:28:35.288847 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.288758    2554 generic.go:356] "Generic (PLEG): container finished" podID="03e5e231-2046-4dd7-8530-65cb7b051884" containerID="4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6" exitCode=0
Sep 18 17:28:35.288847 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.288824    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"
Sep 18 17:28:35.289073 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.288842    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" event={"ID":"03e5e231-2046-4dd7-8530-65cb7b051884","Type":"ContainerDied","Data":"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"}
Sep 18 17:28:35.289073 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.288917    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp" event={"ID":"03e5e231-2046-4dd7-8530-65cb7b051884","Type":"ContainerDied","Data":"32ae9c4e7b0ff4f6567ae33c1003888eda25622e6571a3200bcb0f9bb30a3626"}
Sep 18 17:28:35.289073 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.288949    2554 scope.go:117] "RemoveContainer" containerID="4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"
Sep 18 17:28:35.298845 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.298818    2554 scope.go:117] "RemoveContainer" containerID="4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"
Sep 18 17:28:35.299180 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:28:35.299154    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6\": container with ID starting with 4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6 not found: ID does not exist" containerID="4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"
Sep 18 17:28:35.299247 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.299189    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6"} err="failed to get container status \"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6\": rpc error: code = NotFound desc = could not find container \"4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6\": container with ID starting with 4d485a035e5a704bdabf2b00a8c9e3967979cf37cf6beef891c4ba442f95adb6 not found: ID does not exist"
Sep 18 17:28:35.419285 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.419240    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs\") pod \"03e5e231-2046-4dd7-8530-65cb7b051884\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") "
Sep 18 17:28:35.419482 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.419335    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7d7l\" (UniqueName: \"kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l\") pod \"03e5e231-2046-4dd7-8530-65cb7b051884\" (UID: \"03e5e231-2046-4dd7-8530-65cb7b051884\") "
Sep 18 17:28:35.421546 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.421512    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l" (OuterVolumeSpecName: "kube-api-access-c7d7l") pod "03e5e231-2046-4dd7-8530-65cb7b051884" (UID: "03e5e231-2046-4dd7-8530-65cb7b051884"). InnerVolumeSpecName "kube-api-access-c7d7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:35.421684 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.421567    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03e5e231-2046-4dd7-8530-65cb7b051884" (UID: "03e5e231-2046-4dd7-8530-65cb7b051884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:35.520689 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.520634    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e5e231-2046-4dd7-8530-65cb7b051884-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:28:35.520689 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.520684    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7d7l\" (UniqueName: \"kubernetes.io/projected/03e5e231-2046-4dd7-8530-65cb7b051884-kube-api-access-c7d7l\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:28:35.611431 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.611379    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:28:35.616743 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:35.616716    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-tlskp"]
Sep 18 17:28:36.996555 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:36.996520    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" path="/var/lib/kubelet/pods/03e5e231-2046-4dd7-8530-65cb7b051884/volumes"
Sep 18 17:28:40.057870 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:40.057833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:40.153310 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:40.153277    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:46.346091 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:46.346055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:51.920317 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:51.920223    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:53.347424 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:53.347383    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:54.117197 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.117162    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:54.302386 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.302353    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:28:54.302756 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.302743    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main"
Sep 18 17:28:54.302800 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.302759    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main"
Sep 18 17:28:54.302835 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.302828    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e5e231-2046-4dd7-8530-65cb7b051884" containerName="main"
Sep 18 17:28:54.305976 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.305952    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.309479 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.309448    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-sfd7z\""
Sep 18 17:28:54.309479 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.309455    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 17:28:54.309666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.309490    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-pn4qc\""
Sep 18 17:28:54.309666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.309554    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 17:28:54.309666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.309615    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 17:28:54.315626 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.315600    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:28:54.337380 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.337355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:28:54.488781 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.488745    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.489162 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.488789    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ln6x\" (UniqueName: \"kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.590195 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.590152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.590195 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.590194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ln6x\" (UniqueName: \"kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.592728 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.592700    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.598140 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.598117    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ln6x\" (UniqueName: \"kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.617556 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.617527    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:54.895329 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:54.895306    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:28:54.897639 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:28:54.897612    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0c5c34_a5d3_43b4_a67e_212514012992.slice/crio-405340970af41660e304864586c37a76f938755890ce306f2c52918b2af65983 WatchSource:0}: Error finding container 405340970af41660e304864586c37a76f938755890ce306f2c52918b2af65983: Status 404 returned error can't find the container with id 405340970af41660e304864586c37a76f938755890ce306f2c52918b2af65983
Sep 18 17:28:55.382597 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:55.382554    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" event={"ID":"ec0c5c34-a5d3-43b4-a67e-212514012992","Type":"ContainerStarted","Data":"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"}
Sep 18 17:28:55.382797 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:55.382617    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" event={"ID":"ec0c5c34-a5d3-43b4-a67e-212514012992","Type":"ContainerStarted","Data":"405340970af41660e304864586c37a76f938755890ce306f2c52918b2af65983"}
Sep 18 17:28:55.382797 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:55.382640    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:28:55.399168 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:28:55.399105    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podStartSLOduration=1.399088057 podStartE2EDuration="1.399088057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:55.398383576 +0000 UTC m=+4406.878357134" watchObservedRunningTime="2026-09-18 17:28:55.399088057 +0000 UTC m=+4406.879061643"
Sep 18 17:29:00.690780 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:00.690739    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:29:07.827954 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:07.827918    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:29:07.928668 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:07.928632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:29:26.390445 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:26.390395    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:29:57.833813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:57.833747    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:29:57.935173 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:29:57.935134    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:04.154758 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:04.154717    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:09.807997 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:09.807958    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:09.908651 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:09.908614    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:10.054239 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.054204    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:30:10.057574 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.057547    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.061439 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.061337    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 17:30:10.061981 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.061960    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-q2fmk\""
Sep 18 17:30:10.062076 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.061960    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-86wfl\""
Sep 18 17:30:10.062076 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.061964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 17:30:10.062209 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.061971    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 17:30:10.069319 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.069246    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:30:10.102760 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.102728    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:10.158830 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.158786    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.158992 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.158862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9sx\" (UniqueName: \"kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.260232 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.260193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9sx\" (UniqueName: \"kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.260403 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.260260    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.262721 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.262694    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.268487 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.268463    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9sx\" (UniqueName: \"kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.371517 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.371402    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.644898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.644871    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:30:10.647156 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:30:10.647130    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb17666_aaaa_49fd_9423_ea2d92b96b93.slice/crio-506f0ee4d6e013f0aff4fc8ad94803e1cd7e37ffa81e2f0fe1588be2bd5f6163 WatchSource:0}: Error finding container 506f0ee4d6e013f0aff4fc8ad94803e1cd7e37ffa81e2f0fe1588be2bd5f6163: Status 404 returned error can't find the container with id 506f0ee4d6e013f0aff4fc8ad94803e1cd7e37ffa81e2f0fe1588be2bd5f6163
Sep 18 17:30:10.741436 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.741365    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" event={"ID":"efb17666-aaaa-49fd-9423-ea2d92b96b93","Type":"ContainerStarted","Data":"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"}
Sep 18 17:30:10.741622 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.741461    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" event={"ID":"efb17666-aaaa-49fd-9423-ea2d92b96b93","Type":"ContainerStarted","Data":"506f0ee4d6e013f0aff4fc8ad94803e1cd7e37ffa81e2f0fe1588be2bd5f6163"}
Sep 18 17:30:10.741949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.741885    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:10.761630 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:10.761570    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podStartSLOduration=0.761553649 podStartE2EDuration="761.553649ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:10.760162019 +0000 UTC m=+4482.240135566" watchObservedRunningTime="2026-09-18 17:30:10.761553649 +0000 UTC m=+4482.241527215"
Sep 18 17:30:23.230458 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:23.230335    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:29.361715 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:29.361688    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:30:29.386998 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:29.386970    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:30:41.748159 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:41.748127    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:30:43.345232 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:43.345197    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:30:43.364297 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:43.364259    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:43.462049 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:43.462009    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:30:51.748598 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:51.748559    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:30:58.366692 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:30:58.366642    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main" containerID="cri-o://2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc" gracePeriod=45
Sep 18 17:31:01.748978 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:01.748937    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:11.748942 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:11.748896    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:11.749336 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:11.749041    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:31:21.748640 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:21.748596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:24.944934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:24.944894    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:31:24.979021 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:24.978969    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:25.075885 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:25.075852    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:26.390018 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:26.389979    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:28.620532 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.620500    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:31:28.670495 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.670374    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs\") pod \"efb17666-aaaa-49fd-9423-ea2d92b96b93\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") "
Sep 18 17:31:28.670495 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.670457    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9sx\" (UniqueName: \"kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx\") pod \"efb17666-aaaa-49fd-9423-ea2d92b96b93\" (UID: \"efb17666-aaaa-49fd-9423-ea2d92b96b93\") "
Sep 18 17:31:28.672631 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.672601    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb17666-aaaa-49fd-9423-ea2d92b96b93" (UID: "efb17666-aaaa-49fd-9423-ea2d92b96b93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:31:28.672751 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.672611    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx" (OuterVolumeSpecName: "kube-api-access-ft9sx") pod "efb17666-aaaa-49fd-9423-ea2d92b96b93" (UID: "efb17666-aaaa-49fd-9423-ea2d92b96b93"). InnerVolumeSpecName "kube-api-access-ft9sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:31:28.771792 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.771754    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb17666-aaaa-49fd-9423-ea2d92b96b93-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:31:28.771792 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:28.771787    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9sx\" (UniqueName: \"kubernetes.io/projected/efb17666-aaaa-49fd-9423-ea2d92b96b93-kube-api-access-ft9sx\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:31:29.119678 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.119638    2554 generic.go:356] "Generic (PLEG): container finished" podID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerID="2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc" exitCode=0
Sep 18 17:31:29.119855 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.119685    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"
Sep 18 17:31:29.119855 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.119720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" event={"ID":"efb17666-aaaa-49fd-9423-ea2d92b96b93","Type":"ContainerDied","Data":"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"}
Sep 18 17:31:29.119855 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.119772    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd" event={"ID":"efb17666-aaaa-49fd-9423-ea2d92b96b93","Type":"ContainerDied","Data":"506f0ee4d6e013f0aff4fc8ad94803e1cd7e37ffa81e2f0fe1588be2bd5f6163"}
Sep 18 17:31:29.119855 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.119799    2554 scope.go:117] "RemoveContainer" containerID="2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"
Sep 18 17:31:29.128312 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.128282    2554 scope.go:117] "RemoveContainer" containerID="2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"
Sep 18 17:31:29.128617 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:31:29.128598    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc\": container with ID starting with 2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc not found: ID does not exist" containerID="2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"
Sep 18 17:31:29.128698 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.128624    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc"} err="failed to get container status \"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc\": rpc error: code = NotFound desc = could not find container \"2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc\": container with ID starting with 2a28800e901070d472d544cb8f6afa7c05f7d8789adcc2b695ef6a3b93ed90dc not found: ID does not exist"
Sep 18 17:31:29.138469 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.138443    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:31:29.140039 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:29.140022    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgq9bd"]
Sep 18 17:31:30.996756 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:30.996719    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" path="/var/lib/kubelet/pods/efb17666-aaaa-49fd-9423-ea2d92b96b93/volumes"
Sep 18 17:31:33.415659 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:33.415626    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:36.390240 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:36.390199    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:39.415225 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:39.415179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:39.964265 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:39.964223    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" containerID="cri-o://cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74" gracePeriod=45
Sep 18 17:31:45.039003 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:45.038927    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:45.106892 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:45.106831    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:46.389982 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:46.389940    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:31:46.390466 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:46.390053    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:31:53.214905 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:53.214808    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:31:56.390460 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:31:56.390401    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:32:06.390556 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:06.390511    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:32:10.215693 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.215665    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:32:10.248488 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.248388    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs\") pod \"ec0c5c34-a5d3-43b4-a67e-212514012992\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") "
Sep 18 17:32:10.248488 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.248457    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ln6x\" (UniqueName: \"kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x\") pod \"ec0c5c34-a5d3-43b4-a67e-212514012992\" (UID: \"ec0c5c34-a5d3-43b4-a67e-212514012992\") "
Sep 18 17:32:10.250692 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.250653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec0c5c34-a5d3-43b4-a67e-212514012992" (UID: "ec0c5c34-a5d3-43b4-a67e-212514012992"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:10.250812 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.250746    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x" (OuterVolumeSpecName: "kube-api-access-4ln6x") pod "ec0c5c34-a5d3-43b4-a67e-212514012992" (UID: "ec0c5c34-a5d3-43b4-a67e-212514012992"). InnerVolumeSpecName "kube-api-access-4ln6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:10.320152 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.320113    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerID="cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74" exitCode=0
Sep 18 17:32:10.320323 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.320168    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"
Sep 18 17:32:10.320323 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.320200    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" event={"ID":"ec0c5c34-a5d3-43b4-a67e-212514012992","Type":"ContainerDied","Data":"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"}
Sep 18 17:32:10.320323 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.320265    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp" event={"ID":"ec0c5c34-a5d3-43b4-a67e-212514012992","Type":"ContainerDied","Data":"405340970af41660e304864586c37a76f938755890ce306f2c52918b2af65983"}
Sep 18 17:32:10.320323 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.320288    2554 scope.go:117] "RemoveContainer" containerID="cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"
Sep 18 17:32:10.329801 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.329779    2554 scope.go:117] "RemoveContainer" containerID="cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"
Sep 18 17:32:10.330063 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:32:10.330044    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74\": container with ID starting with cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74 not found: ID does not exist" containerID="cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"
Sep 18 17:32:10.330110 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.330073    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74"} err="failed to get container status \"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74\": rpc error: code = NotFound desc = could not find container \"cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74\": container with ID starting with cbf07c90a6311d108d49808d5654fb6a58242d12d4dacf31143f5c93db85dc74 not found: ID does not exist"
Sep 18 17:32:10.341770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.341741    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:32:10.346014 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.345991    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-t85lp"]
Sep 18 17:32:10.349813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.349796    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0c5c34-a5d3-43b4-a67e-212514012992-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:32:10.349874 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.349816    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ln6x\" (UniqueName: \"kubernetes.io/projected/ec0c5c34-a5d3-43b4-a67e-212514012992-kube-api-access-4ln6x\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:32:10.997014 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:10.996979    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" path="/var/lib/kubelet/pods/ec0c5c34-a5d3-43b4-a67e-212514012992/volumes"
Sep 18 17:32:27.098503 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:27.098470    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:27.192753 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:27.192717    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:27.576114 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:27.576070    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:29.584274 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584230    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:32:29.584800 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584778    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main"
Sep 18 17:32:29.584879 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584803    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main"
Sep 18 17:32:29.584879 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584817    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main"
Sep 18 17:32:29.584879 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584825    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main"
Sep 18 17:32:29.585037 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584906    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0c5c34-a5d3-43b4-a67e-212514012992" containerName="main"
Sep 18 17:32:29.585037 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.584923    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb17666-aaaa-49fd-9423-ea2d92b96b93" containerName="main"
Sep 18 17:32:29.588209 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.588187    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.592039 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.592016    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 17:32:29.592198 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.592078    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 17:32:29.592198 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.592129    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 17:32:29.592198 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.592137    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-657px\""
Sep 18 17:32:29.595613 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.595586    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:32:29.610729 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.610700    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:29.720815 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720777    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.720815 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720821    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.721025 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720853    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.721025 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g744b\" (UniqueName: \"kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.721025 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720947    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.721025 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.720966    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.807190 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.807152    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:29.822338 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822287    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822573 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g744b\" (UniqueName: \"kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822573 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822573 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822573 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822539    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822573 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822569    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822909 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822843    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.822968 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822910    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.823026 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.822990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.824753 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.824732    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.824890 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.824874    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.830333 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.830309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g744b\" (UniqueName: \"kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-pmvjz\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:29.899760 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:29.899660    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:30.169617 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:30.169590    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:32:30.171986 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:32:30.171955    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756d7383_b966_407b_ba5c_a847fb489917.slice/crio-8000f752fd9291140648eb5df350098ad8cbf4a4d45b0b7cd879a6311901700f WatchSource:0}: Error finding container 8000f752fd9291140648eb5df350098ad8cbf4a4d45b0b7cd879a6311901700f: Status 404 returned error can't find the container with id 8000f752fd9291140648eb5df350098ad8cbf4a4d45b0b7cd879a6311901700f
Sep 18 17:32:30.416089 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:30.416043    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" event={"ID":"756d7383-b966-407b-ba5c-a847fb489917","Type":"ContainerStarted","Data":"cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86"}
Sep 18 17:32:30.416266 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:30.416109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" event={"ID":"756d7383-b966-407b-ba5c-a847fb489917","Type":"ContainerStarted","Data":"8000f752fd9291140648eb5df350098ad8cbf4a4d45b0b7cd879a6311901700f"}
Sep 18 17:32:30.435158 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:30.435063    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" podStartSLOduration=1.435043617 podStartE2EDuration="1.435043617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:30.433074602 +0000 UTC m=+4621.913048153" watchObservedRunningTime="2026-09-18 17:32:30.435043617 +0000 UTC m=+4621.915017163"
Sep 18 17:32:30.899789 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:30.899728    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:33.605898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:33.605857    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:39.484079 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:39.484044    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:39.555898 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:39.555862    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:32:39.900797 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:39.900690    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:39.913374 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:39.913340    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:40.480058 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:40.480029    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:32:43.575476 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:32:43.575410    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:33:09.461920 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:09.461880    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:33:14.336666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.336624    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:33:14.346230 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:33:14.346190    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:33:14.346230 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:14.346230 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:14.346230 ip-10-0-142-62 kubenswrapper[2554]:  > containerID="cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86" cmd=["/bin/sleep","15"]
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.346255    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" message="" err=<
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:  >
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:33:14.346272    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main" containerID="cri-o://cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86"
Sep 18 17:33:14.346557 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.346315    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main" containerID="cri-o://cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86" gracePeriod=60
Sep 18 17:33:14.470383 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.470347    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 17:33:14.629236 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.629200    2554 generic.go:356] "Generic (PLEG): container finished" podID="756d7383-b966-407b-ba5c-a847fb489917" containerID="cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86" exitCode=0
Sep 18 17:33:14.629405 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.629275    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" event={"ID":"756d7383-b966-407b-ba5c-a847fb489917","Type":"ContainerDied","Data":"cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86"}
Sep 18 17:33:14.693038 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.693007    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:33:14.728501 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728472    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728694 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728511    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728694 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728550    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728694 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728570    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728694 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728595    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g744b\" (UniqueName: \"kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728694 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728624    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home\") pod \"756d7383-b966-407b-ba5c-a847fb489917\" (UID: \"756d7383-b966-407b-ba5c-a847fb489917\") "
Sep 18 17:33:14.728966 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728890    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache" (OuterVolumeSpecName: "model-cache") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:14.728966 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.728902    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:14.729084 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.729013    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-model-cache\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:14.729084 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.729035    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-tmp-dir\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:14.729183 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.729124    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home" (OuterVolumeSpecName: "home") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:14.730837 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.730812    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:33:14.731132 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.731115    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b" (OuterVolumeSpecName: "kube-api-access-g744b") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "kube-api-access-g744b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:14.731207 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.731119    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm" (OuterVolumeSpecName: "dshm") pod "756d7383-b966-407b-ba5c-a847fb489917" (UID: "756d7383-b966-407b-ba5c-a847fb489917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:14.830070 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.830025    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g744b\" (UniqueName: \"kubernetes.io/projected/756d7383-b966-407b-ba5c-a847fb489917-kube-api-access-g744b\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:14.830070 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.830061    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-home\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:14.830070 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.830070    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756d7383-b966-407b-ba5c-a847fb489917-tls-certs\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:14.830070 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:14.830080    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756d7383-b966-407b-ba5c-a847fb489917-dshm\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:33:15.635387 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:15.635347    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"
Sep 18 17:33:15.635387 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:15.635360    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz" event={"ID":"756d7383-b966-407b-ba5c-a847fb489917","Type":"ContainerDied","Data":"8000f752fd9291140648eb5df350098ad8cbf4a4d45b0b7cd879a6311901700f"}
Sep 18 17:33:15.636081 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:15.635489    2554 scope.go:117] "RemoveContainer" containerID="cd1bda0814712cb14b3869234f0d2feeb2e4c1b407a1953bcf959eba94c98f86"
Sep 18 17:33:15.653073 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:15.653041    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:33:15.656619 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:15.656596    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-pmvjz"]
Sep 18 17:33:16.996388 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:16.996351    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="756d7383-b966-407b-ba5c-a847fb489917" path="/var/lib/kubelet/pods/756d7383-b966-407b-ba5c-a847fb489917/volumes"
Sep 18 17:33:27.088058 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:27.088011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:33:27.147430 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:27.147384    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:33:28.323626 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:28.323591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:33:28.385363 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:33:28.385318    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:15.459077 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:15.459036    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:16.453813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.453772    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nmjsz/must-gather-l4f9q"]
Sep 18 17:34:16.454129 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.454117    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main"
Sep 18 17:34:16.454179 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.454130    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main"
Sep 18 17:34:16.454214 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.454206    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="756d7383-b966-407b-ba5c-a847fb489917" containerName="main"
Sep 18 17:34:16.457452 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.457433    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.459956 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.459930    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmjsz\"/\"kube-root-ca.crt\""
Sep 18 17:34:16.460833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.460814    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmjsz\"/\"openshift-service-ca.crt\""
Sep 18 17:34:16.460833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.460824    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nmjsz\"/\"default-dockercfg-dnsd2\""
Sep 18 17:34:16.466405 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.466378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmjsz/must-gather-l4f9q"]
Sep 18 17:34:16.590884 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.590844    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltk5w\" (UniqueName: \"kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.591065 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.590906    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.691771 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.691726    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltk5w\" (UniqueName: \"kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.691982 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.691795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.692114 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.692078    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.718453 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.718409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltk5w\" (UniqueName: \"kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w\") pod \"must-gather-l4f9q\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") " pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:16.767200 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:16.767160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:17.036454 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:17.036398    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmjsz/must-gather-l4f9q"]
Sep 18 17:34:17.039008 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:34:17.038984    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda97a46ce_cb4d_4442_b645_5852781c060c.slice/crio-aef15c31f08008b8e3f70242915c4224615195a4aa2b04e217a9cc1ce849d19a WatchSource:0}: Error finding container aef15c31f08008b8e3f70242915c4224615195a4aa2b04e217a9cc1ce849d19a: Status 404 returned error can't find the container with id aef15c31f08008b8e3f70242915c4224615195a4aa2b04e217a9cc1ce849d19a
Sep 18 17:34:17.040889 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:17.040872    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:34:17.162458 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:17.162344    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:17.936204 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:17.936119    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" event={"ID":"a97a46ce-cb4d-4442-b645-5852781c060c","Type":"ContainerStarted","Data":"aef15c31f08008b8e3f70242915c4224615195a4aa2b04e217a9cc1ce849d19a"}
Sep 18 17:34:21.467349 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:21.467312    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:23.972916 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:23.972872    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" event={"ID":"a97a46ce-cb4d-4442-b645-5852781c060c","Type":"ContainerStarted","Data":"a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90"}
Sep 18 17:34:23.973298 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:23.972934    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" event={"ID":"a97a46ce-cb4d-4442-b645-5852781c060c","Type":"ContainerStarted","Data":"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"}
Sep 18 17:34:23.991187 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:23.991118    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" podStartSLOduration=1.818987693 podStartE2EDuration="7.991095097s" totalImagesPullingTime="6.172107404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:23.988731353 +0000 UTC m=+4735.468704902" watchObservedRunningTime="2026-09-18 17:34:23.991095097 +0000 UTC m=+4735.471068721"
Sep 18 17:34:27.062969 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:27.062932    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:27.163169 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:27.163127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:41.471490 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:41.471451    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-x62qz_6bc7580a-5ed1-4e06-95de-7715075b6b5a/authorino/0.log"
Sep 18 17:34:41.484588 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:41.484560    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-hvwsj_fbe1c95a-cf4f-4988-ae84-361190df0401/manager/0.log"
Sep 18 17:34:41.498141 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:41.498115    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-lzmkz_8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c/manager/0.log"
Sep 18 17:34:46.095639 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:46.095603    2554 generic.go:356] "Generic (PLEG): container finished" podID="a97a46ce-cb4d-4442-b645-5852781c060c" containerID="70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408" exitCode=0
Sep 18 17:34:46.096059 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:46.095677    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" event={"ID":"a97a46ce-cb4d-4442-b645-5852781c060c","Type":"ContainerDied","Data":"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"}
Sep 18 17:34:46.096059 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:46.095999    2554 scope.go:117] "RemoveContainer" containerID="70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"
Sep 18 17:34:46.968366 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:46.968332    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nmjsz_must-gather-l4f9q_a97a46ce-cb4d-4442-b645-5852781c060c/gather/0.log"
Sep 18 17:34:47.388738 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:47.388657    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:47.541923 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:47.541887    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:47.642132 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:47.642041    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:50.756046 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:50.756012    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vn5kq_25c3b578-0fb3-4d56-bb23-da9ba897582a/global-pull-secret-syncer/0.log"
Sep 18 17:34:50.835675 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:50.835633    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log"
Sep 18 17:34:51.009683 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:51.009570    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-62.ec2.internal_5410219c58050432591751550b16d506/haproxy/0.log"
Sep 18 17:34:52.373208 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.373174    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-nmjsz/must-gather-l4f9q"]
Sep 18 17:34:52.373782 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.373464    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-nmjsz/must-gather-l4f9q" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="copy" containerID="cri-o://a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90" gracePeriod=2
Sep 18 17:34:52.375587 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.375556    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-nmjsz/must-gather-l4f9q"]
Sep 18 17:34:52.375707 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:34:52.375646    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-l4f9q\" is forbidden: User \"system:node:ip-10-0-142-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nmjsz\": no relationship found between node 'ip-10-0-142-62.ec2.internal' and this object" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:52.607705 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.607678    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nmjsz_must-gather-l4f9q_a97a46ce-cb4d-4442-b645-5852781c060c/copy/0.log"
Sep 18 17:34:52.610355 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.610335    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:52.612732 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:34:52.612707    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-l4f9q\" is forbidden: User \"system:node:ip-10-0-142-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nmjsz\": no relationship found between node 'ip-10-0-142-62.ec2.internal' and this object" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:52.727216 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.727183    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output\") pod \"a97a46ce-cb4d-4442-b645-5852781c060c\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") "
Sep 18 17:34:52.727373 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.727237    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltk5w\" (UniqueName: \"kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w\") pod \"a97a46ce-cb4d-4442-b645-5852781c060c\" (UID: \"a97a46ce-cb4d-4442-b645-5852781c060c\") "
Sep 18 17:34:52.729562 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.729528    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w" (OuterVolumeSpecName: "kube-api-access-ltk5w") pod "a97a46ce-cb4d-4442-b645-5852781c060c" (UID: "a97a46ce-cb4d-4442-b645-5852781c060c"). InnerVolumeSpecName "kube-api-access-ltk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:34:52.735139 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.735106    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a97a46ce-cb4d-4442-b645-5852781c060c" (UID: "a97a46ce-cb4d-4442-b645-5852781c060c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:34:52.828100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.828062    2554 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a97a46ce-cb4d-4442-b645-5852781c060c-must-gather-output\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:34:52.828100 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.828093    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltk5w\" (UniqueName: \"kubernetes.io/projected/a97a46ce-cb4d-4442-b645-5852781c060c-kube-api-access-ltk5w\") on node \"ip-10-0-142-62.ec2.internal\" DevicePath \"\""
Sep 18 17:34:52.996772 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:52.996677    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" path="/var/lib/kubelet/pods/a97a46ce-cb4d-4442-b645-5852781c060c/volumes"
Sep 18 17:34:53.130933 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.130901    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nmjsz_must-gather-l4f9q_a97a46ce-cb4d-4442-b645-5852781c060c/copy/0.log"
Sep 18 17:34:53.131269 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.131242    2554 generic.go:356] "Generic (PLEG): container finished" podID="a97a46ce-cb4d-4442-b645-5852781c060c" containerID="a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90" exitCode=143
Sep 18 17:34:53.131327 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.131302    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmjsz/must-gather-l4f9q"
Sep 18 17:34:53.131363 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.131334    2554 scope.go:117] "RemoveContainer" containerID="a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90"
Sep 18 17:34:53.139510 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.139488    2554 scope.go:117] "RemoveContainer" containerID="70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"
Sep 18 17:34:53.154458 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.154397    2554 scope.go:117] "RemoveContainer" containerID="a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90"
Sep 18 17:34:53.154760 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:34:53.154740    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90\": container with ID starting with a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90 not found: ID does not exist" containerID="a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90"
Sep 18 17:34:53.154813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.154769    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90"} err="failed to get container status \"a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90\": rpc error: code = NotFound desc = could not find container \"a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90\": container with ID starting with a3baae3b7d1b35300d93abb8ce72b914a064e442327fb5cd326203ba5d0e8f90 not found: ID does not exist"
Sep 18 17:34:53.154813 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.154786    2554 scope.go:117] "RemoveContainer" containerID="70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"
Sep 18 17:34:53.155043 ip-10-0-142-62 kubenswrapper[2554]: E0918 17:34:53.155003    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408\": container with ID starting with 70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408 not found: ID does not exist" containerID="70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"
Sep 18 17:34:53.155085 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.155050    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408"} err="failed to get container status \"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408\": rpc error: code = NotFound desc = could not find container \"70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408\": container with ID starting with 70fac06afa3f9538e8d4d3db22614fbc35e26bf1ee6db0b3252d571e95ad1408 not found: ID does not exist"
Sep 18 17:34:53.202131 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.202098    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:53.267470 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:53.267377    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2ngq_93619efd-a838-4e61-9443-e9e48874784b/konnectivity-agent/0.log" limit=1024
Sep 18 17:34:55.765638 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:55.765604    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-x62qz_6bc7580a-5ed1-4e06-95de-7715075b6b5a/authorino/0.log"
Sep 18 17:34:55.795166 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:55.795140    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-hvwsj_fbe1c95a-cf4f-4988-ae84-361190df0401/manager/0.log"
Sep 18 17:34:55.825638 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:55.825606    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-lzmkz_8cb9d37a-fe35-41d1-8fb5-80ea9fd65e7c/manager/0.log"
Sep 18 17:34:57.297532 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.297506    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/alertmanager/0.log"
Sep 18 17:34:57.320486 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.320457    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/config-reloader/0.log"
Sep 18 17:34:57.340126 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.340100    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/kube-rbac-proxy-web/0.log"
Sep 18 17:34:57.361099 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.361071    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/kube-rbac-proxy/0.log"
Sep 18 17:34:57.380504 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.380454    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/kube-rbac-proxy-metric/0.log"
Sep 18 17:34:57.402366 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.402333    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/prom-label-proxy/0.log"
Sep 18 17:34:57.431327 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.431302    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_145151be-5eab-4898-b428-dc9cac9250ce/init-config-reloader/0.log"
Sep 18 17:34:57.473359 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.473326    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-s46vk_0953c73e-e2c8-4e23-9917-c19c5325f849/cluster-monitoring-operator/0.log"
Sep 18 17:34:57.565680 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.565573    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-654dcd44d-nl4r8_ef681765-8424-45d8-ba0d-b3797d7fbfe1/metrics-server/0.log"
Sep 18 17:34:57.854314 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.854218    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/node-exporter/0.log"
Sep 18 17:34:57.903949 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.903920    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/kube-rbac-proxy/0.log"
Sep 18 17:34:57.926982 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:57.926953    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5bbs_63849575-d8c1-4f76-8e0f-46d5268ea942/init-textfile/0.log"
Sep 18 17:34:58.048706 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.048671    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/prometheus/0.log"
Sep 18 17:34:58.069657 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.069632    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/config-reloader/0.log"
Sep 18 17:34:58.092295 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.092255    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/thanos-sidecar/0.log"
Sep 18 17:34:58.121849 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.121772    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/kube-rbac-proxy-web/0.log"
Sep 18 17:34:58.155118 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.155091    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/kube-rbac-proxy/0.log"
Sep 18 17:34:58.189260 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.189228    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/kube-rbac-proxy-thanos/0.log"
Sep 18 17:34:58.227260 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.227234    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a23802-3f52-44bf-98c1-36add00a9c41/init-config-reloader/0.log"
Sep 18 17:34:58.266646 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.266618    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w9pfz_42838fd5-f64e-4be8-8665-8969f844f13f/prometheus-operator/0.log"
Sep 18 17:34:58.286065 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.286039    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w9pfz_42838fd5-f64e-4be8-8665-8969f844f13f/kube-rbac-proxy/0.log"
Sep 18 17:34:58.354696 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.354666    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/telemeter-client/0.log"
Sep 18 17:34:58.378961 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.378877    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/reload/0.log"
Sep 18 17:34:58.407542 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:58.407516    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bcd745757-h2jkt_39b86d54-e098-415f-b586-6e72eb9eb34e/kube-rbac-proxy/0.log"
Sep 18 17:34:59.809879 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:34:59.809840    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7m6xk_3ca2d477-1869-49a3-adc2-7f8da9249342/networking-console-plugin/0.log"
Sep 18 17:35:00.770908 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.770877    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bfd6579b6-bqphh_c6e0698a-60f5-4159-9d44-a3f5a9537461/console/0.log"
Sep 18 17:35:00.870150 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870113    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"]
Sep 18 17:35:00.870547 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870473    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="gather"
Sep 18 17:35:00.870547 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870485    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="gather"
Sep 18 17:35:00.870547 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870510    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="copy"
Sep 18 17:35:00.870547 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870516    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="copy"
Sep 18 17:35:00.870732 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870575    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="copy"
Sep 18 17:35:00.870732 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.870583    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a97a46ce-cb4d-4442-b645-5852781c060c" containerName="gather"
Sep 18 17:35:00.876153 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.876130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:00.878778 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.878757    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"openshift-service-ca.crt\""
Sep 18 17:35:00.879773 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.879752    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"kube-root-ca.crt\""
Sep 18 17:35:00.879883 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.879755    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jhnn\"/\"default-dockercfg-mqxfb\""
Sep 18 17:35:00.881861 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:00.881838    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"]
Sep 18 17:35:01.000433 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.000380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-proc\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.000433 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.000435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-podres\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.000669 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.000480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4qn\" (UniqueName: \"kubernetes.io/projected/7935c5ca-ebc2-49c7-97bd-1baf4504c417-kube-api-access-lm4qn\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.000669 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.000516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-lib-modules\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.000669 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.000616    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-sys\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101497 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101378    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-sys\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101497 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-proc\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101497 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-podres\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101497 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101495    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4qn\" (UniqueName: \"kubernetes.io/projected/7935c5ca-ebc2-49c7-97bd-1baf4504c417-kube-api-access-lm4qn\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101519    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-lib-modules\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101524    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-sys\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101587    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-proc\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101635    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-podres\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.101770 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.101676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7935c5ca-ebc2-49c7-97bd-1baf4504c417-lib-modules\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.110132 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.110111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4qn\" (UniqueName: \"kubernetes.io/projected/7935c5ca-ebc2-49c7-97bd-1baf4504c417-kube-api-access-lm4qn\") pod \"perf-node-gather-daemonset-t82q5\" (UID: \"7935c5ca-ebc2-49c7-97bd-1baf4504c417\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.187499 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.187461    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:01.456967 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:01.456940    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"]
Sep 18 17:35:01.459626 ip-10-0-142-62 kubenswrapper[2554]: W0918 17:35:01.459592    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7935c5ca_ebc2_49c7_97bd_1baf4504c417.slice/crio-7447a31bbc03f0c9aac4b376534af4049d78b1310343ca9bdaeb6fdc35132a39 WatchSource:0}: Error finding container 7447a31bbc03f0c9aac4b376534af4049d78b1310343ca9bdaeb6fdc35132a39: Status 404 returned error can't find the container with id 7447a31bbc03f0c9aac4b376534af4049d78b1310343ca9bdaeb6fdc35132a39
Sep 18 17:35:02.175236 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.175199    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5" event={"ID":"7935c5ca-ebc2-49c7-97bd-1baf4504c417","Type":"ContainerStarted","Data":"b19985cf88b0088555177138549def7cb9ac1a6ec519964e89fff5e5a0c8dcdb"}
Sep 18 17:35:02.175679 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.175257    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5" event={"ID":"7935c5ca-ebc2-49c7-97bd-1baf4504c417","Type":"ContainerStarted","Data":"7447a31bbc03f0c9aac4b376534af4049d78b1310343ca9bdaeb6fdc35132a39"}
Sep 18 17:35:02.175679 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.175296    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:02.193434 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.193365    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5" podStartSLOduration=2.193349223 podStartE2EDuration="2.193349223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:02.190907554 +0000 UTC m=+4773.670881101" watchObservedRunningTime="2026-09-18 17:35:02.193349223 +0000 UTC m=+4773.673322768"
Sep 18 17:35:02.505083 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.505051    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgftx_35655ad6-0a9b-41cb-a9ef-031e5880b4c2/dns/0.log"
Sep 18 17:35:02.525219 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.525177    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgftx_35655ad6-0a9b-41cb-a9ef-031e5880b4c2/kube-rbac-proxy/0.log"
Sep 18 17:35:02.547294 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:02.547270    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kjtm7_d83c7a85-6566-44e2-8725-dd1c6429033d/dns-node-resolver/0.log"
Sep 18 17:35:03.182023 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:03.181991    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrts2_dd1bdd77-ade8-4626-b01e-2b3c6fe51d98/node-ca/0.log"
Sep 18 17:35:04.794576 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:04.794544    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6wbnf_56cf29d2-ddb5-4c98-a9d8-693d27a505d7/serve-healthcheck-canary/0.log"
Sep 18 17:35:05.373148 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:05.373120    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvdpc_9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3/kube-rbac-proxy/0.log"
Sep 18 17:35:05.393666 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:05.393634    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvdpc_9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3/exporter/0.log"
Sep 18 17:35:05.413504 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:05.413474    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvdpc_9f70c7e6-2e72-4f4e-bd49-4760ac36aaa3/extractor/0.log"
Sep 18 17:35:08.190934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:08.190901    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-t82q5"
Sep 18 17:35:08.413996 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:08.413963    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fd4df8954-4q8l9_f4236bc3-d243-4e84-be74-d4ef205ebbec/manager/0.log"
Sep 18 17:35:09.049934 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:09.049896    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7dcf4bff6-btz59_fa6da241-6405-42c6-bf14-a629c6f95e06/manager/0.log"
Sep 18 17:35:09.490270 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:09.490243    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-8rthl_d3b80e01-1408-4b29-ad42-13765cff2024/server/0.log"
Sep 18 17:35:10.012760 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:10.012721    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-bcvg2_698b4837-4bb2-4158-afc4-7090f0509ca2/manager/0.log"
Sep 18 17:35:10.109794 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:10.109746    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-f2sqm_23e7a231-5e71-4f02-a799-5a8a3a01883b/seaweedfs/0.log"
Sep 18 17:35:16.733480 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.733445    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/kube-multus-additional-cni-plugins/0.log"
Sep 18 17:35:16.752113 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.752086    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/egress-router-binary-copy/0.log"
Sep 18 17:35:16.774032 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.774001    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/cni-plugins/0.log"
Sep 18 17:35:16.793478 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.793448    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/bond-cni-plugin/0.log"
Sep 18 17:35:16.813542 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.813517    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/routeoverride-cni/0.log"
Sep 18 17:35:16.833184 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.833138    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/whereabouts-cni-bincopy/0.log"
Sep 18 17:35:16.852983 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.852961    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vl7gs_364f7d51-7a38-43f1-a5b3-73756da9c04c/whereabouts-cni/0.log"
Sep 18 17:35:16.934753 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:16.934718    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ppstz_cee9d688-9df4-4669-a39d-cde460288b6b/kube-multus/0.log"
Sep 18 17:35:17.103794 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:17.103709    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fxk2s_023a0f44-e934-492c-a9fd-c88a77e45000/network-metrics-daemon/0.log"
Sep 18 17:35:17.125828 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:17.125798    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fxk2s_023a0f44-e934-492c-a9fd-c88a77e45000/kube-rbac-proxy/0.log"
Sep 18 17:35:17.967011 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:17.966969    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-controller/0.log"
Sep 18 17:35:17.982833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:17.982806    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/0.log"
Sep 18 17:35:17.982833 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:17.982819    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovn-acl-logging/1.log"
Sep 18 17:35:18.002176 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.002134    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/kube-rbac-proxy-node/0.log"
Sep 18 17:35:18.022016 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.021973    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 17:35:18.040344 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.040307    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/northd/0.log"
Sep 18 17:35:18.059794 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.059764    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/nbdb/0.log"
Sep 18 17:35:18.078960 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.078879    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/sbdb/0.log"
Sep 18 17:35:18.135327 ip-10-0-142-62 kubenswrapper[2554]: I0918 17:35:18.135292    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-754q4_a5b4cb8d-c838-48fd-8531-708ca2a3ab4b/ovnkube-controller/0.log"