Sep 09 20:09:21.581097 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:09:21.581114 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:09:21.581127 ip-10-0-128-184 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:09:21.581541 ip-10-0-128-184 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:09:31.825030 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:09:31.825048 ip-10-0-128-184 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f039e5ce71c048f99232ee73d8e86476 --
Sep 09 20:11:17.411657 ip-10-0-128-184 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:17.878467 ip-10-0-128-184 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:17.878467 ip-10-0-128-184 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:17.878467 ip-10-0-128-184 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:17.878467 ip-10-0-128-184 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:17.878467 ip-10-0-128-184 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:17.881452 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.881335    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:17.887791 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887762    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.887791 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887788    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.887791 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887793    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887797    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887801    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887804    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887808    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887811    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887814    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887818    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887823    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887826    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887831    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887834    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887836    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887839    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887842    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887845    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887848    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887854    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887858    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.887916 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887861    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887864    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887867    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887870    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887873    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887876    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887879    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887882    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887885    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887888    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887890    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887903    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887906    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887909    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887912    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887915    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887920    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887924    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887927    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887930    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.888420 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887933    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887935    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887938    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887941    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887943    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887946    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887949    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887951    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887954    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887956    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887959    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887962    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887964    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887967    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887970    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887973    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887976    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887979    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887982    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887985    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.888919 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887987    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887990    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887992    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887995    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.887998    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888002    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888006    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888009    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888013    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888016    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888019    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888022    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888025    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888027    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888030    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888033    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888036    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888038    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888041    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888044    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.889456 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888047    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888050    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888053    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888055    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888058    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888061    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888064    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888067    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888070    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888073    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888076    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888078    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888081    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888084    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888087    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888090    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888094    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888098    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888911    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888919    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.889978 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888922    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888925    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888929    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888933    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888936    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888939    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888942    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888944    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888947    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888950    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888953    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888956    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888958    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888961    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888964    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888968    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888972    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888977    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888980    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888983    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.890501 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888986    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888989    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888991    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888994    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.888997    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889000    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889003    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889005    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889008    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889011    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889013    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889016    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889019    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889021    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889024    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889028    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889032    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889035    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889038    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.891012 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889041    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889044    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889047    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889050    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889052    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889055    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889057    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889060    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889063    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889066    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889069    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889072    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889075    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889078    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889080    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889083    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889086    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889089    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889091    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.891524 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889094    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889097    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889101    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889104    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889108    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889111    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889113    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889116    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889119    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889123    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889126    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889129    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889131    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889134    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889137    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889140    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889142    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889145    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889147    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889150    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.892020 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889153    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889155    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889159    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889162    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889165    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889167    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889170    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889173    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889176    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889178    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889181    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889183    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889186    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889188    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889191    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889193    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889196    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889198    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889201    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889707    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889715    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.892566 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889719    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889721    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889725    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889728    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889731    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889734    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889737    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889740    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889743    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889747    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889751    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889755    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889763    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889767    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889770    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889773    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889776    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889779    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889784    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.893102 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889786    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889789    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889792    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889795    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889797    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889800    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889803    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889806    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889808    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889811    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889814    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889816    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889819    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889822    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889824    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889827    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889830    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889833    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889835    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889838    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.893604 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889841    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889844    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889847    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889849    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889852    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889856    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889859    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889862    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889864    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889867    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889870    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889873    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889875    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889879    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889883    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889886    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889889    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889892    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889895    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.894096 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889898    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889900    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889903    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889906    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889909    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889911    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889914    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889917    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889919    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889921    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889930    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889933    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889937    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889940    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889943    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889945    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889948    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889951    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889955    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889957    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.894619 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889960    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889963    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889965    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889968    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889971    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889973    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889976    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889979    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889982    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889985    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889988    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889991    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889994    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889996    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.889999    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890001    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890004    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890006    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890009    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890102    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890152    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:17.895128 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890160    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890165    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890171    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890175    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890180    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890185    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890188    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890191    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890195    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890200    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890204    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890207    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890210    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890214    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890217    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890220    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890225    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890228    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890231    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890234    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890241    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890245    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890249    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:17.895679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890252    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890255    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890259    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890262    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890265    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890268    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890271    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890275    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890278    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890281    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890298    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890302    2577 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890306    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890311    2577 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890314    2577 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890317    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890320    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890324    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890328    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890331    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890334    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890338    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890341    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890344    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890347    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:17.896400 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890350    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890353    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890356    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890359    2577 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890365    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890368    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890372    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890375    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890379    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890382    2577 flags.go:64] FLAG: --help="false"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890385    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-184.ec2.internal"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890389    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890392    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890395    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890399    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890402    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890405    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890408    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890411    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890415    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890419    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890422    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890425    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890428    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:17.897043 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890431    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890434    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890438    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890441    2577 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890444    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890447    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890450    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890456    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890459    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890462    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890465    2577 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890469    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890475    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890478    2577 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890481    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890486    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890490    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890494    2577 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890497    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890501    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890505    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890508    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890511    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890514    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890518    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:17.897666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890526    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890529    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890533    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890536    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890543    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890549    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890553    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890556    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890559    2577 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890562    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890566    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-096439b07359e4ff1"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890569    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890572    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890575    2577 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890578    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890582    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890585    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890588    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890591    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890597    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890600    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890604    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890607    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890610    2577 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890613    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:17.898587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890616    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890619    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890623    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890625    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890629    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890632    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890635    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890638    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890641    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890644    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890647    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890651    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890655    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890658    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890665    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890668    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890671    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890676    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890679    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890682    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890685    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890688    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890691    2577 flags.go:64] FLAG: --v="2"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890695    2577 flags.go:64] FLAG: --version="false"
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890699    2577 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:17.899699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890703    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.890708    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890851    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890855    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890859    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890862    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890864    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890867    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890870    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890873    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890875    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890878    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890881    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890884    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890887    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890890    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890892    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890895    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890898    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.900775 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890901    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890904    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890909    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890912    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890914    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890918    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890920    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890923    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890926    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890928    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890931    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890934    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890936    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890939    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890943    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890946    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890949    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890952    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890955    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890957    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.901623 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890960    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890962    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890965    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890968    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890970    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890973    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890976    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890978    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890981    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890984    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890986    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890989    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890992    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890994    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.890999    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891002    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891004    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891007    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891010    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891013    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.902398 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891016    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891019    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891022    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891026    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891030    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891034    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891039    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891042    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891045    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891048    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891051    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891054    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891057    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891060    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891063    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891066    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891068    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891071    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891074    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.902984 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891076    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891079    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891082    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891084    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891088    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891091    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891094    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891098    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891101    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891104    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891106    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891110    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891114    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891117    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891120    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891123    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891126    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891129    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891132    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891138    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.903785 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891141    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891144    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.891147    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.891153    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.900417    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.900448    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.900478    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.900488    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900580    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900590    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900595    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900600    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900604    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900609    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.904626 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900613    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900618    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900622    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900626    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900632    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900636    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900641    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900645    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900650    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900654    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900659    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900663    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900670    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900678    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900682    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900687    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900692    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900696    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900700    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900704    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.904992 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900708    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900713    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900717    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900724    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900729    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900734    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900738    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900742    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900747    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900751    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900756    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900760    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900764    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900768    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900772    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900776    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900780    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900784    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900788    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900793    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.905670 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900797    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900801    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900805    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900810    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900814    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900819    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900823    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900827    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900831    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900835    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900840    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900843    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900847    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900852    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900856    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900862    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900867    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900872    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900876    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900880    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.906177 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900885    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900889    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900893    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900897    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900902    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900906    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900910    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900916    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900922    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900926    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900930    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900934    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900940    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900946    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900950    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900955    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900959    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900964    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900968    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.906764 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900974    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900978    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900982    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900987    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900991    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.900996    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901000    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901004    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901009    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901013    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901018    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901023    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901029    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901033    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.901042    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:17.907238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901242    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901252    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901257    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901262    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901266    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901271    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901275    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901279    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901301    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901306    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901310    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901314    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901319    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901323    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901327    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901330    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901335    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901339    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901344    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:17.907811 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901348    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901352    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901356    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901360    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901364    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901369    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901374    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901379    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901383    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901389    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901394    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901398    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901403    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901410    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901418    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901423    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901428    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901432    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901437    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901441    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:17.908306 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901445    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901449    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901454    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901458    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901462    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901466    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901470    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901475    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901479    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901483    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901487    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901491    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901496    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901500    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901504    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901508    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901512    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901517    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901522    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:17.908848 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901526    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901533    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901538    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901544    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901550    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901554    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901558    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901562    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901566    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901570    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901575    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901579    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901583    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901587    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901592    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901599    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901603    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901607    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901612    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:17.909475 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901616    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901620    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901624    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901628    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901632    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901637    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901640    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901644    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901648    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901653    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901656    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901662    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901667    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901671    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901675    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901680    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901685    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901690    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901694    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901699    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:17.910097 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901702    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:17.901707    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.901716    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.901924    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.905345    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.906565    2577 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.906681    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:17.910647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.906727    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:17.912431 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.912402    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:17.915875 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.915843    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:17.927184 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.927154    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:17.931010 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.930988    2577 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:17.931943 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.931925    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:17.934793 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.934762    2577 fs.go:135] Filesystem UUIDs: map[34982ee0-3673-4e71-b356-f66b62bccdc1:/dev/nvme0n1p3 3e9b59b8-9368-4acc-9f95-debc52fcc7ff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:11:17.934869 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.934792    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:17.941078 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.940883    2577 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:17.939408998 +0000 UTC m=+0.388730788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100739 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25238cd93c33d1174124e7b04aec91 SystemUUID:ec25238c-d93c-33d1-1741-24e7b04aec91 BootID:f039e5ce-71c0-48f9-9232-ee73d8e86476 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c1:d0:8f:a7:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c1:d0:8f:a7:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:1d:dc:f0:52:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:17.941078 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.941054    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:17.941281 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.941108    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:17.941281 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.941205    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:17.944912 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.944860    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:17.945122 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.944918    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:17.945912 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.945894    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:17.945955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.945914    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:17.945955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.945923    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:17.945955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.945941    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:17.947976 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.947958    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:17.952563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952544    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:17.952702 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952691    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:17.952734 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952725    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:17.952762 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952737    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:17.952762 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952748    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:17.952762 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952757    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:17.952862 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.952767    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:17.953298 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.953265    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:17.953370 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.953307    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:17.955928 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.955908    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:17.958813 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.958792    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:17.958902 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.958832    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:17.959044 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959032    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:17.959090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959048    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:17.959090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959055    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:17.959090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959061    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:17.959090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959068    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:17.959090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959074    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:17.959654 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959645    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:17.959654 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959654    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:17.959726 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959661    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:17.959726 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959667    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:17.959726 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959676    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:17.959726 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.959683    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:17.960622 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.960609    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:17.960622 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.960623    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:17.965674 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.965651    2577 server.go:1268] "Started kubelet"
Sep 09 20:11:17.965827 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.965747    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:17.965827 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.965824    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:17.965927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.965756    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:17.966601 ip-10-0-128-184 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:17.967841 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.967800    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:17.967966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.967936    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:17.968853 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.968829    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:17.969603 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.969568    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:17.969696 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.969584    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:17.970206 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.970187    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:17.974319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.974261    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:17.974954 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.974935    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:17.975112 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.975088    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:17.975801 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975780    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:17.975801 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975781    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:17.975951 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975808    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:17.975951 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975904    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:17.975951 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975919    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:17.976037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.975969    2577 factory.go:55] Registering systemd factory
Sep 09 20:11:17.976037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976009    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:17.976280 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976266    2577 factory.go:153] Registering CRI-O factory
Sep 09 20:11:17.976357 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976300    2577 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:17.976401 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976356    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:17.976401 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976375    2577 factory.go:103] Registering Raw factory
Sep 09 20:11:17.976401 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.976392    2577 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:17.977158 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.977138    2577 manager.go:349] Starting recovery of all containers
Sep 09 20:11:17.979033 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.978778    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:11:17.979033 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.978820    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:11:17.980208 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.978990    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-184.ec2.internal.18d3c0160ec13e26  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-184.ec2.internal,UID:ip-10-0-128-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-184.ec2.internal,},FirstTimestamp:2026-09-09 20:11:17.965577766 +0000 UTC m=+0.414899536,LastTimestamp:2026-09-09 20:11:17.965577766 +0000 UTC m=+0.414899536,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-184.ec2.internal,}"
Sep 09 20:11:17.988345 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.988039    2577 manager.go:354] Recovery completed
Sep 09 20:11:17.994320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.994281    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4p8k9"
Sep 09 20:11:17.994490 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.994477    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:17.996843 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.996823    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:17.996921 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.996865    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:17.996921 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.996880    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:17.997517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.997496    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:17.997517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.997511    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:17.997712 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.997513    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:17.997712 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.997585    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:17.998253 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.998238    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:17.998374 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.998266    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:17.998374 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.998280    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:17.998831 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:17.998742    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-184.ec2.internal.18d3c016109e5b09  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-184.ec2.internal,UID:ip-10-0-128-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-184.ec2.internal,},FirstTimestamp:2026-09-09 20:11:17.996845833 +0000 UTC m=+0.446167607,LastTimestamp:2026-09-09 20:11:17.996845833 +0000 UTC m=+0.446167607,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-184.ec2.internal,}"
Sep 09 20:11:18.000009 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:17.999993    2577 policy_none.go:49] "None policy: Start"
Sep 09 20:11:18.000118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.000013    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:18.000118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.000028    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:18.002071 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.002056    2577 policy_none.go:47] "Start"
Sep 09 20:11:18.002320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.002302    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4p8k9"
Sep 09 20:11:18.037053 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037028    2577 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.037083    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037117    2577 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037579    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037591    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037711    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037789    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.037821    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.038425    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.038471    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.050553    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.051788    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.051837    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.051860    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.051910    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:18.059766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.053905    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:18.138532 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.138407    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:18.139454 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.139435    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:18.139544 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.139469    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:18.139544 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.139479    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:18.139544 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.139507    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-184.ec2.internal"
Sep 09 20:11:18.148161 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.148133    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-184.ec2.internal"
Sep 09 20:11:18.148354 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.148176    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-184.ec2.internal\": node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:18.151996 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.151966    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"]
Sep 09 20:11:18.152143 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.152049    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:18.152865 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.152844    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:18.152963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.152876    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:18.152963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.152888    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:18.155278 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.155253    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:18.155402 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.155360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:18.155402 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.155389    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:18.156061 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156043    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:18.156140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156071    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:18.156140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156081    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:18.156140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156043    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:18.156226 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156151    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:18.156226 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.156163    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:18.158460 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.158445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:18.158523 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.158473    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:18.159267 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.159251    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:18.159347 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.159300    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:18.159347 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:18.159316    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:18.168924 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.168894    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:18.186359 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.186334    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-184.ec2.internal\" not found" node="ip-10-0-128-184.ec2.internal"
Sep 09 20:11:18.191244 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:18.191223    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-184.ec2.internal\" not found" node="ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.832325 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:19.832293    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:19.839626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.839599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc5e2da99c7ada79b50f8817f3f862d1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"dc5e2da99c7ada79b50f8817f3f862d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.839748 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.839634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.839748 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.839653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.932811 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:19.932765    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:19.940102 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.940170 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.940170 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc5e2da99c7ada79b50f8817f3f862d1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"dc5e2da99c7ada79b50f8817f3f862d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.940238 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.940238 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b24c588bb9bea25a5973040c9c64e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"067b24c588bb9bea25a5973040c9c64e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:19.940238 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:19.940214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc5e2da99c7ada79b50f8817f3f862d1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"dc5e2da99c7ada79b50f8817f3f862d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:20.033615 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.033537    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.053035 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.052991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:20.058558 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.058482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:20.134235 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.134195    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.234843 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.234808    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.335492 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.335388    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.435940 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.435893    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.469529 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.469493    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:20.469998 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.469680    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:20.536951 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.536917    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.539473 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.539240    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:20.568297 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.568246    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:19 +0000 UTC" deadline="2028-04-09 21:52:10.35746163 +0000 UTC"
Sep 09 20:11:20.568297 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.568306    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13873h40m49.78916018s"
Sep 09 20:11:20.637699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:20.637629    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 09 20:11:20.687862 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:20.687829    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067b24c588bb9bea25a5973040c9c64e.slice/crio-114504f4dba4ab67fb78f19c371fb1afad924cb9878b3d9d6e4c5681a40f1694 WatchSource:0}: Error finding container 114504f4dba4ab67fb78f19c371fb1afad924cb9878b3d9d6e4c5681a40f1694: Status 404 returned error can't find the container with id 114504f4dba4ab67fb78f19c371fb1afad924cb9878b3d9d6e4c5681a40f1694
Sep 09 20:11:20.692679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.692661    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:20.725652 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.725614    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:20.738767 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.738736    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:20.751729 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.751691    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:20.753750 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.753727    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 09 20:11:20.763033 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.762995    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:20.929048 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:20.928904    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:21.516882 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.516835    2577 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:21.524597 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.524565    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:21.525905 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.525869    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal","openshift-multus/network-metrics-daemon-9blb8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss","openshift-dns/node-resolver-l4pwf","openshift-multus/multus-additional-cni-plugins-tlr89","openshift-multus/multus-phcgt","openshift-network-diagnostics/network-check-target-c2tnv","openshift-network-operator/iptables-alerter-778bm","openshift-ovn-kubernetes/ovnkube-node-258xn","kube-system/global-pull-secret-syncer-6ptjs","kube-system/konnectivity-agent-x8jwj","kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4vs8l","openshift-image-registry/node-ca-jmz8t"]
Sep 09 20:11:21.528620 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.528578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.530981 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.530942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.531192 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.531140    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:21.531269 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.531211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:21.531536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.531513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:21.531765 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.531715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rr7s2\""
Sep 09 20:11:21.533556 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.533519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.535849 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.535684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.535849 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.535800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:21.536540 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.536201    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n6qqv\""
Sep 09 20:11:21.536540 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.536402    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.536540 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.536412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.538380 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.538096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.538983 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.538959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.539104 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.539068    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:21.539237 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.539218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gtwl\""
Sep 09 20:11:21.539317 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.539303    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.539393 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.539378    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:21.540426 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.540405    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hwxjp\""
Sep 09 20:11:21.540426 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.540425    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:21.540966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.540690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.541235 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.541206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:21.541452 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.541433    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:21.541529 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.541508    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.541657 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.541636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.543188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.542967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:21.543188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.542981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:21.543188 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.543072    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:21.543188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.543094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4shnv\""
Sep 09 20:11:21.545331 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.545297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.547492 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.547469    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bz4t4\""
Sep 09 20:11:21.547810 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.547735    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.547810 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.547772    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.548431 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.548409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.550305 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.550418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tckc5\" (UniqueName: \"kubernetes.io/projected/b05148b1-8474-4106-a452-6fee11123ae9-kube-api-access-tckc5\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.550418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-multus-certs\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.550418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-etc-kubernetes\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.550418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59f4958b-f12d-4efd-ab96-0e71ae498273-konnectivity-ca\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.550418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.550737 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-sys-fs\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.550737 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-cnibin\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.550737 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.550870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-netns\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.550870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-multus\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.550870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-socket-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.550988 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-daemon-config\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.550988 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59f4958b-f12d-4efd-ab96-0e71ae498273-agent-certs\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.550988 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.550966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.551166 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/282b4062-5693-4318-8e23-104b0ec0bb27-host\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.551166 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cnibin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.551285 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cni-binary-copy\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.551285 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-socket-dir-parent\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-k8s-cni-cncf-io\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-bin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.551947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-conf-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-registration-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-system-cni-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-hostroot\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-device-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-kubelet\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.552619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnb4\" (UniqueName: \"kubernetes.io/projected/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-kube-api-access-twnb4\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94t8w\" (UniqueName: \"kubernetes.io/projected/cb7a3927-19b9-4c23-8c72-b245c699f4ee-kube-api-access-94t8w\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6dm\" (UniqueName: \"kubernetes.io/projected/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kube-api-access-bm6dm\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmxj\" (UniqueName: \"kubernetes.io/projected/282b4062-5693-4318-8e23-104b0ec0bb27-kube-api-access-qxmxj\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553160    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.553361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-system-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553368    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/282b4062-5693-4318-8e23-104b0ec0bb27-serviceca\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.554198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.553842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.554592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.554301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:21.554592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.554523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:21.554592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.554585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5jvv2\""
Sep 09 20:11:21.555917 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.554916    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pl5fx\""
Sep 09 20:11:21.555917 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.555324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.556116 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.556033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.556175 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.556142    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:21.556363 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.556326    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:21.558548 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.558526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.561159 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.561133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lmb6g\""
Sep 09 20:11:21.561291 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.561157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:21.561291 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.561164    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:21.561399 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.561133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:21.572581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.572550    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5lswz"
Sep 09 20:11:21.580605 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.580565    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5lswz"
Sep 09 20:11:21.619573 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.619519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" event={"ID":"dc5e2da99c7ada79b50f8817f3f862d1","Type":"ContainerStarted","Data":"8ebf2cfd000e1b7c99710763cc74109db594652cedc97a121da64560aa663c43"}
Sep 09 20:11:21.620730 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.620695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"067b24c588bb9bea25a5973040c9c64e","Type":"ContainerStarted","Data":"114504f4dba4ab67fb78f19c371fb1afad924cb9878b3d9d6e4c5681a40f1694"}
Sep 09 20:11:21.639673 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.639636    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:21.654703 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-conf\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.654703 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-sys\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.654927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-etc-tuned\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.654927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.654927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-daemon-config\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.654927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-netns\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.654927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-systemd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-log-socket\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.654993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-bin\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cnibin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-conf-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-kubernetes\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.655175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-conf-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-var-lib-kubelet\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cnibin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-tmp\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-node-log\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2pc\" (UniqueName: \"kubernetes.io/projected/489520f6-3850-426c-b05a-2955e55cfd8b-kube-api-access-cm2pc\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3cf640-5639-40e0-b251-550ef10be316-hosts-file\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-systemd-units\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-env-overrides\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/489520f6-3850-426c-b05a-2955e55cfd8b-ovn-node-metrics-cert\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655422 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/ac721a5e-4c47-4054-b183-0ee6c936dccd-kube-api-access-n2spf\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-slash\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-daemon-config\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94t8w\" (UniqueName: \"kubernetes.io/projected/cb7a3927-19b9-4c23-8c72-b245c699f4ee-kube-api-access-94t8w\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-kubelet\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-netd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3cf640-5639-40e0-b251-550ef10be316-tmp-dir\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac721a5e-4c47-4054-b183-0ee6c936dccd-iptables-alerter-script\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-modprobe-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsh8g\" (UniqueName: \"kubernetes.io/projected/84ad5e66-ebe8-4198-9688-a0c064038611-kube-api-access-tsh8g\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-dbus\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59f4958b-f12d-4efd-ab96-0e71ae498273-konnectivity-ca\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.655894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-netns\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-multus\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.655883    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysconfig\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-netns\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.655913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.655981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:22.155940701 +0000 UTC m=+3.042530611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-multus\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fjp\" (UniqueName: \"kubernetes.io/projected/4d3cf640-5639-40e0-b251-550ef10be316-kube-api-access-n9fjp\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59f4958b-f12d-4efd-ab96-0e71ae498273-agent-certs\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/282b4062-5693-4318-8e23-104b0ec0bb27-host\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/282b4062-5693-4318-8e23-104b0ec0bb27-host\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-config\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cni-binary-copy\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.656699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-socket-dir-parent\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-k8s-cni-cncf-io\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-bin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-socket-dir-parent\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-k8s-cni-cncf-io\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-registration-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-registration-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59f4958b-f12d-4efd-ab96-0e71ae498273-konnectivity-ca\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-host\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-system-cni-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-cni-bin\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-system-cni-dir\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-hostroot\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-hostroot\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-device-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-etc-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.657555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-script-lib\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-kubelet-config\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac721a5e-4c47-4054-b183-0ee6c936dccd-host-slash\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-cni-binary-copy\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-kubelet\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-device-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnb4\" (UniqueName: \"kubernetes.io/projected/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-kube-api-access-twnb4\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6dm\" (UniqueName: \"kubernetes.io/projected/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kube-api-access-bm6dm\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-var-lib-kubelet\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmxj\" (UniqueName: \"kubernetes.io/projected/282b4062-5693-4318-8e23-104b0ec0bb27-kube-api-access-qxmxj\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-systemd\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656940    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-var-lib-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.656996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-system-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/282b4062-5693-4318-8e23-104b0ec0bb27-serviceca\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.658256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-run\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-lib-modules\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-ovn\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-system-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-multus-cni-dir\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tckc5\" (UniqueName: \"kubernetes.io/projected/b05148b1-8474-4106-a452-6fee11123ae9-kube-api-access-tckc5\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-multus-certs\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-etc-kubernetes\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-etc-kubernetes\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-host-run-multus-certs\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-sys-fs\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/282b4062-5693-4318-8e23-104b0ec0bb27-serviceca\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-sys-fs\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.658898 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-cnibin\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-socket-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07b3a564-f786-4c1b-875d-1d7ced95cc5e-socket-dir\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.657803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b05148b1-8474-4106-a452-6fee11123ae9-cnibin\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.659484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.658622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b05148b1-8474-4106-a452-6fee11123ae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.661823 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.661792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59f4958b-f12d-4efd-ab96-0e71ae498273-agent-certs\") pod \"konnectivity-agent-x8jwj\" (UID: \"59f4958b-f12d-4efd-ab96-0e71ae498273\") " pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.666050 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.666002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnb4\" (UniqueName: \"kubernetes.io/projected/e2824d45-98ee-4a43-94ed-c01bb6a31b6d-kube-api-access-twnb4\") pod \"multus-phcgt\" (UID: \"e2824d45-98ee-4a43-94ed-c01bb6a31b6d\") " pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.666210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.666097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94t8w\" (UniqueName: \"kubernetes.io/projected/cb7a3927-19b9-4c23-8c72-b245c699f4ee-kube-api-access-94t8w\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:21.666816 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.666790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tckc5\" (UniqueName: \"kubernetes.io/projected/b05148b1-8474-4106-a452-6fee11123ae9-kube-api-access-tckc5\") pod \"multus-additional-cni-plugins-tlr89\" (UID: \"b05148b1-8474-4106-a452-6fee11123ae9\") " pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.666923 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.666835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6dm\" (UniqueName: \"kubernetes.io/projected/07b3a564-f786-4c1b-875d-1d7ced95cc5e-kube-api-access-bm6dm\") pod \"aws-ebs-csi-driver-node-mr6ss\" (UID: \"07b3a564-f786-4c1b-875d-1d7ced95cc5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.668169 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.668140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmxj\" (UniqueName: \"kubernetes.io/projected/282b4062-5693-4318-8e23-104b0ec0bb27-kube-api-access-qxmxj\") pod \"node-ca-jmz8t\" (UID: \"282b4062-5693-4318-8e23-104b0ec0bb27\") " pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.758073 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fjp\" (UniqueName: \"kubernetes.io/projected/4d3cf640-5639-40e0-b251-550ef10be316-kube-api-access-n9fjp\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-config\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-host\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-etc-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-script-lib\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-kubelet-config\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.758273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac721a5e-4c47-4054-b183-0ee6c936dccd-host-slash\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-systemd\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-host\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-var-lib-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-var-lib-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-run\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-etc-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-kubelet-config\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac721a5e-4c47-4054-b183-0ee6c936dccd-host-slash\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-lib-modules\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-ovn\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-systemd\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-lib-modules\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-ovn\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-run\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.758678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-conf\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-sys\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-etc-tuned\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.758614    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-netns\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-sys\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.758681    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:22.258662181 +0000 UTC m=+3.145252083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-netns\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-systemd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-log-socket\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysctl-conf\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-log-socket\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-bin\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-systemd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-bin\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.759504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-kubernetes\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-var-lib-kubelet\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-kubernetes\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-tmp\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-node-log\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-var-lib-kubelet\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2pc\" (UniqueName: \"kubernetes.io/projected/489520f6-3850-426c-b05a-2955e55cfd8b-kube-api-access-cm2pc\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-node-log\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.758975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3cf640-5639-40e0-b251-550ef10be316-hosts-file\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-systemd-units\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-env-overrides\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3cf640-5639-40e0-b251-550ef10be316-hosts-file\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/489520f6-3850-426c-b05a-2955e55cfd8b-ovn-node-metrics-cert\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-systemd-units\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/ac721a5e-4c47-4054-b183-0ee6c936dccd-kube-api-access-n2spf\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-slash\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-kubelet\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-netd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.760306 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3cf640-5639-40e0-b251-550ef10be316-tmp-dir\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac721a5e-4c47-4054-b183-0ee6c936dccd-iptables-alerter-script\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-modprobe-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsh8g\" (UniqueName: \"kubernetes.io/projected/84ad5e66-ebe8-4198-9688-a0c064038611-kube-api-access-tsh8g\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-script-lib\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-ovnkube-config\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-kubelet\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-slash\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-dbus\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysconfig\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-cni-netd\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/489520f6-3850-426c-b05a-2955e55cfd8b-env-overrides\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/489520f6-3850-426c-b05a-2955e55cfd8b-run-openvswitch\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31c6b37-d608-46c2-9df9-0032ba3235d0-dbus\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:21.761181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-modprobe-d\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.762003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84ad5e66-ebe8-4198-9688-a0c064038611-etc-sysconfig\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.762003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3cf640-5639-40e0-b251-550ef10be316-tmp-dir\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.762003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.759840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac721a5e-4c47-4054-b183-0ee6c936dccd-iptables-alerter-script\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.762003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.761896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/489520f6-3850-426c-b05a-2955e55cfd8b-ovn-node-metrics-cert\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.762003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.761986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-tmp\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.762218 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.762086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84ad5e66-ebe8-4198-9688-a0c064038611-etc-tuned\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.765208 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.765179    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:21.765208 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.765210    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:21.765409 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.765225    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:21.765409 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:21.765312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:22.265291741 +0000 UTC m=+3.151881629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:21.768084 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.767991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/ac721a5e-4c47-4054-b183-0ee6c936dccd-kube-api-access-n2spf\") pod \"iptables-alerter-778bm\" (UID: \"ac721a5e-4c47-4054-b183-0ee6c936dccd\") " pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:21.768228 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.768102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fjp\" (UniqueName: \"kubernetes.io/projected/4d3cf640-5639-40e0-b251-550ef10be316-kube-api-access-n9fjp\") pod \"node-resolver-l4pwf\" (UID: \"4d3cf640-5639-40e0-b251-550ef10be316\") " pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.768482 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.768456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2pc\" (UniqueName: \"kubernetes.io/projected/489520f6-3850-426c-b05a-2955e55cfd8b-kube-api-access-cm2pc\") pod \"ovnkube-node-258xn\" (UID: \"489520f6-3850-426c-b05a-2955e55cfd8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.768963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.768942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsh8g\" (UniqueName: \"kubernetes.io/projected/84ad5e66-ebe8-4198-9688-a0c064038611-kube-api-access-tsh8g\") pod \"tuned-4vs8l\" (UID: \"84ad5e66-ebe8-4198-9688-a0c064038611\") " pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.814836 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.814803    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:21.842177 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.842138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:21.849920 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.849891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss"
Sep 09 20:11:21.860904 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.860873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jmz8t"
Sep 09 20:11:21.866783 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.866761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tlr89"
Sep 09 20:11:21.874517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.874483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phcgt"
Sep 09 20:11:21.883593 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.883560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4vs8l"
Sep 09 20:11:21.893716 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.893684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:21.902647 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.902608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4pwf"
Sep 09 20:11:21.909550 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:21.909517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-778bm"
Sep 09 20:11:22.162344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.162242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:22.162559 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.162414    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:22.162559 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.162494    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.162471987 +0000 UTC m=+4.049061896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:22.263378 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.263334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:22.263560 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.263467    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:22.263560 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.263525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.263511391 +0000 UTC m=+4.150101292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:22.364270 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.364233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:22.364449 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.364414    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:22.364449 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.364438    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:22.364449 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.364449    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:22.364554 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.364503    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.364489637 +0000 UTC m=+4.251079530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:22.584521 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.584323    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:21 +0000 UTC" deadline="2028-03-23 15:35:31.948422504 +0000 UTC"
Sep 09 20:11:22.584521 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.584496    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13459h24m9.363931798s"
Sep 09 20:11:22.586704 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.586674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2824d45_98ee_4a43_94ed_c01bb6a31b6d.slice/crio-3a16d4f3471521482b40b457857e910dbc3aa2cfdda3c3d622183a3f98c20dda WatchSource:0}: Error finding container 3a16d4f3471521482b40b457857e910dbc3aa2cfdda3c3d622183a3f98c20dda: Status 404 returned error can't find the container with id 3a16d4f3471521482b40b457857e910dbc3aa2cfdda3c3d622183a3f98c20dda
Sep 09 20:11:22.587545 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.587513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ad5e66_ebe8_4198_9688_a0c064038611.slice/crio-f4cfe880f9e31fddde927b2dea92a2d00e0a49b45565da869bfa4fd9427088d5 WatchSource:0}: Error finding container f4cfe880f9e31fddde927b2dea92a2d00e0a49b45565da869bfa4fd9427088d5: Status 404 returned error can't find the container with id f4cfe880f9e31fddde927b2dea92a2d00e0a49b45565da869bfa4fd9427088d5
Sep 09 20:11:22.589444 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.589356    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489520f6_3850_426c_b05a_2955e55cfd8b.slice/crio-1b5a069125c7d0d248c5b0d1bebbcb6c5887ec16920dae47ab54650dd5084de6 WatchSource:0}: Error finding container 1b5a069125c7d0d248c5b0d1bebbcb6c5887ec16920dae47ab54650dd5084de6: Status 404 returned error can't find the container with id 1b5a069125c7d0d248c5b0d1bebbcb6c5887ec16920dae47ab54650dd5084de6
Sep 09 20:11:22.590210 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.590161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac721a5e_4c47_4054_b183_0ee6c936dccd.slice/crio-e60264b2e7474d371aec097e1ed04db67cda5f652b485ac755eb0528e805ccc7 WatchSource:0}: Error finding container e60264b2e7474d371aec097e1ed04db67cda5f652b485ac755eb0528e805ccc7: Status 404 returned error can't find the container with id e60264b2e7474d371aec097e1ed04db67cda5f652b485ac755eb0528e805ccc7
Sep 09 20:11:22.592770 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.592678    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05148b1_8474_4106_a452_6fee11123ae9.slice/crio-63d40bcb759e83ead1a538619f9f1299c6d3261684d391012c4d6ad1f3a2ff47 WatchSource:0}: Error finding container 63d40bcb759e83ead1a538619f9f1299c6d3261684d391012c4d6ad1f3a2ff47: Status 404 returned error can't find the container with id 63d40bcb759e83ead1a538619f9f1299c6d3261684d391012c4d6ad1f3a2ff47
Sep 09 20:11:22.593933 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.593524    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b3a564_f786_4c1b_875d_1d7ced95cc5e.slice/crio-230bb6880c35ed057e67714e72fb761b417e6732c3f04aaa75a45a2c37b8a4b8 WatchSource:0}: Error finding container 230bb6880c35ed057e67714e72fb761b417e6732c3f04aaa75a45a2c37b8a4b8: Status 404 returned error can't find the container with id 230bb6880c35ed057e67714e72fb761b417e6732c3f04aaa75a45a2c37b8a4b8
Sep 09 20:11:22.594568 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.594544    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f4958b_f12d_4efd_ab96_0e71ae498273.slice/crio-2f8663096243bb6a1d221d4d98fd173fd3b9415cf333551c706b558b3e2ff01c WatchSource:0}: Error finding container 2f8663096243bb6a1d221d4d98fd173fd3b9415cf333551c706b558b3e2ff01c: Status 404 returned error can't find the container with id 2f8663096243bb6a1d221d4d98fd173fd3b9415cf333551c706b558b3e2ff01c
Sep 09 20:11:22.596801 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.596532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3cf640_5639_40e0_b251_550ef10be316.slice/crio-22179fa1246e05639083580c42b8e73d9e4b968e482efebb66c371936fdf5da6 WatchSource:0}: Error finding container 22179fa1246e05639083580c42b8e73d9e4b968e482efebb66c371936fdf5da6: Status 404 returned error can't find the container with id 22179fa1246e05639083580c42b8e73d9e4b968e482efebb66c371936fdf5da6
Sep 09 20:11:22.597265 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:22.597178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282b4062_5693_4318_8e23_104b0ec0bb27.slice/crio-6f3ee842a567d1206a1c9c0ae654c728d034d788c47b2264057cd18accddbbc6 WatchSource:0}: Error finding container 6f3ee842a567d1206a1c9c0ae654c728d034d788c47b2264057cd18accddbbc6: Status 404 returned error can't find the container with id 6f3ee842a567d1206a1c9c0ae654c728d034d788c47b2264057cd18accddbbc6
Sep 09 20:11:22.614935 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.614906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:22.615132 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:22.615029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:22.624082 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.624051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerStarted","Data":"63d40bcb759e83ead1a538619f9f1299c6d3261684d391012c4d6ad1f3a2ff47"}
Sep 09 20:11:22.625201 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.625170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4vs8l" event={"ID":"84ad5e66-ebe8-4198-9688-a0c064038611","Type":"ContainerStarted","Data":"f4cfe880f9e31fddde927b2dea92a2d00e0a49b45565da869bfa4fd9427088d5"}
Sep 09 20:11:22.626421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.626398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-778bm" event={"ID":"ac721a5e-4c47-4054-b183-0ee6c936dccd","Type":"ContainerStarted","Data":"e60264b2e7474d371aec097e1ed04db67cda5f652b485ac755eb0528e805ccc7"}
Sep 09 20:11:22.627663 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.627639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"1b5a069125c7d0d248c5b0d1bebbcb6c5887ec16920dae47ab54650dd5084de6"}
Sep 09 20:11:22.630608 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.630577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phcgt" event={"ID":"e2824d45-98ee-4a43-94ed-c01bb6a31b6d","Type":"ContainerStarted","Data":"3a16d4f3471521482b40b457857e910dbc3aa2cfdda3c3d622183a3f98c20dda"}
Sep 09 20:11:22.636065 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.636007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jmz8t" event={"ID":"282b4062-5693-4318-8e23-104b0ec0bb27","Type":"ContainerStarted","Data":"6f3ee842a567d1206a1c9c0ae654c728d034d788c47b2264057cd18accddbbc6"}
Sep 09 20:11:22.643128 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.643094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4pwf" event={"ID":"4d3cf640-5639-40e0-b251-550ef10be316","Type":"ContainerStarted","Data":"22179fa1246e05639083580c42b8e73d9e4b968e482efebb66c371936fdf5da6"}
Sep 09 20:11:22.645454 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.645423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x8jwj" event={"ID":"59f4958b-f12d-4efd-ab96-0e71ae498273","Type":"ContainerStarted","Data":"2f8663096243bb6a1d221d4d98fd173fd3b9415cf333551c706b558b3e2ff01c"}
Sep 09 20:11:22.646542 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:22.646519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss" event={"ID":"07b3a564-f786-4c1b-875d-1d7ced95cc5e","Type":"ContainerStarted","Data":"230bb6880c35ed057e67714e72fb761b417e6732c3f04aaa75a45a2c37b8a4b8"}
Sep 09 20:11:23.170634 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.170475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:23.170817 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.170671    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.170817 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.170749    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.170727836 +0000 UTC m=+6.057317738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.272245 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.272203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:23.272429 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.272394    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.272492 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.272459    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.272441002 +0000 UTC m=+6.159030904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.373581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.373537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:23.373963 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.373764    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:23.373963 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.373786    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:23.373963 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.373799    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:23.373963 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.373858    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.373839322 +0000 UTC m=+6.260429224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:23.585032 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.584890    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:21 +0000 UTC" deadline="2028-02-28 11:51:10.433409777 +0000 UTC"
Sep 09 20:11:23.585032 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.584961    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12879h39m46.848453052s"
Sep 09 20:11:23.617792 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.617751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:23.617951 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.617909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:23.618372 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.618350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:23.618481 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:23.618446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:23.672958 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.672916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" event={"ID":"dc5e2da99c7ada79b50f8817f3f862d1","Type":"ContainerStarted","Data":"6ad07bce99a72b024f461b781fa51652ac62d8fc27edbfd74a39dd3a6368536f"}
Sep 09 20:11:23.688092 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.687933    2577 generic.go:356] "Generic (PLEG): container finished" podID="067b24c588bb9bea25a5973040c9c64e" containerID="c7c86efb2ad694b4652ab690238d6985cc5cf81caac57490872134e35ffd2f29" exitCode=0
Sep 09 20:11:23.688092 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.688000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"067b24c588bb9bea25a5973040c9c64e","Type":"ContainerDied","Data":"c7c86efb2ad694b4652ab690238d6985cc5cf81caac57490872134e35ffd2f29"}
Sep 09 20:11:23.689337 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:23.689267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" podStartSLOduration=3.689246438 podStartE2EDuration="3.689246438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:23.688736954 +0000 UTC m=+4.575326891" watchObservedRunningTime="2026-09-09 20:11:23.689246438 +0000 UTC m=+4.575836350"
Sep 09 20:11:24.615086 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:24.615047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:24.615695 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:24.615182    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:24.704155 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:24.704078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"067b24c588bb9bea25a5973040c9c64e","Type":"ContainerStarted","Data":"8b64874bf4dc39fe58362e6d023ee21f8b17f1d835b1b7af159ad9a79476af26"}
Sep 09 20:11:24.725269 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:24.724762    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" podStartSLOduration=4.7247379259999995 podStartE2EDuration="4.724737926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:24.723943333 +0000 UTC m=+5.610533245" watchObservedRunningTime="2026-09-09 20:11:24.724737926 +0000 UTC m=+5.611327837"
Sep 09 20:11:25.193324 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:25.193176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:25.193505 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.193348    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:25.193505 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.193429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:29.193408035 +0000 UTC m=+10.079997934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:25.293905 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:25.293859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:25.294138 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.294115    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:25.294218 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.294196    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:29.294173332 +0000 UTC m=+10.180763233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:25.395339 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:25.395296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:25.395638 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.395615    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:25.395740 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.395648    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:25.395740 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.395663    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:25.395839 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.395748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:29.39572891 +0000 UTC m=+10.282318812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:25.615801 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:25.614977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:25.615801 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.615141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:25.615801 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:25.615590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:25.615801 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:25.615700    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:26.615667 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:26.615630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:26.615868 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:26.615775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:27.615235 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:27.615203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:27.615423 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:27.615352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:27.615883 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:27.615861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:27.616293 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:27.615982    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:28.615160 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:28.615124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:28.615364 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:28.615260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:29.226460 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:29.226410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:29.226927 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.226584    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:29.226927 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.226678    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:37.226655597 +0000 UTC m=+18.113245499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:29.327063 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:29.326989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:29.327257 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.327164    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:29.327257 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.327240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:37.327220887 +0000 UTC m=+18.213810791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:29.428499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:29.428428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:29.428699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.428599    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:29.428699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.428632    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:29.428699 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.428648    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:29.428868 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.428715    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:37.428696059 +0000 UTC m=+18.315285951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:29.616327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:29.616251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:29.618904 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:29.618870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:29.619085 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.619065    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:29.623071 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:29.619536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:30.615152 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:30.615113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:30.615631 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:30.615261    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:31.615038 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:31.614979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:31.615307 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:31.615130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:31.615307 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:31.615184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:31.615307 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:31.615289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:32.615211 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:32.615168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:32.615405 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:32.615296    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:33.614933 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:33.614893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:33.615161 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:33.614947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:33.615161 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:33.615052    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:33.615161 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:33.615114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:34.616106 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:34.615819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:34.616503 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:34.616210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:35.615100 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:35.615060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:35.615296 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:35.615110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:35.615296 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:35.615196    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:35.615402 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:35.615322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:36.615140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:36.615102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:36.615662 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:36.615233    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:37.290531 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:37.290490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:37.290717 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.290646    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:37.290717 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.290710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:11:53.290693573 +0000 UTC m=+34.177283462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:37.391138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:37.391093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:37.391318 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.391262    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:37.391372 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.391347    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:53.391322532 +0000 UTC m=+34.277912420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:37.491631 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:37.491594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:37.491828 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.491794    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:37.491828 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.491827    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:37.491930 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.491842    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:37.491930 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.491909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:53.491889317 +0000 UTC m=+34.378479220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:37.615273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:37.615179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:37.615756 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.615357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:37.615756 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:37.615179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:37.615756 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:37.615511    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:38.615899 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:38.615856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:38.616356 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:38.615994    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:39.616049 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:39.615999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:39.616528 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:39.616127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:39.616528 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:39.616198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:39.616528 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:39.616318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:40.615264 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:40.615224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:40.615426 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:40.615339    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:41.615513 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.615256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:41.616397 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.615256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:41.616397 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:41.615625    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:41.616397 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:41.615707    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:41.740960 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.740880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss" event={"ID":"07b3a564-f786-4c1b-875d-1d7ced95cc5e","Type":"ContainerStarted","Data":"261ca2c2232c68a8eadb9a0b3f1c9ed9e4bd134d51494c5d7786baf78ea8d9eb"}
Sep 09 20:11:41.742459 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.742430    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="dbfc1668097917c7553c514ea28df5526e154dc39ab4ee4faa5501eaa2c749f9" exitCode=0
Sep 09 20:11:41.742596 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.742512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"dbfc1668097917c7553c514ea28df5526e154dc39ab4ee4faa5501eaa2c749f9"}
Sep 09 20:11:41.744270 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.744243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4vs8l" event={"ID":"84ad5e66-ebe8-4198-9688-a0c064038611","Type":"ContainerStarted","Data":"d799a8ce2005699d8a0e53e097dececb2fc433573c1fea7e238b762372801569"}
Sep 09 20:11:41.747513 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"7f9a93b32e65489b3769692b8965eee851b05047b66c494274504723d2574bdb"}
Sep 09 20:11:41.747632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"3c021c27213d03b54f50c5e61198203127e3825b3afc1c3c716abeccb5904cc9"}
Sep 09 20:11:41.747632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"28589f3c58cefa5d22545df81dcb7dbb2b8e844cb79c33cd21c7378c7c7994df"}
Sep 09 20:11:41.747763 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"ae37e17c1a109814261ac3a39485d064a7ed4500b7b2c604f2050f3e5e48882b"}
Sep 09 20:11:41.747763 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"4e3d694c6db8c3dac209b2584ff8e61cafa49ab8fdd69b9de87dfcdc19063ce6"}
Sep 09 20:11:41.747876 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.747767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"a3b60e039853472db7c790f4b8a7804fa4f44e5d15781d1a4c1eff2b9550a027"}
Sep 09 20:11:41.749081 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.749055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phcgt" event={"ID":"e2824d45-98ee-4a43-94ed-c01bb6a31b6d","Type":"ContainerStarted","Data":"4132dbd002657bcd86744d2f8e7ee5d5a8c2c5f66de7afe26e3f6dfb0ec007aa"}
Sep 09 20:11:41.750576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.750548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jmz8t" event={"ID":"282b4062-5693-4318-8e23-104b0ec0bb27","Type":"ContainerStarted","Data":"978ab8d79f69fe1e90f15532c02ae8b3f4b217eb6346402af58dec3bf35bf03a"}
Sep 09 20:11:41.752190 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.752166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4pwf" event={"ID":"4d3cf640-5639-40e0-b251-550ef10be316","Type":"ContainerStarted","Data":"f10341f904bed6104c3bb55894edc7828f6191ee6492a0c872513b04c67c1afc"}
Sep 09 20:11:41.753785 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.753754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x8jwj" event={"ID":"59f4958b-f12d-4efd-ab96-0e71ae498273","Type":"ContainerStarted","Data":"51c9f044885e32f14ec45959685e55fe21c422fb03db30f8c444bfb1c0b03b7f"}
Sep 09 20:11:41.782002 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.781947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x8jwj" podStartSLOduration=4.390401356 podStartE2EDuration="22.781928632s" totalImagesPullingTime="18.391527276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:41.781663881 +0000 UTC m=+22.668253790" watchObservedRunningTime="2026-09-09 20:11:41.781928632 +0000 UTC m=+22.668518543"
Sep 09 20:11:41.797393 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.797338    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4vs8l" podStartSLOduration=4.371908214 podStartE2EDuration="22.797320117s" totalImagesPullingTime="18.425411903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:41.797312859 +0000 UTC m=+22.683902769" watchObservedRunningTime="2026-09-09 20:11:41.797320117 +0000 UTC m=+22.683910026"
Sep 09 20:11:41.838364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.838305    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jmz8t" podStartSLOduration=4.449450556 podStartE2EDuration="22.838289025s" totalImagesPullingTime="18.388838469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:41.811415368 +0000 UTC m=+22.698005289" watchObservedRunningTime="2026-09-09 20:11:41.838289025 +0000 UTC m=+22.724878937"
Sep 09 20:11:41.838629 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.838600    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-phcgt" podStartSLOduration=4.342232087 podStartE2EDuration="22.838594249s" totalImagesPullingTime="18.496362162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:41.838228979 +0000 UTC m=+22.724818889" watchObservedRunningTime="2026-09-09 20:11:41.838594249 +0000 UTC m=+22.725184160"
Sep 09 20:11:41.860194 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:41.860138    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l4pwf" podStartSLOduration=3.423029376 podStartE2EDuration="21.860120016s" totalImagesPullingTime="18.43709064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:41.859571809 +0000 UTC m=+22.746161729" watchObservedRunningTime="2026-09-09 20:11:41.860120016 +0000 UTC m=+22.746709926"
Sep 09 20:11:42.614849 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:42.614814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:42.615159 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:42.614938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:42.758406 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:42.758125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-778bm" event={"ID":"ac721a5e-4c47-4054-b183-0ee6c936dccd","Type":"ContainerStarted","Data":"c1e89cecc79cef87f5be316d065c14b2a52519844d471dfe21bbdf1ffb4021ba"}
Sep 09 20:11:42.824068 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:42.824010    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:43.613048 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.612879    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:42.824054915Z","UUID":"4aab4e7a-5d01-4256-a442-fd4de7676cec","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:43.614850 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.614823    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:43.614999 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.614863    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:43.614999 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.614885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:43.614999 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.614885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:43.615190 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:43.615040    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:43.615190 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:43.615094    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:43.763316 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:43.763283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss" event={"ID":"07b3a564-f786-4c1b-875d-1d7ced95cc5e","Type":"ContainerStarted","Data":"dab5a2937e498ded0dfabb76ff412be780332480c75dc8b1e55c593cfb61e0ea"}
Sep 09 20:11:44.615190 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:44.615146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:44.615386 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:44.615271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:44.768640 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:44.768599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss" event={"ID":"07b3a564-f786-4c1b-875d-1d7ced95cc5e","Type":"ContainerStarted","Data":"39d6b8fac5d127a433de2ac5c19aa50df9959eb738f6ee2047dc4cf55f6af192"}
Sep 09 20:11:44.772599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:44.772558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"37fe34de47932fbe715e10aea2f340af413c2a8d3d5a53c3b310606e8e58ac94"}
Sep 09 20:11:44.786830 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:44.786775    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mr6ss" podStartSLOduration=4.764605864 podStartE2EDuration="25.786759487s" totalImagesPullingTime="21.022153623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.786619303 +0000 UTC m=+25.673209215" watchObservedRunningTime="2026-09-09 20:11:44.786759487 +0000 UTC m=+25.673349396"
Sep 09 20:11:44.787188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:44.787153    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-778bm" podStartSLOduration=6.348052616 podStartE2EDuration="24.787143077s" totalImagesPullingTime="18.439090461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:42.77594438 +0000 UTC m=+23.662534308" watchObservedRunningTime="2026-09-09 20:11:44.787143077 +0000 UTC m=+25.673733023"
Sep 09 20:11:45.615579 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:45.615540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:45.615801 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:45.615676    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:45.615801 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:45.615733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:45.615907 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:45.615863    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:45.870453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:45.870360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:45.870944 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:45.870923    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:46.615179 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.614962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:46.615439 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:46.615263    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:46.788753 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.788714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" event={"ID":"489520f6-3850-426c-b05a-2955e55cfd8b","Type":"ContainerStarted","Data":"7c6f9d53cf0500e6544e3ee791b8ef88b525b62277dabc3a01db8294ebd50308"}
Sep 09 20:11:46.789083 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.789060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:46.791505 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.791473    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="ba26aff470be7b882f95ee7980c2e8ec69ea8d8fcb7d61d238772df824c70330" exitCode=0
Sep 09 20:11:46.791648 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.791545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"ba26aff470be7b882f95ee7980c2e8ec69ea8d8fcb7d61d238772df824c70330"}
Sep 09 20:11:46.791834 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.791815    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:46.792363 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.792348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x8jwj"
Sep 09 20:11:46.806229 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.806205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:46.815907 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:46.815855    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-258xn" podStartSLOduration=9.367443872 podStartE2EDuration="27.815840358s" totalImagesPullingTime="18.448396486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:46.814594285 +0000 UTC m=+27.701184195" watchObservedRunningTime="2026-09-09 20:11:46.815840358 +0000 UTC m=+27.702430267"
Sep 09 20:11:47.615446 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:47.615411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:47.615971 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:47.615411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:47.615971 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:47.615565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:47.615971 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:47.615595    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:47.794457 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:47.794422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:47.794991 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:47.794963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:47.810268 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:47.810233    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:11:48.142164 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.142135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ptjs"]
Sep 09 20:11:48.142351 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.142241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:48.142351 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:48.142335    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:48.146241 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.146192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c2tnv"]
Sep 09 20:11:48.146404 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.146339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:48.146488 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:48.146445    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:48.147004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.146978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9blb8"]
Sep 09 20:11:48.147156 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.147094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:48.147223 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:48.147193    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:48.799180 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.799149    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="b343593fe5e34d1c4f5f209e7fa1b18f84d66c1008a953133464840a4d74ae9e" exitCode=0
Sep 09 20:11:48.799736 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:48.799239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"b343593fe5e34d1c4f5f209e7fa1b18f84d66c1008a953133464840a4d74ae9e"}
Sep 09 20:11:49.616707 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:49.616610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:49.616857 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:49.616717    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:49.616857 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:49.616774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:49.616929 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:49.616895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:49.804257 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:49.804223    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="286814c26d46dfbbc5b1081b46afb938664dcd4f63c28ea5b37bd66829ca7364" exitCode=0
Sep 09 20:11:49.804601 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:49.804301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"286814c26d46dfbbc5b1081b46afb938664dcd4f63c28ea5b37bd66829ca7364"}
Sep 09 20:11:50.615149 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:50.614836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:50.615377 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:50.615246    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:51.615624 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:51.615591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:51.616062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:51.615591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:51.616062 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:51.615808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:51.616301 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:51.616270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:52.615374 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:52.615283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:52.615576 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:52.615443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2tnv" podUID="cd5da601-b4d8-48d8-9899-bd65af4d19d1"
Sep 09 20:11:53.317905 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.317864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:53.318416 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.318013    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:53.318416 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.318111    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.318087238 +0000 UTC m=+66.204677133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:53.418381 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.418333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:53.418572 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.418466    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:53.418572 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.418556    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret podName:b31c6b37-d608-46c2-9df9-0032ba3235d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.418532312 +0000 UTC m=+66.305122240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret") pod "global-pull-secret-syncer-6ptjs" (UID: "b31c6b37-d608-46c2-9df9-0032ba3235d0") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:53.519463 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.519422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:53.519633 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.519572    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:53.519633 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.519599    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:53.519633 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.519612    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jrsbf for pod openshift-network-diagnostics/network-check-target-c2tnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:53.519754 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.519683    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf podName:cd5da601-b4d8-48d8-9899-bd65af4d19d1 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.519661865 +0000 UTC m=+66.406251756 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrsbf" (UniqueName: "kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf") pod "network-check-target-c2tnv" (UID: "cd5da601-b4d8-48d8-9899-bd65af4d19d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:53.615727 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.615690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:53.615919 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.615701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:53.615919 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.615827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ptjs" podUID="b31c6b37-d608-46c2-9df9-0032ba3235d0"
Sep 09 20:11:53.615919 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:53.615891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:11:53.911131 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.911045    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeReady"
Sep 09 20:11:53.911292 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.911231    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:53.944209 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.944168    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-h8qd8"]
Sep 09 20:11:53.968978 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.968947    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:11:53.969173 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.969106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:53.976369 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.973115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 20:11:53.976369 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.973185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6pvwv\""
Sep 09 20:11:53.976369 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.973254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 20:11:53.989149 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.989111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-h8qd8"]
Sep 09 20:11:53.989149 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.989150    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dg8zk"]
Sep 09 20:11:53.989335 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.989229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:53.991938 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.991915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:11:53.991938 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.991920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ctkpk\""
Sep 09 20:11:53.992167 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.992149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:11:53.994241 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.994219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:11:53.997578 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:53.997554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:11:54.011313 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.011282    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fd489"]
Sep 09 20:11:54.011483 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.011417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.014082 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.014059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzq8l\""
Sep 09 20:11:54.014228 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.014110    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:54.014228 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.014061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:54.014228 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.014067    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:54.035620 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.035585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:11:54.035620 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.035624    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dg8zk"]
Sep 09 20:11:54.035812 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.035714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.035812 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.035727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fd489"]
Sep 09 20:11:54.038406 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.038388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:54.038492 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.038463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nht2s\""
Sep 09 20:11:54.038538 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.038466    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:54.124173 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66vs\" (UniqueName: \"kubernetes.io/projected/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-kube-api-access-h66vs\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.124173 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded7752f-71da-4bf8-9d82-ec219a56d7bb-cert\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.124383 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8qc\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124383 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124383 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmc4\" (UniqueName: \"kubernetes.io/projected/ded7752f-71da-4bf8-9d82-ec219a56d7bb-kube-api-access-hnmc4\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.124383 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124508 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124508 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124508 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.124599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-config-volume\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.124686 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.124686 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-tmp-dir\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.124748 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.124748 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.124708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-nginx-conf\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.231109 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66vs\" (UniqueName: \"kubernetes.io/projected/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-kube-api-access-h66vs\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.231222 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded7752f-71da-4bf8-9d82-ec219a56d7bb-cert\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.231222 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8qc\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231353 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmc4\" (UniqueName: \"kubernetes.io/projected/ded7752f-71da-4bf8-9d82-ec219a56d7bb-kube-api-access-hnmc4\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.231425 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231472 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231472 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231568 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.231568 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231668 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231668 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-config-volume\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.231668 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.231668 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-tmp-dir\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.231668 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.231655    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:54.231902 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.231902 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.231714    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:54.731694659 +0000 UTC m=+35.618284566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:54.231902 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-nginx-conf\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.231902 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.231816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.232131 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.232101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.232185 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.232167    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.232237 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.232191    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:54.232237 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.232203    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:11:54.232237 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.232233    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:54.732214344 +0000 UTC m=+35.618804235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.232384 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.232256    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:54.732240816 +0000 UTC m=+35.618830718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:11:54.232512 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.232488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-nginx-conf\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.232576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.232518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-tmp-dir\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.232630 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.232588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.232762 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.232742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-config-volume\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.235861 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.235840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.235952 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.235847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.235952 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.235895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded7752f-71da-4bf8-9d82-ec219a56d7bb-cert\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.241139 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.241104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66vs\" (UniqueName: \"kubernetes.io/projected/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-kube-api-access-h66vs\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.241139 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.241127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8qc\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.241399 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.241342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.241893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.241871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmc4\" (UniqueName: \"kubernetes.io/projected/ded7752f-71da-4bf8-9d82-ec219a56d7bb-kube-api-access-hnmc4\") pod \"ingress-canary-dg8zk\" (UID: \"ded7752f-71da-4bf8-9d82-ec219a56d7bb\") " pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.319853 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.319808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dg8zk"
Sep 09 20:11:54.614956 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.614777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:11:54.617759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.617741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg729\""
Sep 09 20:11:54.617875 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.617766    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:54.617875 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.617828    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:54.676754 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.676723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dg8zk"]
Sep 09 20:11:54.689088 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:11:54.689051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded7752f_71da_4bf8_9d82_ec219a56d7bb.slice/crio-b7610296bce938d0cea1f421739d41606db310372569f01e9a3f6fd7ee107c96 WatchSource:0}: Error finding container b7610296bce938d0cea1f421739d41606db310372569f01e9a3f6fd7ee107c96: Status 404 returned error can't find the container with id b7610296bce938d0cea1f421739d41606db310372569f01e9a3f6fd7ee107c96
Sep 09 20:11:54.736358 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.736270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:54.736358 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.736314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.736373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736436    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736457    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736471    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736472    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736511    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:55.736496978 +0000 UTC m=+36.623086866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:11:54.736526 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736526    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:55.736520566 +0000 UTC m=+36.623110455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:54.736744 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:54.736538    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:55.73653293 +0000 UTC m=+36.623122818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.824122 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.824086    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="acf0344e362d984d303a49c490c9a79d4e5c02787ff020fdb88aa545b3538440" exitCode=0
Sep 09 20:11:54.824297 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.824172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"acf0344e362d984d303a49c490c9a79d4e5c02787ff020fdb88aa545b3538440"}
Sep 09 20:11:54.825335 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:54.825311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dg8zk" event={"ID":"ded7752f-71da-4bf8-9d82-ec219a56d7bb","Type":"ContainerStarted","Data":"b7610296bce938d0cea1f421739d41606db310372569f01e9a3f6fd7ee107c96"}
Sep 09 20:11:55.614953 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.614912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:11:55.615572 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.615114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:11:55.617916 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.617889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:55.618983 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.618824    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:55.618983 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.618862    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb455\""
Sep 09 20:11:55.745907 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.745872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.745938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.745965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746039    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746082    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746099    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:55.746113 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746118    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:11:55.746417 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746125    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:57.746104274 +0000 UTC m=+38.632694186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:55.746417 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746146    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:57.746136401 +0000 UTC m=+38.632726292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:55.746417 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:55.746163    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:57.746152481 +0000 UTC m=+38.632742390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:11:55.832700 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.832665    2577 generic.go:356] "Generic (PLEG): container finished" podID="b05148b1-8474-4106-a452-6fee11123ae9" containerID="be9bd9c9ef40eec2a63c6edb7c590580cf540dc5794c2edec38199989a0b363e" exitCode=0
Sep 09 20:11:55.832897 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:55.832711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerDied","Data":"be9bd9c9ef40eec2a63c6edb7c590580cf540dc5794c2edec38199989a0b363e"}
Sep 09 20:11:56.839829 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:56.839787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlr89" event={"ID":"b05148b1-8474-4106-a452-6fee11123ae9","Type":"ContainerStarted","Data":"2a2aefb3d6e237f0f3d660c357c8a7cf8ecb8c6a0f5ca280fa334544e37a7ad5"}
Sep 09 20:11:56.862560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:56.862508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tlr89" podStartSLOduration=8.947474945 podStartE2EDuration="37.862484061s" totalImagesPullingTime="28.915009116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:56.860702024 +0000 UTC m=+37.747291969" watchObservedRunningTime="2026-09-09 20:11:56.862484061 +0000 UTC m=+37.749073971"
Sep 09 20:11:57.767824 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:57.767786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:11:57.767945 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:57.767859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:11:57.767945 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:57.767898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:11:57.767945 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.767932    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.767986    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.767997    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.768004    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:01.767987531 +0000 UTC m=+42.654577418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.768004    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.768061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:01.768049689 +0000 UTC m=+42.654639578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:11:57.768089 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:11:57.768076    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:01.768069641 +0000 UTC m=+42.654659529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:57.843703 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:57.843662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dg8zk" event={"ID":"ded7752f-71da-4bf8-9d82-ec219a56d7bb","Type":"ContainerStarted","Data":"89b8450e2ff0fbb83e347fa3387f3244c937d3467b9045469e86bd9d08f47b62"}
Sep 09 20:11:57.860365 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:11:57.860307    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dg8zk" podStartSLOduration=2.338579755 podStartE2EDuration="4.86029069s" totalImagesPullingTime="2.521710935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:57.85981971 +0000 UTC m=+38.746409643" watchObservedRunningTime="2026-09-09 20:11:57.86029069 +0000 UTC m=+38.746880600"
Sep 09 20:12:01.794876 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:01.794654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:01.794902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:01.794929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.794812    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.794998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.794982631 +0000 UTC m=+50.681572519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.795060    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.795083    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.795126    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.79511249 +0000 UTC m=+50.681702382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.795063    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:01.795321 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:01.795158    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.79515255 +0000 UTC m=+50.681742437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:09.858391 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:09.858348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:09.858406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:09.858429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858504    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858525    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858529    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858553    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858574    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.858561152 +0000 UTC m=+66.745151039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858586    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.858580601 +0000 UTC m=+66.745170488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:12:09.858899 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:09.858623    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:25.858596094 +0000 UTC m=+66.745186005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:12:19.814758 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:19.814727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-258xn"
Sep 09 20:12:25.383159 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.383107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:12:25.385899 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.385872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:25.393595 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.393564    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:12:25.393719 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.393676    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:13:29.393650806 +0000 UTC m=+130.280240711 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : secret "metrics-daemon-secret" not found
Sep 09 20:12:25.484407 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.484371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:12:25.487318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.487294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:25.498907 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.498875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31c6b37-d608-46c2-9df9-0032ba3235d0-original-pull-secret\") pod \"global-pull-secret-syncer-6ptjs\" (UID: \"b31c6b37-d608-46c2-9df9-0032ba3235d0\") " pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:12:25.584811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.584767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:12:25.587604 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.587582    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:25.597935 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.597909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:25.608462 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.608435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsbf\" (UniqueName: \"kubernetes.io/projected/cd5da601-b4d8-48d8-9899-bd65af4d19d1-kube-api-access-jrsbf\") pod \"network-check-target-c2tnv\" (UID: \"cd5da601-b4d8-48d8-9899-bd65af4d19d1\") " pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:12:25.628515 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.628471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ptjs"
Sep 09 20:12:25.826319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.826285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg729\""
Sep 09 20:12:25.834432 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.834406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:12:25.886319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.886282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:12:25.886511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.886338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:12:25.886511 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886448    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:12:25.886511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.886472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886515    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:57.886498628 +0000 UTC m=+98.773088542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886451    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886554    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886560    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886618    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:57.886602258 +0000 UTC m=+98.773192162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:25.886688 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:25.886647    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:57.886636734 +0000 UTC m=+98.773226626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:12:25.932562 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:25.932520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ptjs"]
Sep 09 20:12:25.936223 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:12:25.936192    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31c6b37_d608_46c2_9df9_0032ba3235d0.slice/crio-8f3131251561245807a6c810f983d91f49d5e57e80e8c79abd785270e7c53ec6 WatchSource:0}: Error finding container 8f3131251561245807a6c810f983d91f49d5e57e80e8c79abd785270e7c53ec6: Status 404 returned error can't find the container with id 8f3131251561245807a6c810f983d91f49d5e57e80e8c79abd785270e7c53ec6
Sep 09 20:12:26.141351 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:26.141319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c2tnv"]
Sep 09 20:12:26.144283 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:12:26.144255    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5da601_b4d8_48d8_9899_bd65af4d19d1.slice/crio-fbe849feebe738a1d4619144b3eccefd28448812ae2b1d4fcf6458134075e69a WatchSource:0}: Error finding container fbe849feebe738a1d4619144b3eccefd28448812ae2b1d4fcf6458134075e69a: Status 404 returned error can't find the container with id fbe849feebe738a1d4619144b3eccefd28448812ae2b1d4fcf6458134075e69a
Sep 09 20:12:26.914298 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:26.914256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c2tnv" event={"ID":"cd5da601-b4d8-48d8-9899-bd65af4d19d1","Type":"ContainerStarted","Data":"fbe849feebe738a1d4619144b3eccefd28448812ae2b1d4fcf6458134075e69a"}
Sep 09 20:12:26.916275 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:26.916226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ptjs" event={"ID":"b31c6b37-d608-46c2-9df9-0032ba3235d0","Type":"ContainerStarted","Data":"8f3131251561245807a6c810f983d91f49d5e57e80e8c79abd785270e7c53ec6"}
Sep 09 20:12:30.929242 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:30.929207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c2tnv" event={"ID":"cd5da601-b4d8-48d8-9899-bd65af4d19d1","Type":"ContainerStarted","Data":"a0bf52806e8d0f7e1514c55398f53b784a402bb598e1afbe288f01d61e1ae554"}
Sep 09 20:12:30.929681 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:30.929340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:12:30.931001 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:30.930969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ptjs" event={"ID":"b31c6b37-d608-46c2-9df9-0032ba3235d0","Type":"ContainerStarted","Data":"4f0e69fe21b3ed9ab815869d1cbf270d5aef6ea40ef4ed944675df10cf788cc7"}
Sep 09 20:12:30.945037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:30.944962    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c2tnv" podStartSLOduration=67.641111906 podStartE2EDuration="1m11.944944369s" totalImagesPullingTime="4.303832463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:30.944620192 +0000 UTC m=+71.831210119" watchObservedRunningTime="2026-09-09 20:12:30.944944369 +0000 UTC m=+71.831534280"
Sep 09 20:12:30.959498 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:30.959430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6ptjs" podStartSLOduration=66.441327388 podStartE2EDuration="1m10.95940705s" totalImagesPullingTime="4.518079662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:30.958813636 +0000 UTC m=+71.845403546" watchObservedRunningTime="2026-09-09 20:12:30.95940705 +0000 UTC m=+71.845996963"
Sep 09 20:12:57.920806 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:57.920767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:12:57.920806 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:57.920810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:12:57.920868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.920937    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.920961    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.920977    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.921046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:01.921014414 +0000 UTC m=+162.807604306 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.920977    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.921068    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:01.921058024 +0000 UTC m=+162.807647912 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:12:57.921327 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:12:57.921082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:01.921076466 +0000 UTC m=+162.807666361 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:13:01.936404 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:13:01.936368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c2tnv"
Sep 09 20:13:29.458245 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:13:29.458198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:13:29.458741 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:29.458327    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:13:29.458741 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:29.458384    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:15:31.458370658 +0000 UTC m=+252.344960552 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : secret "metrics-daemon-secret" not found
Sep 09 20:13:56.984975 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:56.984923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8" podUID="bc3e0c62-325f-47d0-a395-0feacc9c4dc0"
Sep 09 20:13:56.999284 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:56.999236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" podUID="961a2f93-759d-4ba6-95b7-601913f069f7"
Sep 09 20:13:57.044742 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:57.044704    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fd489" podUID="04021b1b-4eaf-4de5-b3d5-6446103d6ba1"
Sep 09 20:13:57.152136 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:13:57.152107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fd489"
Sep 09 20:13:57.152283 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:13:57.152108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:13:57.152283 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:13:57.152108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:13:58.634300 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:13:58.634246    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:14:01.793733 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.793696    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"]
Sep 09 20:14:01.796560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.796537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.800744 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.800712    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:14:01.800893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.800712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:14:01.800893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.800713    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:14:01.800893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.800755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:14:01.810661 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.810633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"]
Sep 09 20:14:01.896532 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.896489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhk76\" (UniqueName: \"kubernetes.io/projected/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-kube-api-access-qhk76\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.896532 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.896529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-klusterlet-config\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.896746 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.896557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-tmp\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.997294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:14:01.997294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhk76\" (UniqueName: \"kubernetes.io/projected/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-kube-api-access-qhk76\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-klusterlet-config\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-tmp\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997404    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997430    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bbbcc864-pd5t7: secret "image-registry-tls" not found
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997459    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls podName:961a2f93-759d-4ba6-95b7-601913f069f7 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:03.997471298 +0000 UTC m=+284.884061187 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls") pod "image-registry-64bbbcc864-pd5t7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7") : secret "image-registry-tls" not found
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls podName:04021b1b-4eaf-4de5-b3d5-6446103d6ba1 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:03.997506811 +0000 UTC m=+284.884096698 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls") pod "dns-default-fd489" (UID: "04021b1b-4eaf-4de5-b3d5-6446103d6ba1") : secret "dns-default-metrics-tls" not found
Sep 09 20:14:01.997581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:14:01.998006 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997636    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:14:01.998006 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:14:01.997678    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert podName:bc3e0c62-325f-47d0-a395-0feacc9c4dc0 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:03.997667101 +0000 UTC m=+284.884256990 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-h8qd8" (UID: "bc3e0c62-325f-47d0-a395-0feacc9c4dc0") : secret "networking-console-plugin-cert" not found
Sep 09 20:14:01.998006 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.997858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-tmp\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:01.999979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:01.999955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-klusterlet-config\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:02.005686 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:02.005663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhk76\" (UniqueName: \"kubernetes.io/projected/52923dd2-8daa-4f60-aaa7-1aabcb49e29f-kube-api-access-qhk76\") pod \"klusterlet-addon-workmgr-845b968d58-blp4n\" (UID: \"52923dd2-8daa-4f60-aaa7-1aabcb49e29f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:02.105882 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:02.105787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:02.373131 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:02.373047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"]
Sep 09 20:14:02.376212 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:14:02.376175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52923dd2_8daa_4f60_aaa7_1aabcb49e29f.slice/crio-2e27a60d329b3de9631f45f9745c9e444575dc6a84d897e2026b3945df55bc16 WatchSource:0}: Error finding container 2e27a60d329b3de9631f45f9745c9e444575dc6a84d897e2026b3945df55bc16: Status 404 returned error can't find the container with id 2e27a60d329b3de9631f45f9745c9e444575dc6a84d897e2026b3945df55bc16
Sep 09 20:14:03.169460 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:03.169416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n" event={"ID":"52923dd2-8daa-4f60-aaa7-1aabcb49e29f","Type":"ContainerStarted","Data":"2e27a60d329b3de9631f45f9745c9e444575dc6a84d897e2026b3945df55bc16"}
Sep 09 20:14:07.181706 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:07.181670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n" event={"ID":"52923dd2-8daa-4f60-aaa7-1aabcb49e29f","Type":"ContainerStarted","Data":"19ad603bdd18b6fd4317520ff09bb3f15a5256bd117f11457ac671850f408aab"}
Sep 09 20:14:07.182210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:07.181820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:07.183441 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:07.183421    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:14:07.197318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:07.197261    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n" podStartSLOduration=2.205860251 podStartE2EDuration="6.197245706s" totalImagesPullingTime="3.991385455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:07.197144637 +0000 UTC m=+168.083734566" watchObservedRunningTime="2026-09-09 20:14:07.197245706 +0000 UTC m=+168.083835616"
Sep 09 20:14:13.615267 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:14:13.615180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:15:31.559090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:31.559041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:15:31.559656 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:31.559222    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:15:31.559656 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:31.559324    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs podName:cb7a3927-19b9-4c23-8c72-b245c699f4ee nodeName:}" failed. No retries permitted until 2026-09-09 20:17:33.559300814 +0000 UTC m=+374.445890710 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs") pod "network-metrics-daemon-9blb8" (UID: "cb7a3927-19b9-4c23-8c72-b245c699f4ee") : secret "metrics-daemon-secret" not found
Sep 09 20:15:39.867352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.867315    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"]
Sep 09 20:15:39.870448 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.870429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:39.873719 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.873485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:15:39.873719 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.873553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.873719 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.873636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:15:39.873974 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.873913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.874955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.874933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sxrbg\""
Sep 09 20:15:39.875809 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.875788    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55f5778b68-h5lt7"]
Sep 09 20:15:39.878737 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.878713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:39.883188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.883152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:15:39.883323 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.883219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:15:39.883392 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.883322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mgtkg\""
Sep 09 20:15:39.883446 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.883389    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:15:39.883883 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.883853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:15:39.884490 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.884474    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.884585 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.884520    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.884585 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.884527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"]
Sep 09 20:15:39.895121 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.895084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f5778b68-h5lt7"]
Sep 09 20:15:39.965767 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.965736    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"]
Sep 09 20:15:39.968706 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.968687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:39.971752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.971727    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:15:39.971953 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.971936    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:15:39.972012 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.971969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:15:39.972012 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.971995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kbwmx\""
Sep 09 20:15:39.978415 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:39.978386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"]
Sep 09 20:15:40.018028 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.017992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.018205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.018205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-default-certificate\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.018205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c8d445d-fe1b-4900-9a72-fe48105d4c91-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.018205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-stats-auth\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.018205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5rr\" (UniqueName: \"kubernetes.io/projected/9789ae06-35b6-4a0c-8222-a466e0b46d3d-kube-api-access-bp5rr\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.018362 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctfs\" (UniqueName: \"kubernetes.io/projected/1c8d445d-fe1b-4900-9a72-fe48105d4c91-kube-api-access-9ctfs\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.018362 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.018299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c8d445d-fe1b-4900-9a72-fe48105d4c91-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.119649 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c8d445d-fe1b-4900-9a72-fe48105d4c91-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.119649 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-stats-auth\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.119649 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.119878 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5rr\" (UniqueName: \"kubernetes.io/projected/9789ae06-35b6-4a0c-8222-a466e0b46d3d-kube-api-access-bp5rr\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.119878 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c8d445d-fe1b-4900-9a72-fe48105d4c91-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.119878 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-default-certificate\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.119878 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.119776    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:40.619755288 +0000 UTC m=+261.506345194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:40.120101 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.119938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctfs\" (UniqueName: \"kubernetes.io/projected/1c8d445d-fe1b-4900-9a72-fe48105d4c91-kube-api-access-9ctfs\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.120169 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.120098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.120217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.120187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.120264 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.120223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9ng\" (UniqueName: \"kubernetes.io/projected/643ee664-f8f9-46f5-9c48-e2546af50327-kube-api-access-xb9ng\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.120264 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.120254    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:40.120358 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.120308    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:40.620290797 +0000 UTC m=+261.506880690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:40.120358 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.120337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c8d445d-fe1b-4900-9a72-fe48105d4c91-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.122074 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.122052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c8d445d-fe1b-4900-9a72-fe48105d4c91-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.122192 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.122175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-default-certificate\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.122192 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.122184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-stats-auth\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.130605 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.130570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctfs\" (UniqueName: \"kubernetes.io/projected/1c8d445d-fe1b-4900-9a72-fe48105d4c91-kube-api-access-9ctfs\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fn6dq\" (UID: \"1c8d445d-fe1b-4900-9a72-fe48105d4c91\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.130787 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.130770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5rr\" (UniqueName: \"kubernetes.io/projected/9789ae06-35b6-4a0c-8222-a466e0b46d3d-kube-api-access-bp5rr\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.179692 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.179655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"
Sep 09 20:15:40.221310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.221266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.221481 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.221344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9ng\" (UniqueName: \"kubernetes.io/projected/643ee664-f8f9-46f5-9c48-e2546af50327-kube-api-access-xb9ng\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.221481 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.221439    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:15:40.221563 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.221536    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls podName:643ee664-f8f9-46f5-9c48-e2546af50327 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:40.721512411 +0000 UTC m=+261.608102314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4lh8c" (UID: "643ee664-f8f9-46f5-9c48-e2546af50327") : secret "samples-operator-tls" not found
Sep 09 20:15:40.231459 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.231433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9ng\" (UniqueName: \"kubernetes.io/projected/643ee664-f8f9-46f5-9c48-e2546af50327-kube-api-access-xb9ng\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.451778 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.451701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq"]
Sep 09 20:15:40.455245 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:15:40.455206    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8d445d_fe1b_4900_9a72_fe48105d4c91.slice/crio-fed3c83957499db51296d18d1e8e453b8191bfed4a39755b2d67c91e0af0b751 WatchSource:0}: Error finding container fed3c83957499db51296d18d1e8e453b8191bfed4a39755b2d67c91e0af0b751: Status 404 returned error can't find the container with id fed3c83957499db51296d18d1e8e453b8191bfed4a39755b2d67c91e0af0b751
Sep 09 20:15:40.626131 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.626085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.626308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.626161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:40.626308 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.626254    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:40.626378 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.626309    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:41.626293695 +0000 UTC m=+262.512883583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:40.626378 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.626325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:41.626318981 +0000 UTC m=+262.512908869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:40.727420 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:40.727349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:40.727571 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.727548    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:15:40.727650 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:40.727639    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls podName:643ee664-f8f9-46f5-9c48-e2546af50327 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:41.727616567 +0000 UTC m=+262.614206471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4lh8c" (UID: "643ee664-f8f9-46f5-9c48-e2546af50327") : secret "samples-operator-tls" not found
Sep 09 20:15:41.409510 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:41.409423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq" event={"ID":"1c8d445d-fe1b-4900-9a72-fe48105d4c91","Type":"ContainerStarted","Data":"fed3c83957499db51296d18d1e8e453b8191bfed4a39755b2d67c91e0af0b751"}
Sep 09 20:15:41.634212 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:41.634178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:41.634394 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:41.634292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:41.634394 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:41.634370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:43.634347045 +0000 UTC m=+264.520936952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:41.634524 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:41.634427    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:41.634524 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:41.634479    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:43.634467656 +0000 UTC m=+264.521057543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:41.735236 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:41.735140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:41.735396 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:41.735303    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:15:41.735396 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:41.735381    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls podName:643ee664-f8f9-46f5-9c48-e2546af50327 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:43.735359537 +0000 UTC m=+264.621949449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4lh8c" (UID: "643ee664-f8f9-46f5-9c48-e2546af50327") : secret "samples-operator-tls" not found
Sep 09 20:15:43.415846 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:43.415763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq" event={"ID":"1c8d445d-fe1b-4900-9a72-fe48105d4c91","Type":"ContainerStarted","Data":"7c6762ed26c0390f03d7c5a8734579bb53b2d1e54cfe4a038978058ee68ee04b"}
Sep 09 20:15:43.431188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:43.431131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq" podStartSLOduration=2.042527986 podStartE2EDuration="4.431110256s" totalImagesPullingTime="2.38858227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:43.43052288 +0000 UTC m=+264.317112792" watchObservedRunningTime="2026-09-09 20:15:43.431110256 +0000 UTC m=+264.317700171"
Sep 09 20:15:43.650516 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:43.650474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:43.650686 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:43.650575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:43.650686 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:43.650662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.650642592 +0000 UTC m=+268.537232480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:43.650806 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:43.650693    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:43.650806 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:43.650735    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.65072387 +0000 UTC m=+268.537313758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:43.751323 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:43.751225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:43.751466 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:43.751357    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:15:43.751466 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:43.751440    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls podName:643ee664-f8f9-46f5-9c48-e2546af50327 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.751419486 +0000 UTC m=+268.638009387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4lh8c" (UID: "643ee664-f8f9-46f5-9c48-e2546af50327") : secret "samples-operator-tls" not found
Sep 09 20:15:45.558150 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.558116    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cllf7"]
Sep 09 20:15:45.561353 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.561334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.564365 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.564341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6kgz\""
Sep 09 20:15:45.564555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.564529    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:15:45.565344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.565323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:15:45.565446 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.565347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:15:45.565446 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.565404    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:15:45.571432 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.571387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cllf7"]
Sep 09 20:15:45.665308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.665272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjv5\" (UniqueName: \"kubernetes.io/projected/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-api-access-wsjv5\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.665308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.665310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3d559a3-895a-4d80-9d36-c9cb061384e6-crio-socket\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.665528 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.665348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d559a3-895a-4d80-9d36-c9cb061384e6-data-volume\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.665528 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.665428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.665528 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.665456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766378 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjv5\" (UniqueName: \"kubernetes.io/projected/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-api-access-wsjv5\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766378 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3d559a3-895a-4d80-9d36-c9cb061384e6-crio-socket\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766612 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d559a3-895a-4d80-9d36-c9cb061384e6-data-volume\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766612 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766612 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3d559a3-895a-4d80-9d36-c9cb061384e6-crio-socket\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766612 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.766612 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:45.766567    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:45.766785 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:45.766620    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:46.266605181 +0000 UTC m=+267.153195069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:45.766785 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:45.766630    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:45.766785 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:45.766692    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:46.266675082 +0000 UTC m=+267.153264972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:45.766895 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.766804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d559a3-895a-4d80-9d36-c9cb061384e6-data-volume\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:45.780522 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:45.780489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjv5\" (UniqueName: \"kubernetes.io/projected/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-api-access-wsjv5\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:46.271274 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:46.271237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:46.271466 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:46.271285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:46.271466 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:46.271372    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:46.271466 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:46.271426    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:46.271466 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:46.271431    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.271416117 +0000 UTC m=+268.158006004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:46.271639 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:46.271533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:47.271520183 +0000 UTC m=+268.158110075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:47.279808 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.279763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:47.279808 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.279807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:47.280253 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.279893    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:47.280253 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.279918    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:47.280253 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.279962    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:49.279943384 +0000 UTC m=+270.166533286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:47.280253 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.279975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:49.279969443 +0000 UTC m=+270.166559332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:47.385308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.385271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-l8hwn"]
Sep 09 20:15:47.388519 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.388502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.391156 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.391132    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tb6gh\""
Sep 09 20:15:47.391271 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.391162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 20:15:47.392270 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.392246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 20:15:47.392504 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.392489    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 20:15:47.392581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.392498    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 20:15:47.395606 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.395583    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-l8hwn"]
Sep 09 20:15:47.481183 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.481139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-key\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.481366 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.481198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-cabundle\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.481366 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.481260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqst\" (UniqueName: \"kubernetes.io/projected/40542d6b-6f81-4269-a419-63eed1d9f35d-kube-api-access-blqst\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.582447 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.582353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-key\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.582447 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.582409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-cabundle\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.582447 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.582448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqst\" (UniqueName: \"kubernetes.io/projected/40542d6b-6f81-4269-a419-63eed1d9f35d-kube-api-access-blqst\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.583144 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.583106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-cabundle\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.584811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.584788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40542d6b-6f81-4269-a419-63eed1d9f35d-signing-key\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.590819 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.590793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqst\" (UniqueName: \"kubernetes.io/projected/40542d6b-6f81-4269-a419-63eed1d9f35d-kube-api-access-blqst\") pod \"service-ca-75d99c484-l8hwn\" (UID: \"40542d6b-6f81-4269-a419-63eed1d9f35d\") " pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.683652 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.683609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:47.683857 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.683705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:47.683857 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.683753    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:47.683857 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.683834    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:55.68381572 +0000 UTC m=+276.570405612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:47.684007 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.683862    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:15:55.683844202 +0000 UTC m=+276.570434090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:47.698208 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.698183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-l8hwn"
Sep 09 20:15:47.785198 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.785156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:47.785373 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.785287    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:15:47.785373 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:47.785355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls podName:643ee664-f8f9-46f5-9c48-e2546af50327 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:55.785339569 +0000 UTC m=+276.671929457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4lh8c" (UID: "643ee664-f8f9-46f5-9c48-e2546af50327") : secret "samples-operator-tls" not found
Sep 09 20:15:47.974690 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:47.974635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-l8hwn"]
Sep 09 20:15:47.977206 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:15:47.977181    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40542d6b_6f81_4269_a419_63eed1d9f35d.slice/crio-ea0dca2a3f09e7bf8493da0f3d57acbbfe2c7d261769363000e05deccae007a5 WatchSource:0}: Error finding container ea0dca2a3f09e7bf8493da0f3d57acbbfe2c7d261769363000e05deccae007a5: Status 404 returned error can't find the container with id ea0dca2a3f09e7bf8493da0f3d57acbbfe2c7d261769363000e05deccae007a5
Sep 09 20:15:48.337160 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:48.337132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4pwf_4d3cf640-5639-40e0-b251-550ef10be316/dns-node-resolver/0.log"
Sep 09 20:15:48.431637 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:48.431599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-l8hwn" event={"ID":"40542d6b-6f81-4269-a419-63eed1d9f35d","Type":"ContainerStarted","Data":"ea0dca2a3f09e7bf8493da0f3d57acbbfe2c7d261769363000e05deccae007a5"}
Sep 09 20:15:49.297970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:49.297924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:49.297970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:49.297968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:49.298230 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:49.298078    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:49.298230 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:49.298139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:53.298124643 +0000 UTC m=+274.184714531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:49.298230 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:49.298188    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:49.298336 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:49.298246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:53.298232067 +0000 UTC m=+274.184821967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:49.336426 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:49.336394    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jmz8t_282b4062-5693-4318-8e23-104b0ec0bb27/node-ca/0.log"
Sep 09 20:15:49.936430 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:49.936394    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dg8zk_ded7752f-71da-4bf8-9d82-ec219a56d7bb/serve-healthcheck-canary/0.log"
Sep 09 20:15:50.439277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:50.439236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-l8hwn" event={"ID":"40542d6b-6f81-4269-a419-63eed1d9f35d","Type":"ContainerStarted","Data":"c3fe6da2441e7e66765666fe16cf3e62b02ea387abbe7d52441cff7208e562c3"}
Sep 09 20:15:50.455013 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:50.454931    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-l8hwn" podStartSLOduration=1.585481686 podStartE2EDuration="3.45491451s" totalImagesPullingTime="1.869432824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:50.454151573 +0000 UTC m=+271.340741506" watchObservedRunningTime="2026-09-09 20:15:50.45491451 +0000 UTC m=+271.341504463"
Sep 09 20:15:50.938412 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:50.938356    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fn6dq_1c8d445d-fe1b-4900-9a72-fe48105d4c91/kube-storage-version-migrator-operator/0.log"
Sep 09 20:15:53.335665 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:53.335601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:53.335665 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:53.335658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:15:53.336274 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:53.335737    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:15:53.336274 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:53.335788    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:53.336274 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:53.335800    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:01.335784762 +0000 UTC m=+282.222374650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : configmap "kube-rbac-proxy" not found
Sep 09 20:15:53.336274 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:53.335873    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:01.335854518 +0000 UTC m=+282.222444412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:15:55.758822 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:55.758783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:55.759241 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:55.758844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:15:55.759241 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:55.758948    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:15:55.759241 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:55.759041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:11.759003534 +0000 UTC m=+292.645593422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : configmap references non-existent config key: service-ca.crt
Sep 09 20:15:55.759241 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:15:55.759068    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs podName:9789ae06-35b6-4a0c-8222-a466e0b46d3d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:11.759056516 +0000 UTC m=+292.645646421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs") pod "router-default-55f5778b68-h5lt7" (UID: "9789ae06-35b6-4a0c-8222-a466e0b46d3d") : secret "router-metrics-certs-default" not found
Sep 09 20:15:55.859833 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:55.859793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:55.862212 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:55.862190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/643ee664-f8f9-46f5-9c48-e2546af50327-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4lh8c\" (UID: \"643ee664-f8f9-46f5-9c48-e2546af50327\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:55.877231 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:55.877196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"
Sep 09 20:15:56.148154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:56.148118    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c"]
Sep 09 20:15:56.462534 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:56.462447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c" event={"ID":"643ee664-f8f9-46f5-9c48-e2546af50327","Type":"ContainerStarted","Data":"bd21d7b284bee57f7120c3a7ba4428b8ede8a397652d95cc9445d78a10d47ee5"}
Sep 09 20:15:58.471121 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:58.471013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c" event={"ID":"643ee664-f8f9-46f5-9c48-e2546af50327","Type":"ContainerStarted","Data":"a71ab164e60f842311c9e46887860a83ec48b1265dc773fca54fc8cd2dd1fcad"}
Sep 09 20:15:58.471121 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:58.471087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c" event={"ID":"643ee664-f8f9-46f5-9c48-e2546af50327","Type":"ContainerStarted","Data":"c5e9ea3e3d2de8ad6770891689ed02e47b7ed2a971d42365820d82b8b7cd0cf5"}
Sep 09 20:15:58.486341 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:15:58.486286    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4lh8c" podStartSLOduration=17.535393813 podStartE2EDuration="19.486269193s" totalImagesPullingTime="1.95087538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:58.485985722 +0000 UTC m=+279.372575660" watchObservedRunningTime="2026-09-09 20:15:58.486269193 +0000 UTC m=+279.372859137"
Sep 09 20:16:00.152858 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:00.152810    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" podUID="961a2f93-759d-4ba6-95b7-601913f069f7"
Sep 09 20:16:00.152858 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:00.152810    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8" podUID="bc3e0c62-325f-47d0-a395-0feacc9c4dc0"
Sep 09 20:16:00.152858 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:00.152810    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fd489" podUID="04021b1b-4eaf-4de5-b3d5-6446103d6ba1"
Sep 09 20:16:00.476857 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:00.476774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:00.477004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:00.476774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:00.477004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:00.476774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:16:01.405568 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:01.405517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:01.405568 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:01.405570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:01.405988 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:01.405676    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:16:01.405988 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:01.405755    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls podName:e3d559a3-895a-4d80-9d36-c9cb061384e6 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:17.405739185 +0000 UTC m=+298.292329092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cllf7" (UID: "e3d559a3-895a-4d80-9d36-c9cb061384e6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:16:01.406182 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:01.406162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3d559a3-895a-4d80-9d36-c9cb061384e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:04.026426 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.026385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:04.026839 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.026447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:16:04.026839 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.026474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:04.028896 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.028867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3e0c62-325f-47d0-a395-0feacc9c4dc0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-h8qd8\" (UID: \"bc3e0c62-325f-47d0-a395-0feacc9c4dc0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:16:04.029335 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.029310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04021b1b-4eaf-4de5-b3d5-6446103d6ba1-metrics-tls\") pod \"dns-default-fd489\" (UID: \"04021b1b-4eaf-4de5-b3d5-6446103d6ba1\") " pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:04.029421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.029372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"image-registry-64bbbcc864-pd5t7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") " pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:04.081998 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.081965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nht2s\""
Sep 09 20:16:04.082181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.082046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6pvwv\""
Sep 09 20:16:04.082181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.082050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ctkpk\""
Sep 09 20:16:04.088476 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.088450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8"
Sep 09 20:16:04.088564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.088480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:04.088596 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.088561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:04.367851 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.367774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fd489"]
Sep 09 20:16:04.370674 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:04.370635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04021b1b_4eaf_4de5_b3d5_6446103d6ba1.slice/crio-59980b8e0369260c1e9d1d87159625b99d66ae27b2f5b404c3127d2698a0abef WatchSource:0}: Error finding container 59980b8e0369260c1e9d1d87159625b99d66ae27b2f5b404c3127d2698a0abef: Status 404 returned error can't find the container with id 59980b8e0369260c1e9d1d87159625b99d66ae27b2f5b404c3127d2698a0abef
Sep 09 20:16:04.374599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.374572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-h8qd8"]
Sep 09 20:16:04.377602 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:04.377572    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3e0c62_325f_47d0_a395_0feacc9c4dc0.slice/crio-432b801c83da02c270be1936bf2c75ae384515093f40a2794c5edb6dcd7e9823 WatchSource:0}: Error finding container 432b801c83da02c270be1936bf2c75ae384515093f40a2794c5edb6dcd7e9823: Status 404 returned error can't find the container with id 432b801c83da02c270be1936bf2c75ae384515093f40a2794c5edb6dcd7e9823
Sep 09 20:16:04.398170 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.398104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:16:04.400509 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:04.400482    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961a2f93_759d_4ba6_95b7_601913f069f7.slice/crio-a112c3142134963a57a44c044d6b931f69e264d3a57c34de08797a2ad86221fa WatchSource:0}: Error finding container a112c3142134963a57a44c044d6b931f69e264d3a57c34de08797a2ad86221fa: Status 404 returned error can't find the container with id a112c3142134963a57a44c044d6b931f69e264d3a57c34de08797a2ad86221fa
Sep 09 20:16:04.490667 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.490611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" event={"ID":"961a2f93-759d-4ba6-95b7-601913f069f7","Type":"ContainerStarted","Data":"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"}
Sep 09 20:16:04.490873 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.490688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:04.490873 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.490713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" event={"ID":"961a2f93-759d-4ba6-95b7-601913f069f7","Type":"ContainerStarted","Data":"a112c3142134963a57a44c044d6b931f69e264d3a57c34de08797a2ad86221fa"}
Sep 09 20:16:04.491991 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.491957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fd489" event={"ID":"04021b1b-4eaf-4de5-b3d5-6446103d6ba1","Type":"ContainerStarted","Data":"59980b8e0369260c1e9d1d87159625b99d66ae27b2f5b404c3127d2698a0abef"}
Sep 09 20:16:04.493186 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.493166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8" event={"ID":"bc3e0c62-325f-47d0-a395-0feacc9c4dc0","Type":"ContainerStarted","Data":"432b801c83da02c270be1936bf2c75ae384515093f40a2794c5edb6dcd7e9823"}
Sep 09 20:16:04.510463 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:04.510402    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" podStartSLOduration=284.510387183 podStartE2EDuration="4m44.510387183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:04.509322291 +0000 UTC m=+285.395912201" watchObservedRunningTime="2026-09-09 20:16:04.510387183 +0000 UTC m=+285.396977092"
Sep 09 20:16:05.499873 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:05.499831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8" event={"ID":"bc3e0c62-325f-47d0-a395-0feacc9c4dc0","Type":"ContainerStarted","Data":"95af8923a40ccf0340ac92b3f0b9c9216d25bd02bda9839edb5e8f51104589a5"}
Sep 09 20:16:05.516117 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:05.516058    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-h8qd8" podStartSLOduration=263.507063642 podStartE2EDuration="4m24.516035898s" totalImagesPullingTime="1.008972256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:05.514927972 +0000 UTC m=+286.401517903" watchObservedRunningTime="2026-09-09 20:16:05.516035898 +0000 UTC m=+286.402625807"
Sep 09 20:16:06.497947 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.497868    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-vk7b8"]
Sep 09 20:16:06.500901 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.500880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:06.504638 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.504014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kb2ql\""
Sep 09 20:16:06.504638 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.504519    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:16:06.507706 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.505865    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:16:06.510831 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.510800    2577 generic.go:356] "Generic (PLEG): container finished" podID="52923dd2-8daa-4f60-aaa7-1aabcb49e29f" containerID="19ad603bdd18b6fd4317520ff09bb3f15a5256bd117f11457ac671850f408aab" exitCode=1
Sep 09 20:16:06.511003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.510873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n" event={"ID":"52923dd2-8daa-4f60-aaa7-1aabcb49e29f","Type":"ContainerDied","Data":"19ad603bdd18b6fd4317520ff09bb3f15a5256bd117f11457ac671850f408aab"}
Sep 09 20:16:06.511334 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.511313    2577 scope.go:117] "RemoveContainer" containerID="19ad603bdd18b6fd4317520ff09bb3f15a5256bd117f11457ac671850f408aab"
Sep 09 20:16:06.513090 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.513061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fd489" event={"ID":"04021b1b-4eaf-4de5-b3d5-6446103d6ba1","Type":"ContainerStarted","Data":"58ef35593a9020abae27da76f2adba619f3cb63ab1e7ea0729655301f42cb898"}
Sep 09 20:16:06.513220 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.513119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fd489" event={"ID":"04021b1b-4eaf-4de5-b3d5-6446103d6ba1","Type":"ContainerStarted","Data":"fd84d8fa55cce07842ee6ba4611da2151575befc093c4c46cb2a01fb83ce7886"}
Sep 09 20:16:06.514590 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.514571    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vk7b8"]
Sep 09 20:16:06.548138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.548109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp6k\" (UniqueName: \"kubernetes.io/projected/768305c7-e43f-481e-bc7f-d68400d5ddcd-kube-api-access-zwp6k\") pod \"downloads-85d98c998f-vk7b8\" (UID: \"768305c7-e43f-481e-bc7f-d68400d5ddcd\") " pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:06.648852 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.648800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp6k\" (UniqueName: \"kubernetes.io/projected/768305c7-e43f-481e-bc7f-d68400d5ddcd-kube-api-access-zwp6k\") pod \"downloads-85d98c998f-vk7b8\" (UID: \"768305c7-e43f-481e-bc7f-d68400d5ddcd\") " pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:06.659998 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.659960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp6k\" (UniqueName: \"kubernetes.io/projected/768305c7-e43f-481e-bc7f-d68400d5ddcd-kube-api-access-zwp6k\") pod \"downloads-85d98c998f-vk7b8\" (UID: \"768305c7-e43f-481e-bc7f-d68400d5ddcd\") " pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:06.848583 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:06.848545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:07.119466 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.119340    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fd489" podStartSLOduration=252.246944729 podStartE2EDuration="4m14.119318577s" totalImagesPullingTime="1.872373848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:06.620758983 +0000 UTC m=+287.507348916" watchObservedRunningTime="2026-09-09 20:16:07.119318577 +0000 UTC m=+288.005908488"
Sep 09 20:16:07.119886 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.119870    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vk7b8"]
Sep 09 20:16:07.122158 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:07.122130    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768305c7_e43f_481e_bc7f_d68400d5ddcd.slice/crio-4d32a15b275eec1ae2f4dcb4111b00ec82526b8cb6ed3e05f8cf3f04e396afb9 WatchSource:0}: Error finding container 4d32a15b275eec1ae2f4dcb4111b00ec82526b8cb6ed3e05f8cf3f04e396afb9: Status 404 returned error can't find the container with id 4d32a15b275eec1ae2f4dcb4111b00ec82526b8cb6ed3e05f8cf3f04e396afb9
Sep 09 20:16:07.182110 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.182007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:16:07.517385 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.517294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vk7b8" event={"ID":"768305c7-e43f-481e-bc7f-d68400d5ddcd","Type":"ContainerStarted","Data":"4d32a15b275eec1ae2f4dcb4111b00ec82526b8cb6ed3e05f8cf3f04e396afb9"}
Sep 09 20:16:07.519122 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.519088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n" event={"ID":"52923dd2-8daa-4f60-aaa7-1aabcb49e29f","Type":"ContainerStarted","Data":"a2b5cc7d5173f7f72330dc551e7d9f6310ca1ce0f07812a53837af71d3dcc9d2"}
Sep 09 20:16:07.519410 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.519389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:16:07.519494 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.519442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:07.519966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:07.519949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-845b968d58-blp4n"
Sep 09 20:16:11.790122 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:11.790078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:11.790740 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:11.790190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:11.790879 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:11.790852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9789ae06-35b6-4a0c-8222-a466e0b46d3d-service-ca-bundle\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:11.792912 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:11.792884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789ae06-35b6-4a0c-8222-a466e0b46d3d-metrics-certs\") pod \"router-default-55f5778b68-h5lt7\" (UID: \"9789ae06-35b6-4a0c-8222-a466e0b46d3d\") " pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:11.987730 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:11.987688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:12.272691 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.272659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f5778b68-h5lt7"]
Sep 09 20:16:12.276830 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:12.276795    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9789ae06_35b6_4a0c_8222_a466e0b46d3d.slice/crio-d9574b8c52c5e9e59fc3552f0a6ac9cb83deef803a404bb1a3e2bc1ef2311c92 WatchSource:0}: Error finding container d9574b8c52c5e9e59fc3552f0a6ac9cb83deef803a404bb1a3e2bc1ef2311c92: Status 404 returned error can't find the container with id d9574b8c52c5e9e59fc3552f0a6ac9cb83deef803a404bb1a3e2bc1ef2311c92
Sep 09 20:16:12.546277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.546187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f5778b68-h5lt7" event={"ID":"9789ae06-35b6-4a0c-8222-a466e0b46d3d","Type":"ContainerStarted","Data":"ff86ca8637b1b7b2cdb4f5bc5d3e55b1e3e0be5613197f33939a7dd82f0c27da"}
Sep 09 20:16:12.546277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.546246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f5778b68-h5lt7" event={"ID":"9789ae06-35b6-4a0c-8222-a466e0b46d3d","Type":"ContainerStarted","Data":"d9574b8c52c5e9e59fc3552f0a6ac9cb83deef803a404bb1a3e2bc1ef2311c92"}
Sep 09 20:16:12.564116 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.564060    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55f5778b68-h5lt7" podStartSLOduration=33.564014214 podStartE2EDuration="33.564014214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:12.563896456 +0000 UTC m=+293.450486389" watchObservedRunningTime="2026-09-09 20:16:12.564014214 +0000 UTC m=+293.450604129"
Sep 09 20:16:12.988003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.987913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:12.990963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:12.990935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:13.550294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:13.550260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:13.551678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:13.551651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55f5778b68-h5lt7"
Sep 09 20:16:16.616262 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:16.616224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9blb8" podUID="cb7a3927-19b9-4c23-8c72-b245c699f4ee"
Sep 09 20:16:17.440630 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:17.440585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:17.443324 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:17.443290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3d559a3-895a-4d80-9d36-c9cb061384e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cllf7\" (UID: \"e3d559a3-895a-4d80-9d36-c9cb061384e6\") " pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:17.526549 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:17.526512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fd489"
Sep 09 20:16:17.669962 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:17.669918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cllf7"
Sep 09 20:16:23.548170 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.548133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gsr9d"]
Sep 09 20:16:23.550870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.550842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.554945 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.554727    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:16:23.555708 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.555298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:16:23.555708 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.555537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5js7b\""
Sep 09 20:16:23.556994 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.556158    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:16:23.556994 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.556407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:16:23.556994 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.556590    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:16:23.556994 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.556775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:16:23.694886 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.694846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.694886 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.694902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-sys\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.694945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.694990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-metrics-client-ca\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.695034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.695113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-root\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.695166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srwn\" (UniqueName: \"kubernetes.io/projected/b251bc25-f3ac-423a-a6f2-39c05318135e-kube-api-access-2srwn\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.695195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-wtmp\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.695493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.695270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-textfile\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.796840 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-metrics-client-ca\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-root\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srwn\" (UniqueName: \"kubernetes.io/projected/b251bc25-f3ac-423a-a6f2-39c05318135e-kube-api-access-2srwn\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.796988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-wtmp\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-textfile\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797379 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797379 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-sys\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797379 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-sys\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797379 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-root\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797667 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797724 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-wtmp\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.797780 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:23.797750    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:16:23.797833 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:23.797806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls podName:b251bc25-f3ac-423a-a6f2-39c05318135e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.297786082 +0000 UTC m=+305.184375986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls") pod "node-exporter-gsr9d" (UID: "b251bc25-f3ac-423a-a6f2-39c05318135e") : secret "node-exporter-tls" not found
Sep 09 20:16:23.797914 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.797894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-textfile\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.798560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.798470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b251bc25-f3ac-423a-a6f2-39c05318135e-metrics-client-ca\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.806690 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.806613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.808075 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.807965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srwn\" (UniqueName: \"kubernetes.io/projected/b251bc25-f3ac-423a-a6f2-39c05318135e-kube-api-access-2srwn\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:23.889114 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:23.889065    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:24.093566 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.093466    2577 patch_prober.go:28] interesting pod/image-registry-64bbbcc864-pd5t7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:16:24.093566 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.093531    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:16:24.145103 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.145064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cllf7"]
Sep 09 20:16:24.148982 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:24.148951    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d559a3_895a_4d80_9d36_c9cb061384e6.slice/crio-0e9890a250be5895f58880cb56f062f6f8da7cf0ed28551b7162c2fbf95fe981 WatchSource:0}: Error finding container 0e9890a250be5895f58880cb56f062f6f8da7cf0ed28551b7162c2fbf95fe981: Status 404 returned error can't find the container with id 0e9890a250be5895f58880cb56f062f6f8da7cf0ed28551b7162c2fbf95fe981
Sep 09 20:16:24.217817 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.217767    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:24.301054 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.300999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:24.303841 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.303802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b251bc25-f3ac-423a-a6f2-39c05318135e-node-exporter-tls\") pod \"node-exporter-gsr9d\" (UID: \"b251bc25-f3ac-423a-a6f2-39c05318135e\") " pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:24.464498 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.464400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gsr9d"
Sep 09 20:16:24.479413 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:24.479370    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb251bc25_f3ac_423a_a6f2_39c05318135e.slice/crio-88f9faebc4032af8fb72fca923930f5715b67cf00d0324bddd9105a8326ed851 WatchSource:0}: Error finding container 88f9faebc4032af8fb72fca923930f5715b67cf00d0324bddd9105a8326ed851: Status 404 returned error can't find the container with id 88f9faebc4032af8fb72fca923930f5715b67cf00d0324bddd9105a8326ed851
Sep 09 20:16:24.595745 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.595699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vk7b8" event={"ID":"768305c7-e43f-481e-bc7f-d68400d5ddcd","Type":"ContainerStarted","Data":"8e70651caebc2ad1d6a7016fa45f281dfc8193a71601b14f70134b2b0fb23fc9"}
Sep 09 20:16:24.596365 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.596238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:24.599086 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.599050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cllf7" event={"ID":"e3d559a3-895a-4d80-9d36-c9cb061384e6","Type":"ContainerStarted","Data":"5e54db49126369160769189ab3774a5265313fb250a10d084d5cff325024ed70"}
Sep 09 20:16:24.599266 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.599126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cllf7" event={"ID":"e3d559a3-895a-4d80-9d36-c9cb061384e6","Type":"ContainerStarted","Data":"0e9890a250be5895f58880cb56f062f6f8da7cf0ed28551b7162c2fbf95fe981"}
Sep 09 20:16:24.600611 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.600584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsr9d" event={"ID":"b251bc25-f3ac-423a-a6f2-39c05318135e","Type":"ContainerStarted","Data":"88f9faebc4032af8fb72fca923930f5715b67cf00d0324bddd9105a8326ed851"}
Sep 09 20:16:24.610300 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.610260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-vk7b8"
Sep 09 20:16:24.614436 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.614379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-vk7b8" podStartSLOduration=1.767551032 podStartE2EDuration="18.614360002s" totalImagesPullingTime="16.84680897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:24.612627029 +0000 UTC m=+305.499216986" watchObservedRunningTime="2026-09-09 20:16:24.614360002 +0000 UTC m=+305.500949903"
Sep 09 20:16:24.622130 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.622094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:16:24.627640 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.627604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.630512 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.630485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:16:24.630860 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.630825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:16:24.631223 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:16:24.631308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:16:24.631308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:16:24.631593 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:16:24.631676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dcrts\""
Sep 09 20:16:24.631676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:16:24.631676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:16:24.631814 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.631573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:16:24.644927 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.644894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:16:24.705485 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.705722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.706215 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rsz\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-kube-api-access-c9rsz\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.706215 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-out\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.706215 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-web-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.706215 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.706215 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.705882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807143 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807312 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807312 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807312 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807312 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807312 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rsz\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-kube-api-access-c9rsz\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-out\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.807584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.807538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-web-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.808584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.808550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.808714 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.808668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.809508 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:24.808981    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 20:16:24.809508 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:24.809079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls podName:31873c93-4339-46e7-87b2-05fa7d7bc3e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:25.309055803 +0000 UTC m=+306.195645720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "31873c93-4339-46e7-87b2-05fa7d7bc3e8") : secret "alertmanager-main-tls" not found
Sep 09 20:16:24.809508 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.809191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.810613 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.810551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-web-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.810804 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.810779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.811560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.811513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.812879 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.812636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.813979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.813170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-out\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.813979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.813424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.814538 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.814483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.815244 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.815224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:24.817066 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:24.817044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rsz\" (UniqueName: \"kubernetes.io/projected/31873c93-4339-46e7-87b2-05fa7d7bc3e8-kube-api-access-c9rsz\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:25.312175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:25.312132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:25.326553 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:25.326520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31873c93-4339-46e7-87b2-05fa7d7bc3e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31873c93-4339-46e7-87b2-05fa7d7bc3e8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:25.506976 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:25.506930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:25.541328 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:25.541094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:16:26.137181 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.137096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:16:26.140809 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:26.140772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31873c93_4339_46e7_87b2_05fa7d7bc3e8.slice/crio-39fb4c772f3a007c7cf2633ccadd2b7de34cfa9c0b66d66aff858368a4b2d2a8 WatchSource:0}: Error finding container 39fb4c772f3a007c7cf2633ccadd2b7de34cfa9c0b66d66aff858368a4b2d2a8: Status 404 returned error can't find the container with id 39fb4c772f3a007c7cf2633ccadd2b7de34cfa9c0b66d66aff858368a4b2d2a8
Sep 09 20:16:26.548792 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.548751    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"]
Sep 09 20:16:26.583970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.583931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"]
Sep 09 20:16:26.584172 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.584124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.587423    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-szpsk\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.587693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.587920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.588190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.588386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.588657    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-abfb1qjvj4nv9\""
Sep 09 20:16:26.589931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.588882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:16:26.614753 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.614710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cllf7" event={"ID":"e3d559a3-895a-4d80-9d36-c9cb061384e6","Type":"ContainerStarted","Data":"bd803ef3d1a5c5d4d3034ac0bd7751e312a8a4c64853ed7b138d3e97e291e0ce"}
Sep 09 20:16:26.617227 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.617186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"39fb4c772f3a007c7cf2633ccadd2b7de34cfa9c0b66d66aff858368a4b2d2a8"}
Sep 09 20:16:26.620219 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.620188    2577 generic.go:356] "Generic (PLEG): container finished" podID="b251bc25-f3ac-423a-a6f2-39c05318135e" containerID="f37047eb05a2b65d1bca5ed9abed89bf7ee19fb24a855831e039a76a91fbf8a6" exitCode=0
Sep 09 20:16:26.620606 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.620575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsr9d" event={"ID":"b251bc25-f3ac-423a-a6f2-39c05318135e","Type":"ContainerDied","Data":"f37047eb05a2b65d1bca5ed9abed89bf7ee19fb24a855831e039a76a91fbf8a6"}
Sep 09 20:16:26.728952 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.728901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729162 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729162 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729162 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f593b73b-69b3-4d66-b857-29ac46a03f5d-metrics-client-ca\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-grpc-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.729473 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.729383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t8t\" (UniqueName: \"kubernetes.io/projected/f593b73b-69b3-4d66-b857-29ac46a03f5d-kube-api-access-p7t8t\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t8t\" (UniqueName: \"kubernetes.io/projected/f593b73b-69b3-4d66-b857-29ac46a03f5d-kube-api-access-p7t8t\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f593b73b-69b3-4d66-b857-29ac46a03f5d-metrics-client-ca\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.834196 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.833992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-grpc-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.837231 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.835813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f593b73b-69b3-4d66-b857-29ac46a03f5d-metrics-client-ca\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.837839 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.837711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.838847 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.838794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.838847 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.838809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-grpc-tls\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.838847 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.838710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.840238 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.840191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.843169 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.843124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f593b73b-69b3-4d66-b857-29ac46a03f5d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.844622 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.844568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t8t\" (UniqueName: \"kubernetes.io/projected/f593b73b-69b3-4d66-b857-29ac46a03f5d-kube-api-access-p7t8t\") pod \"thanos-querier-56d8fffdd5-v6jgr\" (UID: \"f593b73b-69b3-4d66-b857-29ac46a03f5d\") " pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:26.901240 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:26.901065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:27.274957 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:27.274842    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"]
Sep 09 20:16:27.279868 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:27.279821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf593b73b_69b3_4d66_b857_29ac46a03f5d.slice/crio-acb0a0df8b2e2010890dd434ff924d8350943838f5cfb7e570e859c64edacc7b WatchSource:0}: Error finding container acb0a0df8b2e2010890dd434ff924d8350943838f5cfb7e570e859c64edacc7b: Status 404 returned error can't find the container with id acb0a0df8b2e2010890dd434ff924d8350943838f5cfb7e570e859c64edacc7b
Sep 09 20:16:27.629208 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:27.629115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsr9d" event={"ID":"b251bc25-f3ac-423a-a6f2-39c05318135e","Type":"ContainerStarted","Data":"ed4c0cc67286de0a6d09f9c6654d5dea1abab9b16ce1c946b252e2bf0b297371"}
Sep 09 20:16:27.629651 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:27.629189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsr9d" event={"ID":"b251bc25-f3ac-423a-a6f2-39c05318135e","Type":"ContainerStarted","Data":"aa0027c358e2fe824622cd5442f0ba35c18a2bcadd7e762f68994b0018e439dd"}
Sep 09 20:16:27.638685 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:27.638597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"acb0a0df8b2e2010890dd434ff924d8350943838f5cfb7e570e859c64edacc7b"}
Sep 09 20:16:28.325541 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.325466    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gsr9d" podStartSLOduration=3.880007545 podStartE2EDuration="5.325440471s" totalImagesPullingTime="1.362692125s" totalInitContainerRuntime="82.740801ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:27.652824737 +0000 UTC m=+308.539414674" watchObservedRunningTime="2026-09-09 20:16:28.325440471 +0000 UTC m=+309.212030381"
Sep 09 20:16:28.326754 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.326726    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"]
Sep 09 20:16:28.345728 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.345689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"]
Sep 09 20:16:28.345909 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.345827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.351315 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.350869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-z2h56\""
Sep 09 20:16:28.351315 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.351151    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 20:16:28.454878 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.454709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.454878 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.454771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bvlk\" (UniqueName: \"kubernetes.io/projected/89604b42-04d8-4094-af6a-a5e1da80a2bb-kube-api-access-2bvlk\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.556564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.555758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.556564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.555813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bvlk\" (UniqueName: \"kubernetes.io/projected/89604b42-04d8-4094-af6a-a5e1da80a2bb-kube-api-access-2bvlk\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.556564 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:28.556277    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 20:16:28.556564 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:28.556344    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert podName:89604b42-04d8-4094-af6a-a5e1da80a2bb nodeName:}" failed. No retries permitted until 2026-09-09 20:16:29.056321863 +0000 UTC m=+309.942911757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-7c828" (UID: "89604b42-04d8-4094-af6a-a5e1da80a2bb") : secret "monitoring-plugin-cert" not found
Sep 09 20:16:28.567337 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.567271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bvlk\" (UniqueName: \"kubernetes.io/projected/89604b42-04d8-4094-af6a-a5e1da80a2bb-kube-api-access-2bvlk\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:28.791094 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.791053    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54d5986f85-lpjcp"]
Sep 09 20:16:28.809590 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.809312    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54d5986f85-lpjcp"]
Sep 09 20:16:28.809590 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.809435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.817364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.812366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q8kqv\""
Sep 09 20:16:28.817364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.812366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:16:28.817364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.812525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:16:28.817364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.812370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:16:28.819010 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.818111    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:16:28.819010 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.818345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:16:28.822779 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.822743    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:16:28.960617 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-serving-certs-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.960794 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.960794 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.960794 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.960794 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-federate-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.961070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.961070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-metrics-client-ca\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:28.961070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:28.960936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qc4\" (UniqueName: \"kubernetes.io/projected/ab4c2a5a-414d-4ec4-8e05-f39734703b06-kube-api-access-c7qc4\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062497 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-serving-certs-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062497 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-federate-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.062759 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-metrics-client-ca\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.063071 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.062822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qc4\" (UniqueName: \"kubernetes.io/projected/ab4c2a5a-414d-4ec4-8e05-f39734703b06-kube-api-access-c7qc4\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.063410 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.063273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-serving-certs-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.063820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.063768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-metrics-client-ca\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.064095 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.064038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.065850 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.065821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-federate-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.065966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.065926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.066046 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.065981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-secret-telemeter-client\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.066142 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.066120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab4c2a5a-414d-4ec4-8e05-f39734703b06-telemeter-client-tls\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.066564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.066543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/89604b42-04d8-4094-af6a-a5e1da80a2bb-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7c828\" (UID: \"89604b42-04d8-4094-af6a-a5e1da80a2bb\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:29.072054 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.072009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qc4\" (UniqueName: \"kubernetes.io/projected/ab4c2a5a-414d-4ec4-8e05-f39734703b06-kube-api-access-c7qc4\") pod \"telemeter-client-54d5986f85-lpjcp\" (UID: \"ab4c2a5a-414d-4ec4-8e05-f39734703b06\") " pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.133635 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.133595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp"
Sep 09 20:16:29.262595 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.262492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:29.359456 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.359338    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:16:29.444729 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.444680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54d5986f85-lpjcp"]
Sep 09 20:16:29.447985 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:29.447950    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4c2a5a_414d_4ec4_8e05_f39734703b06.slice/crio-ca1422101138de925edb18e357656d4975b128ab5d86d4aa30eaed52ff5f74e3 WatchSource:0}: Error finding container ca1422101138de925edb18e357656d4975b128ab5d86d4aa30eaed52ff5f74e3: Status 404 returned error can't find the container with id ca1422101138de925edb18e357656d4975b128ab5d86d4aa30eaed52ff5f74e3
Sep 09 20:16:29.584134 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.584093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"]
Sep 09 20:16:29.588884 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:29.588849    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89604b42_04d8_4094_af6a_a5e1da80a2bb.slice/crio-4fb7456abc24b4482e2f08a4341200ab979ded874bc033a4ba99b3fbe3949379 WatchSource:0}: Error finding container 4fb7456abc24b4482e2f08a4341200ab979ded874bc033a4ba99b3fbe3949379: Status 404 returned error can't find the container with id 4fb7456abc24b4482e2f08a4341200ab979ded874bc033a4ba99b3fbe3949379
Sep 09 20:16:29.653146 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.653011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cllf7" event={"ID":"e3d559a3-895a-4d80-9d36-c9cb061384e6","Type":"ContainerStarted","Data":"42e84e7f9ee74124428d2560d0e4c69fcb7d149681fd21e9d4c4a35725864676"}
Sep 09 20:16:29.656436 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.656380    2577 generic.go:356] "Generic (PLEG): container finished" podID="31873c93-4339-46e7-87b2-05fa7d7bc3e8" containerID="a60ab5ba7d32b06d021abb53a056019d7e778e0332c7a89d58ae6bf6dfe18cec" exitCode=0
Sep 09 20:16:29.656624 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.656492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerDied","Data":"a60ab5ba7d32b06d021abb53a056019d7e778e0332c7a89d58ae6bf6dfe18cec"}
Sep 09 20:16:29.658940 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.658883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828" event={"ID":"89604b42-04d8-4094-af6a-a5e1da80a2bb","Type":"ContainerStarted","Data":"4fb7456abc24b4482e2f08a4341200ab979ded874bc033a4ba99b3fbe3949379"}
Sep 09 20:16:29.661328 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.661293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp" event={"ID":"ab4c2a5a-414d-4ec4-8e05-f39734703b06","Type":"ContainerStarted","Data":"ca1422101138de925edb18e357656d4975b128ab5d86d4aa30eaed52ff5f74e3"}
Sep 09 20:16:29.671328 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:29.671267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cllf7" podStartSLOduration=40.17388376 podStartE2EDuration="44.671246831s" totalImagesPullingTime="4.497363071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:15:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:29.670707232 +0000 UTC m=+310.557297145" watchObservedRunningTime="2026-09-09 20:16:29.671246831 +0000 UTC m=+310.557836742"
Sep 09 20:16:30.996880 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:30.995255    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:16:31.020402 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.020349    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:16:31.020603 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.020486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.024536    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.024859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.025126    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.025367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.025632    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:16:31.028070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.027568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tmbk2\""
Sep 09 20:16:31.032969 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.032541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr5n\" (UniqueName: \"kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.185564 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.185473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr5n\" (UniqueName: \"kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.286352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.286334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.288092 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.287407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.288092 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.287557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.288092 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.287958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.288453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.288427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.293293 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.293255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.293815 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.293760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.298439 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.298379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr5n\" (UniqueName: \"kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n\") pod \"console-54fdd6988d-q6ktx\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") " pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.343699 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.343622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:31.615255 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.615218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:16:31.677454 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.677411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"cd95c6a9847181862866ec0dc0b01f1512f5ce697255b33b24db04a6f9388083"}
Sep 09 20:16:31.677639 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.677508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"68e986ea22a0a1aa2d1c3c8dabe6260ae18bf80b972ba286b18537b719987b4b"}
Sep 09 20:16:31.677639 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.677557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"7a379e9304073744c8456604c032536c85e34217c1ddba697de9d522bebc4729"}
Sep 09 20:16:31.685120 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:31.685030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:16:32.768867 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:16:32.768820    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1615a6_4ff3_47bf_ba75_2e7597ab2557.slice/crio-d73be726a33f9608fdc90403b1cc1f8a9adc1a98e8b2f54b3c2c256d1c113936 WatchSource:0}: Error finding container d73be726a33f9608fdc90403b1cc1f8a9adc1a98e8b2f54b3c2c256d1c113936: Status 404 returned error can't find the container with id d73be726a33f9608fdc90403b1cc1f8a9adc1a98e8b2f54b3c2c256d1c113936
Sep 09 20:16:33.687266 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:33.687227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fdd6988d-q6ktx" event={"ID":"fb1615a6-4ff3-47bf-ba75-2e7597ab2557","Type":"ContainerStarted","Data":"d73be726a33f9608fdc90403b1cc1f8a9adc1a98e8b2f54b3c2c256d1c113936"}
Sep 09 20:16:34.695681 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.695644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"473b48854a7f9b22b9de4c5bd6f744858a7efa22263f82bae425e6b00d0876d5"}
Sep 09 20:16:34.696103 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.695734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"a4847b123963cf7ea784198bad6804d6aed7f83954fbc01544dbb5939e1ae401"}
Sep 09 20:16:34.697908 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.697882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828" event={"ID":"89604b42-04d8-4094-af6a-a5e1da80a2bb","Type":"ContainerStarted","Data":"cae154ecae07c352d3f9a0e99d8ca656626273d19a3d3840bc70cd5f6be9b8ff"}
Sep 09 20:16:34.698058 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.698046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:34.702394 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.702234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp" event={"ID":"ab4c2a5a-414d-4ec4-8e05-f39734703b06","Type":"ContainerStarted","Data":"e49c0995d6c4990e1171ec0b3a5b07853ecf809f413bbf73343bdc70c16452ed"}
Sep 09 20:16:34.702394 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.702305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp" event={"ID":"ab4c2a5a-414d-4ec4-8e05-f39734703b06","Type":"ContainerStarted","Data":"d779bd69f0abe7c3ccf9bcbf62a82222ccc820c5bf07c19a6e6da8c0e6f67197"}
Sep 09 20:16:34.702394 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.702334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp" event={"ID":"ab4c2a5a-414d-4ec4-8e05-f39734703b06","Type":"ContainerStarted","Data":"6704d6e16e618db2da0a5cfc37653b4976c597eb9ba2be4423c1d05deeefa2ac"}
Sep 09 20:16:34.705574 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.705509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828"
Sep 09 20:16:34.714410 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:34.714352    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7c828" podStartSLOduration=2.183296857 podStartE2EDuration="6.71433329s" totalImagesPullingTime="4.531036433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:34.71256422 +0000 UTC m=+315.599154165" watchObservedRunningTime="2026-09-09 20:16:34.71433329 +0000 UTC m=+315.600923199"
Sep 09 20:16:35.711537 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.711495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"38d90b0e3d42cae2e4a529d6c185707657945243b545ffe8e7c1ad67877b1ff0"}
Sep 09 20:16:35.712109 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.711577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"838ea6e413aaebcd113bc73400ed3fedb98023ee6c2923283c8efe2e71becf3f"}
Sep 09 20:16:35.712109 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.711619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"9f6cf7d0b6aa36ae7af90dafe03a9b47616703479b500c44a21ccdfde3722f3c"}
Sep 09 20:16:35.712109 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.711656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31873c93-4339-46e7-87b2-05fa7d7bc3e8","Type":"ContainerStarted","Data":"6cf3aefc5a7365613ebc2ddabfc9bd20a4fef9777c14e29223985e5ce3e5cea4"}
Sep 09 20:16:35.715183 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.715146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"bc9f63d857a62336591a2d603ac2ac3f97e3b35b46ab6c70db3bb754f40be0ba"}
Sep 09 20:16:35.715335 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.715235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"8d4f8793f0c392665d952f3678562d99b1cb4204c2c558af6e9aaa6c418e7416"}
Sep 09 20:16:35.715335 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.715281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" event={"ID":"f593b73b-69b3-4d66-b857-29ac46a03f5d","Type":"ContainerStarted","Data":"1853f485648b6ac840d24a391c58dd7f4bc7c0fa15127c7828a541050f8546e0"}
Sep 09 20:16:35.715435 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.715390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:35.739201 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.739122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.592726432 podStartE2EDuration="11.73908827s" totalImagesPullingTime="7.119728728s" totalInitContainerRuntime="26.63311ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:35.737787246 +0000 UTC m=+316.624377211" watchObservedRunningTime="2026-09-09 20:16:35.73908827 +0000 UTC m=+316.625678183"
Sep 09 20:16:35.765491 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.765437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr" podStartSLOduration=3.029901335 podStartE2EDuration="9.765418626s" totalImagesPullingTime="6.735517291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:35.763045084 +0000 UTC m=+316.649635011" watchObservedRunningTime="2026-09-09 20:16:35.765418626 +0000 UTC m=+316.652008550"
Sep 09 20:16:35.787848 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:35.787785    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54d5986f85-lpjcp" podStartSLOduration=3.116388407 podStartE2EDuration="7.787760753s" totalImagesPullingTime="4.671372346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:35.784161303 +0000 UTC m=+316.670751245" watchObservedRunningTime="2026-09-09 20:16:35.787760753 +0000 UTC m=+316.674350666"
Sep 09 20:16:36.002778 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:36.002702    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:16:36.726872 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:36.726841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56d8fffdd5-v6jgr"
Sep 09 20:16:37.725469 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:37.725417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fdd6988d-q6ktx" event={"ID":"fb1615a6-4ff3-47bf-ba75-2e7597ab2557","Type":"ContainerStarted","Data":"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"}
Sep 09 20:16:37.744889 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:37.744825    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54fdd6988d-q6ktx" podStartSLOduration=3.519776779 podStartE2EDuration="7.744805397s" totalImagesPullingTime="4.225028618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.742902531 +0000 UTC m=+318.629492453" watchObservedRunningTime="2026-09-09 20:16:37.744805397 +0000 UTC m=+318.631395301"
Sep 09 20:16:41.344493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:41.344447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:16:54.395223 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.395144    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" containerName="registry" containerID="cri-o://c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98" gracePeriod=30
Sep 09 20:16:54.700361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.700328    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:54.804180 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.804140    2577 generic.go:356] "Generic (PLEG): container finished" podID="961a2f93-759d-4ba6-95b7-601913f069f7" containerID="c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98" exitCode=0
Sep 09 20:16:54.804388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.804196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7"
Sep 09 20:16:54.804388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.804209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" event={"ID":"961a2f93-759d-4ba6-95b7-601913f069f7","Type":"ContainerDied","Data":"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"}
Sep 09 20:16:54.804388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.804288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bbbcc864-pd5t7" event={"ID":"961a2f93-759d-4ba6-95b7-601913f069f7","Type":"ContainerDied","Data":"a112c3142134963a57a44c044d6b931f69e264d3a57c34de08797a2ad86221fa"}
Sep 09 20:16:54.804388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.804328    2577 scope.go:117] "RemoveContainer" containerID="c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"
Sep 09 20:16:54.813009 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.812990    2577 scope.go:117] "RemoveContainer" containerID="c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"
Sep 09 20:16:54.813419 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:16:54.813393    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98\": container with ID starting with c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98 not found: ID does not exist" containerID="c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"
Sep 09 20:16:54.813496 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.813427    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98"} err="failed to get container status \"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98\": rpc error: code = NotFound desc = could not find container \"c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98\": container with ID starting with c3798ab6bdda3b457759dec658f352f37a081c56682b5f878699a855c94aae98 not found: ID does not exist"
Sep 09 20:16:54.821726 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821694    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q8qc\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821961 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821961 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.821961 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.822157 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.821966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") pod \"961a2f93-759d-4ba6-95b7-601913f069f7\" (UID: \"961a2f93-759d-4ba6-95b7-601913f069f7\") "
Sep 09 20:16:54.822412 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.822340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:54.822506 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.822429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:54.824841 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.824807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:54.824975 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.824872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:54.824975 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.824919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:54.825110 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.824990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:54.825110 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.825033    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc" (OuterVolumeSpecName: "kube-api-access-2q8qc") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "kube-api-access-2q8qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:54.830527 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.830490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "961a2f93-759d-4ba6-95b7-601913f069f7" (UID: "961a2f93-759d-4ba6-95b7-601913f069f7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923403    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-registry-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923442    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-bound-sa-token\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923455    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-trusted-ca\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923466    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-installation-pull-secrets\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923480    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8qc\" (UniqueName: \"kubernetes.io/projected/961a2f93-759d-4ba6-95b7-601913f069f7-kube-api-access-2q8qc\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923493    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/961a2f93-759d-4ba6-95b7-601913f069f7-ca-trust-extracted\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923506    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/961a2f93-759d-4ba6-95b7-601913f069f7-registry-certificates\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:54.923837 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:54.923519    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/961a2f93-759d-4ba6-95b7-601913f069f7-image-registry-private-configuration\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:16:55.126416 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:55.126378    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:16:55.128195 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:55.128168    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bbbcc864-pd5t7"]
Sep 09 20:16:55.620465 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:16:55.620432    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" path="/var/lib/kubelet/pods/961a2f93-759d-4ba6-95b7-601913f069f7/volumes"
Sep 09 20:17:02.745852 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:02.745787    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54fdd6988d-q6ktx" podUID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" containerName="console" containerID="cri-o://002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9" gracePeriod=15
Sep 09 20:17:03.031458 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.031418    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:17:03.199314 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrr5n\" (UniqueName: \"kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199499 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199779 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert\") pod \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\" (UID: \"fb1615a6-4ff3-47bf-ba75-2e7597ab2557\") "
Sep 09 20:17:03.199922 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:03.200080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199981    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:03.200080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.199980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config" (OuterVolumeSpecName: "console-config") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:03.200080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.200045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:03.201826 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.201801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:03.201936 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.201823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:03.201936 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.201857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n" (OuterVolumeSpecName: "kube-api-access-lrr5n") pod "fb1615a6-4ff3-47bf-ba75-2e7597ab2557" (UID: "fb1615a6-4ff3-47bf-ba75-2e7597ab2557"). InnerVolumeSpecName "kube-api-access-lrr5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:03.300463 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300422    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-trusted-ca-bundle\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300463 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300463    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300476    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr5n\" (UniqueName: \"kubernetes.io/projected/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-kube-api-access-lrr5n\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300490    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-oauth-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300502    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-oauth-serving-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300513    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-console-serving-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.300676 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.300524    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1615a6-4ff3-47bf-ba75-2e7597ab2557-service-ca\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:17:03.841672 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.841633    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" containerID="002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9" exitCode=0
Sep 09 20:17:03.842084 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.841688    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fdd6988d-q6ktx"
Sep 09 20:17:03.842084 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.841722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fdd6988d-q6ktx" event={"ID":"fb1615a6-4ff3-47bf-ba75-2e7597ab2557","Type":"ContainerDied","Data":"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"}
Sep 09 20:17:03.842084 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.841800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fdd6988d-q6ktx" event={"ID":"fb1615a6-4ff3-47bf-ba75-2e7597ab2557","Type":"ContainerDied","Data":"d73be726a33f9608fdc90403b1cc1f8a9adc1a98e8b2f54b3c2c256d1c113936"}
Sep 09 20:17:03.842084 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.841823    2577 scope.go:117] "RemoveContainer" containerID="002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"
Sep 09 20:17:03.858590 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.858557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:17:03.861916 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.861889    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54fdd6988d-q6ktx"]
Sep 09 20:17:03.861998 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.861909    2577 scope.go:117] "RemoveContainer" containerID="002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"
Sep 09 20:17:03.862279 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:17:03.862259    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9\": container with ID starting with 002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9 not found: ID does not exist" containerID="002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"
Sep 09 20:17:03.862320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:03.862290    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9"} err="failed to get container status \"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9\": rpc error: code = NotFound desc = could not find container \"002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9\": container with ID starting with 002bde3f927d72c1bca84f14d21408cf356d73fa387f89e9f79837de5805fbc9 not found: ID does not exist"
Sep 09 20:17:05.620656 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:05.620613    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" path="/var/lib/kubelet/pods/fb1615a6-4ff3-47bf-ba75-2e7597ab2557/volumes"
Sep 09 20:17:08.866116 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:08.866083    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c8d445d-fe1b-4900-9a72-fe48105d4c91" containerID="7c6762ed26c0390f03d7c5a8734579bb53b2d1e54cfe4a038978058ee68ee04b" exitCode=0
Sep 09 20:17:08.866623 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:08.866156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq" event={"ID":"1c8d445d-fe1b-4900-9a72-fe48105d4c91","Type":"ContainerDied","Data":"7c6762ed26c0390f03d7c5a8734579bb53b2d1e54cfe4a038978058ee68ee04b"}
Sep 09 20:17:08.866623 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:08.866493    2577 scope.go:117] "RemoveContainer" containerID="7c6762ed26c0390f03d7c5a8734579bb53b2d1e54cfe4a038978058ee68ee04b"
Sep 09 20:17:09.872611 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:09.872574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fn6dq" event={"ID":"1c8d445d-fe1b-4900-9a72-fe48105d4c91","Type":"ContainerStarted","Data":"005a52fc09e20e3c78ed0ba35d707fdf88d8fac4f9049f7a9fa827486034a015"}
Sep 09 20:17:33.653954 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:33.653909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:17:33.656661 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:33.656630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7a3927-19b9-4c23-8c72-b245c699f4ee-metrics-certs\") pod \"network-metrics-daemon-9blb8\" (UID: \"cb7a3927-19b9-4c23-8c72-b245c699f4ee\") " pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:17:33.719777 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:33.719740    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb455\""
Sep 09 20:17:33.727010 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:33.726977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9blb8"
Sep 09 20:17:34.013165 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:34.013134    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9blb8"]
Sep 09 20:17:34.015557 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:17:34.015527    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7a3927_19b9_4c23_8c72_b245c699f4ee.slice/crio-32f3fcf1381cfd17327076e25540bf276a892fe8d76ea653ea50f151bab0148f WatchSource:0}: Error finding container 32f3fcf1381cfd17327076e25540bf276a892fe8d76ea653ea50f151bab0148f: Status 404 returned error can't find the container with id 32f3fcf1381cfd17327076e25540bf276a892fe8d76ea653ea50f151bab0148f
Sep 09 20:17:34.979069 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:34.979012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9blb8" event={"ID":"cb7a3927-19b9-4c23-8c72-b245c699f4ee","Type":"ContainerStarted","Data":"32f3fcf1381cfd17327076e25540bf276a892fe8d76ea653ea50f151bab0148f"}
Sep 09 20:17:35.985590 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:35.985550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9blb8" event={"ID":"cb7a3927-19b9-4c23-8c72-b245c699f4ee","Type":"ContainerStarted","Data":"a43af1b0eeb8e80af6689519ecc5beeee8cc3316320239a47c20f855d18dfe5c"}
Sep 09 20:17:35.986097 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:35.985606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9blb8" event={"ID":"cb7a3927-19b9-4c23-8c72-b245c699f4ee","Type":"ContainerStarted","Data":"53950f2d3a7e67f17bcb3e344e37e3138efa9eea2e2ec693e04068091ebad901"}
Sep 09 20:17:36.003419 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:36.003342    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9blb8" podStartSLOduration=376.084084094 podStartE2EDuration="6m17.003325763s" totalImagesPullingTime="919.241669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:36.002209951 +0000 UTC m=+376.888799881" watchObservedRunningTime="2026-09-09 20:17:36.003325763 +0000 UTC m=+376.889915674"
Sep 09 20:17:40.050848 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.050811    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:17:40.051254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051171    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" containerName="registry"
Sep 09 20:17:40.051254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051186    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" containerName="registry"
Sep 09 20:17:40.051254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051195    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" containerName="console"
Sep 09 20:17:40.051254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051201    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" containerName="console"
Sep 09 20:17:40.051382 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051271    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="961a2f93-759d-4ba6-95b7-601913f069f7" containerName="registry"
Sep 09 20:17:40.051382 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.051278    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1615a6-4ff3-47bf-ba75-2e7597ab2557" containerName="console"
Sep 09 20:17:40.054242 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.054221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.057115 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.057080    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:17:40.058235 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.058194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tmbk2\""
Sep 09 20:17:40.058385 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.058205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:17:40.058385 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.058206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:17:40.058385 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.058295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:17:40.058560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.058354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:17:40.062270 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.062249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:17:40.067151 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.067123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:17:40.111563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv558\" (UniqueName: \"kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.111811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.111697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212633 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv558\" (UniqueName: \"kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.212752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.213007 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.212808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.213551 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.213522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.213551 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.213539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.213691 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.213555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.213787 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.213761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.215424 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.215396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.215424 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.215417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.221369 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.221337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv558\" (UniqueName: \"kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558\") pod \"console-5f749cd6c8-q2467\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") " pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.364922 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.364813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:40.639872 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:40.639789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:17:40.643238 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:17:40.643205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2f4372_48e1_4e22_9f0c_ff4a7ac2da2d.slice/crio-101a28c771d22e0e4cc71c9c7759f36d73cc0a09c3e72825ebe5b9a848cdd6af WatchSource:0}: Error finding container 101a28c771d22e0e4cc71c9c7759f36d73cc0a09c3e72825ebe5b9a848cdd6af: Status 404 returned error can't find the container with id 101a28c771d22e0e4cc71c9c7759f36d73cc0a09c3e72825ebe5b9a848cdd6af
Sep 09 20:17:41.008102 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:41.007998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f749cd6c8-q2467" event={"ID":"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d","Type":"ContainerStarted","Data":"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"}
Sep 09 20:17:41.008102 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:41.008073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f749cd6c8-q2467" event={"ID":"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d","Type":"ContainerStarted","Data":"101a28c771d22e0e4cc71c9c7759f36d73cc0a09c3e72825ebe5b9a848cdd6af"}
Sep 09 20:17:41.025888 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:41.025820    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f749cd6c8-q2467" podStartSLOduration=1.025797062 podStartE2EDuration="1.025797062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:41.024747435 +0000 UTC m=+381.911337358" watchObservedRunningTime="2026-09-09 20:17:41.025797062 +0000 UTC m=+381.912386971"
Sep 09 20:17:50.365479 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:50.365440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:50.365876 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:50.365513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:50.370512 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:50.370480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:17:51.053775 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:17:51.053745    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:19:42.655949 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.655906    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"]
Sep 09 20:19:42.659579 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.659552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.662705 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.662680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:19:42.662884 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.662727    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rwk4w\""
Sep 09 20:19:42.663733 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.663707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:19:42.669039 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.668991    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"]
Sep 09 20:19:42.722920 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.722879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.722920 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.722920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.723175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.722955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hf8l\" (UniqueName: \"kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.824154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.824108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.824154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.824155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.824370 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.824192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hf8l\" (UniqueName: \"kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.824587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.824560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.824651 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.824567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.833531 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.833495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hf8l\" (UniqueName: \"kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:42.968879 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:42.968773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:19:43.245860 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:43.245734    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"]
Sep 09 20:19:43.249077 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:19:43.249044    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfc55b0_7609_41de_93d6_5e106a9e5532.slice/crio-de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18 WatchSource:0}: Error finding container de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18: Status 404 returned error can't find the container with id de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18
Sep 09 20:19:43.505491 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:43.505406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerStarted","Data":"de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18"}
Sep 09 20:19:50.543121 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:50.543081    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerID="801d64774ecae0c18d59e292782ceb87d513226dd5632cb4961e04bbab36192a" exitCode=0
Sep 09 20:19:50.543505 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:50.543166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerDied","Data":"801d64774ecae0c18d59e292782ceb87d513226dd5632cb4961e04bbab36192a"}
Sep 09 20:19:53.557666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:53.557631    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerID="45a6ff3f7fdc7636ff2e32e8d0a019e33a8610b0b3160c8e5bad2e5a95db382d" exitCode=0
Sep 09 20:19:53.558082 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:19:53.557725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerDied","Data":"45a6ff3f7fdc7636ff2e32e8d0a019e33a8610b0b3160c8e5bad2e5a95db382d"}
Sep 09 20:20:00.591537 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:00.591493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerStarted","Data":"b2101b62073c1de440fe8024dc468f260905f56a71c05da23d35c979299cf041"}
Sep 09 20:20:00.613129 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:00.613064    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" podStartSLOduration=3.197245496 podStartE2EDuration="18.613010463s" totalImagesPullingTime="15.415764967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:00.610512774 +0000 UTC m=+521.497102730" watchObservedRunningTime="2026-09-09 20:20:00.613010463 +0000 UTC m=+521.499600375"
Sep 09 20:20:01.597761 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:01.597715    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerID="b2101b62073c1de440fe8024dc468f260905f56a71c05da23d35c979299cf041" exitCode=0
Sep 09 20:20:01.598186 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:01.597800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerDied","Data":"b2101b62073c1de440fe8024dc468f260905f56a71c05da23d35c979299cf041"}
Sep 09 20:20:02.730893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.730859    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:20:02.799770 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.799736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util\") pod \"4bfc55b0-7609-41de-93d6-5e106a9e5532\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") "
Sep 09 20:20:02.799916 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.799785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hf8l\" (UniqueName: \"kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l\") pod \"4bfc55b0-7609-41de-93d6-5e106a9e5532\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") "
Sep 09 20:20:02.799916 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.799906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle\") pod \"4bfc55b0-7609-41de-93d6-5e106a9e5532\" (UID: \"4bfc55b0-7609-41de-93d6-5e106a9e5532\") "
Sep 09 20:20:02.800544 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.800507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle" (OuterVolumeSpecName: "bundle") pod "4bfc55b0-7609-41de-93d6-5e106a9e5532" (UID: "4bfc55b0-7609-41de-93d6-5e106a9e5532"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:02.802098 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.802072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l" (OuterVolumeSpecName: "kube-api-access-9hf8l") pod "4bfc55b0-7609-41de-93d6-5e106a9e5532" (UID: "4bfc55b0-7609-41de-93d6-5e106a9e5532"). InnerVolumeSpecName "kube-api-access-9hf8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:02.805242 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.805207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util" (OuterVolumeSpecName: "util") pod "4bfc55b0-7609-41de-93d6-5e106a9e5532" (UID: "4bfc55b0-7609-41de-93d6-5e106a9e5532"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:02.900709 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.900617    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-util\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:20:02.900709 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.900650    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hf8l\" (UniqueName: \"kubernetes.io/projected/4bfc55b0-7609-41de-93d6-5e106a9e5532-kube-api-access-9hf8l\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:20:02.900709 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:02.900661    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc55b0-7609-41de-93d6-5e106a9e5532-bundle\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:20:03.608320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:03.608285    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw"
Sep 09 20:20:03.608472 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:03.608284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ampwtw" event={"ID":"4bfc55b0-7609-41de-93d6-5e106a9e5532","Type":"ContainerDied","Data":"de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18"}
Sep 09 20:20:03.608472 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:03.608448    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de80f5654a367ec8069d12fe399a5707b50b5ef568d250de55339c2fd5831f18"
Sep 09 20:20:15.603834 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.603787    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wkz7h"]
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604272    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="util"
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="util"
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604309    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="pull"
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="pull"
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604342    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="extract"
Sep 09 20:20:15.604421 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604351    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="extract"
Sep 09 20:20:15.604705 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.604443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfc55b0-7609-41de-93d6-5e106a9e5532" containerName="extract"
Sep 09 20:20:15.607553 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.607528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.610285 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.610238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:20:15.610466 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.610308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:20:15.610539 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.610512    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:20:15.610539 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.610515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7l8zv\""
Sep 09 20:20:15.611479 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.611460    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:20:15.611607 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.611462    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:20:15.621538 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.621496    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wkz7h"]
Sep 09 20:20:15.721679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.721585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmn9k\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-kube-api-access-lmn9k\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.721838 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.721701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-cabundle0\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.721880 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.721836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.822514 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.822471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-cabundle0\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.822527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.822622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmn9k\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-kube-api-access-lmn9k\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:15.822706    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:15.822733    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:15.822742    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-wkz7h: references non-existent secret key: ca.crt
Sep 09 20:20:15.822808 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:15.822813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates podName:c125d4c1-88b4-4812-8510-8d6c1d0e2f2d nodeName:}" failed. No retries permitted until 2026-09-09 20:20:16.322796585 +0000 UTC m=+537.209386472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates") pod "keda-operator-579c8958f7-wkz7h" (UID: "c125d4c1-88b4-4812-8510-8d6c1d0e2f2d") : references non-existent secret key: ca.crt
Sep 09 20:20:15.823328 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.823305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-cabundle0\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:15.848471 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:15.848440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmn9k\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-kube-api-access-lmn9k\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:16.327554 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.327470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:16.327764 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.327636    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:20:16.327764 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.327661    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:20:16.327764 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.327673    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-wkz7h: references non-existent secret key: ca.crt
Sep 09 20:20:16.327764 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.327752    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates podName:c125d4c1-88b4-4812-8510-8d6c1d0e2f2d nodeName:}" failed. No retries permitted until 2026-09-09 20:20:17.327732488 +0000 UTC m=+538.214322402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates") pod "keda-operator-579c8958f7-wkz7h" (UID: "c125d4c1-88b4-4812-8510-8d6c1d0e2f2d") : references non-existent secret key: ca.crt
Sep 09 20:20:16.378638 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.378593    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f75dd"]
Sep 09 20:20:16.382063 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.382012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:16.384638 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.384614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:20:16.392039 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.391981    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f75dd"]
Sep 09 20:20:16.529314 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.529276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpp4\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-kube-api-access-xhpp4\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:16.529521 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.529329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:16.630399 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.630290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:16.630865 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.630415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpp4\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-kube-api-access-xhpp4\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:16.630865 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.630459    2577 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 09 20:20:16.630865 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.630500    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-f75dd: secret "keda-admission-webhooks-certs" not found
Sep 09 20:20:16.630865 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:20:16.630556    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates podName:73c0479d-b0f4-4768-8186-e0e2c8e817ae nodeName:}" failed. No retries permitted until 2026-09-09 20:20:17.13053996 +0000 UTC m=+538.017129848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates") pod "keda-admission-6c4d864b86-f75dd" (UID: "73c0479d-b0f4-4768-8186-e0e2c8e817ae") : secret "keda-admission-webhooks-certs" not found
Sep 09 20:20:16.641116 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:16.641077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpp4\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-kube-api-access-xhpp4\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:17.135107 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.135060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:17.137693 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.137664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73c0479d-b0f4-4768-8186-e0e2c8e817ae-certificates\") pod \"keda-admission-6c4d864b86-f75dd\" (UID: \"73c0479d-b0f4-4768-8186-e0e2c8e817ae\") " pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:17.294664 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.294626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:17.337209 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.337166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:17.339743 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.339708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c125d4c1-88b4-4812-8510-8d6c1d0e2f2d-certificates\") pod \"keda-operator-579c8958f7-wkz7h\" (UID: \"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d\") " pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:17.419662 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.419565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:17.571702 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.571652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f75dd"]
Sep 09 20:20:17.574742 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:20:17.574708    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c0479d_b0f4_4768_8186_e0e2c8e817ae.slice/crio-4d2169855812f7845b00ff0388616aec10b9c957990068204fa59722d2c51280 WatchSource:0}: Error finding container 4d2169855812f7845b00ff0388616aec10b9c957990068204fa59722d2c51280: Status 404 returned error can't find the container with id 4d2169855812f7845b00ff0388616aec10b9c957990068204fa59722d2c51280
Sep 09 20:20:17.667860 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.667822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-f75dd" event={"ID":"73c0479d-b0f4-4768-8186-e0e2c8e817ae","Type":"ContainerStarted","Data":"4d2169855812f7845b00ff0388616aec10b9c957990068204fa59722d2c51280"}
Sep 09 20:20:17.703156 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:17.703120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wkz7h"]
Sep 09 20:20:17.705824 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:20:17.705794    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc125d4c1_88b4_4812_8510_8d6c1d0e2f2d.slice/crio-59e5ab4155b852485107942c3e39cf314efb6f9ca1a40d0f940430322962b965 WatchSource:0}: Error finding container 59e5ab4155b852485107942c3e39cf314efb6f9ca1a40d0f940430322962b965: Status 404 returned error can't find the container with id 59e5ab4155b852485107942c3e39cf314efb6f9ca1a40d0f940430322962b965
Sep 09 20:20:18.676137 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:18.676099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-wkz7h" event={"ID":"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d","Type":"ContainerStarted","Data":"59e5ab4155b852485107942c3e39cf314efb6f9ca1a40d0f940430322962b965"}
Sep 09 20:20:20.688072 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:20.688036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-f75dd" event={"ID":"73c0479d-b0f4-4768-8186-e0e2c8e817ae","Type":"ContainerStarted","Data":"2a41a835205a35a8340d313cf6516725f2020b5cfb23c2c41e86a226b931fbc0"}
Sep 09 20:20:20.688554 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:20.688147    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:20.706091 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:20.705996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-f75dd" podStartSLOduration=2.4105099340000002 podStartE2EDuration="4.705972286s" totalImagesPullingTime="2.295462352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:20.704468761 +0000 UTC m=+541.591058692" watchObservedRunningTime="2026-09-09 20:20:20.705972286 +0000 UTC m=+541.592562195"
Sep 09 20:20:23.703738 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:23.703696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-wkz7h" event={"ID":"c125d4c1-88b4-4812-8510-8d6c1d0e2f2d","Type":"ContainerStarted","Data":"0b806fbec14e4a15ebabbbdc0f295566b9df362c71b4549bbcd035a455f62d58"}
Sep 09 20:20:23.704203 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:23.703844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:20:23.723541 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:23.723487    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-wkz7h" podStartSLOduration=3.572058402 podStartE2EDuration="8.723470569s" totalImagesPullingTime="5.151412167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:23.722250203 +0000 UTC m=+544.608840124" watchObservedRunningTime="2026-09-09 20:20:23.723470569 +0000 UTC m=+544.610060479"
Sep 09 20:20:41.696126 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:41.696081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-f75dd"
Sep 09 20:20:44.709166 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:20:44.709126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-wkz7h"
Sep 09 20:21:33.693218 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:33.693179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:21:35.014844 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.014809    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:21:35.018153 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.018134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.021290 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.021248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:21:35.021500 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.021344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:21:35.021500 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.021413    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-t9mbn\""
Sep 09 20:21:35.022204 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.022189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:21:35.030512 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.030481    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"]
Sep 09 20:21:35.034099 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.034069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:21:35.034250 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.034179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.037210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.037177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:21:35.037210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.037207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-b4f7n\""
Sep 09 20:21:35.050254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.050219    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"]
Sep 09 20:21:35.079646 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.079605    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gv28n"]
Sep 09 20:21:35.082944 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.082918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.086138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.086111    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:21:35.086560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.086543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jfl48\""
Sep 09 20:21:35.089301 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.089274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98rh\" (UniqueName: \"kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.089465 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.089324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx86z\" (UniqueName: \"kubernetes.io/projected/b70e67ea-06c4-46fa-90eb-0f905bb453f8-kube-api-access-sx86z\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.089465 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.089353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.089465 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.089376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70e67ea-06c4-46fa-90eb-0f905bb453f8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.094931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.094901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gv28n"]
Sep 09 20:21:35.190651 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98rh\" (UniqueName: \"kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.190870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx86z\" (UniqueName: \"kubernetes.io/projected/b70e67ea-06c4-46fa-90eb-0f905bb453f8-kube-api-access-sx86z\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.190870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.190870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70e67ea-06c4-46fa-90eb-0f905bb453f8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.190870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-data\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.190870 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.190762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhb4\" (UniqueName: \"kubernetes.io/projected/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-kube-api-access-4mhb4\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.193409 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.193382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70e67ea-06c4-46fa-90eb-0f905bb453f8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.193540 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.193516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.205136 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.205097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx86z\" (UniqueName: \"kubernetes.io/projected/b70e67ea-06c4-46fa-90eb-0f905bb453f8-kube-api-access-sx86z\") pod \"llmisvc-controller-manager-645f9c8fd6-gtnbt\" (UID: \"b70e67ea-06c4-46fa-90eb-0f905bb453f8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.205270 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.205131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98rh\" (UniqueName: \"kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh\") pod \"kserve-controller-manager-55b9566966-jd969\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") " pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.291970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.291913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-data\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.291970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.291979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhb4\" (UniqueName: \"kubernetes.io/projected/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-kube-api-access-4mhb4\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.292363 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.292342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-data\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.301729 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.301695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhb4\" (UniqueName: \"kubernetes.io/projected/cc00a022-2e7f-42d6-8e3b-31ae2f78513d-kube-api-access-4mhb4\") pod \"seaweedfs-64568bcd49-gv28n\" (UID: \"cc00a022-2e7f-42d6-8e3b-31ae2f78513d\") " pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.330617 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.330575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:35.345859 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.345770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:35.392956 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.392867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:35.627227 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.627082    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:21:35.629998 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:21:35.629962    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd224cb_19d2_4bef_bd91_6cc9f14810cf.slice/crio-b2d0e9223a93c76aed5ce7e9c38903c230524ceae13671c501be562032b196ed WatchSource:0}: Error finding container b2d0e9223a93c76aed5ce7e9c38903c230524ceae13671c501be562032b196ed: Status 404 returned error can't find the container with id b2d0e9223a93c76aed5ce7e9c38903c230524ceae13671c501be562032b196ed
Sep 09 20:21:35.631403 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.631381    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:21:35.655381 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.655348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"]
Sep 09 20:21:35.659693 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:21:35.659652    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb70e67ea_06c4_46fa_90eb_0f905bb453f8.slice/crio-f9b4b7af88e241e5cebaebc5af1d723efc01e21139d2db8534ffa3006d195ac2 WatchSource:0}: Error finding container f9b4b7af88e241e5cebaebc5af1d723efc01e21139d2db8534ffa3006d195ac2: Status 404 returned error can't find the container with id f9b4b7af88e241e5cebaebc5af1d723efc01e21139d2db8534ffa3006d195ac2
Sep 09 20:21:35.679268 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:35.679074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gv28n"]
Sep 09 20:21:35.681931 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:21:35.681898    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc00a022_2e7f_42d6_8e3b_31ae2f78513d.slice/crio-2a293c4d60954ad4e64b7fb52106829d1715be27de0723cd04fe60b1048e840c WatchSource:0}: Error finding container 2a293c4d60954ad4e64b7fb52106829d1715be27de0723cd04fe60b1048e840c: Status 404 returned error can't find the container with id 2a293c4d60954ad4e64b7fb52106829d1715be27de0723cd04fe60b1048e840c
Sep 09 20:21:36.020701 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:36.020661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt" event={"ID":"b70e67ea-06c4-46fa-90eb-0f905bb453f8","Type":"ContainerStarted","Data":"f9b4b7af88e241e5cebaebc5af1d723efc01e21139d2db8534ffa3006d195ac2"}
Sep 09 20:21:36.021904 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:36.021873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gv28n" event={"ID":"cc00a022-2e7f-42d6-8e3b-31ae2f78513d","Type":"ContainerStarted","Data":"2a293c4d60954ad4e64b7fb52106829d1715be27de0723cd04fe60b1048e840c"}
Sep 09 20:21:36.022997 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:36.022978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jd969" event={"ID":"4bd224cb-19d2-4bef-bd91-6cc9f14810cf","Type":"ContainerStarted","Data":"b2d0e9223a93c76aed5ce7e9c38903c230524ceae13671c501be562032b196ed"}
Sep 09 20:21:41.056037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.055988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt" event={"ID":"b70e67ea-06c4-46fa-90eb-0f905bb453f8","Type":"ContainerStarted","Data":"b3fe9666d4d5e7a16fe39c44fe58b50d84b9cc04f56220b2fd2474cf7f9753fe"}
Sep 09 20:21:41.056470 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.056083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:21:41.057598 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.057574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gv28n" event={"ID":"cc00a022-2e7f-42d6-8e3b-31ae2f78513d","Type":"ContainerStarted","Data":"7c6e24187f23a46864400150461e6ec739cceffd58fd9cc241f002129ae7d7ae"}
Sep 09 20:21:41.057722 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.057621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:41.059002 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.058982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jd969" event={"ID":"4bd224cb-19d2-4bef-bd91-6cc9f14810cf","Type":"ContainerStarted","Data":"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"}
Sep 09 20:21:41.059133 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.059115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:21:41.088245 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.088194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt" podStartSLOduration=1.5337202570000001 podStartE2EDuration="6.088179585s" totalImagesPullingTime="4.554459328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:41.087734114 +0000 UTC m=+621.974324023" watchObservedRunningTime="2026-09-09 20:21:41.088179585 +0000 UTC m=+621.974769494"
Sep 09 20:21:41.123534 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.123459    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gv28n" podStartSLOduration=1.377364463 podStartE2EDuration="6.123436175s" totalImagesPullingTime="4.746071712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:41.123073088 +0000 UTC m=+622.009663018" watchObservedRunningTime="2026-09-09 20:21:41.123436175 +0000 UTC m=+622.010026085"
Sep 09 20:21:41.157674 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:41.157608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-jd969" podStartSLOduration=2.412086868 podStartE2EDuration="7.157582506s" totalImagesPullingTime="4.745495638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:41.157226973 +0000 UTC m=+622.043816923" watchObservedRunningTime="2026-09-09 20:21:41.157582506 +0000 UTC m=+622.044172417"
Sep 09 20:21:47.071146 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:47.071109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gv28n"
Sep 09 20:21:58.714493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:58.714451    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f749cd6c8-q2467" podUID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" containerName="console" containerID="cri-o://301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d" gracePeriod=15
Sep 09 20:21:58.968702 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:58.968629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:21:59.109912 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.109870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.109930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.109975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110118 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110287 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110287 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110388 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv558\" (UniqueName: \"kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558\") pod \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\" (UID: \"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d\") "
Sep 09 20:21:59.110577 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config" (OuterVolumeSpecName: "console-config") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:59.110577 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:59.110577 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:59.110751 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.110611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca" (OuterVolumeSpecName: "service-ca") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:59.112236 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.112197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:59.112352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.112279    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:59.112401 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.112372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558" (OuterVolumeSpecName: "kube-api-access-sv558") pod "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" (UID: "3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d"). InnerVolumeSpecName "kube-api-access-sv558". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:59.147126 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.147089    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" containerID="301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d" exitCode=0
Sep 09 20:21:59.147294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.147135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f749cd6c8-q2467"
Sep 09 20:21:59.147294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.147175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f749cd6c8-q2467" event={"ID":"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d","Type":"ContainerDied","Data":"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"}
Sep 09 20:21:59.147294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.147229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f749cd6c8-q2467" event={"ID":"3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d","Type":"ContainerDied","Data":"101a28c771d22e0e4cc71c9c7759f36d73cc0a09c3e72825ebe5b9a848cdd6af"}
Sep 09 20:21:59.147294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.147249    2577 scope.go:117] "RemoveContainer" containerID="301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"
Sep 09 20:21:59.157072 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.157053    2577 scope.go:117] "RemoveContainer" containerID="301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"
Sep 09 20:21:59.157412 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:21:59.157389    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d\": container with ID starting with 301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d not found: ID does not exist" containerID="301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"
Sep 09 20:21:59.157520 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.157418    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d"} err="failed to get container status \"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d\": rpc error: code = NotFound desc = could not find container \"301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d\": container with ID starting with 301a7effe449608ff3a0e1636758aaaa502637cc90b81b870e4526411babf16d not found: ID does not exist"
Sep 09 20:21:59.171064 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.171031    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:21:59.176195 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.176159    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f749cd6c8-q2467"]
Sep 09 20:21:59.211408 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211365    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv558\" (UniqueName: \"kubernetes.io/projected/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-kube-api-access-sv558\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211408 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211402    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-serving-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211408 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211414    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211423    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-oauth-serving-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211432    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-trusted-ca-bundle\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211440    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-console-oauth-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.211632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.211448    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d-service-ca\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.620185 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:21:59.620151    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" path="/var/lib/kubelet/pods/3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d/volumes"
Sep 09 20:22:12.071261 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:12.071229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-gtnbt"
Sep 09 20:22:12.074358 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:12.074331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:22:19.849899 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.849809    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:22:19.850362 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.850005    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-55b9566966-jd969" podUID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" containerName="manager" containerID="cri-o://919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9" gracePeriod=10
Sep 09 20:22:19.874965 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.874931    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-98795"]
Sep 09 20:22:19.875336 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.875322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" containerName="console"
Sep 09 20:22:19.875379 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.875338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" containerName="console"
Sep 09 20:22:19.875412 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.875407    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2f4372-48e1-4e22-9f0c-ff4a7ac2da2d" containerName="console"
Sep 09 20:22:19.878317 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.878296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:19.885787 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.885754    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-98795"]
Sep 09 20:22:19.991492 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.991449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93c1a534-82e5-4447-8553-04f6bec0107f-cert\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:19.991678 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:19.991506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmvq\" (UniqueName: \"kubernetes.io/projected/93c1a534-82e5-4447-8553-04f6bec0107f-kube-api-access-rrmvq\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.092984 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.092953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93c1a534-82e5-4447-8553-04f6bec0107f-cert\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.093149 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.093004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmvq\" (UniqueName: \"kubernetes.io/projected/93c1a534-82e5-4447-8553-04f6bec0107f-kube-api-access-rrmvq\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.095425 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.095391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93c1a534-82e5-4447-8553-04f6bec0107f-cert\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.101471 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.101403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmvq\" (UniqueName: \"kubernetes.io/projected/93c1a534-82e5-4447-8553-04f6bec0107f-kube-api-access-rrmvq\") pod \"kserve-controller-manager-55b9566966-98795\" (UID: \"93c1a534-82e5-4447-8553-04f6bec0107f\") " pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.117971 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.117937    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:22:20.193391 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.193347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert\") pod \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") "
Sep 09 20:22:20.193578 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.193405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98rh\" (UniqueName: \"kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh\") pod \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\" (UID: \"4bd224cb-19d2-4bef-bd91-6cc9f14810cf\") "
Sep 09 20:22:20.195660 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.195635    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh" (OuterVolumeSpecName: "kube-api-access-k98rh") pod "4bd224cb-19d2-4bef-bd91-6cc9f14810cf" (UID: "4bd224cb-19d2-4bef-bd91-6cc9f14810cf"). InnerVolumeSpecName "kube-api-access-k98rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:20.195720 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.195633    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert" (OuterVolumeSpecName: "cert") pod "4bd224cb-19d2-4bef-bd91-6cc9f14810cf" (UID: "4bd224cb-19d2-4bef-bd91-6cc9f14810cf"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:20.233881 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.233823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:20.246887 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.246857    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" containerID="919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9" exitCode=0
Sep 09 20:22:20.247004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.246902    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jd969"
Sep 09 20:22:20.247004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.246938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jd969" event={"ID":"4bd224cb-19d2-4bef-bd91-6cc9f14810cf","Type":"ContainerDied","Data":"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"}
Sep 09 20:22:20.247004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.246990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jd969" event={"ID":"4bd224cb-19d2-4bef-bd91-6cc9f14810cf","Type":"ContainerDied","Data":"b2d0e9223a93c76aed5ce7e9c38903c230524ceae13671c501be562032b196ed"}
Sep 09 20:22:20.247145 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.247030    2577 scope.go:117] "RemoveContainer" containerID="919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"
Sep 09 20:22:20.259241 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.259208    2577 scope.go:117] "RemoveContainer" containerID="919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"
Sep 09 20:22:20.259602 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:22:20.259577    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9\": container with ID starting with 919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9 not found: ID does not exist" containerID="919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"
Sep 09 20:22:20.259691 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.259610    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9"} err="failed to get container status \"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9\": rpc error: code = NotFound desc = could not find container \"919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9\": container with ID starting with 919164d279b5947b55caef1e983611a182729ecda3601605c377554cdc6888c9 not found: ID does not exist"
Sep 09 20:22:20.271811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.271768    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:22:20.274626 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.274597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jd969"]
Sep 09 20:22:20.294827 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.294669    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:22:20.294827 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.294695    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98rh\" (UniqueName: \"kubernetes.io/projected/4bd224cb-19d2-4bef-bd91-6cc9f14810cf-kube-api-access-k98rh\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:22:20.509625 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:20.509593    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-98795"]
Sep 09 20:22:20.511982 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:22:20.511954    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c1a534_82e5_4447_8553_04f6bec0107f.slice/crio-cc939055715474a87270e4154090eb96a86088be098af32399b1bb8170500546 WatchSource:0}: Error finding container cc939055715474a87270e4154090eb96a86088be098af32399b1bb8170500546: Status 404 returned error can't find the container with id cc939055715474a87270e4154090eb96a86088be098af32399b1bb8170500546
Sep 09 20:22:21.254584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:21.254485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-98795" event={"ID":"93c1a534-82e5-4447-8553-04f6bec0107f","Type":"ContainerStarted","Data":"02a3f6ebe45321f5c72fde392c232dcb3c5431bc3975455927788c492787e078"}
Sep 09 20:22:21.254584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:21.254538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-98795" event={"ID":"93c1a534-82e5-4447-8553-04f6bec0107f","Type":"ContainerStarted","Data":"cc939055715474a87270e4154090eb96a86088be098af32399b1bb8170500546"}
Sep 09 20:22:21.255136 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:21.254625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:21.273955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:21.273895    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-98795" podStartSLOduration=1.9398022350000002 podStartE2EDuration="2.273875854s" totalImagesPullingTime="334.073619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:21.273394491 +0000 UTC m=+662.159984401" watchObservedRunningTime="2026-09-09 20:22:21.273875854 +0000 UTC m=+662.160465817"
Sep 09 20:22:21.620896 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:21.620853    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" path="/var/lib/kubelet/pods/4bd224cb-19d2-4bef-bd91-6cc9f14810cf/volumes"
Sep 09 20:22:52.263608 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:52.263572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-98795"
Sep 09 20:22:53.708512 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.708466    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-m5ssq"]
Sep 09 20:22:53.709044 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.709011    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" containerName="manager"
Sep 09 20:22:53.709096 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.709045    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" containerName="manager"
Sep 09 20:22:53.709151 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.709141    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bd224cb-19d2-4bef-bd91-6cc9f14810cf" containerName="manager"
Sep 09 20:22:53.712721 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.712693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.715752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.715720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rtp2h\""
Sep 09 20:22:53.715752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.715747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:22:53.715944 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.715767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:22:53.719831 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.719801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-m5ssq"]
Sep 09 20:22:53.875443 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.875388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqjd\" (UniqueName: \"kubernetes.io/projected/0d270fc9-177e-42c9-82e8-9469928913f0-kube-api-access-ndqjd\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.875637 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.875476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-metrics-certs\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.875637 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.875537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-cert\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.976592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.976499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-metrics-certs\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.976592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.976561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-cert\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.976797 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.976604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqjd\" (UniqueName: \"kubernetes.io/projected/0d270fc9-177e-42c9-82e8-9469928913f0-kube-api-access-ndqjd\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.979188 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.979159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-cert\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.979327 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.979209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d270fc9-177e-42c9-82e8-9469928913f0-metrics-certs\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:53.989437 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:53.989398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqjd\" (UniqueName: \"kubernetes.io/projected/0d270fc9-177e-42c9-82e8-9469928913f0-kube-api-access-ndqjd\") pod \"odh-model-controller-7b966d7984-m5ssq\" (UID: \"0d270fc9-177e-42c9-82e8-9469928913f0\") " pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:54.027517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:54.027470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:54.304654 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:54.304625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-m5ssq"]
Sep 09 20:22:54.306827 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:22:54.306783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d270fc9_177e_42c9_82e8_9469928913f0.slice/crio-131c199c49229416894d8e00797a0a99911045d795fa8d29cce35f1b55ff0fd6 WatchSource:0}: Error finding container 131c199c49229416894d8e00797a0a99911045d795fa8d29cce35f1b55ff0fd6: Status 404 returned error can't find the container with id 131c199c49229416894d8e00797a0a99911045d795fa8d29cce35f1b55ff0fd6
Sep 09 20:22:54.415141 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:54.415100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-m5ssq" event={"ID":"0d270fc9-177e-42c9-82e8-9469928913f0","Type":"ContainerStarted","Data":"131c199c49229416894d8e00797a0a99911045d795fa8d29cce35f1b55ff0fd6"}
Sep 09 20:22:57.433845 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:57.433808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-m5ssq" event={"ID":"0d270fc9-177e-42c9-82e8-9469928913f0","Type":"ContainerStarted","Data":"2c642655c40dae38482deb0cbb1b2cb1124f533b16c0d728dab644aa62c7d246"}
Sep 09 20:22:57.434348 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:57.433922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:22:57.450123 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:22:57.450060    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-m5ssq" podStartSLOduration=1.873170679 podStartE2EDuration="4.449991163s" totalImagesPullingTime="2.576820484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:57.449500113 +0000 UTC m=+698.336090042" watchObservedRunningTime="2026-09-09 20:22:57.449991163 +0000 UTC m=+698.336581072"
Sep 09 20:23:08.441203 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:23:08.441167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-m5ssq"
Sep 09 20:24:41.625769 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.625681    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:24:41.629823 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.629795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.632818 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.632791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:24:41.632968 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.632791    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-510d5-predictor-serving-cert\""
Sep 09 20:24:41.634138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.633754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:24:41.634138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.633783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4tmvf\""
Sep 09 20:24:41.634138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.633797    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-510d5-kube-rbac-proxy-sar-config\""
Sep 09 20:24:41.641462 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.641433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:24:41.722483 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.722446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.722688 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.722496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.722688 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.722653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttk68\" (UniqueName: \"kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.823314 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.823277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.823581 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.823562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.823731 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.823717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttk68\" (UniqueName: \"kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.825063 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:24:41.825013    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-510d5-predictor-serving-cert: secret "error-404-isvc-510d5-predictor-serving-cert" not found
Sep 09 20:24:41.825298 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:24:41.825287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls podName:4d076f53-441c-4eb3-9d76-11f8352cd9b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:42.32525376 +0000 UTC m=+803.211843653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls") pod "error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" (UID: "4d076f53-441c-4eb3-9d76-11f8352cd9b6") : secret "error-404-isvc-510d5-predictor-serving-cert" not found
Sep 09 20:24:41.826711 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.826683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:41.833974 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:41.833935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttk68\" (UniqueName: \"kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:42.328680 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:42.328638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:42.331308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:42.331278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") pod \"error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") " pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:42.546103 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:42.546050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:42.818733 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:42.818702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:24:42.821617 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:24:42.821579    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d076f53_441c_4eb3_9d76_11f8352cd9b6.slice/crio-74cde35b9043f13b7d129e02de1f660c86dff2ed3f7c84c7c4dbd61ff4d3996a WatchSource:0}: Error finding container 74cde35b9043f13b7d129e02de1f660c86dff2ed3f7c84c7c4dbd61ff4d3996a: Status 404 returned error can't find the container with id 74cde35b9043f13b7d129e02de1f660c86dff2ed3f7c84c7c4dbd61ff4d3996a
Sep 09 20:24:42.940618 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:42.940522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerStarted","Data":"74cde35b9043f13b7d129e02de1f660c86dff2ed3f7c84c7c4dbd61ff4d3996a"}
Sep 09 20:24:55.015785 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:55.015747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerStarted","Data":"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"}
Sep 09 20:24:58.041813 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:58.041768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerStarted","Data":"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"}
Sep 09 20:24:58.042246 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:58.041869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:58.042246 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:58.041902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:24:58.043474 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:58.043439    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:24:58.059865 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:58.059806    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podStartSLOduration=2.622002187 podStartE2EDuration="17.059787413s" totalImagesPullingTime="14.437785226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:58.0584074 +0000 UTC m=+818.944997344" watchObservedRunningTime="2026-09-09 20:24:58.059787413 +0000 UTC m=+818.946377322"
Sep 09 20:24:59.047136 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:24:59.047097    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:04.051323 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:04.051289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:25:04.051753 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:04.051724    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:14.052669 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:14.052622    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:24.052244 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:24.052193    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:34.052432 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:34.052381    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:44.053154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:25:44.053120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:26:11.396169 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.396126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:26:11.396755 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.396373    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" containerID="cri-o://9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5" gracePeriod=30
Sep 09 20:26:11.396755 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.396431    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kube-rbac-proxy" containerID="cri-o://08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0" gracePeriod=30
Sep 09 20:26:11.572068 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.572011    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:26:11.576210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.576176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.578821 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.578793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e273-kube-rbac-proxy-sar-config\""
Sep 09 20:26:11.578966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.578791    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e273-predictor-serving-cert\""
Sep 09 20:26:11.584396 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.584367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:26:11.708903 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.708809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.708903 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.708860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.708903 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.708896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd45s\" (UniqueName: \"kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.809855 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.809813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.810083 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.809873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd45s\" (UniqueName: \"kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.810083 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.809925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.810592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.810568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.812786 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.812756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.818094 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.818066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd45s\" (UniqueName: \"kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s\") pod \"error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:11.889738 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:11.889691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:12.164115 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.164085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:26:12.166410 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:26:12.166365    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d9c6fa_303f_490f_a07a_70a48e0d5ea4.slice/crio-548fa8d2547d6058810d77c81a06676ac2f46cb3b7ee7599b0879c97f5b6ada9 WatchSource:0}: Error finding container 548fa8d2547d6058810d77c81a06676ac2f46cb3b7ee7599b0879c97f5b6ada9: Status 404 returned error can't find the container with id 548fa8d2547d6058810d77c81a06676ac2f46cb3b7ee7599b0879c97f5b6ada9
Sep 09 20:26:12.407809 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.407767    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerID="08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0" exitCode=2
Sep 09 20:26:12.408277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.407841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerDied","Data":"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"}
Sep 09 20:26:12.410546 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.410508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerStarted","Data":"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"}
Sep 09 20:26:12.410727 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.410583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerStarted","Data":"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"}
Sep 09 20:26:12.410727 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.410614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerStarted","Data":"548fa8d2547d6058810d77c81a06676ac2f46cb3b7ee7599b0879c97f5b6ada9"}
Sep 09 20:26:12.410945 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.410774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:12.428568 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:12.428448    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podStartSLOduration=1.428428364 podStartE2EDuration="1.428428364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:12.426900532 +0000 UTC m=+893.313490460" watchObservedRunningTime="2026-09-09 20:26:12.428428364 +0000 UTC m=+893.315018274"
Sep 09 20:26:13.415095 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:13.415059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:13.416560 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:13.416528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:14.048277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:14.048233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 09 20:26:14.052144 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:14.052110    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:26:14.419680 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:14.419582    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:15.156702 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.156631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:26:15.345227 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.345187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") pod \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") "
Sep 09 20:26:15.345227 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.345230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttk68\" (UniqueName: \"kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68\") pod \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") "
Sep 09 20:26:15.345489 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.345373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config\") pod \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\" (UID: \"4d076f53-441c-4eb3-9d76-11f8352cd9b6\") "
Sep 09 20:26:15.345701 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.345668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d076f53-441c-4eb3-9d76-11f8352cd9b6" (UID: "4d076f53-441c-4eb3-9d76-11f8352cd9b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:15.347679 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.347652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d076f53-441c-4eb3-9d76-11f8352cd9b6" (UID: "4d076f53-441c-4eb3-9d76-11f8352cd9b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:15.347813 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.347699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68" (OuterVolumeSpecName: "kube-api-access-ttk68") pod "4d076f53-441c-4eb3-9d76-11f8352cd9b6" (UID: "4d076f53-441c-4eb3-9d76-11f8352cd9b6"). InnerVolumeSpecName "kube-api-access-ttk68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:15.425863 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.425774    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerID="9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5" exitCode=0
Sep 09 20:26:15.425863 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.425826    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"
Sep 09 20:26:15.426343 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.425860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerDied","Data":"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"}
Sep 09 20:26:15.426343 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.425928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l" event={"ID":"4d076f53-441c-4eb3-9d76-11f8352cd9b6","Type":"ContainerDied","Data":"74cde35b9043f13b7d129e02de1f660c86dff2ed3f7c84c7c4dbd61ff4d3996a"}
Sep 09 20:26:15.426343 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.425963    2577 scope.go:117] "RemoveContainer" containerID="08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"
Sep 09 20:26:15.435953 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.435928    2577 scope.go:117] "RemoveContainer" containerID="9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"
Sep 09 20:26:15.444660 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.444629    2577 scope.go:117] "RemoveContainer" containerID="08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"
Sep 09 20:26:15.445069 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:26:15.445043    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0\": container with ID starting with 08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0 not found: ID does not exist" containerID="08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"
Sep 09 20:26:15.445138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.445087    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0"} err="failed to get container status \"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0\": rpc error: code = NotFound desc = could not find container \"08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0\": container with ID starting with 08d0aae85926f1f500cc9face3e9224e043a71fe8b0934a278a3c7fedd1ebbc0 not found: ID does not exist"
Sep 09 20:26:15.445138 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.445108    2577 scope.go:117] "RemoveContainer" containerID="9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"
Sep 09 20:26:15.445597 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:26:15.445551    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5\": container with ID starting with 9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5 not found: ID does not exist" containerID="9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"
Sep 09 20:26:15.445597 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.445589    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5"} err="failed to get container status \"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5\": rpc error: code = NotFound desc = could not find container \"9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5\": container with ID starting with 9e719b853d5b8c9300a4f712ec857b758d16a9608a0cf3e410c6c61f3b76e6a5 not found: ID does not exist"
Sep 09 20:26:15.446290 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.446264    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:26:15.446290 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.446292    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d076f53-441c-4eb3-9d76-11f8352cd9b6-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:26:15.446435 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.446303    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttk68\" (UniqueName: \"kubernetes.io/projected/4d076f53-441c-4eb3-9d76-11f8352cd9b6-kube-api-access-ttk68\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:26:15.447875 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.447855    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:26:15.453982 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.453953    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-510d5-predictor-5ccb9788d8-cqw5l"]
Sep 09 20:26:15.621080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:15.621013    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" path="/var/lib/kubelet/pods/4d076f53-441c-4eb3-9d76-11f8352cd9b6/volumes"
Sep 09 20:26:19.424410 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:19.424324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:26:19.425066 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:19.425011    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:29.425315 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:29.425264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:39.425855 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:39.425807    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:49.425243 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:49.425198    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:59.425201 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:26:59.425164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:27:01.557078 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557036    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:27:01.557453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557400    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kube-rbac-proxy"
Sep 09 20:27:01.557453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557413    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kube-rbac-proxy"
Sep 09 20:27:01.557453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557424    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container"
Sep 09 20:27:01.557453 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557430    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container"
Sep 09 20:27:01.557582 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557506    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kube-rbac-proxy"
Sep 09 20:27:01.557582 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.557517    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d076f53-441c-4eb3-9d76-11f8352cd9b6" containerName="kserve-container"
Sep 09 20:27:01.560825 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.560803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.563798 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.563756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53cb1-predictor-serving-cert\""
Sep 09 20:27:01.563798 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.563789    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53cb1-kube-rbac-proxy-sar-config\""
Sep 09 20:27:01.570234 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.570204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:27:01.639154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.639120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.639154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.639164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7g9\" (UniqueName: \"kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.639418 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.639259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.740587 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.740539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7g9\" (UniqueName: \"kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.740802 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.740607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.740802 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.740664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.741576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.741543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.743272 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.743249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.749828 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.749798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7g9\" (UniqueName: \"kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9\") pod \"error-404-isvc-53cb1-predictor-7578994fd6-m8xvq\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") " pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:01.873665 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:01.873572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:02.150428 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.150394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:27:02.153124 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:27:02.153094    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2101f46d_8265_461b_af6d_f5b24632d9b9.slice/crio-7a5ee108df44d55e4724991b785923e552587e5124f8c4617403aca098a2c32b WatchSource:0}: Error finding container 7a5ee108df44d55e4724991b785923e552587e5124f8c4617403aca098a2c32b: Status 404 returned error can't find the container with id 7a5ee108df44d55e4724991b785923e552587e5124f8c4617403aca098a2c32b
Sep 09 20:27:02.654695 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.654648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerStarted","Data":"c2ff64b6bd10227dd99528d2c71480d3cf5203a3a9b63babcc8ceae3e50ae602"}
Sep 09 20:27:02.655140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.654719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerStarted","Data":"653c6c1af8a8ef88c56b4839faabb8856e99bd63340b8e31c5637542efa5fb08"}
Sep 09 20:27:02.655140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.654739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerStarted","Data":"7a5ee108df44d55e4724991b785923e552587e5124f8c4617403aca098a2c32b"}
Sep 09 20:27:02.655140 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.654761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:02.672907 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:02.672846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podStartSLOduration=1.6728278300000001 podStartE2EDuration="1.67282783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:02.670848772 +0000 UTC m=+943.557438699" watchObservedRunningTime="2026-09-09 20:27:02.67282783 +0000 UTC m=+943.559417740"
Sep 09 20:27:03.665279 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:03.665236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:03.666107 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:03.665948    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:04.667332 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:04.667278    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:09.672363 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:09.672332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:27:09.672857 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:09.672828    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:19.673811 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:19.673774    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:29.673762 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:29.673720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:39.673594 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:39.673548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:49.674051 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:27:49.673951    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:35:26.321591 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.321507    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:35:26.322060 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.321756    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" containerID="cri-o://b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460" gracePeriod=30
Sep 09 20:35:26.322060 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.321863    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kube-rbac-proxy" containerID="cri-o://598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828" gracePeriod=30
Sep 09 20:35:26.405668 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.405633    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:35:26.409404 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.409385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.412563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.412540    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b3d7b-predictor-serving-cert\""
Sep 09 20:35:26.412921 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.412904    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b3d7b-kube-rbac-proxy-sar-config\""
Sep 09 20:35:26.429774 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.429744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:35:26.496348 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.496312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.496536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.496410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.496536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.496439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqz8z\" (UniqueName: \"kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.597318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.597228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.597318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.597275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqz8z\" (UniqueName: \"kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.597318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.597300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.597935 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.597908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.599990 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.599968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.606163 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.606139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqz8z\" (UniqueName: \"kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z\") pod \"error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") " pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.719955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.719919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:26.993319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:26.993294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:35:26.995580 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:35:26.995547    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fe67e5_60c4_4d79_b684_d8f424dda060.slice/crio-b81148ae3a494aa311b50dc5ca759a0f348170cb57475714990fb8275c528402 WatchSource:0}: Error finding container b81148ae3a494aa311b50dc5ca759a0f348170cb57475714990fb8275c528402: Status 404 returned error can't find the container with id b81148ae3a494aa311b50dc5ca759a0f348170cb57475714990fb8275c528402
Sep 09 20:35:27.079667 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:27.079635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerStarted","Data":"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"}
Sep 09 20:35:27.079757 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:27.079700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerStarted","Data":"b81148ae3a494aa311b50dc5ca759a0f348170cb57475714990fb8275c528402"}
Sep 09 20:35:27.081426 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:27.081404    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerID="598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828" exitCode=2
Sep 09 20:35:27.081506 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:27.081477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerDied","Data":"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"}
Sep 09 20:35:28.087671 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:28.087630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerStarted","Data":"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"}
Sep 09 20:35:28.088111 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:28.087713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:28.106820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:28.106751    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podStartSLOduration=2.106730457 podStartE2EDuration="2.106730457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:28.106729869 +0000 UTC m=+1448.993319798" watchObservedRunningTime="2026-09-09 20:35:28.106730457 +0000 UTC m=+1448.993320585"
Sep 09 20:35:29.091989 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.091956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:29.093572 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.093541    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:35:29.420947 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.420839    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 09 20:35:29.425310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.425281    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:35:29.672521 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.672468    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:35:29.723877 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.723839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd45s\" (UniqueName: \"kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s\") pod \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") "
Sep 09 20:35:29.724080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.723913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config\") pod \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") "
Sep 09 20:35:29.724080 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.723943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls\") pod \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\" (UID: \"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4\") "
Sep 09 20:35:29.724302 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.724270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" (UID: "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:29.726097 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.726069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s" (OuterVolumeSpecName: "kube-api-access-vd45s") pod "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" (UID: "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4"). InnerVolumeSpecName "kube-api-access-vd45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:29.726210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.726170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" (UID: "e5d9c6fa-303f-490f-a07a-70a48e0d5ea4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:29.824931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.824899    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:35:29.824931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.824927    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:35:29.825117 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:29.824940    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd45s\" (UniqueName: \"kubernetes.io/projected/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4-kube-api-access-vd45s\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:35:30.097869 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.097836    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerID="b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460" exitCode=0
Sep 09 20:35:30.098344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.097887    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"
Sep 09 20:35:30.098344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.097917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerDied","Data":"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"}
Sep 09 20:35:30.098344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.097988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2" event={"ID":"e5d9c6fa-303f-490f-a07a-70a48e0d5ea4","Type":"ContainerDied","Data":"548fa8d2547d6058810d77c81a06676ac2f46cb3b7ee7599b0879c97f5b6ada9"}
Sep 09 20:35:30.098344 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.098049    2577 scope.go:117] "RemoveContainer" containerID="598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"
Sep 09 20:35:30.098621 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.098346    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:35:30.107643 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.107444    2577 scope.go:117] "RemoveContainer" containerID="b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"
Sep 09 20:35:30.115729 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.115712    2577 scope.go:117] "RemoveContainer" containerID="598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"
Sep 09 20:35:30.116010 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:35:30.115993    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828\": container with ID starting with 598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828 not found: ID does not exist" containerID="598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"
Sep 09 20:35:30.116110 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.116032    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828"} err="failed to get container status \"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828\": rpc error: code = NotFound desc = could not find container \"598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828\": container with ID starting with 598c1e83d6e7f15548ea0407de711b44eec728cdfaa571f014af42c2dcf97828 not found: ID does not exist"
Sep 09 20:35:30.116110 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.116056    2577 scope.go:117] "RemoveContainer" containerID="b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"
Sep 09 20:35:30.116326 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:35:30.116309    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460\": container with ID starting with b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460 not found: ID does not exist" containerID="b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"
Sep 09 20:35:30.116361 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.116332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460"} err="failed to get container status \"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460\": rpc error: code = NotFound desc = could not find container \"b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460\": container with ID starting with b70020e84981fe4f72e14a4e02e82751472970f976aefdb61d4ccbe583515460 not found: ID does not exist"
Sep 09 20:35:30.120259 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.120239    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:35:30.124221 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:30.124203    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e273-predictor-6db6bf4f6-6q6b2"]
Sep 09 20:35:31.622308 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:31.622275    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" path="/var/lib/kubelet/pods/e5d9c6fa-303f-490f-a07a-70a48e0d5ea4/volumes"
Sep 09 20:35:35.103153 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:35.103121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:35:35.103652 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:35.103618    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:35:45.104217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:45.104176    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:35:55.104683 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:35:55.104636    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:36:05.103979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:05.103937    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:36:15.105178 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:15.105148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:36:16.205154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.205121    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:36:16.205565 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.205372    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container" containerID="cri-o://653c6c1af8a8ef88c56b4839faabb8856e99bd63340b8e31c5637542efa5fb08" gracePeriod=30
Sep 09 20:36:16.205565 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.205432    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kube-rbac-proxy" containerID="cri-o://c2ff64b6bd10227dd99528d2c71480d3cf5203a3a9b63babcc8ceae3e50ae602" gracePeriod=30
Sep 09 20:36:16.386584 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.386543    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:36:16.387180 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kube-rbac-proxy"
Sep 09 20:36:16.387243 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387183    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kube-rbac-proxy"
Sep 09 20:36:16.387243 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387222    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container"
Sep 09 20:36:16.387243 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387230    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container"
Sep 09 20:36:16.387346 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387308    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kserve-container"
Sep 09 20:36:16.387346 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.387325    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d9c6fa-303f-490f-a07a-70a48e0d5ea4" containerName="kube-rbac-proxy"
Sep 09 20:36:16.391014 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.390992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.393813 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.393784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-90077-kube-rbac-proxy-sar-config\""
Sep 09 20:36:16.393982 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.393900    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-90077-predictor-serving-cert\""
Sep 09 20:36:16.398981 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.398952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:36:16.439978 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.439932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rghg\" (UniqueName: \"kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.440200 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.440064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.440279 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.440205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.541330 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.541296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.541536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.541345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rghg\" (UniqueName: \"kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.541536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.541474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.542258 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.542234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.544247 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.544223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.554814 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.554792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rghg\" (UniqueName: \"kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg\") pod \"error-404-isvc-90077-predictor-7767f7d476-wsrwq\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") " pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.704417 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.704371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:16.976658 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:16.976630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:36:16.978710 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:36:16.978676    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd51f79_73b9_4af0_8f79_d50c7b74f377.slice/crio-677cf0c471c547736b640db4ce4d3846f81e283b009b9ce8ccbcd82f047ff25e WatchSource:0}: Error finding container 677cf0c471c547736b640db4ce4d3846f81e283b009b9ce8ccbcd82f047ff25e: Status 404 returned error can't find the container with id 677cf0c471c547736b640db4ce4d3846f81e283b009b9ce8ccbcd82f047ff25e
Sep 09 20:36:17.322682 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.322643    2577 generic.go:356] "Generic (PLEG): container finished" podID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerID="c2ff64b6bd10227dd99528d2c71480d3cf5203a3a9b63babcc8ceae3e50ae602" exitCode=2
Sep 09 20:36:17.323109 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.322724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerDied","Data":"c2ff64b6bd10227dd99528d2c71480d3cf5203a3a9b63babcc8ceae3e50ae602"}
Sep 09 20:36:17.324641 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.324618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerStarted","Data":"8cfb7bb4eb30bee55e672b8d4b07fc13b7f89b52bdbdb409e6ff5caa1a2ced3e"}
Sep 09 20:36:17.324768 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.324665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerStarted","Data":"8e10eb2473232f05002748e361df3ab74af580e5ed6faae057665d5d09224b77"}
Sep 09 20:36:17.324768 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.324686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerStarted","Data":"677cf0c471c547736b640db4ce4d3846f81e283b009b9ce8ccbcd82f047ff25e"}
Sep 09 20:36:17.324768 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.324724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:17.341767 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:17.341722    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podStartSLOduration=1.341708504 podStartE2EDuration="1.341708504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:17.340341525 +0000 UTC m=+1498.226931480" watchObservedRunningTime="2026-09-09 20:36:17.341708504 +0000 UTC m=+1498.228298415"
Sep 09 20:36:18.329182 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:18.329141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:18.330507 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:18.330476    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:19.337175 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.337146    2577 generic.go:356] "Generic (PLEG): container finished" podID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerID="653c6c1af8a8ef88c56b4839faabb8856e99bd63340b8e31c5637542efa5fb08" exitCode=0
Sep 09 20:36:19.337565 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.337212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerDied","Data":"653c6c1af8a8ef88c56b4839faabb8856e99bd63340b8e31c5637542efa5fb08"}
Sep 09 20:36:19.337565 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.337540    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:19.458798 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.458776    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:36:19.568551 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.568471    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls\") pod \"2101f46d-8265-461b-af6d-f5b24632d9b9\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") "
Sep 09 20:36:19.568551 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.568521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7g9\" (UniqueName: \"kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9\") pod \"2101f46d-8265-461b-af6d-f5b24632d9b9\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") "
Sep 09 20:36:19.568551 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.568541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config\") pod \"2101f46d-8265-461b-af6d-f5b24632d9b9\" (UID: \"2101f46d-8265-461b-af6d-f5b24632d9b9\") "
Sep 09 20:36:19.568922 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.568894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2101f46d-8265-461b-af6d-f5b24632d9b9" (UID: "2101f46d-8265-461b-af6d-f5b24632d9b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:19.570531 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.570506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2101f46d-8265-461b-af6d-f5b24632d9b9" (UID: "2101f46d-8265-461b-af6d-f5b24632d9b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:19.570621 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.570584    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9" (OuterVolumeSpecName: "kube-api-access-8c7g9") pod "2101f46d-8265-461b-af6d-f5b24632d9b9" (UID: "2101f46d-8265-461b-af6d-f5b24632d9b9"). InnerVolumeSpecName "kube-api-access-8c7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:19.669217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.669178    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2101f46d-8265-461b-af6d-f5b24632d9b9-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.669217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.669210    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7g9\" (UniqueName: \"kubernetes.io/projected/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-api-access-8c7g9\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.669217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.669222    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2101f46d-8265-461b-af6d-f5b24632d9b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.709460 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.709427    2577 scope.go:117] "RemoveContainer" containerID="c2ff64b6bd10227dd99528d2c71480d3cf5203a3a9b63babcc8ceae3e50ae602"
Sep 09 20:36:19.718011 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:19.717994    2577 scope.go:117] "RemoveContainer" containerID="653c6c1af8a8ef88c56b4839faabb8856e99bd63340b8e31c5637542efa5fb08"
Sep 09 20:36:20.341493 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:20.341462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"
Sep 09 20:36:20.341996 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:20.341454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq" event={"ID":"2101f46d-8265-461b-af6d-f5b24632d9b9","Type":"ContainerDied","Data":"7a5ee108df44d55e4724991b785923e552587e5124f8c4617403aca098a2c32b"}
Sep 09 20:36:20.369419 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:20.369383    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:36:20.373460 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:20.373430    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53cb1-predictor-7578994fd6-m8xvq"]
Sep 09 20:36:21.620781 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:21.620743    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" path="/var/lib/kubelet/pods/2101f46d-8265-461b-af6d-f5b24632d9b9/volumes"
Sep 09 20:36:24.342364 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:24.342327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:36:24.342820 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:24.342791    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:34.343658 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:34.343617    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:36.485563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.485527    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:36:36.486062 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.485849    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container" containerID="cri-o://0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc" gracePeriod=30
Sep 09 20:36:36.486208 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.486086    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kube-rbac-proxy" containerID="cri-o://df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134" gracePeriod=30
Sep 09 20:36:36.663570 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.663537    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:36:36.663938 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.663925    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kube-rbac-proxy"
Sep 09 20:36:36.663938 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.663939    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kube-rbac-proxy"
Sep 09 20:36:36.664037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.663962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container"
Sep 09 20:36:36.664037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.663968    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container"
Sep 09 20:36:36.664037 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.664033    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kserve-container"
Sep 09 20:36:36.664139 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.664047    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2101f46d-8265-461b-af6d-f5b24632d9b9" containerName="kube-rbac-proxy"
Sep 09 20:36:36.667298 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.667278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.669850 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.669822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-81279-predictor-serving-cert\""
Sep 09 20:36:36.669972 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.669897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-81279-kube-rbac-proxy-sar-config\""
Sep 09 20:36:36.677151 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.677129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:36:36.821472 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.821431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248tb\" (UniqueName: \"kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.821632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.821493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.821632 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.821604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.922082 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.922035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248tb\" (UniqueName: \"kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.922252 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.922115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.922252 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.922174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.922386 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:36:36.922291    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-81279-predictor-serving-cert: secret "error-404-isvc-81279-predictor-serving-cert" not found
Sep 09 20:36:36.922386 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:36:36.922352    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls podName:f224ccfc-8bb2-436f-8deb-2c370420fef8 nodeName:}" failed. No retries permitted until 2026-09-09 20:36:37.422331352 +0000 UTC m=+1518.308921254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls") pod "error-404-isvc-81279-predictor-84575cb8cb-pllk2" (UID: "f224ccfc-8bb2-436f-8deb-2c370420fef8") : secret "error-404-isvc-81279-predictor-serving-cert" not found
Sep 09 20:36:36.922844 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.922817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:36.931529 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:36.931506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-248tb\" (UniqueName: \"kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:37.424205 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.424165    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerID="df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134" exitCode=2
Sep 09 20:36:37.424392 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.424246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerDied","Data":"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"}
Sep 09 20:36:37.426586 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.426562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:37.428966 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.428948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") pod \"error-404-isvc-81279-predictor-84575cb8cb-pllk2\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") " pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:37.579806 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.579769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:37.856141 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:37.856117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:36:37.857748 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:36:37.857718    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf224ccfc_8bb2_436f_8deb_2c370420fef8.slice/crio-a919bbb5d64a5011cc78e19ace724a0d18d8c8e4d5b156de9aa3c1ebf86cdb12 WatchSource:0}: Error finding container a919bbb5d64a5011cc78e19ace724a0d18d8c8e4d5b156de9aa3c1ebf86cdb12: Status 404 returned error can't find the container with id a919bbb5d64a5011cc78e19ace724a0d18d8c8e4d5b156de9aa3c1ebf86cdb12
Sep 09 20:36:38.431150 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.431107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerStarted","Data":"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"}
Sep 09 20:36:38.431342 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.431259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:38.431417 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.431398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:38.431487 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.431447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerStarted","Data":"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"}
Sep 09 20:36:38.431487 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.431470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerStarted","Data":"a919bbb5d64a5011cc78e19ace724a0d18d8c8e4d5b156de9aa3c1ebf86cdb12"}
Sep 09 20:36:38.432752 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.432727    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:36:38.447930 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:38.447879    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podStartSLOduration=2.447862101 podStartE2EDuration="2.447862101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:38.447200626 +0000 UTC m=+1519.333790560" watchObservedRunningTime="2026-09-09 20:36:38.447862101 +0000 UTC m=+1519.334452012"
Sep 09 20:36:39.436381 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.436343    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:36:39.850113 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.850088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:36:39.947402 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.947354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls\") pod \"a2fe67e5-60c4-4d79-b684-d8f424dda060\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") "
Sep 09 20:36:39.947571 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.947426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config\") pod \"a2fe67e5-60c4-4d79-b684-d8f424dda060\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") "
Sep 09 20:36:39.947571 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.947454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqz8z\" (UniqueName: \"kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z\") pod \"a2fe67e5-60c4-4d79-b684-d8f424dda060\" (UID: \"a2fe67e5-60c4-4d79-b684-d8f424dda060\") "
Sep 09 20:36:39.947790 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.947764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2fe67e5-60c4-4d79-b684-d8f424dda060" (UID: "a2fe67e5-60c4-4d79-b684-d8f424dda060"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:39.949580 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.949517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2fe67e5-60c4-4d79-b684-d8f424dda060" (UID: "a2fe67e5-60c4-4d79-b684-d8f424dda060"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:39.949685 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:39.949662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z" (OuterVolumeSpecName: "kube-api-access-lqz8z") pod "a2fe67e5-60c4-4d79-b684-d8f424dda060" (UID: "a2fe67e5-60c4-4d79-b684-d8f424dda060"). InnerVolumeSpecName "kube-api-access-lqz8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:40.048594 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.048556    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2fe67e5-60c4-4d79-b684-d8f424dda060-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:40.048594 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.048589    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:40.048594 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.048601    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqz8z\" (UniqueName: \"kubernetes.io/projected/a2fe67e5-60c4-4d79-b684-d8f424dda060-kube-api-access-lqz8z\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:36:40.442572 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.442532    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerID="0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc" exitCode=0
Sep 09 20:36:40.443065 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.442587    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"
Sep 09 20:36:40.443065 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.442621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerDied","Data":"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"}
Sep 09 20:36:40.443065 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.442709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt" event={"ID":"a2fe67e5-60c4-4d79-b684-d8f424dda060","Type":"ContainerDied","Data":"b81148ae3a494aa311b50dc5ca759a0f348170cb57475714990fb8275c528402"}
Sep 09 20:36:40.443065 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.442754    2577 scope.go:117] "RemoveContainer" containerID="df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"
Sep 09 20:36:40.451555 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.451537    2577 scope.go:117] "RemoveContainer" containerID="0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"
Sep 09 20:36:40.459330 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.459305    2577 scope.go:117] "RemoveContainer" containerID="df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"
Sep 09 20:36:40.459638 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:36:40.459615    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134\": container with ID starting with df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134 not found: ID does not exist" containerID="df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"
Sep 09 20:36:40.459692 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.459648    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134"} err="failed to get container status \"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134\": rpc error: code = NotFound desc = could not find container \"df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134\": container with ID starting with df110ea08fb4dabb51d6ea752a38752ba8a99b698b5d6d4b4302acf9730ce134 not found: ID does not exist"
Sep 09 20:36:40.459692 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.459667    2577 scope.go:117] "RemoveContainer" containerID="0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"
Sep 09 20:36:40.459917 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:36:40.459902    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc\": container with ID starting with 0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc not found: ID does not exist" containerID="0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"
Sep 09 20:36:40.459954 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.459924    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc"} err="failed to get container status \"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc\": rpc error: code = NotFound desc = could not find container \"0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc\": container with ID starting with 0744316a90da72c1610647deb05d2a0a7f1ff666251b3c91223ad98dfeb6c9cc not found: ID does not exist"
Sep 09 20:36:40.463846 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.463822    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:36:40.467887 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:40.467861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3d7b-predictor-544b6dfd74-mbzlt"]
Sep 09 20:36:41.621600 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:41.621526    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" path="/var/lib/kubelet/pods/a2fe67e5-60c4-4d79-b684-d8f424dda060/volumes"
Sep 09 20:36:44.342771 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:44.342731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:44.440704 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:44.440672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:36:44.441154 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:44.441122    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:36:54.343563 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:54.343522    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:36:54.441271 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:36:54.441225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:37:04.343195 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:04.343159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:37:04.441446 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:04.441403    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:37:14.441137 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:14.441086    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:37:24.442187 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:24.442155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:37:26.475352 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.475315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:37:26.475869 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.475579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container" containerID="cri-o://8e10eb2473232f05002748e361df3ab74af580e5ed6faae057665d5d09224b77" gracePeriod=30
Sep 09 20:37:26.475869 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.475621    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kube-rbac-proxy" containerID="cri-o://8cfb7bb4eb30bee55e672b8d4b07fc13b7f89b52bdbdb409e6ff5caa1a2ced3e" gracePeriod=30
Sep 09 20:37:26.632490 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.632452    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:37:26.632930 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.632911    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kube-rbac-proxy"
Sep 09 20:37:26.633064 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.632934    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kube-rbac-proxy"
Sep 09 20:37:26.633064 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.632964    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container"
Sep 09 20:37:26.633064 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.632974    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container"
Sep 09 20:37:26.633235 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.633093    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kube-rbac-proxy"
Sep 09 20:37:26.633235 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.633109    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2fe67e5-60c4-4d79-b684-d8f424dda060" containerName="kserve-container"
Sep 09 20:37:26.636445 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.636422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.639236 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.639206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-28e88-predictor-serving-cert\""
Sep 09 20:37:26.639236 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.639224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-28e88-kube-rbac-proxy-sar-config\""
Sep 09 20:37:26.644732 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.644701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:37:26.678827 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.678783    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerID="8cfb7bb4eb30bee55e672b8d4b07fc13b7f89b52bdbdb409e6ff5caa1a2ced3e" exitCode=2
Sep 09 20:37:26.679007 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.678865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerDied","Data":"8cfb7bb4eb30bee55e672b8d4b07fc13b7f89b52bdbdb409e6ff5caa1a2ced3e"}
Sep 09 20:37:26.711756 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.711708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.711964 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.711874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.711964 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.711951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fjm\" (UniqueName: \"kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.812772 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.812732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.812979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.812792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fjm\" (UniqueName: \"kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.812979 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.812858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.812979 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:37:26.812901    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-28e88-predictor-serving-cert: secret "error-404-isvc-28e88-predictor-serving-cert" not found
Sep 09 20:37:26.813181 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:37:26.812991    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls podName:0586df93-682b-4686-9b9c-d6b03fae9893 nodeName:}" failed. No retries permitted until 2026-09-09 20:37:27.312967714 +0000 UTC m=+1568.199557602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls") pod "error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" (UID: "0586df93-682b-4686-9b9c-d6b03fae9893") : secret "error-404-isvc-28e88-predictor-serving-cert" not found
Sep 09 20:37:26.813637 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.813617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:26.821781 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:26.821757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fjm\" (UniqueName: \"kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:27.318423 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:27.318382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:27.320837 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:27.320802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") pod \"error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") " pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:27.549288 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:27.549252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:27.822528 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:27.822491    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:37:27.824542 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:37:27.824503    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0586df93_682b_4686_9b9c_d6b03fae9893.slice/crio-5e680a4129a9f589d81b7b888c7a792d074cee441928e47b8d6d71c3a28722e1 WatchSource:0}: Error finding container 5e680a4129a9f589d81b7b888c7a792d074cee441928e47b8d6d71c3a28722e1: Status 404 returned error can't find the container with id 5e680a4129a9f589d81b7b888c7a792d074cee441928e47b8d6d71c3a28722e1
Sep 09 20:37:28.691149 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:28.691113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerStarted","Data":"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"}
Sep 09 20:37:28.691559 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:28.691184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerStarted","Data":"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"}
Sep 09 20:37:28.691559 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:28.691218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerStarted","Data":"5e680a4129a9f589d81b7b888c7a792d074cee441928e47b8d6d71c3a28722e1"}
Sep 09 20:37:28.691559 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:28.691253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:28.709389 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:28.709323    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podStartSLOduration=2.709305108 podStartE2EDuration="2.709305108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:28.706677509 +0000 UTC m=+1569.593267434" watchObservedRunningTime="2026-09-09 20:37:28.709305108 +0000 UTC m=+1569.595895017"
Sep 09 20:37:29.337809 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.337768    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:37:29.697891 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.697858    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerID="8e10eb2473232f05002748e361df3ab74af580e5ed6faae057665d5d09224b77" exitCode=0
Sep 09 20:37:29.698416 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.697927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerDied","Data":"8e10eb2473232f05002748e361df3ab74af580e5ed6faae057665d5d09224b77"}
Sep 09 20:37:29.698416 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.698219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:29.699514 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.699489    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:37:29.836162 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.836133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:37:29.940918 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.940810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config\") pod \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") "
Sep 09 20:37:29.940918 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.940860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls\") pod \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") "
Sep 09 20:37:29.940918 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.940894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rghg\" (UniqueName: \"kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg\") pod \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\" (UID: \"5bd51f79-73b9-4af0-8f79-d50c7b74f377\") "
Sep 09 20:37:29.941320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.941289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bd51f79-73b9-4af0-8f79-d50c7b74f377" (UID: "5bd51f79-73b9-4af0-8f79-d50c7b74f377"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:29.943217 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.943183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bd51f79-73b9-4af0-8f79-d50c7b74f377" (UID: "5bd51f79-73b9-4af0-8f79-d50c7b74f377"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:29.943328 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:29.943217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg" (OuterVolumeSpecName: "kube-api-access-5rghg") pod "5bd51f79-73b9-4af0-8f79-d50c7b74f377" (UID: "5bd51f79-73b9-4af0-8f79-d50c7b74f377"). InnerVolumeSpecName "kube-api-access-5rghg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:30.042495 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.042452    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:37:30.042495 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.042488    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd51f79-73b9-4af0-8f79-d50c7b74f377-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:37:30.042495 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.042502    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rghg\" (UniqueName: \"kubernetes.io/projected/5bd51f79-73b9-4af0-8f79-d50c7b74f377-kube-api-access-5rghg\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:37:30.704156 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.704122    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"
Sep 09 20:37:30.704604 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.704160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq" event={"ID":"5bd51f79-73b9-4af0-8f79-d50c7b74f377","Type":"ContainerDied","Data":"677cf0c471c547736b640db4ce4d3846f81e283b009b9ce8ccbcd82f047ff25e"}
Sep 09 20:37:30.704604 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.704243    2577 scope.go:117] "RemoveContainer" containerID="8cfb7bb4eb30bee55e672b8d4b07fc13b7f89b52bdbdb409e6ff5caa1a2ced3e"
Sep 09 20:37:30.704931 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.704903    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:37:30.713340 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.713324    2577 scope.go:117] "RemoveContainer" containerID="8e10eb2473232f05002748e361df3ab74af580e5ed6faae057665d5d09224b77"
Sep 09 20:37:30.724932 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.724911    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:37:30.730256 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:30.730233    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-90077-predictor-7767f7d476-wsrwq"]
Sep 09 20:37:31.620976 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:31.620942    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" path="/var/lib/kubelet/pods/5bd51f79-73b9-4af0-8f79-d50c7b74f377/volumes"
Sep 09 20:37:35.708911 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:35.708875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:37:35.709457 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:35.709428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:37:45.710313 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:45.710269    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:37:55.710210 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:37:55.710163    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:38:05.710013 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:38:05.709952    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:38:15.710218 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:38:15.710136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:45:51.609481 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.609411    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:45:51.609963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.609858    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kube-rbac-proxy"
Sep 09 20:45:51.609963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.609877    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kube-rbac-proxy"
Sep 09 20:45:51.609963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.609906    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container"
Sep 09 20:45:51.609963 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.609915    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container"
Sep 09 20:45:51.610222 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.610005    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kserve-container"
Sep 09 20:45:51.610222 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.610037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd51f79-73b9-4af0-8f79-d50c7b74f377" containerName="kube-rbac-proxy"
Sep 09 20:45:51.613337 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.613316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.616148 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.616123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-326f5-predictor-serving-cert\""
Sep 09 20:45:51.616259 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.616130    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-326f5-kube-rbac-proxy-sar-config\""
Sep 09 20:45:51.626346 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.626321    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:45:51.643318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.643291    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:45:51.643578 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.643527    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container" containerID="cri-o://ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048" gracePeriod=30
Sep 09 20:45:51.643666 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.643550    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kube-rbac-proxy" containerID="cri-o://2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0" gracePeriod=30
Sep 09 20:45:51.702278 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.702248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.702376 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.702286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvd7w\" (UniqueName: \"kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.702427 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.702374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.803415 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.803390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.803533 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.803426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvd7w\" (UniqueName: \"kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.803533 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.803463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.803643 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:45:51.803621    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-326f5-predictor-serving-cert: secret "error-404-isvc-326f5-predictor-serving-cert" not found
Sep 09 20:45:51.803712 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:45:51.803701    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls podName:60fc950f-f50c-4f79-9417-be52f2740af7 nodeName:}" failed. No retries permitted until 2026-09-09 20:45:52.30368185 +0000 UTC m=+2073.190271746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls") pod "error-404-isvc-326f5-predictor-5f77687698-df8ms" (UID: "60fc950f-f50c-4f79-9417-be52f2740af7") : secret "error-404-isvc-326f5-predictor-serving-cert" not found
Sep 09 20:45:51.804003 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.803984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:51.812063 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:51.812041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvd7w\" (UniqueName: \"kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:52.035805 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.035779    2577 generic.go:356] "Generic (PLEG): container finished" podID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerID="2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0" exitCode=2
Sep 09 20:45:52.035940 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.035860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerDied","Data":"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"}
Sep 09 20:45:52.307336 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.307256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:52.309756 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.309732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") pod \"error-404-isvc-326f5-predictor-5f77687698-df8ms\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") " pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:52.528967 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.528938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:52.801302 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:52.801270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:45:52.804743 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:45:52.804713    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fc950f_f50c_4f79_9417_be52f2740af7.slice/crio-fc1fc6c1c16a380b7af35bedf818a9ca4273adcbb3a4e086baafec8cfed57e8b WatchSource:0}: Error finding container fc1fc6c1c16a380b7af35bedf818a9ca4273adcbb3a4e086baafec8cfed57e8b: Status 404 returned error can't find the container with id fc1fc6c1c16a380b7af35bedf818a9ca4273adcbb3a4e086baafec8cfed57e8b
Sep 09 20:45:53.042700 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.042664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerStarted","Data":"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"}
Sep 09 20:45:53.042855 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.042788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:53.042855 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.042823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerStarted","Data":"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"}
Sep 09 20:45:53.042855 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.042853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:53.042980 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.042879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerStarted","Data":"fc1fc6c1c16a380b7af35bedf818a9ca4273adcbb3a4e086baafec8cfed57e8b"}
Sep 09 20:45:53.044336 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.044308    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:45:53.060903 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:53.060859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podStartSLOduration=2.060842487 podStartE2EDuration="2.060842487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:53.059408638 +0000 UTC m=+2073.945998580" watchObservedRunningTime="2026-09-09 20:45:53.060842487 +0000 UTC m=+2073.947432400"
Sep 09 20:45:54.047516 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.047477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:45:54.395612 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.395587    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:45:54.425500 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.425470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config\") pod \"f224ccfc-8bb2-436f-8deb-2c370420fef8\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") "
Sep 09 20:45:54.425621 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.425526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-248tb\" (UniqueName: \"kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb\") pod \"f224ccfc-8bb2-436f-8deb-2c370420fef8\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") "
Sep 09 20:45:54.425621 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.425555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") pod \"f224ccfc-8bb2-436f-8deb-2c370420fef8\" (UID: \"f224ccfc-8bb2-436f-8deb-2c370420fef8\") "
Sep 09 20:45:54.425789 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.425765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f224ccfc-8bb2-436f-8deb-2c370420fef8" (UID: "f224ccfc-8bb2-436f-8deb-2c370420fef8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:54.427484 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.427461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb" (OuterVolumeSpecName: "kube-api-access-248tb") pod "f224ccfc-8bb2-436f-8deb-2c370420fef8" (UID: "f224ccfc-8bb2-436f-8deb-2c370420fef8"). InnerVolumeSpecName "kube-api-access-248tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:54.427618 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.427598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f224ccfc-8bb2-436f-8deb-2c370420fef8" (UID: "f224ccfc-8bb2-436f-8deb-2c370420fef8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:54.526766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.526739    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:45:54.526766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.526763    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248tb\" (UniqueName: \"kubernetes.io/projected/f224ccfc-8bb2-436f-8deb-2c370420fef8-kube-api-access-248tb\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:45:54.526894 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:54.526773    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f224ccfc-8bb2-436f-8deb-2c370420fef8-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:45:55.053060 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.053008    2577 generic.go:356] "Generic (PLEG): container finished" podID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerID="ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048" exitCode=0
Sep 09 20:45:55.053060 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.053055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerDied","Data":"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"}
Sep 09 20:45:55.053511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.053068    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"
Sep 09 20:45:55.053511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.053122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2" event={"ID":"f224ccfc-8bb2-436f-8deb-2c370420fef8","Type":"ContainerDied","Data":"a919bbb5d64a5011cc78e19ace724a0d18d8c8e4d5b156de9aa3c1ebf86cdb12"}
Sep 09 20:45:55.053511 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.053157    2577 scope.go:117] "RemoveContainer" containerID="2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"
Sep 09 20:45:55.061859 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.061844    2577 scope.go:117] "RemoveContainer" containerID="ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"
Sep 09 20:45:55.069706 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.069688    2577 scope.go:117] "RemoveContainer" containerID="2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"
Sep 09 20:45:55.069949 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:45:55.069928    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0\": container with ID starting with 2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0 not found: ID does not exist" containerID="2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"
Sep 09 20:45:55.070120 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.069961    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0"} err="failed to get container status \"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0\": rpc error: code = NotFound desc = could not find container \"2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0\": container with ID starting with 2bbb0b61b097b56150b2b9f9933284de0ef962b602b3bab2ae95c68250f9d2e0 not found: ID does not exist"
Sep 09 20:45:55.070120 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.069983    2577 scope.go:117] "RemoveContainer" containerID="ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"
Sep 09 20:45:55.070279 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:45:55.070261    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048\": container with ID starting with ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048 not found: ID does not exist" containerID="ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"
Sep 09 20:45:55.070318 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.070286    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048"} err="failed to get container status \"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048\": rpc error: code = NotFound desc = could not find container \"ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048\": container with ID starting with ddd39ca89d29999c6383af5370e6bc091a7ad68da47ac6b9ccd00e035ec2b048 not found: ID does not exist"
Sep 09 20:45:55.074255 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.074232    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:45:55.078187 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.078165    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-81279-predictor-84575cb8cb-pllk2"]
Sep 09 20:45:55.621129 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:55.621093    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" path="/var/lib/kubelet/pods/f224ccfc-8bb2-436f-8deb-2c370420fef8/volumes"
Sep 09 20:45:59.052273 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:59.052247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:45:59.052689 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:45:59.052664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:46:09.053611 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:09.053568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:46:19.053036 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:19.052979    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:46:29.052889 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:29.052851    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:46:39.054168 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:39.054135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:46:41.300837 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.300805    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:46:41.301310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.301043    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container" containerID="cri-o://c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a" gracePeriod=30
Sep 09 20:46:41.301310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.301081    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kube-rbac-proxy" containerID="cri-o://114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2" gracePeriod=30
Sep 09 20:46:41.493292 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493263    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 20:46:41.493712 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container"
Sep 09 20:46:41.493774 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container"
Sep 09 20:46:41.493774 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493731    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kube-rbac-proxy"
Sep 09 20:46:41.493774 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493737    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kube-rbac-proxy"
Sep 09 20:46:41.493992 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kube-rbac-proxy"
Sep 09 20:46:41.493992 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.493817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f224ccfc-8bb2-436f-8deb-2c370420fef8" containerName="kserve-container"
Sep 09 20:46:41.496983 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.496963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.499419 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.499397    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6a9b1-predictor-serving-cert\""
Sep 09 20:46:41.499536 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.499398    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6a9b1-kube-rbac-proxy-sar-config\""
Sep 09 20:46:41.507085 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.507061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 20:46:41.558534 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.558471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.558646 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.558577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.558646 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.558636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbzn\" (UniqueName: \"kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.659659 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.659634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.659776 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.659687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.659776 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.659729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbzn\" (UniqueName: \"kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.659776 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:46:41.659770    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-serving-cert: secret "error-404-isvc-6a9b1-predictor-serving-cert" not found
Sep 09 20:46:41.659941 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:46:41.659826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls podName:99df6d45-1d47-4165-9988-59ab7a964aae nodeName:}" failed. No retries permitted until 2026-09-09 20:46:42.159811351 +0000 UTC m=+2123.046401263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls") pod "error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" (UID: "99df6d45-1d47-4165-9988-59ab7a964aae") : secret "error-404-isvc-6a9b1-predictor-serving-cert" not found
Sep 09 20:46:41.660387 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.660368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:41.667936 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:41.667915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbzn\" (UniqueName: \"kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:42.164481 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.164450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:42.166913 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.166886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") pod \"error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") " pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:42.280673 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.280644    2577 generic.go:356] "Generic (PLEG): container finished" podID="0586df93-682b-4686-9b9c-d6b03fae9893" containerID="114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2" exitCode=2
Sep 09 20:46:42.280792 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.280709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerDied","Data":"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"}
Sep 09 20:46:42.409694 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.409656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:42.679593 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:42.679564    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 20:46:42.681372 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:46:42.681344    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99df6d45_1d47_4165_9988_59ab7a964aae.slice/crio-78ae011b618c12a4c9cb2d5f001e2ae681311794fb68a051bf97ac45853da8a9 WatchSource:0}: Error finding container 78ae011b618c12a4c9cb2d5f001e2ae681311794fb68a051bf97ac45853da8a9: Status 404 returned error can't find the container with id 78ae011b618c12a4c9cb2d5f001e2ae681311794fb68a051bf97ac45853da8a9
Sep 09 20:46:43.287126 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.287071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerStarted","Data":"656b048876bc56609f54f89bd51fe1fbdd9f044ae1a3c43ed8e0deb0b70782c7"}
Sep 09 20:46:43.287310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.287152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerStarted","Data":"5225940b277695fa46df8627c09b17cb8ec9969ea22908e930b432276898d04f"}
Sep 09 20:46:43.287310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.287181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerStarted","Data":"78ae011b618c12a4c9cb2d5f001e2ae681311794fb68a051bf97ac45853da8a9"}
Sep 09 20:46:43.287310 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.287268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:43.305119 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.305070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podStartSLOduration=2.305052618 podStartE2EDuration="2.305052618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:43.303439341 +0000 UTC m=+2124.190029252" watchObservedRunningTime="2026-09-09 20:46:43.305052618 +0000 UTC m=+2124.191642529"
Sep 09 20:46:43.950150 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:43.950126    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:46:44.079575 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.079551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6fjm\" (UniqueName: \"kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm\") pod \"0586df93-682b-4686-9b9c-d6b03fae9893\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") "
Sep 09 20:46:44.079727 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.079599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config\") pod \"0586df93-682b-4686-9b9c-d6b03fae9893\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") "
Sep 09 20:46:44.079727 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.079646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") pod \"0586df93-682b-4686-9b9c-d6b03fae9893\" (UID: \"0586df93-682b-4686-9b9c-d6b03fae9893\") "
Sep 09 20:46:44.079995 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.079966    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0586df93-682b-4686-9b9c-d6b03fae9893" (UID: "0586df93-682b-4686-9b9c-d6b03fae9893"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:44.081669 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.081652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm" (OuterVolumeSpecName: "kube-api-access-h6fjm") pod "0586df93-682b-4686-9b9c-d6b03fae9893" (UID: "0586df93-682b-4686-9b9c-d6b03fae9893"). InnerVolumeSpecName "kube-api-access-h6fjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:44.081782 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.081762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0586df93-682b-4686-9b9c-d6b03fae9893" (UID: "0586df93-682b-4686-9b9c-d6b03fae9893"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:44.180920 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.180899    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0586df93-682b-4686-9b9c-d6b03fae9893-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:46:44.180920 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.180919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fjm\" (UniqueName: \"kubernetes.io/projected/0586df93-682b-4686-9b9c-d6b03fae9893-kube-api-access-h6fjm\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:46:44.181070 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.180929    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0586df93-682b-4686-9b9c-d6b03fae9893-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:46:44.293809 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.293778    2577 generic.go:356] "Generic (PLEG): container finished" podID="0586df93-682b-4686-9b9c-d6b03fae9893" containerID="c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a" exitCode=0
Sep 09 20:46:44.293913 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.293830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"
Sep 09 20:46:44.293913 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.293856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerDied","Data":"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"}
Sep 09 20:46:44.293986 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.293919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns" event={"ID":"0586df93-682b-4686-9b9c-d6b03fae9893","Type":"ContainerDied","Data":"5e680a4129a9f589d81b7b888c7a792d074cee441928e47b8d6d71c3a28722e1"}
Sep 09 20:46:44.293986 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.293945    2577 scope.go:117] "RemoveContainer" containerID="114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"
Sep 09 20:46:44.294370 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.294349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:44.295412 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.295386    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:46:44.303179 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.303161    2577 scope.go:117] "RemoveContainer" containerID="c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"
Sep 09 20:46:44.313688 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.313638    2577 scope.go:117] "RemoveContainer" containerID="114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"
Sep 09 20:46:44.314322 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:46:44.314294    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2\": container with ID starting with 114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2 not found: ID does not exist" containerID="114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"
Sep 09 20:46:44.314429 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.314331    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2"} err="failed to get container status \"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2\": rpc error: code = NotFound desc = could not find container \"114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2\": container with ID starting with 114b44e78472143987582c5f81547f07a0c0b512c45404b6fc264c471f6e56b2 not found: ID does not exist"
Sep 09 20:46:44.314429 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.314348    2577 scope.go:117] "RemoveContainer" containerID="c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"
Sep 09 20:46:44.314700 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:46:44.314675    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a\": container with ID starting with c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a not found: ID does not exist" containerID="c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"
Sep 09 20:46:44.314768 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.314711    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a"} err="failed to get container status \"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a\": rpc error: code = NotFound desc = could not find container \"c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a\": container with ID starting with c2eaf7059c63ec0c7d1442412fee5a6939bfe2ec777407b1bef8da821ae8ff6a not found: ID does not exist"
Sep 09 20:46:44.315922 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.315903    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:46:44.318710 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:44.318687    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-28e88-predictor-6bb6f499b4-jg8ns"]
Sep 09 20:46:45.299004 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:45.298964    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:46:45.621782 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:45.621707    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" path="/var/lib/kubelet/pods/0586df93-682b-4686-9b9c-d6b03fae9893/volumes"
Sep 09 20:46:50.303137 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:50.303110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:46:50.303592 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:46:50.303490    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:47:00.303557 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:00.303519    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:47:01.831449 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:01.831412    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:47:01.831886 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:01.831728    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container" containerID="cri-o://034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989" gracePeriod=30
Sep 09 20:47:01.831886 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:01.831792    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kube-rbac-proxy" containerID="cri-o://b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007" gracePeriod=30
Sep 09 20:47:02.009658 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.009623    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:47:02.010012 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.009999    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kube-rbac-proxy"
Sep 09 20:47:02.010094 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.010013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kube-rbac-proxy"
Sep 09 20:47:02.010094 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.010050    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container"
Sep 09 20:47:02.010094 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.010056    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container"
Sep 09 20:47:02.010254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.010120    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kube-rbac-proxy"
Sep 09 20:47:02.010254 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.010130    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0586df93-682b-4686-9b9c-d6b03fae9893" containerName="kserve-container"
Sep 09 20:47:02.013276 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.013251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.015879 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.015852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7dad6-predictor-serving-cert\""
Sep 09 20:47:02.015991 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.015856    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7dad6-kube-rbac-proxy-sar-config\""
Sep 09 20:47:02.024172 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.024146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:47:02.114918 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.114853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9lj\" (UniqueName: \"kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.115051 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.114920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.115051 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.114947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.215911 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.215883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.216071 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.215924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.216071 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.215957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9lj\" (UniqueName: \"kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.216573 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.216548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.218408 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.218388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.226314 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.226291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9lj\" (UniqueName: \"kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj\") pod \"error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") " pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.325277 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.325252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:02.380532 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.380503    2577 generic.go:356] "Generic (PLEG): container finished" podID="60fc950f-f50c-4f79-9417-be52f2740af7" containerID="b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007" exitCode=2
Sep 09 20:47:02.380672 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.380545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerDied","Data":"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"}
Sep 09 20:47:02.595167 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:02.595138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:47:02.597104 ip-10-0-128-184 kubenswrapper[2577]: W0909 20:47:02.597077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4987542a_4749_4692_95cb_58f083be47ab.slice/crio-68e5693dd55ecd4ecf751fa74979b2e1218d33b1c4ffb825f1e94d4907b63fe6 WatchSource:0}: Error finding container 68e5693dd55ecd4ecf751fa74979b2e1218d33b1c4ffb825f1e94d4907b63fe6: Status 404 returned error can't find the container with id 68e5693dd55ecd4ecf751fa74979b2e1218d33b1c4ffb825f1e94d4907b63fe6
Sep 09 20:47:03.387208 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.387165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerStarted","Data":"66755aee5f80bb4f1b7a863c8713dfccf7db3ed1381b65ab0751af8f533b254b"}
Sep 09 20:47:03.387576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.387245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:03.387576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.387269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:03.387576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.387286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerStarted","Data":"b72fb8c5f0722b11a6ad8b909acb70e10b0bfe27c16608bb822c84c6b43ba2d9"}
Sep 09 20:47:03.387576 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.387311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerStarted","Data":"68e5693dd55ecd4ecf751fa74979b2e1218d33b1c4ffb825f1e94d4907b63fe6"}
Sep 09 20:47:03.388955 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.388925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:03.403575 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:03.403516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podStartSLOduration=2.403498119 podStartE2EDuration="2.403498119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:03.402958501 +0000 UTC m=+2144.289548437" watchObservedRunningTime="2026-09-09 20:47:03.403498119 +0000 UTC m=+2144.290088032"
Sep 09 20:47:04.048203 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.048164    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:47:04.391713 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.391627    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:04.688616 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.688593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:47:04.839842 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.839816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvd7w\" (UniqueName: \"kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w\") pod \"60fc950f-f50c-4f79-9417-be52f2740af7\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") "
Sep 09 20:47:04.839969 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.839904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config\") pod \"60fc950f-f50c-4f79-9417-be52f2740af7\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") "
Sep 09 20:47:04.839969 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.839934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") pod \"60fc950f-f50c-4f79-9417-be52f2740af7\" (UID: \"60fc950f-f50c-4f79-9417-be52f2740af7\") "
Sep 09 20:47:04.840250 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.840217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60fc950f-f50c-4f79-9417-be52f2740af7" (UID: "60fc950f-f50c-4f79-9417-be52f2740af7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:04.841860 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.841835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w" (OuterVolumeSpecName: "kube-api-access-tvd7w") pod "60fc950f-f50c-4f79-9417-be52f2740af7" (UID: "60fc950f-f50c-4f79-9417-be52f2740af7"). InnerVolumeSpecName "kube-api-access-tvd7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:04.841970 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.841891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60fc950f-f50c-4f79-9417-be52f2740af7" (UID: "60fc950f-f50c-4f79-9417-be52f2740af7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:04.940549 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.940497    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60fc950f-f50c-4f79-9417-be52f2740af7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:47:04.940549 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.940519    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60fc950f-f50c-4f79-9417-be52f2740af7-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:47:04.940549 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:04.940528    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvd7w\" (UniqueName: \"kubernetes.io/projected/60fc950f-f50c-4f79-9417-be52f2740af7-kube-api-access-tvd7w\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:47:05.398054 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.398003    2577 generic.go:356] "Generic (PLEG): container finished" podID="60fc950f-f50c-4f79-9417-be52f2740af7" containerID="034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989" exitCode=0
Sep 09 20:47:05.398517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.398062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerDied","Data":"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"}
Sep 09 20:47:05.398517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.398068    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"
Sep 09 20:47:05.398517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.398111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms" event={"ID":"60fc950f-f50c-4f79-9417-be52f2740af7","Type":"ContainerDied","Data":"fc1fc6c1c16a380b7af35bedf818a9ca4273adcbb3a4e086baafec8cfed57e8b"}
Sep 09 20:47:05.398517 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.398136    2577 scope.go:117] "RemoveContainer" containerID="b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"
Sep 09 20:47:05.409171 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.409154    2577 scope.go:117] "RemoveContainer" containerID="034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"
Sep 09 20:47:05.417248 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.417231    2577 scope.go:117] "RemoveContainer" containerID="b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"
Sep 09 20:47:05.417533 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:47:05.417499    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007\": container with ID starting with b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007 not found: ID does not exist" containerID="b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"
Sep 09 20:47:05.417599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.417529    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007"} err="failed to get container status \"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007\": rpc error: code = NotFound desc = could not find container \"b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007\": container with ID starting with b080bbbc5268ca11114f7e9f52adfd82c7bc138212b60502e4d2c434a4930007 not found: ID does not exist"
Sep 09 20:47:05.417599 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.417545    2577 scope.go:117] "RemoveContainer" containerID="034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"
Sep 09 20:47:05.418203 ip-10-0-128-184 kubenswrapper[2577]: E0909 20:47:05.417886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989\": container with ID starting with 034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989 not found: ID does not exist" containerID="034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"
Sep 09 20:47:05.418203 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.417924    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989"} err="failed to get container status \"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989\": rpc error: code = NotFound desc = could not find container \"034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989\": container with ID starting with 034e2f7263dbad10da8d05a1a647986ab53ecbb09dcdd59ce156ee17e060a989 not found: ID does not exist"
Sep 09 20:47:05.419897 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.419875    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:47:05.429220 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.424245    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-326f5-predictor-5f77687698-df8ms"]
Sep 09 20:47:05.625660 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:05.625610    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" path="/var/lib/kubelet/pods/60fc950f-f50c-4f79-9417-be52f2740af7/volumes"
Sep 09 20:47:09.396320 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:09.396293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:47:09.396790 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:09.396760    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:10.303718 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:10.303672    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:47:19.397294 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:19.397214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:20.304289 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:20.304246    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:47:29.397007 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:29.396965    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:30.304201 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:30.304159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 20:47:39.397766 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:39.397723    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:47:49.397992 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:47:49.397955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:56:16.668585 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:16.668512    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:56:16.669366 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:16.668740    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container" containerID="cri-o://b72fb8c5f0722b11a6ad8b909acb70e10b0bfe27c16608bb822c84c6b43ba2d9" gracePeriod=30
Sep 09 20:56:16.669366 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:16.668790    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kube-rbac-proxy" containerID="cri-o://66755aee5f80bb4f1b7a863c8713dfccf7db3ed1381b65ab0751af8f533b254b" gracePeriod=30
Sep 09 20:56:16.959912 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:16.959830    2577 generic.go:356] "Generic (PLEG): container finished" podID="4987542a-4749-4692-95cb-58f083be47ab" containerID="66755aee5f80bb4f1b7a863c8713dfccf7db3ed1381b65ab0751af8f533b254b" exitCode=2
Sep 09 20:56:16.960058 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:16.959913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerDied","Data":"66755aee5f80bb4f1b7a863c8713dfccf7db3ed1381b65ab0751af8f533b254b"}
Sep 09 20:56:19.222780 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.222752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:56:19.296797 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.296774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9lj\" (UniqueName: \"kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj\") pod \"4987542a-4749-4692-95cb-58f083be47ab\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") "
Sep 09 20:56:19.296935 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.296854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config\") pod \"4987542a-4749-4692-95cb-58f083be47ab\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") "
Sep 09 20:56:19.296935 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.296903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls\") pod \"4987542a-4749-4692-95cb-58f083be47ab\" (UID: \"4987542a-4749-4692-95cb-58f083be47ab\") "
Sep 09 20:56:19.297243 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.297220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4987542a-4749-4692-95cb-58f083be47ab" (UID: "4987542a-4749-4692-95cb-58f083be47ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:19.298893 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.298862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4987542a-4749-4692-95cb-58f083be47ab" (UID: "4987542a-4749-4692-95cb-58f083be47ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.298984 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.298912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj" (OuterVolumeSpecName: "kube-api-access-qx9lj") pod "4987542a-4749-4692-95cb-58f083be47ab" (UID: "4987542a-4749-4692-95cb-58f083be47ab"). InnerVolumeSpecName "kube-api-access-qx9lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:19.398147 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.398123    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4987542a-4749-4692-95cb-58f083be47ab-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.398147 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.398145    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9lj\" (UniqueName: \"kubernetes.io/projected/4987542a-4749-4692-95cb-58f083be47ab-kube-api-access-qx9lj\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.398275 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.398156    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4987542a-4749-4692-95cb-58f083be47ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.898857 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.898787    2577 scope.go:117] "RemoveContainer" containerID="66755aee5f80bb4f1b7a863c8713dfccf7db3ed1381b65ab0751af8f533b254b"
Sep 09 20:56:19.907134 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.907114    2577 scope.go:117] "RemoveContainer" containerID="b72fb8c5f0722b11a6ad8b909acb70e10b0bfe27c16608bb822c84c6b43ba2d9"
Sep 09 20:56:19.978201 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.978162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerDied","Data":"b72fb8c5f0722b11a6ad8b909acb70e10b0bfe27c16608bb822c84c6b43ba2d9"}
Sep 09 20:56:19.978319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.978257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm" event={"ID":"4987542a-4749-4692-95cb-58f083be47ab","Type":"ContainerDied","Data":"68e5693dd55ecd4ecf751fa74979b2e1218d33b1c4ffb825f1e94d4907b63fe6"}
Sep 09 20:56:19.978319 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:19.978180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"
Sep 09 20:56:20.007186 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:20.007155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:56:20.010558 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:20.010538    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7dad6-predictor-6c8d884d74-7hlfm"]
Sep 09 20:56:21.621621 ip-10-0-128-184 kubenswrapper[2577]: I0909 20:56:21.621590    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4987542a-4749-4692-95cb-58f083be47ab" path="/var/lib/kubelet/pods/4987542a-4749-4692-95cb-58f083be47ab/volumes"
Sep 09 21:04:01.320630 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:01.320536    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 21:04:01.323262 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:01.320812    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container" containerID="cri-o://5225940b277695fa46df8627c09b17cb8ec9969ea22908e930b432276898d04f" gracePeriod=30
Sep 09 21:04:01.323262 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:01.320871    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kube-rbac-proxy" containerID="cri-o://656b048876bc56609f54f89bd51fe1fbdd9f044ae1a3c43ed8e0deb0b70782c7" gracePeriod=30
Sep 09 21:04:02.091011 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:02.090976    2577 generic.go:356] "Generic (PLEG): container finished" podID="99df6d45-1d47-4165-9988-59ab7a964aae" containerID="656b048876bc56609f54f89bd51fe1fbdd9f044ae1a3c43ed8e0deb0b70782c7" exitCode=2
Sep 09 21:04:02.091196 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:02.091051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerDied","Data":"656b048876bc56609f54f89bd51fe1fbdd9f044ae1a3c43ed8e0deb0b70782c7"}
Sep 09 21:04:04.111848 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.111813    2577 generic.go:356] "Generic (PLEG): container finished" podID="99df6d45-1d47-4165-9988-59ab7a964aae" containerID="5225940b277695fa46df8627c09b17cb8ec9969ea22908e930b432276898d04f" exitCode=0
Sep 09 21:04:04.112250 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.111877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerDied","Data":"5225940b277695fa46df8627c09b17cb8ec9969ea22908e930b432276898d04f"}
Sep 09 21:04:04.172483 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.172461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 21:04:04.248258 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.248227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") pod \"99df6d45-1d47-4165-9988-59ab7a964aae\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") "
Sep 09 21:04:04.248387 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.248280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config\") pod \"99df6d45-1d47-4165-9988-59ab7a964aae\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") "
Sep 09 21:04:04.248387 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.248367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xbzn\" (UniqueName: \"kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn\") pod \"99df6d45-1d47-4165-9988-59ab7a964aae\" (UID: \"99df6d45-1d47-4165-9988-59ab7a964aae\") "
Sep 09 21:04:04.248692 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.248659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99df6d45-1d47-4165-9988-59ab7a964aae" (UID: "99df6d45-1d47-4165-9988-59ab7a964aae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:04.250415 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.250392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99df6d45-1d47-4165-9988-59ab7a964aae" (UID: "99df6d45-1d47-4165-9988-59ab7a964aae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:04.250502 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.250438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn" (OuterVolumeSpecName: "kube-api-access-2xbzn") pod "99df6d45-1d47-4165-9988-59ab7a964aae" (UID: "99df6d45-1d47-4165-9988-59ab7a964aae"). InnerVolumeSpecName "kube-api-access-2xbzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:04.349305 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.349276    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xbzn\" (UniqueName: \"kubernetes.io/projected/99df6d45-1d47-4165-9988-59ab7a964aae-kube-api-access-2xbzn\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.349305 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.349303    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99df6d45-1d47-4165-9988-59ab7a964aae-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.349497 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:04.349317    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99df6d45-1d47-4165-9988-59ab7a964aae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 09 21:04:05.118516 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.118482    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"
Sep 09 21:04:05.119042 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.118481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk" event={"ID":"99df6d45-1d47-4165-9988-59ab7a964aae","Type":"ContainerDied","Data":"78ae011b618c12a4c9cb2d5f001e2ae681311794fb68a051bf97ac45853da8a9"}
Sep 09 21:04:05.119042 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.118659    2577 scope.go:117] "RemoveContainer" containerID="656b048876bc56609f54f89bd51fe1fbdd9f044ae1a3c43ed8e0deb0b70782c7"
Sep 09 21:04:05.131771 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.131746    2577 scope.go:117] "RemoveContainer" containerID="5225940b277695fa46df8627c09b17cb8ec9969ea22908e930b432276898d04f"
Sep 09 21:04:05.153092 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.153066    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 21:04:05.161152 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.161122    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a9b1-predictor-8496c76f6d-6c6kk"]
Sep 09 21:04:05.621064 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:05.621035    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" path="/var/lib/kubelet/pods/99df6d45-1d47-4165-9988-59ab7a964aae/volumes"
Sep 09 21:04:36.633896 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:36.633861    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6ptjs_b31c6b37-d608-46c2-9df9-0032ba3235d0/global-pull-secret-syncer/0.log"
Sep 09 21:04:36.851064 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:36.851009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x8jwj_59f4958b-f12d-4efd-ab96-0e71ae498273/konnectivity-agent/0.log"
Sep 09 21:04:36.877177 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:36.877149    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-184.ec2.internal_dc5e2da99c7ada79b50f8817f3f862d1/haproxy/0.log"
Sep 09 21:04:40.835601 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:40.835572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/alertmanager/0.log"
Sep 09 21:04:40.863744 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:40.863718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/config-reloader/0.log"
Sep 09 21:04:40.895178 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:40.895154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/kube-rbac-proxy-web/0.log"
Sep 09 21:04:40.934886 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:40.934863    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/kube-rbac-proxy/0.log"
Sep 09 21:04:40.972238 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:40.972218    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/kube-rbac-proxy-metric/0.log"
Sep 09 21:04:41.006750 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.006727    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/prom-label-proxy/0.log"
Sep 09 21:04:41.050174 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.050155    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31873c93-4339-46e7-87b2-05fa7d7bc3e8/init-config-reloader/0.log"
Sep 09 21:04:41.321575 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.321550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-7c828_89604b42-04d8-4094-af6a-a5e1da80a2bb/monitoring-plugin/0.log"
Sep 09 21:04:41.543519 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.543496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsr9d_b251bc25-f3ac-423a-a6f2-39c05318135e/node-exporter/0.log"
Sep 09 21:04:41.616537 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.616469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsr9d_b251bc25-f3ac-423a-a6f2-39c05318135e/kube-rbac-proxy/0.log"
Sep 09 21:04:41.703681 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:41.703655    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsr9d_b251bc25-f3ac-423a-a6f2-39c05318135e/init-textfile/0.log"
Sep 09 21:04:42.325273 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.325242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d5986f85-lpjcp_ab4c2a5a-414d-4ec4-8e05-f39734703b06/telemeter-client/0.log"
Sep 09 21:04:42.377996 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.377973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d5986f85-lpjcp_ab4c2a5a-414d-4ec4-8e05-f39734703b06/reload/0.log"
Sep 09 21:04:42.420219 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.420195    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d5986f85-lpjcp_ab4c2a5a-414d-4ec4-8e05-f39734703b06/kube-rbac-proxy/0.log"
Sep 09 21:04:42.474552 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.474529    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/thanos-query/0.log"
Sep 09 21:04:42.508133 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.508115    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/kube-rbac-proxy-web/0.log"
Sep 09 21:04:42.536965 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.536949    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/kube-rbac-proxy/0.log"
Sep 09 21:04:42.567428 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.567392    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/prom-label-proxy/0.log"
Sep 09 21:04:42.604889 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.604844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/kube-rbac-proxy-rules/0.log"
Sep 09 21:04:42.631924 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:42.631904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d8fffdd5-v6jgr_f593b73b-69b3-4d66-b857-29ac46a03f5d/kube-rbac-proxy-metrics/0.log"
Sep 09 21:04:43.767676 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:43.767644    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-h8qd8_bc3e0c62-325f-47d0-a395-0feacc9c4dc0/networking-console-plugin/0.log"
Sep 09 21:04:44.654711 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.654678    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-vk7b8_768305c7-e43f-481e-bc7f-d68400d5ddcd/download-server/0.log"
Sep 09 21:04:44.947871 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.947791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"]
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948171    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948180    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948186    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948198    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948204    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948212    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948240    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container"
Sep 09 21:04:44.948259 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948245    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948298    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948307    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948315    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99df6d45-1d47-4165-9988-59ab7a964aae" containerName="kserve-container"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948321    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kube-rbac-proxy"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948327    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4987542a-4749-4692-95cb-58f083be47ab" containerName="kserve-container"
Sep 09 21:04:44.948669 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.948333    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60fc950f-f50c-4f79-9417-be52f2740af7" containerName="kserve-container"
Sep 09 21:04:44.951531 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.951513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:44.954321 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.954303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cmgmq\"/\"default-dockercfg-fpm2c\""
Sep 09 21:04:44.954407 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.954308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cmgmq\"/\"kube-root-ca.crt\""
Sep 09 21:04:44.955430 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.955413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cmgmq\"/\"openshift-service-ca.crt\""
Sep 09 21:04:44.961897 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:44.961878    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"]
Sep 09 21:04:45.059533 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.059500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-proc\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.059691 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.059543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-sys\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.059691 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.059562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-podres\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.059691 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.059675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8xz\" (UniqueName: \"kubernetes.io/projected/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-kube-api-access-tb8xz\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.059799 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.059710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-lib-modules\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.160822 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8xz\" (UniqueName: \"kubernetes.io/projected/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-kube-api-access-tb8xz\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.160954 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-lib-modules\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.160954 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-proc\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.160954 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-sys\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.160954 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-podres\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.161134 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.160995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-sys\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.161134 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.161001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-proc\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.161134 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.161001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-lib-modules\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.161134 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.161071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-podres\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.169246 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.169222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8xz\" (UniqueName: \"kubernetes.io/projected/753d9c03-8498-46ee-8bfb-0f1ae782dd3c-kube-api-access-tb8xz\") pod \"perf-node-gather-daemonset-n8jbs\" (UID: \"753d9c03-8498-46ee-8bfb-0f1ae782dd3c\") " pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.261997 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.261933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:45.539981 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:45.539958    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"]
Sep 09 21:04:45.542874 ip-10-0-128-184 kubenswrapper[2577]: W0909 21:04:45.542847    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod753d9c03_8498_46ee_8bfb_0f1ae782dd3c.slice/crio-06dfa833573e90f1106bae5b1cdc8b10c1e64fdeda2fbad8ca6ede58b79aaf43 WatchSource:0}: Error finding container 06dfa833573e90f1106bae5b1cdc8b10c1e64fdeda2fbad8ca6ede58b79aaf43: Status 404 returned error can't find the container with id 06dfa833573e90f1106bae5b1cdc8b10c1e64fdeda2fbad8ca6ede58b79aaf43
Sep 09 21:04:46.131354 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.131327    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fd489_04021b1b-4eaf-4de5-b3d5-6446103d6ba1/dns/0.log"
Sep 09 21:04:46.156969 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.156943    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fd489_04021b1b-4eaf-4de5-b3d5-6446103d6ba1/kube-rbac-proxy/0.log"
Sep 09 21:04:46.319896 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.319876    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4pwf_4d3cf640-5639-40e0-b251-550ef10be316/dns-node-resolver/0.log"
Sep 09 21:04:46.331908 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.331883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs" event={"ID":"753d9c03-8498-46ee-8bfb-0f1ae782dd3c","Type":"ContainerStarted","Data":"dec74d8dd96177d24fcdcb3a4dce05b9b09841d340b91937365050d1be1bb8b8"}
Sep 09 21:04:46.332009 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.331932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs" event={"ID":"753d9c03-8498-46ee-8bfb-0f1ae782dd3c","Type":"ContainerStarted","Data":"06dfa833573e90f1106bae5b1cdc8b10c1e64fdeda2fbad8ca6ede58b79aaf43"}
Sep 09 21:04:46.332009 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.332002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:46.351099 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.351057    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs" podStartSLOduration=2.351044008 podStartE2EDuration="2.351044008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:46.348934191 +0000 UTC m=+3207.235524117" watchObservedRunningTime="2026-09-09 21:04:46.351044008 +0000 UTC m=+3207.237633917"
Sep 09 21:04:46.856176 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:46.856141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jmz8t_282b4062-5693-4318-8e23-104b0ec0bb27/node-ca/0.log"
Sep 09 21:04:47.637886 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:47.637859    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f5778b68-h5lt7_9789ae06-35b6-4a0c-8222-a466e0b46d3d/router/0.log"
Sep 09 21:04:47.980186 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:47.980111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dg8zk_ded7752f-71da-4bf8-9d82-ec219a56d7bb/serve-healthcheck-canary/0.log"
Sep 09 21:04:48.499878 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:48.499846    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cllf7_e3d559a3-895a-4d80-9d36-c9cb061384e6/kube-rbac-proxy/0.log"
Sep 09 21:04:48.523195 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:48.523172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cllf7_e3d559a3-895a-4d80-9d36-c9cb061384e6/exporter/0.log"
Sep 09 21:04:48.547407 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:48.547381    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cllf7_e3d559a3-895a-4d80-9d36-c9cb061384e6/extractor/0.log"
Sep 09 21:04:50.884357 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:50.884324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-55b9566966-98795_93c1a534-82e5-4447-8553-04f6bec0107f/manager/0.log"
Sep 09 21:04:50.912387 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:50.912363    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-gtnbt_b70e67ea-06c4-46fa-90eb-0f905bb453f8/manager/0.log"
Sep 09 21:04:51.020552 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:51.020530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-m5ssq_0d270fc9-177e-42c9-82e8-9469928913f0/manager/0.log"
Sep 09 21:04:51.100599 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:51.100575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gv28n_cc00a022-2e7f-42d6-8e3b-31ae2f78513d/seaweedfs/0.log"
Sep 09 21:04:52.346622 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:52.346598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cmgmq/perf-node-gather-daemonset-n8jbs"
Sep 09 21:04:55.352502 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:55.352475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fn6dq_1c8d445d-fe1b-4900-9a72-fe48105d4c91/kube-storage-version-migrator-operator/1.log"
Sep 09 21:04:55.353799 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:55.353775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fn6dq_1c8d445d-fe1b-4900-9a72-fe48105d4c91/kube-storage-version-migrator-operator/0.log"
Sep 09 21:04:56.647952 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.647927    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:04:56.676984 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.676954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/egress-router-binary-copy/0.log"
Sep 09 21:04:56.698522 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.698506    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/cni-plugins/0.log"
Sep 09 21:04:56.724233 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.724205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/bond-cni-plugin/0.log"
Sep 09 21:04:56.749207 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.749185    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/routeoverride-cni/0.log"
Sep 09 21:04:56.786843 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.786820    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/whereabouts-cni-bincopy/0.log"
Sep 09 21:04:56.810757 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.810736    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlr89_b05148b1-8474-4106-a452-6fee11123ae9/whereabouts-cni/0.log"
Sep 09 21:04:56.888459 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.888440    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-phcgt_e2824d45-98ee-4a43-94ed-c01bb6a31b6d/kube-multus/0.log"
Sep 09 21:04:56.937367 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.937301    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9blb8_cb7a3927-19b9-4c23-8c72-b245c699f4ee/network-metrics-daemon/0.log"
Sep 09 21:04:56.959121 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:56.959101    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9blb8_cb7a3927-19b9-4c23-8c72-b245c699f4ee/kube-rbac-proxy/0.log"
Sep 09 21:04:57.806837 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.806799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/ovn-controller/0.log"
Sep 09 21:04:57.824663 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.824643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/ovn-acl-logging/0.log"
Sep 09 21:04:57.850485 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.850464    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/kube-rbac-proxy-node/0.log"
Sep 09 21:04:57.872653 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.872636    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:04:57.890190 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.890164    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/northd/0.log"
Sep 09 21:04:57.910693 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.910668    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/nbdb/0.log"
Sep 09 21:04:57.934047 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.934009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/sbdb/0.log"
Sep 09 21:04:57.974895 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:57.974876    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-258xn_489520f6-3850-426c-b05a-2955e55cfd8b/ovnkube-controller/0.log"
Sep 09 21:04:59.407608 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:04:59.407566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-c2tnv_cd5da601-b4d8-48d8-9899-bd65af4d19d1/network-check-target-container/0.log"
Sep 09 21:05:00.424546 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:05:00.424518    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-778bm_ac721a5e-4c47-4054-b183-0ee6c936dccd/iptables-alerter/0.log"
Sep 09 21:05:01.157132 ip-10-0-128-184 kubenswrapper[2577]: I0909 21:05:01.157102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4vs8l_84ad5e66-ebe8-4198-9688-a0c064038611/tuned/0.log"