2026-07-16T22:16:35.249635818Z [disk usage checker] Started 2026-07-16T22:16:35.253388765Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:16:35.454135590Z Gathering data for ns/openshift-cluster-version... 2026-07-16T22:16:35.595108140Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-16T22:16:35.692321935Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-16T22:16:35.779101342Z Gathering data for ns/default... 2026-07-16T22:16:36.117133054Z Gathering data for ns/openshift... 2026-07-16T22:16:36.541748747Z Gathering data for ns/kube-system... 2026-07-16T22:16:37.440022844Z Gathering data for ns/openshift-etcd... 2026-07-16T22:16:37.760775211Z Wrote inspect data to must-gather. 2026-07-16T22:16:37.760813617Z error: inspection completed with the errors occurred while gathering data: 2026-07-16T22:16:37.760813617Z namespaces "assisted-installer" not found 2026-07-16T22:16:38.122511728Z Waiting on subprocesses to finish execution. 2026-07-16T22:16:38.131527062Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-16T22:16:38.135999926Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-16T22:16:38.141322687Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-16T22:16:38.152826461Z INFO: Gathering HAProxy config files 2026-07-16T22:16:38.160558573Z INFO: Waiting for node performance related collection to complete ... 2026-07-16T22:16:38.163742053Z INFO: Collecting host service logs for crio 2026-07-16T22:16:38.164131576Z INFO: Collecting host service logs for kubelet 2026-07-16T22:16:38.164525024Z INFO: Collecting host service logs for rpm-ostreed 2026-07-16T22:16:38.164909572Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-16T22:16:38.165301457Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-16T22:16:38.165682695Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-16T22:16:38.166054324Z INFO: Collecting host service logs for NetworkManager 2026-07-16T22:16:38.166525206Z INFO: Collecting host service logs for openvswitch 2026-07-16T22:16:38.166887726Z INFO: Collecting host service logs for ovs-configuration 2026-07-16T22:16:38.167256394Z INFO: Collecting host service logs for ovsdb-server 2026-07-16T22:16:38.167650534Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-16T22:16:38.168101205Z INFO: Waiting for worker host service log collection to complete ... 2026-07-16T22:16:38.189497525Z Executing Istio gather script 2026-07-16T22:16:38.586139060Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-16T22:16:38.817085205Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-qkcbn 2026-07-16T22:16:38.817131317Z insights-runtime-extractor-cmvdd 2026-07-16T22:16:38.817150880Z insights-runtime-extractor-jdmzl 2026-07-16T22:16:38.817169658Z insights-runtime-extractor-mbqr7 2026-07-16T22:16:39.014051798Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-16T22:16:39.070333257Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-16T22:16:39.174056857Z INFO: "metallb-operator" not detected. Skipping. 2026-07-16T22:16:39.176324123Z No resources found in openshift-etcd namespace. 2026-07-16T22:16:39.289993891Z Wrote inspect data to must-gather. 2026-07-16T22:16:39.393371052Z ERROR: No running kube-apiserver pods found 2026-07-16T22:16:39.516688660Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-16T22:16:39.546292150Z No resources found 2026-07-16T22:16:39.554584103Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-16T22:16:39.554674222Z INFO: on-disk MachineConfig config collection complete. 2026-07-16T22:16:39.560803789Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-16T22:16:39.560803789Z See 'oc rsync -h' for help and examples 2026-07-16T22:16:39.670198368Z error: the server doesn't have a resource type "performanceprofile" 2026-07-16T22:16:39.679052166Z INFO: Worker host service log collection to complete. 2026-07-16T22:16:39.696990480Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-16T22:16:40.049468111Z tar: Removing leading `/' from member names 2026-07-16T22:16:40.062752566Z INFO: HAProxy config collection complete. 2026-07-16T22:16:40.094986041Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.097893101Z No resources found 2026-07-16T22:16:40.298185727Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.461720882Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:16:40.466618031Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-16T22:16:40.528061539Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.663870656Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-16T22:16:40.689971772Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-16T22:16:40.744509666Z INFO: 'previous-logs' folder not found on ip-10-0-138-41.ec2.internal, skipping... 2026-07-16T22:16:40.761600612Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.788043851Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.830848500Z Wrote inspect data to must-gather. 2026-07-16T22:16:40.892388713Z Gathering data for ns/openshift-monitoring... 2026-07-16T22:16:41.027781827Z Wrote inspect data to must-gather. 2026-07-16T22:16:41.041302548Z error: the server doesn't have a resource type "machineconfigs" 2026-07-16T22:16:41.097383202Z INFO: 'previous-logs' folder not found on ip-10-0-139-28.ec2.internal, skipping... 2026-07-16T22:16:41.251574595Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-16T22:16:41.255859109Z INFO: Getting rules from prometheus-k8s-0 2026-07-16T22:16:41.347670321Z Wrote inspect data to must-gather. 2026-07-16T22:16:41.470427733Z INFO: 'previous-logs' folder not found on ip-10-0-140-201.ec2.internal, skipping... 2026-07-16T22:16:41.470468950Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-16T22:16:41.470493035Z INFO: Machine Config Daemon termination log collection complete. 2026-07-16T22:16:41.528819568Z Wrote inspect data to must-gather. 2026-07-16T22:16:41.682756374Z Wrote inspect data to must-gather. 2026-07-16T22:16:41.705078888Z INFO: Getting status/config from prometheus-k8s-0 2026-07-16T22:16:41.739033314Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-16T22:16:41.848750110Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-16T22:16:41.987969136Z Wrote inspect data to must-gather. 2026-07-16T22:16:42.009928648Z INFO: INTERCONNECT MODE 2026-07-16T22:16:42.009960801Z INFO: Gathering ovn-kubernetes DBs 2026-07-16T22:16:42.142374979Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-16T22:16:42.198869719Z INFO: Gathering OVN_Northbound from ovnkube-node-dtgfw... 2026-07-16T22:16:42.199210318Z INFO: Gathering OVN_Northbound from ovnkube-node-pzvqm... 2026-07-16T22:16:42.199500461Z INFO: Gathering OVN_Northbound from ovnkube-node-t77lx... 2026-07-16T22:16:42.199803254Z INFO: Gathering OVN_Southbound from ovnkube-node-dtgfw... 2026-07-16T22:16:42.201183141Z INFO: Gathering OVN_Southbound from ovnkube-node-pzvqm... 2026-07-16T22:16:42.201441839Z INFO: Gathering OVN_Southbound from ovnkube-node-t77lx... 2026-07-16T22:16:42.258316691Z Wrote inspect data to must-gather. 2026-07-16T22:16:42.496242478Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-16T22:16:42.517508186Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-16T22:16:42.537021330Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.563475386Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.565586081Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.596548279Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.606608832Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.622477864Z tar: Removing leading `/' from member names 2026-07-16T22:16:42.681073900Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-16T22:16:42.681771070Z Wrote inspect data to must-gather. 2026-07-16T22:16:42.784489172Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-16T22:16:42.917187530Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:42.952114679Z Wrote inspect data to must-gather. 2026-07-16T22:16:43.079704046Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-16T22:16:43.127635559Z INFO: Waiting for network log collection to complete ... 2026-07-16T22:16:43.127749470Z INFO: Waiting for ovnk database copies to complete ... 2026-07-16T22:16:43.128290070Z INFO: Copying ovnk databases complete. 2026-07-16T22:16:43.130401555Z 3.2M must-gather/network_logs/ovnk_database_store 2026-07-16T22:16:43.132797527Z ovnk_database_store/ 2026-07-16T22:16:43.132797527Z ovnk_database_store/ovnkube-node-pzvqm_nbdb 2026-07-16T22:16:43.141059618Z ovnk_database_store/ovnkube-node-dtgfw_sbdb 2026-07-16T22:16:43.164614583Z ovnk_database_store/ovnkube-node-pzvqm_sbdb 2026-07-16T22:16:43.182242277Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:43.185247016Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-16T22:16:43.189707643Z ovnk_database_store/ovnkube-node-t77lx_nbdb 2026-07-16T22:16:43.198365678Z ovnk_database_store/ovnkube-node-dtgfw_nbdb 2026-07-16T22:16:43.207183105Z ovnk_database_store/ovnkube-node-t77lx_sbdb 2026-07-16T22:16:43.228738055Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-16T22:16:43.232987214Z INFO: Network log collection complete. 2026-07-16T22:16:43.315433285Z Wrote inspect data to must-gather. 2026-07-16T22:16:43.423618218Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:43.423618218Z Gathering data for ns/openshift-network-console... 2026-07-16T22:16:43.547830929Z INFO: Getting status from alertmanager-main-0 2026-07-16T22:16:43.645312878Z Wrote inspect data to must-gather. 2026-07-16T22:16:43.661302819Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:43.665103132Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-16T22:16:43.893035372Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:43.909262096Z Gathering data for ns/openshift-console-operator... 2026-07-16T22:16:43.978959910Z Wrote inspect data to must-gather. 2026-07-16T22:16:44.109436242Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:44.112175685Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-16T22:16:44.286579105Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-16T22:16:44.290583303Z INFO: Fallback to identify the container image from release info 2026-07-16T22:16:44.352646103Z Gathering data for ns/openshift-console... 2026-07-16T22:16:44.358214332Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:44.562627991Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:44.565511026Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-16T22:16:44.745548545Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:44.897147695Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-16T22:16:44.949187882Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:44.953943611Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-16T22:16:45.039394017Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb 2026-07-16T22:16:45.159904951Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:45.188547206Z daemonset.apps/perf-node-gather-daemonset created 2026-07-16T22:16:45.348831895Z Waiting for performance profile collector pods to become ready: 1 2026-07-16T22:16:45.393002532Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:45.395694623Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-16T22:16:45.480872234Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:16:45.481547110Z Gathering data for ns/openshift-dns-operator... 2026-07-16T22:16:45.598399034Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:45.816584255Z Gathering data for ns/openshift-dns... 2026-07-16T22:16:45.838040094Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:45.840302212Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-16T22:16:45.992974474Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:46.186315567Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:46.188788411Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-16T22:16:46.372587476Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:46.463373321Z Gathering data for ns/openshift-image-registry... 2026-07-16T22:16:46.505908017Z Waiting for performance profile collector pods to become ready: 2 2026-07-16T22:16:46.568892147Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:46.570866975Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-16T22:16:46.752784588Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:46.950822502Z Gathering data for ns/openshift-ingress-operator... 2026-07-16T22:16:46.957197632Z Wrote inspect data to /must-gather/istio. 2026-07-16T22:16:47.344980589Z Gathering data for ns/openshift-ingress... 2026-07-16T22:16:47.652922326Z Waiting for performance profile collector pods to become ready: 3 2026-07-16T22:16:47.720548465Z Gathering data for ns/openshift-ingress-canary... 2026-07-16T22:16:47.735062568Z Done executing Istio gather script 2026-07-16T22:16:48.095790385Z Gathering data for ns/openshift-insights... 2026-07-16T22:16:48.790069299Z Waiting for performance profile collector pods to become ready: 4 2026-07-16T22:16:49.919713908Z Waiting for performance profile collector pods to become ready: 5 2026-07-16T22:16:50.485622698Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:16:50.511824074Z Gathering data for ns/kserve... 2026-07-16T22:16:51.057969064Z Waiting for performance profile collector pods to become ready: 6 2026-07-16T22:16:52.169330120Z Gathering data for ns/openshift-config... 2026-07-16T22:16:52.181508176Z Waiting for performance profile collector pods to become ready: 7 2026-07-16T22:16:52.439455927Z Gathering data for ns/openshift-config-managed... 2026-07-16T22:16:52.852456424Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-16T22:16:53.145361136Z Gathering data for ns/openshift-kube-apiserver... 2026-07-16T22:16:53.303916737Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-16T22:16:53.451006711Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-16T22:16:53.592081845Z Collecting performance related data for node ip-10-0-139-28.ec2.internal 2026-07-16T22:16:53.601080949Z Collecting performance related data for node ip-10-0-140-201.ec2.internal 2026-07-16T22:16:53.611491590Z Collecting performance related data for node ip-10-0-138-41.ec2.internal 2026-07-16T22:16:53.794617499Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-16T22:16:54.138702404Z Gathering data for ns/openshift-kube-scheduler... 2026-07-16T22:16:54.472756352Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-16T22:16:54.866992194Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-16T22:16:55.272167820Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-16T22:16:55.490139807Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:16:55.595059834Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-16T22:16:56.164456397Z Gathering data for ns/openshift-multus... 2026-07-16T22:16:57.667340591Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-16T22:16:59.007348708Z Gathering data for ns/openshift-host-network... 2026-07-16T22:16:59.324591634Z Gathering data for ns/openshift-network-diagnostics... 2026-07-16T22:16:59.868590693Z Gathering data for ns/openshift-network-node-identity... 2026-07-16T22:17:00.338917169Z Gathering data for ns/openshift-network-operator... 2026-07-16T22:17:00.496784177Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:17:00.791843433Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-16T22:17:01.061794490Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-16T22:17:01.594807069Z Gathering data for ns/openshift-apiserver-operator... 2026-07-16T22:17:01.905613234Z Gathering data for ns/openshift-apiserver... 2026-07-16T22:17:01.953763152Z Collecting kubelet logs for node ip-10-0-139-28.ec2.internal 2026-07-16T22:17:01.956056582Z Collecting kubelet logs for node ip-10-0-140-201.ec2.internal 2026-07-16T22:17:01.959081168Z Collecting kubelet logs for node ip-10-0-138-41.ec2.internal 2026-07-16T22:17:02.329454098Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wsh68 namespace 2026-07-16T22:17:02.334161535Z INFO: Node performance data collection complete. 2026-07-16T22:17:02.394674477Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-16T22:17:02.666861120Z Gathering data for ns/openshift-controller-manager... 2026-07-16T22:17:02.924952573Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-16T22:17:03.554365535Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-16T22:17:03.831481381Z Gathering data for ns/openshift-service-ca-operator... 2026-07-16T22:17:04.127570033Z Gathering data for ns/openshift-service-ca... 2026-07-16T22:17:04.485358069Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-16T22:17:05.375509272Z Wrote inspect data to must-gather. 2026-07-16T22:17:05.375697271Z error: inspection completed with the errors occurred while gathering data: 2026-07-16T22:17:05.375697271Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-16T22:17:05.502621622Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-16T22:17:05.510743723Z error: the server doesn't have a resource type "clusters" 2026-07-16T22:17:05.660335831Z Caches written to disk