2026-07-22T12:08:45.541213423Z [disk usage checker] Started
2026-07-22T12:08:45.545050635Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:08:45.744199121Z Gathering data for ns/openshift-cluster-version...
2026-07-22T12:08:45.895121974Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-22T12:08:45.994502345Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-22T12:08:46.085573452Z Gathering data for ns/default...
2026-07-22T12:08:46.393148527Z Gathering data for ns/openshift...
2026-07-22T12:08:46.812518613Z Gathering data for ns/kube-system...
2026-07-22T12:08:47.573505089Z Gathering data for ns/openshift-etcd...
2026-07-22T12:08:47.879596018Z Wrote inspect data to must-gather.
2026-07-22T12:08:47.879628798Z error: inspection completed with the errors occurred while gathering data:
2026-07-22T12:08:47.879628798Z     namespaces "assisted-installer" not found
2026-07-22T12:08:48.086812099Z Waiting on subprocesses to finish execution.
2026-07-22T12:08:48.090685698Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-22T12:08:48.093955883Z INFO: Gathering HAProxy config files
2026-07-22T12:08:48.106510355Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-22T12:08:48.108237120Z Executing Istio gather script
2026-07-22T12:08:48.115424072Z INFO: Collecting host service logs for crio
2026-07-22T12:08:48.115764244Z INFO: Collecting host service logs for kubelet
2026-07-22T12:08:48.116095898Z INFO: Collecting host service logs for rpm-ostreed
2026-07-22T12:08:48.116423057Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-22T12:08:48.116773103Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-22T12:08:48.117103210Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-22T12:08:48.117424872Z INFO: Collecting host service logs for NetworkManager
2026-07-22T12:08:48.117767018Z INFO: Collecting host service logs for openvswitch
2026-07-22T12:08:48.118085175Z INFO: Collecting host service logs for ovs-configuration
2026-07-22T12:08:48.118410787Z INFO: Collecting host service logs for ovsdb-server
2026-07-22T12:08:48.118755047Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-22T12:08:48.119150802Z INFO: Waiting for worker host service log collection to complete ...
2026-07-22T12:08:48.122461195Z INFO: Waiting for node performance related collection to complete ...
2026-07-22T12:08:48.126870467Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-22T12:08:48.412010604Z No resources found in openshift-etcd namespace.
2026-07-22T12:08:48.470454755Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-22T12:08:48.699204946Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-kzsvj
2026-07-22T12:08:48.699249717Z insights-runtime-extractor-bxdpv
2026-07-22T12:08:48.699269256Z insights-runtime-extractor-d69p4
2026-07-22T12:08:48.699287799Z insights-runtime-extractor-v4wgf
2026-07-22T12:08:48.750334790Z Wrote inspect data to must-gather.
2026-07-22T12:08:48.851151200Z INFO: "metallb-operator" not detected. Skipping.
2026-07-22T12:08:48.909630062Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-22T12:08:48.915489228Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-22T12:08:48.915597992Z INFO: on-disk MachineConfig config collection complete.
2026-07-22T12:08:48.929191384Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-22T12:08:49.115179611Z error: the server doesn't have a resource type "performanceprofile"
2026-07-22T12:08:49.189548914Z No resources found
2026-07-22T12:08:49.386098837Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-22T12:08:49.386098837Z See 'oc rsync -h' for help and examples
2026-07-22T12:08:49.396822777Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-22T12:08:49.524156986Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-22T12:08:49.527982873Z INFO: Worker host service log collection to complete.
2026-07-22T12:08:49.707039714Z Wrote inspect data to must-gather.
2026-07-22T12:08:49.882702742Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.272865424Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.274975575Z tar: Removing leading `/' from member names
2026-07-22T12:08:50.283341424Z INFO: HAProxy config collection complete.
2026-07-22T12:08:50.308970581Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-22T12:08:50.343314574Z ERROR: No running kube-apiserver pods found
2026-07-22T12:08:50.362878046Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-22T12:08:50.439488332Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-22T12:08:50.462620416Z INFO: 'previous-logs' folder not found on ip-10-0-130-22.ec2.internal, skipping...
2026-07-22T12:08:50.468983974Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.499023972Z No resources found
2026-07-22T12:08:50.528707939Z error: the server doesn't have a resource type "machineconfigs"
2026-07-22T12:08:50.530852927Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.535712271Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-22T12:08:50.563915146Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:08:50.625926855Z Gathering data for ns/openshift-monitoring...
2026-07-22T12:08:50.732293511Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-22T12:08:50.753636910Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.811067358Z Wrote inspect data to must-gather.
2026-07-22T12:08:50.817665499Z INFO: 'previous-logs' folder not found on ip-10-0-132-210.ec2.internal, skipping...
2026-07-22T12:08:50.959049652Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.004641029Z INFO: Getting rules from prometheus-k8s-0
2026-07-22T12:08:51.069772763Z INFO: 'previous-logs' folder not found on ip-10-0-139-45.ec2.internal, skipping...
2026-07-22T12:08:51.069807277Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-22T12:08:51.069807277Z INFO: Machine Config Daemon termination log collection complete.
2026-07-22T12:08:51.077833510Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.140189273Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-22T12:08:51.324198759Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.345625864Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.396367866Z INFO: Getting status/config from prometheus-k8s-0
2026-07-22T12:08:51.544069780Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.597861683Z INFO: INTERCONNECT MODE
2026-07-22T12:08:51.597901376Z INFO: Gathering ovn-kubernetes DBs
2026-07-22T12:08:51.756696549Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-22T12:08:51.772960320Z Wrote inspect data to must-gather.
2026-07-22T12:08:51.778055409Z INFO: Gathering OVN_Northbound from ovnkube-node-5sctd...
2026-07-22T12:08:51.778309825Z INFO: Gathering OVN_Northbound from ovnkube-node-c8nl2...
2026-07-22T12:08:51.778569993Z INFO: Gathering OVN_Northbound from ovnkube-node-fsvpl...
2026-07-22T12:08:51.778889767Z INFO: Gathering OVN_Southbound from ovnkube-node-5sctd...
2026-07-22T12:08:51.779094143Z INFO: Gathering OVN_Southbound from ovnkube-node-c8nl2...
2026-07-22T12:08:51.779379238Z INFO: Gathering OVN_Southbound from ovnkube-node-fsvpl...
2026-07-22T12:08:51.808238253Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-22T12:08:52.053709441Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:52.095117120Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.099738179Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.139380002Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.143997507Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.145634049Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.152957907Z tar: Removing leading `/' from member names
2026-07-22T12:08:52.176672531Z Wrote inspect data to must-gather.
2026-07-22T12:08:52.257079503Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-22T12:08:52.281257090Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-22T12:08:52.318562680Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:52.321547884Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-22T12:08:52.411113478Z Wrote inspect data to must-gather.
2026-07-22T12:08:52.518115589Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-22T12:08:52.554110110Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:52.629444107Z INFO: Waiting for network log collection to complete ...
2026-07-22T12:08:52.629484003Z INFO: Waiting for ovnk database copies to complete ...
2026-07-22T12:08:52.629875069Z INFO: Copying ovnk databases complete.
2026-07-22T12:08:52.631516022Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-22T12:08:52.633723123Z ovnk_database_store/
2026-07-22T12:08:52.633743875Z ovnk_database_store/ovnkube-node-5sctd_nbdb
2026-07-22T12:08:52.642158302Z Wrote inspect data to must-gather.
2026-07-22T12:08:52.650094718Z ovnk_database_store/ovnkube-node-fsvpl_nbdb
2026-07-22T12:08:52.657095355Z ovnk_database_store/ovnkube-node-c8nl2_nbdb
2026-07-22T12:08:52.665767556Z ovnk_database_store/ovnkube-node-fsvpl_sbdb
2026-07-22T12:08:52.688632842Z ovnk_database_store/ovnkube-node-c8nl2_sbdb
2026-07-22T12:08:52.707202079Z ovnk_database_store/ovnkube-node-5sctd_sbdb
2026-07-22T12:08:52.732532505Z INFO: Network log collection complete.
2026-07-22T12:08:52.772316831Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:52.776179087Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-22T12:08:52.780901973Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-22T12:08:52.867908110Z W0722 12:08:52.867854    1501 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-22T12:08:52.876058926Z Wrote inspect data to must-gather.
2026-07-22T12:08:52.940410300Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-22T12:08:52.978893052Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:53.142755115Z Gathering data for ns/openshift-network-console...
2026-07-22T12:08:53.151900453Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-22T12:08:53.153816606Z INFO: Fallback to identify the container image from release info
2026-07-22T12:08:53.160145305Z INFO: Getting status from alertmanager-main-0
2026-07-22T12:08:53.205078507Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:53.209625794Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-22T12:08:53.436963380Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:53.606610066Z Gathering data for ns/openshift-console-operator...
2026-07-22T12:08:53.626882068Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:53.629195213Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-22T12:08:53.752018349Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-22T12:08:53.803428457Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:53.875045894Z daemonset.apps/perf-node-gather-daemonset created
2026-07-22T12:08:54.004348450Z Wrote inspect data to /must-gather/istio.
2026-07-22T12:08:54.012298270Z Waiting for performance profile collector pods to become ready: 1
2026-07-22T12:08:54.049413178Z Gathering data for ns/openshift-console...
2026-07-22T12:08:54.623610901Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-22T12:08:54.662924814Z Done executing Istio gather script
2026-07-22T12:08:55.083175151Z Gathering data for ns/openshift-dns-operator...
2026-07-22T12:08:55.137986501Z Waiting for performance profile collector pods to become ready: 2
2026-07-22T12:08:55.385929109Z Gathering data for ns/openshift-dns...
2026-07-22T12:08:55.569856609Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:08:55.954876787Z Gathering data for ns/openshift-image-registry...
2026-07-22T12:08:56.241834617Z Waiting for performance profile collector pods to become ready: 3
2026-07-22T12:08:56.402245397Z Gathering data for ns/openshift-ingress-operator...
2026-07-22T12:08:56.698149341Z Gathering data for ns/openshift-ingress...
2026-07-22T12:08:56.990210836Z Gathering data for ns/openshift-ingress-canary...
2026-07-22T12:08:57.366728266Z Waiting for performance profile collector pods to become ready: 4
2026-07-22T12:08:57.368261489Z Gathering data for ns/openshift-insights...
2026-07-22T12:08:58.496124375Z Waiting for performance profile collector pods to become ready: 5
2026-07-22T12:08:59.310788038Z Gathering data for ns/openshift-config...
2026-07-22T12:08:59.573842019Z Gathering data for ns/openshift-config-managed...
2026-07-22T12:08:59.631395715Z Waiting for performance profile collector pods to become ready: 6
2026-07-22T12:09:00.041706707Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-22T12:09:00.290379903Z Gathering data for ns/openshift-kube-apiserver...
2026-07-22T12:09:00.574934391Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:09:00.584571521Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-22T12:09:00.735861521Z Waiting for performance profile collector pods to become ready: 7
2026-07-22T12:09:00.901294995Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-22T12:09:01.179870283Z Gathering data for ns/openshift-kube-scheduler...
2026-07-22T12:09:01.420350815Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-22T12:09:01.702196032Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-22T12:09:01.859965019Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-22T12:09:02.074484696Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-22T12:09:02.103717126Z Collecting performance related data for node ip-10-0-132-210.ec2.internal
2026-07-22T12:09:02.115470204Z Collecting performance related data for node ip-10-0-130-22.ec2.internal
2026-07-22T12:09:02.123097679Z Collecting performance related data for node ip-10-0-139-45.ec2.internal
2026-07-22T12:09:02.459987961Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-22T12:09:03.093075723Z Gathering data for ns/openshift-multus...
2026-07-22T12:09:04.696846691Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-22T12:09:05.580803154Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:09:06.123767198Z Gathering data for ns/openshift-host-network...
2026-07-22T12:09:06.426400180Z Gathering data for ns/openshift-network-diagnostics...
2026-07-22T12:09:06.931308970Z Gathering data for ns/openshift-network-node-identity...
2026-07-22T12:09:07.329659519Z Gathering data for ns/openshift-network-operator...
2026-07-22T12:09:07.806487296Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-22T12:09:08.081975410Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-22T12:09:08.608978845Z Gathering data for ns/openshift-apiserver-operator...
2026-07-22T12:09:08.990151126Z Gathering data for ns/openshift-apiserver...
2026-07-22T12:09:09.446036607Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-22T12:09:09.727527345Z Gathering data for ns/openshift-controller-manager...
2026-07-22T12:09:10.074122074Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-22T12:09:10.478029485Z Collecting kubelet logs for node ip-10-0-132-210.ec2.internal
2026-07-22T12:09:10.479880795Z Collecting kubelet logs for node ip-10-0-130-22.ec2.internal
2026-07-22T12:09:10.482098132Z Collecting kubelet logs for node ip-10-0-139-45.ec2.internal
2026-07-22T12:09:10.586899865Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-22T12:09:10.810445706Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-22T12:09:10.830246027Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x7fzg namespace
2026-07-22T12:09:10.836316628Z INFO: Node performance data collection complete.
2026-07-22T12:09:11.160891906Z Gathering data for ns/openshift-service-ca-operator...
2026-07-22T12:09:11.497222610Z Gathering data for ns/openshift-service-ca...
2026-07-22T12:09:11.864147464Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-22T12:09:12.799753163Z Wrote inspect data to must-gather.
2026-07-22T12:09:12.799793956Z error: inspection completed with the errors occurred while gathering data:
2026-07-22T12:09:12.799793956Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-22T12:09:12.922231074Z error: the server doesn't have a resource type "clusters"
2026-07-22T12:09:13.051747387Z Caches written to disk