2026-07-09T18:05:56.019097868Z [disk usage checker] Started
2026-07-09T18:05:56.022491117Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-09T18:05:56.219787322Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:05:56.366456083Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:05:56.478002360Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:05:56.564365984Z Gathering data for ns/default...
2026-07-09T18:05:56.930288947Z Gathering data for ns/openshift...
2026-07-09T18:05:57.354413469Z Gathering data for ns/kube-system...
2026-07-09T18:05:58.066652186Z Gathering data for ns/openshift-etcd...
2026-07-09T18:05:58.421214467Z Wrote inspect data to must-gather.
2026-07-09T18:05:58.421244883Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:05:58.421244883Z     namespaces "assisted-installer" not found
2026-07-09T18:05:58.801000538Z Waiting on subprocesses to finish execution.
2026-07-09T18:05:58.809232710Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:05:58.810870037Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:05:58.817443966Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:05:58.821069489Z Executing Istio gather script
2026-07-09T18:05:58.821256452Z INFO: Gathering HAProxy config files
2026-07-09T18:05:58.827660998Z INFO: Collecting host service logs for crio
2026-07-09T18:05:58.827965418Z INFO: Collecting host service logs for kubelet
2026-07-09T18:05:58.828273378Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:05:58.828637938Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:05:58.829030307Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:05:58.829325076Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:05:58.829621904Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:05:58.829950987Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:05:58.830238326Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:05:58.830530518Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:05:58.830908677Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:05:58.831315127Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:05:58.998219607Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:05:59.380497126Z No resources found in openshift-etcd namespace.
2026-07-09T18:05:59.545555872Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:05:59.545593327Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:05:59.592663286Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-xkn6j
2026-07-09T18:05:59.592733766Z insights-runtime-extractor-54d9n
2026-07-09T18:05:59.592758515Z insights-runtime-extractor-gz4jw
2026-07-09T18:05:59.592777932Z insights-runtime-extractor-xtb8z
2026-07-09T18:06:00.215736484Z No resources found
2026-07-09T18:06:00.310909979Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:06:00.534470212Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:06:00.579486451Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:06:00.781885252Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:06:00.786870661Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:06:00.819646758Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:06:00.893194860Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:06:00.908571778Z INFO: Worker host service log collection to complete.
2026-07-09T18:06:00.946383090Z Wrote inspect data to must-gather.
2026-07-09T18:06:00.959650881Z Wrote inspect data to must-gather.
2026-07-09T18:06:00.975701303Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.001073720Z No resources found
2026-07-09T18:06:01.050798442Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.057754898Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:06:01.069171586Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-09T18:06:01.085529558Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:06:01.085529558Z See 'oc rsync -h' for help and examples
2026-07-09T18:06:01.147872135Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:06:01.164867544Z ERROR: No running kube-apiserver pods found
2026-07-09T18:06:01.217920372Z INFO: 'previous-logs' folder not found on ip-10-0-128-245.ec2.internal, skipping...
2026-07-09T18:06:01.250949699Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.333793367Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:06:01.421270568Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.460911906Z tar: Removing leading `/' from member names
2026-07-09T18:06:01.469700862Z INFO: HAProxy config collection complete.
2026-07-09T18:06:01.476044009Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.614093019Z INFO: 'previous-logs' folder not found on ip-10-0-135-252.ec2.internal, skipping...
2026-07-09T18:06:01.646220041Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:06:01.706815265Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:06:01.755112241Z Wrote inspect data to must-gather.
2026-07-09T18:06:01.861081675Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:06:01.912597291Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:06:01.944457955Z INFO: 'previous-logs' folder not found on ip-10-0-143-142.ec2.internal, skipping...
2026-07-09T18:06:01.944493600Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:06:01.944531303Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:06:02.036483274Z Wrote inspect data to must-gather.
2026-07-09T18:06:02.189538884Z Wrote inspect data to must-gather.
2026-07-09T18:06:02.311615115Z Wrote inspect data to must-gather.
2026-07-09T18:06:02.330902874Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:06:02.382088236Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:06:02.465514463Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:06:02.614988303Z Wrote inspect data to must-gather.
2026-07-09T18:06:02.618546611Z INFO: INTERCONNECT MODE
2026-07-09T18:06:02.618568135Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:06:02.771681589Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:06:02.792864428Z INFO: Gathering OVN_Northbound from ovnkube-node-64kgs...
2026-07-09T18:06:02.793258154Z INFO: Gathering OVN_Northbound from ovnkube-node-jxmsd...
2026-07-09T18:06:02.793554603Z INFO: Gathering OVN_Northbound from ovnkube-node-zpzmp...
2026-07-09T18:06:02.793874672Z INFO: Gathering OVN_Southbound from ovnkube-node-64kgs...
2026-07-09T18:06:02.794155077Z INFO: Gathering OVN_Southbound from ovnkube-node-jxmsd...
2026-07-09T18:06:02.794421602Z INFO: Gathering OVN_Southbound from ovnkube-node-zpzmp...
2026-07-09T18:06:02.888016595Z Wrote inspect data to must-gather.
2026-07-09T18:06:03.086869599Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.117936216Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.127402567Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.141438715Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:06:03.150330026Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.184680168Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.190424607Z tar: Removing leading `/' from member names
2026-07-09T18:06:03.265595461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:06:03.272089314Z Wrote inspect data to must-gather.
2026-07-09T18:06:03.339255919Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:06:03.403147959Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:06:03.545027713Z Wrote inspect data to must-gather.
2026-07-09T18:06:03.573882744Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:03.696948223Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:06:03.733968823Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:06:03.734089491Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:06:03.734883586Z INFO: Copying ovnk databases complete.
2026-07-09T18:06:03.737010726Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:06:03.740051784Z ovnk_database_store/
2026-07-09T18:06:03.740098344Z ovnk_database_store/ovnkube-node-zpzmp_sbdb
2026-07-09T18:06:03.762677348Z ovnk_database_store/ovnkube-node-jxmsd_nbdb
2026-07-09T18:06:03.771489768Z ovnk_database_store/ovnkube-node-64kgs_sbdb
2026-07-09T18:06:03.791830265Z ovnk_database_store/ovnkube-node-64kgs_nbdb
2026-07-09T18:06:03.794576917Z W0709 18:06:03.794524    1416 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-09T18:06:03.801890309Z ovnk_database_store/ovnkube-node-jxmsd_sbdb
2026-07-09T18:06:03.802404429Z Wrote inspect data to must-gather.
2026-07-09T18:06:03.823702332Z ovnk_database_store/ovnkube-node-zpzmp_nbdb
2026-07-09T18:06:03.824063772Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:03.826390225Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:06:03.836637574Z INFO: Network log collection complete.
2026-07-09T18:06:03.939990620Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:06:04.046194474Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:04.131651568Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:06:04.132238981Z Wrote inspect data to must-gather.
2026-07-09T18:06:04.134420753Z Gathering data for ns/openshift-network-console...
2026-07-09T18:06:04.270391575Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:04.274575853Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:06:04.445393027Z Wrote inspect data to must-gather.
2026-07-09T18:06:04.485408350Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:04.708420405Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:06:04.711256900Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:04.714537398Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:06:04.766130741Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:06:04.768883352Z INFO: Fallback to identify the container image from release info
2026-07-09T18:06:04.948244665Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.135866246Z Gathering data for ns/openshift-console...
2026-07-09T18:06:05.143634131Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.145950027Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T18:06:05.313291986Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.455516116Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:06:05.528039557Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.530750704Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T18:06:05.604852993Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:06:05.743270248Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.745320515Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:06:05.764529392Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:06:05.935589325Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:05.938030563Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T18:06:06.081927992Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-09T18:06:06.144949113Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:06.347944888Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:06:06.358762140Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:06.362379708Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:06:06.536267911Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:06.698335132Z Gathering data for ns/openshift-dns...
2026-07-09T18:06:06.734847122Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:06.737843886Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:06:06.888465269Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:06:06.903402079Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:07.091313202Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:07.093755524Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:06:07.276956170Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:07.372145342Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:06:07.469941936Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:06:07.908410623Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:06:08.039787274Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:06:08.179734999Z Done executing Istio gather script
2026-07-09T18:06:08.285505882Z Gathering data for ns/openshift-ingress...
2026-07-09T18:06:08.635232470Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:06:09.033457835Z Gathering data for ns/openshift-insights...
2026-07-09T18:06:09.159987897Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:06:10.274352528Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:06:11.086783577Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-09T18:06:11.241496079Z Gathering data for ns/kserve...
2026-07-09T18:06:11.384154763Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:06:12.497684878Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:06:12.759934914Z Gathering data for ns/openshift-config...
2026-07-09T18:06:13.045467523Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:06:13.494446170Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:06:13.624769147Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:06:13.802872708Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:06:13.905300701Z Collecting performance related data for node ip-10-0-128-245.ec2.internal
2026-07-09T18:06:13.911921705Z Collecting performance related data for node ip-10-0-135-252.ec2.internal
2026-07-09T18:06:13.919124388Z Collecting performance related data for node ip-10-0-143-142.ec2.internal
2026-07-09T18:06:14.148359767Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:06:14.552843859Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:06:14.935422382Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:06:15.267163574Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:06:15.648972721Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:06:15.989101250Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:06:16.091484883Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-09T18:06:16.374484578Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:06:16.944481603Z Gathering data for ns/openshift-multus...
2026-07-09T18:06:18.437974061Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:06:19.786365491Z Gathering data for ns/openshift-host-network...
2026-07-09T18:06:20.105344796Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:06:20.648290044Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:06:21.096129340Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-09T18:06:21.112018122Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:06:21.593424815Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:06:21.718864737Z Collecting kubelet logs for node ip-10-0-128-245.ec2.internal
2026-07-09T18:06:21.720525390Z Collecting kubelet logs for node ip-10-0-135-252.ec2.internal
2026-07-09T18:06:21.722179583Z Collecting kubelet logs for node ip-10-0-143-142.ec2.internal
2026-07-09T18:06:21.922649857Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:06:22.103081724Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-rgx2c namespace
2026-07-09T18:06:22.107786708Z INFO: Node performance data collection complete.
2026-07-09T18:06:22.447762380Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:06:22.794624300Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:06:23.208958217Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:06:23.505394125Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:06:23.868551966Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:06:24.587673148Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:06:24.910566604Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:06:25.261186300Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:06:25.661229737Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:06:26.101305477Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-09T18:06:26.590065924Z Wrote inspect data to must-gather.
2026-07-09T18:06:26.590109056Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:06:26.590109056Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:06:26.714951667Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:06:26.885490309Z Caches written to disk