2026-07-16T18:25:00.554210623Z [disk usage checker] Started
2026-07-16T18:25:00.557561975Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:00.753649369Z Gathering data for ns/openshift-cluster-version...
2026-07-16T18:25:00.905388405Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T18:25:01.009162560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:25:01.103770567Z Gathering data for ns/default...
2026-07-16T18:25:01.459274605Z Gathering data for ns/openshift...
2026-07-16T18:25:01.934299279Z Gathering data for ns/kube-system...
2026-07-16T18:25:02.715242547Z Gathering data for ns/openshift-etcd...
2026-07-16T18:25:03.090164865Z Wrote inspect data to must-gather.
2026-07-16T18:25:03.090197041Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T18:25:03.090197041Z     namespaces "assisted-installer" not found
2026-07-16T18:25:03.287358918Z Waiting on subprocesses to finish execution.
2026-07-16T18:25:03.294136093Z INFO: Gathering HAProxy config files
2026-07-16T18:25:03.304242692Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T18:25:03.311022054Z INFO: Collecting host service logs for crio
2026-07-16T18:25:03.311363575Z INFO: Collecting host service logs for kubelet
2026-07-16T18:25:03.311722850Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T18:25:03.312053976Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T18:25:03.312382587Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T18:25:03.312737782Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T18:25:03.313086323Z INFO: Collecting host service logs for NetworkManager
2026-07-16T18:25:03.313404816Z INFO: Collecting host service logs for openvswitch
2026-07-16T18:25:03.313735166Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T18:25:03.314084522Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T18:25:03.314423773Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T18:25:03.314820771Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T18:25:03.317935094Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T18:25:03.344025908Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T18:25:03.357506012Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T18:25:03.380081253Z Executing Istio gather script
2026-07-16T18:25:03.579532684Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-dhckd
2026-07-16T18:25:03.579592612Z insights-runtime-extractor-gfv6q
2026-07-16T18:25:03.579613820Z insights-runtime-extractor-hsd9q
2026-07-16T18:25:03.579634457Z insights-runtime-extractor-tcfbb
2026-07-16T18:25:03.680888540Z No resources found in openshift-etcd namespace.
2026-07-16T18:25:04.139986705Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T18:25:04.262695384Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T18:25:04.312679536Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T18:25:04.312679536Z See 'oc rsync -h' for help and examples
2026-07-16T18:25:04.337508174Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T18:25:04.381924444Z No resources found
2026-07-16T18:25:04.549001987Z Wrote inspect data to must-gather.
2026-07-16T18:25:04.739081177Z ERROR: No running kube-apiserver pods found
2026-07-16T18:25:04.766514358Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T18:25:04.849629217Z Wrote inspect data to must-gather.
2026-07-16T18:25:04.893319150Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T18:25:04.894884445Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T18:25:04.902221059Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T18:25:04.902295098Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T18:25:05.029644607Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T18:25:05.165930458Z INFO: Worker host service log collection to complete.
2026-07-16T18:25:05.192035358Z No resources found
2026-07-16T18:25:05.254221111Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T18:25:05.346347764Z Wrote inspect data to must-gather.
2026-07-16T18:25:05.530611610Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:25:05.681727440Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:05.733544568Z tar: Removing leading `/' from member names
2026-07-16T18:25:05.734034799Z Wrote inspect data to must-gather.
2026-07-16T18:25:05.741276876Z INFO: HAProxy config collection complete.
2026-07-16T18:25:05.749738273Z Wrote inspect data to must-gather.
2026-07-16T18:25:05.787361553Z Wrote inspect data to must-gather.
2026-07-16T18:25:05.797762941Z INFO: 'previous-logs' folder not found on ip-10-0-135-249.ec2.internal, skipping...
2026-07-16T18:25:05.809436134Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T18:25:05.929714412Z Wrote inspect data to must-gather.
2026-07-16T18:25:06.081216475Z Wrote inspect data to must-gather.
2026-07-16T18:25:06.115226252Z INFO: 'previous-logs' folder not found on ip-10-0-139-103.ec2.internal, skipping...
2026-07-16T18:25:06.145887895Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T18:25:06.224971952Z Gathering data for ns/openshift-monitoring...
2026-07-16T18:25:06.245521730Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T18:25:06.300412235Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T18:25:06.358241075Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T18:25:06.398092054Z Wrote inspect data to must-gather.
2026-07-16T18:25:06.426027416Z INFO: 'previous-logs' folder not found on ip-10-0-141-151.ec2.internal, skipping...
2026-07-16T18:25:06.426193143Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T18:25:06.426244401Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T18:25:06.594674276Z Wrote inspect data to must-gather.
2026-07-16T18:25:06.646189035Z Wrote inspect data to must-gather.
2026-07-16T18:25:06.737562241Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T18:25:06.763316538Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T18:25:06.970138848Z INFO: INTERCONNECT MODE
2026-07-16T18:25:06.970169211Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T18:25:06.982061046Z Wrote inspect data to must-gather.
2026-07-16T18:25:07.148539080Z INFO: Gathering OVN_Northbound from ovnkube-node-4lfqt...
2026-07-16T18:25:07.148760458Z INFO: Gathering OVN_Northbound from ovnkube-node-jcjq6...
2026-07-16T18:25:07.149005905Z INFO: Gathering OVN_Northbound from ovnkube-node-vfz5t...
2026-07-16T18:25:07.149318370Z INFO: Gathering OVN_Southbound from ovnkube-node-4lfqt...
2026-07-16T18:25:07.149710047Z INFO: Gathering OVN_Southbound from ovnkube-node-jcjq6...
2026-07-16T18:25:07.150020001Z INFO: Gathering OVN_Southbound from ovnkube-node-vfz5t...
2026-07-16T18:25:07.174373476Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T18:25:07.252771924Z Wrote inspect data to must-gather.
2026-07-16T18:25:07.444836746Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.458695725Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.498815947Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.505983489Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.528715000Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.531518616Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T18:25:07.540277286Z tar: Removing leading `/' from member names
2026-07-16T18:25:07.584109419Z Wrote inspect data to must-gather.
2026-07-16T18:25:07.647374389Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T18:25:07.801674779Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T18:25:07.838304513Z Wrote inspect data to must-gather.
2026-07-16T18:25:07.904523010Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:07.923703993Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T18:25:08.063898755Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T18:25:08.071394712Z Wrote inspect data to must-gather.
2026-07-16T18:25:08.087136451Z INFO: Waiting for network log collection to complete ...
2026-07-16T18:25:08.087189811Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T18:25:08.087732469Z INFO: Copying ovnk databases complete.
2026-07-16T18:25:08.089831183Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-07-16T18:25:08.092868918Z ovnk_database_store/
2026-07-16T18:25:08.092938017Z ovnk_database_store/ovnkube-node-jcjq6_sbdb
2026-07-16T18:25:08.118045164Z ovnk_database_store/ovnkube-node-vfz5t_nbdb
2026-07-16T18:25:08.128064825Z ovnk_database_store/ovnkube-node-vfz5t_sbdb
2026-07-16T18:25:08.153361532Z ovnk_database_store/ovnkube-node-4lfqt_nbdb
2026-07-16T18:25:08.160294780Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:08.163215160Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T18:25:08.163792909Z ovnk_database_store/ovnkube-node-jcjq6_nbdb
2026-07-16T18:25:08.172867832Z ovnk_database_store/ovnkube-node-4lfqt_sbdb
2026-07-16T18:25:08.203855599Z INFO: Network log collection complete.
2026-07-16T18:25:08.308267276Z Wrote inspect data to must-gather.
2026-07-16T18:25:08.400736474Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:08.484446996Z INFO: Getting status from alertmanager-main-0
2026-07-16T18:25:08.524427790Z W0716 18:25:08.524383    1502 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-16T18:25:08.530162269Z Wrote inspect data to must-gather.
2026-07-16T18:25:08.589939095Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:25:08.650097436Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:08.653559278Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T18:25:08.784298558Z Gathering data for ns/openshift-network-console...
2026-07-16T18:25:08.814999400Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T18:25:08.817128732Z INFO: Fallback to identify the container image from release info
2026-07-16T18:25:08.874361324Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.081788291Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.085060092Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T18:25:09.239072527Z Gathering data for ns/openshift-console-operator...
2026-07-16T18:25:09.311667152Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.467053170Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-16T18:25:09.522994278Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.525950739Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-16T18:25:09.623020358Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T18:25:09.651407608Z Gathering data for ns/openshift-console...
2026-07-16T18:25:09.713729133Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.769020697Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T18:25:09.910533557Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:09.913076017Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-16T18:25:10.101049126Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:10.254599282Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T18:25:10.304213306Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:10.307234359Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-16T18:25:10.487325613Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:10.686381482Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:10.689762212Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-16T18:25:10.721019547Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:10.808781559Z Gathering data for ns/openshift-dns-operator...
2026-07-16T18:25:10.896922998Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:10.919308106Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T18:25:11.092569761Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:11.096595896Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-16T18:25:11.179598808Z Gathering data for ns/openshift-dns...
2026-07-16T18:25:11.272013123Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:11.446014662Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:11.448653728Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T18:25:11.613985809Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:11.804879651Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:25:11.909166530Z Gathering data for ns/openshift-image-registry...
2026-07-16T18:25:12.047908055Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T18:25:12.431799057Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T18:25:12.511615451Z Done executing Istio gather script
2026-07-16T18:25:12.770398018Z Gathering data for ns/openshift-ingress...
2026-07-16T18:25:13.119184216Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T18:25:13.168262932Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T18:25:13.514509874Z Gathering data for ns/openshift-insights...
2026-07-16T18:25:14.286802404Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T18:25:15.410536925Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T18:25:15.714494636Z Gathering data for ns/kserve...
2026-07-16T18:25:15.725774215Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:16.527878014Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T18:25:17.280704735Z Gathering data for ns/openshift-config...
2026-07-16T18:25:17.598120214Z Gathering data for ns/openshift-config-managed...
2026-07-16T18:25:17.662719210Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T18:25:17.915358335Z Collecting performance related data for node ip-10-0-139-103.ec2.internal
2026-07-16T18:25:17.922310219Z Collecting performance related data for node ip-10-0-141-151.ec2.internal
2026-07-16T18:25:17.929721355Z Collecting performance related data for node ip-10-0-135-249.ec2.internal
2026-07-16T18:25:18.136953682Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T18:25:18.467761853Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T18:25:18.830680633Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T18:25:19.212441295Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T18:25:19.570044666Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T18:25:19.880942591Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T18:25:20.227339208Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T18:25:20.646296176Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T18:25:20.732827051Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:20.989891866Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T18:25:21.521697765Z Gathering data for ns/openshift-multus...
2026-07-16T18:25:23.011286059Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T18:25:24.353919708Z Gathering data for ns/openshift-host-network...
2026-07-16T18:25:24.664444449Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T18:25:25.238496370Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T18:25:25.671573397Z Gathering data for ns/openshift-network-operator...
2026-07-16T18:25:25.738381933Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:25.987212057Z Collecting kubelet logs for node ip-10-0-139-103.ec2.internal
2026-07-16T18:25:25.988997943Z Collecting kubelet logs for node ip-10-0-141-151.ec2.internal
2026-07-16T18:25:25.990844199Z Collecting kubelet logs for node ip-10-0-135-249.ec2.internal
2026-07-16T18:25:26.099066629Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T18:25:26.374831488Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-hpk2v namespace
2026-07-16T18:25:26.380352336Z INFO: Node performance data collection complete.
2026-07-16T18:25:26.420489181Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T18:25:26.938656426Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T18:25:27.283225491Z Gathering data for ns/openshift-apiserver...
2026-07-16T18:25:27.693926251Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T18:25:27.959486506Z Gathering data for ns/openshift-controller-manager...
2026-07-16T18:25:28.235715178Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T18:25:28.893287752Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T18:25:29.223741185Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T18:25:29.538000906Z Gathering data for ns/openshift-service-ca...
2026-07-16T18:25:29.918095153Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T18:25:30.743066492Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-16T18:25:30.820332921Z Wrote inspect data to must-gather.
2026-07-16T18:25:30.820375459Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T18:25:30.820375459Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T18:25:30.947772769Z error: the server doesn't have a resource type "clusters"
2026-07-16T18:25:31.088378039Z Caches written to disk