2026-08-05T16:31:05.115297790Z [disk usage checker] Started
2026-08-05T16:31:05.118784700Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:05.314852808Z Gathering data for ns/openshift-cluster-version...
2026-08-05T16:31:05.468051877Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T16:31:05.593196212Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:31:05.678745703Z Gathering data for ns/default...
2026-08-05T16:31:06.002185376Z Gathering data for ns/openshift...
2026-08-05T16:31:06.455807716Z Gathering data for ns/kube-system...
2026-08-05T16:31:07.210780441Z Gathering data for ns/openshift-etcd...
2026-08-05T16:31:07.550850400Z Wrote inspect data to must-gather.
2026-08-05T16:31:07.550883963Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T16:31:07.550883963Z     namespaces "assisted-installer" not found
2026-08-05T16:31:07.713278754Z Waiting on subprocesses to finish execution.
2026-08-05T16:31:07.721075994Z INFO: Collecting host service logs for crio
2026-08-05T16:31:07.721529404Z INFO: Collecting host service logs for kubelet
2026-08-05T16:31:07.721854372Z INFO: Collecting host service logs for rpm-ostreed
2026-08-05T16:31:07.722198425Z INFO: Collecting host service logs for ostree-finalize-staged
2026-08-05T16:31:07.722613715Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-08-05T16:31:07.722675999Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-08-05T16:31:07.723022025Z INFO: Collecting host service logs for machine-config-daemon-host
2026-08-05T16:31:07.723396799Z INFO: Collecting host service logs for NetworkManager
2026-08-05T16:31:07.723733886Z INFO: Collecting host service logs for openvswitch
2026-08-05T16:31:07.724074524Z INFO: Collecting host service logs for ovs-configuration
2026-08-05T16:31:07.724436571Z INFO: Collecting host service logs for ovsdb-server
2026-08-05T16:31:07.724787299Z INFO: Collecting host service logs for ovs-vswitchd
2026-08-05T16:31:07.725202556Z INFO: Waiting for worker host service log collection to complete ...
2026-08-05T16:31:07.726383658Z INFO: Gathering HAProxy config files
2026-08-05T16:31:07.729284354Z INFO: Gathering machine config daemon's old logs from all nodes
2026-08-05T16:31:07.730618384Z Executing Istio gather script
2026-08-05T16:31:07.745380572Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-08-05T16:31:07.749892552Z INFO: Waiting for node performance related collection to complete ...
2026-08-05T16:31:08.216154544Z INFO: Collecting Insights Archives from insights-operator-77696685b-h4zqk
2026-08-05T16:31:08.216154544Z insights-runtime-extractor-5fzhj
2026-08-05T16:31:08.216154544Z insights-runtime-extractor-5tlmw
2026-08-05T16:31:08.216154544Z insights-runtime-extractor-xzwmw
2026-08-05T16:31:08.412783982Z error: the server doesn't have a resource type "performanceprofile"
2026-08-05T16:31:08.514697753Z No resources found
2026-08-05T16:31:08.627039770Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-08-05T16:31:08.811314423Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-08-05T16:31:08.902037761Z No resources found in openshift-etcd namespace.
2026-08-05T16:31:08.914393388Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-08-05T16:31:08.914393388Z See 'oc rsync -h' for help and examples
2026-08-05T16:31:09.029397495Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-08-05T16:31:09.029488656Z INFO: on-disk MachineConfig config collection complete.
2026-08-05T16:31:09.121035269Z INFO: "metallb-operator" not detected. Skipping.
2026-08-05T16:31:09.122782924Z INFO: "sriov-network-operator" not detected. Skipping.
2026-08-05T16:31:09.161998170Z Wrote inspect data to must-gather.
2026-08-05T16:31:09.180740176Z INFO: Worker host service log collection to complete.
2026-08-05T16:31:09.207107888Z INFO: Found 1 replicas - prometheus-k8s-0
2026-08-05T16:31:09.288338442Z ERROR: No running kube-apiserver pods found
2026-08-05T16:31:09.298539415Z INFO: Waiting for HAProxy config collection to complete ...
2026-08-05T16:31:09.837134692Z Wrote inspect data to must-gather.
2026-08-05T16:31:09.858904339Z Wrote inspect data to must-gather.
2026-08-05T16:31:09.866003338Z No resources found
2026-08-05T16:31:09.995691786Z tar: Removing leading `/' from member names
2026-08-05T16:31:10.003791890Z INFO: HAProxy config collection complete.
2026-08-05T16:31:10.012799851Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:31:10.018301215Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.027724730Z Gathering data for ns/openshift-monitoring...
2026-08-05T16:31:10.191664479Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-08-05T16:31:10.217999989Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-08-05T16:31:10.232043303Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:10.255221059Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.267954569Z INFO: 'previous-logs' folder not found on ip-10-0-129-231.ec2.internal, skipping...
2026-08-05T16:31:10.313728313Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.318685481Z error: the server doesn't have a resource type "machineconfigs"
2026-08-05T16:31:10.436929172Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.502107894Z error: the server doesn't have a resource type "machineconfigpools"
2026-08-05T16:31:10.567476134Z INFO: 'previous-logs' folder not found on ip-10-0-136-221.ec2.internal, skipping...
2026-08-05T16:31:10.717830955Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.747812120Z INFO: Getting rules from prometheus-k8s-0
2026-08-05T16:31:10.769378471Z Wrote inspect data to must-gather.
2026-08-05T16:31:10.840876575Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-08-05T16:31:10.857615182Z INFO: 'previous-logs' folder not found on ip-10-0-138-158.ec2.internal, skipping...
2026-08-05T16:31:10.857715285Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-08-05T16:31:10.857766677Z INFO: Machine Config Daemon termination log collection complete.
2026-08-05T16:31:10.928854067Z error: the server doesn't have a resource type "kubeletconfigs"
2026-08-05T16:31:10.991487862Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.135558359Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.202137098Z INFO: Getting status/config from prometheus-k8s-0
2026-08-05T16:31:11.236389856Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.363333802Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.521571055Z INFO: INTERCONNECT MODE
2026-08-05T16:31:11.521684227Z INFO: Gathering ovn-kubernetes DBs
2026-08-05T16:31:11.548877399Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-08-05T16:31:11.568899177Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.582140529Z INFO: Getting status/flags from prometheus-k8s-0
2026-08-05T16:31:11.702566002Z INFO: Gathering OVN_Northbound from ovnkube-node-2x2k6...
2026-08-05T16:31:11.702852627Z INFO: Gathering OVN_Northbound from ovnkube-node-6flds...
2026-08-05T16:31:11.703171643Z INFO: Gathering OVN_Northbound from ovnkube-node-m2psv...
2026-08-05T16:31:11.703518795Z INFO: Gathering OVN_Southbound from ovnkube-node-2x2k6...
2026-08-05T16:31:11.703826593Z INFO: Gathering OVN_Southbound from ovnkube-node-6flds...
2026-08-05T16:31:11.704121106Z INFO: Gathering OVN_Southbound from ovnkube-node-m2psv...
2026-08-05T16:31:11.786185990Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:11.855356289Z Wrote inspect data to must-gather.
2026-08-05T16:31:11.907722309Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T16:31:11.940160710Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-08-05T16:31:12.031436581Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.036244398Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.049946895Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.050103172Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.084115941Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.087716144Z tar: Removing leading `/' from member names
2026-08-05T16:31:12.140655215Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:12.143164565Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-08-05T16:31:12.185453862Z Wrote inspect data to must-gather.
2026-08-05T16:31:12.236330534Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-08-05T16:31:12.401830761Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:12.439574706Z Wrote inspect data to must-gather.
2026-08-05T16:31:12.497106328Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-08-05T16:31:12.600936524Z INFO: Waiting for network log collection to complete ...
2026-08-05T16:31:12.600975374Z INFO: Waiting for ovnk database copies to complete ...
2026-08-05T16:31:12.601492351Z INFO: Copying ovnk databases complete.
2026-08-05T16:31:12.604306683Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-08-05T16:31:12.606751220Z ovnk_database_store/
2026-08-05T16:31:12.606805900Z ovnk_database_store/ovnkube-node-6flds_nbdb
2026-08-05T16:31:12.623474841Z ovnk_database_store/ovnkube-node-2x2k6_nbdb
2026-08-05T16:31:12.635718217Z ovnk_database_store/ovnkube-node-m2psv_nbdb
2026-08-05T16:31:12.642975260Z ovnk_database_store/ovnkube-node-2x2k6_sbdb
2026-08-05T16:31:12.648446569Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:12.651333589Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:31:12.651851331Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-08-05T16:31:12.660934082Z ovnk_database_store/ovnkube-node-m2psv_sbdb
2026-08-05T16:31:12.677917513Z ovnk_database_store/ovnkube-node-6flds_sbdb
2026-08-05T16:31:12.701286942Z INFO: Network log collection complete.
2026-08-05T16:31:12.740279175Z Wrote inspect data to must-gather.
2026-08-05T16:31:12.851969350Z Gathering data for ns/openshift-network-console...
2026-08-05T16:31:12.864604744Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:12.886935230Z INFO: Getting status from alertmanager-main-0
2026-08-05T16:31:13.020403264Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-08-05T16:31:13.023338523Z INFO: Fallback to identify the container image from release info
2026-08-05T16:31:13.072378263Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:13.074924059Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-08-05T16:31:13.293164971Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:13.309324329Z Gathering data for ns/openshift-console-operator...
2026-08-05T16:31:13.473135261Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:13.475590224Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-08-05T16:31:13.626167974Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:13.701638570Z Gathering data for ns/openshift-console...
2026-08-05T16:31:13.820268358Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:31:14.198017280Z Gathering data for ns/openshift-cluster-storage-operator...
2026-08-05T16:31:14.463662533Z Done executing Istio gather script
2026-08-05T16:31:14.557967621Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675
2026-08-05T16:31:14.685660938Z daemonset.apps/perf-node-gather-daemonset created
2026-08-05T16:31:14.744856522Z Gathering data for ns/openshift-dns-operator...
2026-08-05T16:31:14.821427642Z Waiting for performance profile collector pods to become ready: 1
2026-08-05T16:31:15.070667694Z Gathering data for ns/openshift-dns...
2026-08-05T16:31:15.243246446Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:15.744198135Z Gathering data for ns/openshift-image-registry...
2026-08-05T16:31:15.946768506Z Waiting for performance profile collector pods to become ready: 2
2026-08-05T16:31:16.233573018Z Gathering data for ns/openshift-ingress-operator...
2026-08-05T16:31:16.565713033Z Gathering data for ns/openshift-ingress...
2026-08-05T16:31:16.891033110Z Gathering data for ns/openshift-ingress-canary...
2026-08-05T16:31:17.069304408Z Waiting for performance profile collector pods to become ready: 3
2026-08-05T16:31:17.366502022Z Gathering data for ns/openshift-insights...
2026-08-05T16:31:18.179769564Z Waiting for performance profile collector pods to become ready: 4
2026-08-05T16:31:19.185952684Z Gathering data for ns/openshift-jobset-operator...
2026-08-05T16:31:19.301045890Z Waiting for performance profile collector pods to become ready: 5
2026-08-05T16:31:19.968808831Z Gathering data for ns/openshift-config...
2026-08-05T16:31:20.247599446Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:20.247940419Z Gathering data for ns/openshift-config-managed...
2026-08-05T16:31:20.404373757Z Waiting for performance profile collector pods to become ready: 6
2026-08-05T16:31:20.687811751Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-08-05T16:31:20.952254273Z Gathering data for ns/openshift-kube-apiserver...
2026-08-05T16:31:21.241116911Z Gathering data for ns/openshift-kube-controller-manager...
2026-08-05T16:31:21.525733859Z Waiting for performance profile collector pods to become ready: 7
2026-08-05T16:31:21.542291747Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-08-05T16:31:21.825743407Z Gathering data for ns/openshift-kube-scheduler...
2026-08-05T16:31:22.119900065Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-08-05T16:31:22.412256645Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-08-05T16:31:22.631888826Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-08-05T16:31:22.744467326Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-08-05T16:31:22.888830432Z Collecting performance related data for node ip-10-0-129-231.ec2.internal
2026-08-05T16:31:22.895689513Z Collecting performance related data for node ip-10-0-138-158.ec2.internal
2026-08-05T16:31:22.903030092Z Collecting performance related data for node ip-10-0-136-221.ec2.internal
2026-08-05T16:31:23.109685410Z Gathering data for ns/openshift-user-workload-monitoring...
2026-08-05T16:31:23.775559115Z Gathering data for ns/openshift-multus...
2026-08-05T16:31:25.254836873Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:25.282057733Z Gathering data for ns/openshift-ovn-kubernetes...
2026-08-05T16:31:26.579005031Z Gathering data for ns/openshift-host-network...
2026-08-05T16:31:26.947167788Z Gathering data for ns/openshift-network-diagnostics...
2026-08-05T16:31:27.515866985Z Gathering data for ns/openshift-network-node-identity...
2026-08-05T16:31:27.988248879Z Gathering data for ns/openshift-network-operator...
2026-08-05T16:31:28.423183048Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-08-05T16:31:28.713350679Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-08-05T16:31:29.215930574Z Gathering data for ns/openshift-apiserver-operator...
2026-08-05T16:31:29.550934408Z Gathering data for ns/openshift-apiserver...
2026-08-05T16:31:29.978800049Z Gathering data for ns/openshift-controller-manager-operator...
2026-08-05T16:31:30.261588000Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:31:30.268241140Z Gathering data for ns/openshift-controller-manager...
2026-08-05T16:31:30.595726963Z Gathering data for ns/openshift-cluster-samples-operator...
2026-08-05T16:31:30.643835140Z Collecting kubelet logs for node ip-10-0-129-231.ec2.internal
2026-08-05T16:31:30.645623343Z Collecting kubelet logs for node ip-10-0-138-158.ec2.internal
2026-08-05T16:31:30.647712853Z Collecting kubelet logs for node ip-10-0-136-221.ec2.internal
2026-08-05T16:31:30.988177506Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-xcszw namespace
2026-08-05T16:31:30.993737305Z INFO: Node performance data collection complete.
2026-08-05T16:31:31.318569358Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-08-05T16:31:31.657090398Z Gathering data for ns/openshift-service-ca-operator...
2026-08-05T16:31:31.970678357Z Gathering data for ns/openshift-service-ca...
2026-08-05T16:31:32.335204720Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-08-05T16:31:33.221024090Z Wrote inspect data to must-gather.
2026-08-05T16:31:33.221065963Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T16:31:33.221065963Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-08-05T16:31:33.347719272Z error: the server doesn't have a resource type "clusters"
2026-08-05T16:31:33.508623948Z Caches written to disk