2026-09-05T03:19:15.015855807Z [disk usage checker] Started
2026-09-05T03:19:15.020043475Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:15.212377463Z Gathering data for ns/openshift-cluster-version...
2026-09-05T03:19:15.340111204Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-05T03:19:15.428223148Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:19:15.510446977Z Gathering data for ns/default...
2026-09-05T03:19:15.778748746Z Gathering data for ns/openshift...
2026-09-05T03:19:16.213279404Z Gathering data for ns/kube-system...
2026-09-05T03:19:17.014289931Z Gathering data for ns/openshift-etcd...
2026-09-05T03:19:17.324584702Z Wrote inspect data to must-gather.
2026-09-05T03:19:17.324618227Z error: inspection completed with the errors occurred while gathering data:
2026-09-05T03:19:17.324618227Z     namespaces "assisted-installer" not found
2026-09-05T03:19:17.598098844Z Waiting on subprocesses to finish execution.
2026-09-05T03:19:17.605929824Z INFO: Gathering HAProxy config files
2026-09-05T03:19:17.608155692Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-05T03:19:17.610552410Z Executing Istio gather script
2026-09-05T03:19:17.614673825Z INFO: Collecting host service logs for crio
2026-09-05T03:19:17.615106235Z INFO: Collecting host service logs for kubelet
2026-09-05T03:19:17.615445342Z INFO: Collecting host service logs for rpm-ostreed
2026-09-05T03:19:17.615837193Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-05T03:19:17.616289180Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-05T03:19:17.616637522Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-05T03:19:17.617006135Z INFO: Collecting host service logs for NetworkManager
2026-09-05T03:19:17.617366481Z INFO: Collecting host service logs for openvswitch
2026-09-05T03:19:17.617848082Z INFO: Collecting host service logs for ovs-configuration
2026-09-05T03:19:17.618217653Z INFO: Collecting host service logs for ovsdb-server
2026-09-05T03:19:17.618548088Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-05T03:19:17.618968017Z INFO: Waiting for worker host service log collection to complete ...
2026-09-05T03:19:17.625590186Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-05T03:19:17.636217560Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-05T03:19:17.668742467Z INFO: Waiting for node performance related collection to complete ...
2026-09-05T03:19:18.112994828Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-05T03:19:18.113110024Z INFO: on-disk MachineConfig config collection complete.
2026-09-05T03:19:18.352194435Z Wrote inspect data to must-gather.
2026-09-05T03:19:18.457795959Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-05T03:19:18.585565093Z No resources found
2026-09-05T03:19:18.682911466Z error: the server doesn't have a resource type "performanceprofile"
2026-09-05T03:19:18.782378670Z No resources found in openshift-etcd namespace.
2026-09-05T03:19:18.790510273Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-05T03:19:18.808620671Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-05T03:19:18.851692761Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-05T03:19:19.265886099Z No resources found
2026-09-05T03:19:19.353647061Z INFO: "metallb-operator" not detected. Skipping.
2026-09-05T03:19:19.390990146Z tar: Removing leading `/' from member names
2026-09-05T03:19:19.399372613Z INFO: HAProxy config collection complete.
2026-09-05T03:19:19.484566873Z Wrote inspect data to must-gather.
2026-09-05T03:19:19.569031182Z Wrote inspect data to must-gather.
2026-09-05T03:19:19.640115483Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-05T03:19:19.658400577Z Wrote inspect data to must-gather.
2026-09-05T03:19:19.765067482Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-vhhw8
2026-09-05T03:19:19.765120567Z insights-runtime-extractor-gnjb5
2026-09-05T03:19:19.765142126Z insights-runtime-extractor-lfdsd
2026-09-05T03:19:19.765162104Z insights-runtime-extractor-vqs7s
2026-09-05T03:19:19.787158760Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-05T03:19:19.854886623Z Wrote inspect data to must-gather.
2026-09-05T03:19:19.988601996Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-05T03:19:20.012341413Z Wrote inspect data to must-gather.
2026-09-05T03:19:20.030790113Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-05T03:19:20.030790113Z See 'oc rsync -h' for help and examples
2026-09-05T03:19:20.071155379Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:20.095412115Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:19:20.137961034Z INFO: Worker host service log collection to complete.
2026-09-05T03:19:20.143914288Z Gathering data for ns/openshift-monitoring...
2026-09-05T03:19:20.227954256Z ERROR: No running kube-apiserver pods found
2026-09-05T03:19:20.264884086Z error: the server doesn't have a resource type "machineconfigs"
2026-09-05T03:19:20.270916400Z INFO: 'previous-logs' folder not found on ip-10-0-129-239.ec2.internal, skipping...
2026-09-05T03:19:20.342772442Z Wrote inspect data to must-gather.
2026-09-05T03:19:20.373566151Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-05T03:19:20.435382211Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-05T03:19:20.551879284Z INFO: 'previous-logs' folder not found on ip-10-0-136-251.ec2.internal, skipping...
2026-09-05T03:19:20.564014326Z INFO: Getting rules from prometheus-k8s-0
2026-09-05T03:19:20.605902666Z Wrote inspect data to must-gather.
2026-09-05T03:19:20.687041527Z Wrote inspect data to must-gather.
2026-09-05T03:19:20.815012677Z INFO: 'previous-logs' folder not found on ip-10-0-143-31.ec2.internal, skipping...
2026-09-05T03:19:20.815117475Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-05T03:19:20.815146961Z INFO: Machine Config Daemon termination log collection complete.
2026-09-05T03:19:20.852748031Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-05T03:19:20.873648643Z Wrote inspect data to must-gather.
2026-09-05T03:19:20.987914133Z INFO: Getting status/config from prometheus-k8s-0
2026-09-05T03:19:21.063799788Z Wrote inspect data to must-gather.
2026-09-05T03:19:21.126318307Z Wrote inspect data to must-gather.
2026-09-05T03:19:21.288110162Z Wrote inspect data to must-gather.
2026-09-05T03:19:21.376321781Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-05T03:19:21.384107920Z INFO: INTERCONNECT MODE
2026-09-05T03:19:21.384219819Z INFO: Gathering ovn-kubernetes DBs
2026-09-05T03:19:21.506287993Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-05T03:19:21.547545798Z Wrote inspect data to must-gather.
2026-09-05T03:19:21.559849545Z INFO: Gathering OVN_Northbound from ovnkube-node-8pz9w...
2026-09-05T03:19:21.560120231Z INFO: Gathering OVN_Northbound from ovnkube-node-c87qs...
2026-09-05T03:19:21.560385555Z INFO: Gathering OVN_Northbound from ovnkube-node-nq7k4...
2026-09-05T03:19:21.560723640Z INFO: Gathering OVN_Southbound from ovnkube-node-8pz9w...
2026-09-05T03:19:21.561030339Z INFO: Gathering OVN_Southbound from ovnkube-node-c87qs...
2026-09-05T03:19:21.561315268Z INFO: Gathering OVN_Southbound from ovnkube-node-nq7k4...
2026-09-05T03:19:21.566082618Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-05T03:19:21.718291384Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-05T03:19:21.801248228Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:21.889616338Z tar: Removing leading `/' from member names
2026-09-05T03:19:21.893090854Z tar: Removing leading `/' from member names
2026-09-05T03:19:21.920786328Z tar: Removing leading `/' from member names
2026-09-05T03:19:21.944751452Z tar: Removing leading `/' from member names
2026-09-05T03:19:21.963681088Z tar: Removing leading `/' from member names
2026-09-05T03:19:21.996344445Z Wrote inspect data to must-gather.
2026-09-05T03:19:22.009917255Z tar: Removing leading `/' from member names
2026-09-05T03:19:22.085084380Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-05T03:19:22.085323347Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:22.087630638Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-05T03:19:22.252485613Z W0905 03:19:22.252437    1416 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-05T03:19:22.257996433Z Wrote inspect data to must-gather.
2026-09-05T03:19:22.339252526Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:22.340603745Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:19:22.356999518Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-05T03:19:22.450988761Z INFO: Waiting for network log collection to complete ...
2026-09-05T03:19:22.451115379Z INFO: Waiting for ovnk database copies to complete ...
2026-09-05T03:19:22.451545465Z INFO: Copying ovnk databases complete.
2026-09-05T03:19:22.453675273Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-09-05T03:19:22.456462458Z ovnk_database_store/
2026-09-05T03:19:22.456462458Z ovnk_database_store/ovnkube-node-nq7k4_sbdb
2026-09-05T03:19:22.478480916Z ovnk_database_store/ovnkube-node-c87qs_nbdb
2026-09-05T03:19:22.485157600Z ovnk_database_store/ovnkube-node-8pz9w_sbdb
2026-09-05T03:19:22.505812107Z ovnk_database_store/ovnkube-node-8pz9w_nbdb
2026-09-05T03:19:22.513506167Z ovnk_database_store/ovnkube-node-c87qs_sbdb
2026-09-05T03:19:22.534699512Z ovnk_database_store/ovnkube-node-nq7k4_nbdb
2026-09-05T03:19:22.545242986Z INFO: Network log collection complete.
2026-09-05T03:19:22.563951758Z Gathering data for ns/openshift-network-console...
2026-09-05T03:19:22.584652093Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:22.586948018Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-05T03:19:22.594992433Z Wrote inspect data to must-gather.
2026-09-05T03:19:22.787886294Z INFO: Getting status from alertmanager-main-0
2026-09-05T03:19:22.804851590Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:22.927775683Z Wrote inspect data to must-gather.
2026-09-05T03:19:23.035417151Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:23.037747249Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-05T03:19:23.038796906Z Gathering data for ns/openshift-console-operator...
2026-09-05T03:19:23.194907229Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-05T03:19:23.196862467Z INFO: Fallback to identify the container image from release info
2026-09-05T03:19:23.254463556Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:23.438324180Z Gathering data for ns/openshift-console...
2026-09-05T03:19:23.447209019Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:23.451092215Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-05T03:19:23.615192534Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:23.803521949Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:19:23.863119151Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399
2026-09-05T03:19:23.959773809Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-05T03:19:23.995674855Z daemonset.apps/perf-node-gather-daemonset created
2026-09-05T03:19:24.136650297Z Waiting for performance profile collector pods to become ready: 1
2026-09-05T03:19:24.527496551Z Gathering data for ns/openshift-dns-operator...
2026-09-05T03:19:24.542474454Z Done executing Istio gather script
2026-09-05T03:19:24.836872600Z Gathering data for ns/openshift-dns...
2026-09-05T03:19:25.077769933Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:25.258026073Z Waiting for performance profile collector pods to become ready: 2
2026-09-05T03:19:25.510860575Z Gathering data for ns/openshift-image-registry...
2026-09-05T03:19:25.956893257Z Gathering data for ns/openshift-ingress-operator...
2026-09-05T03:19:26.328375428Z Gathering data for ns/openshift-ingress...
2026-09-05T03:19:26.376164700Z Waiting for performance profile collector pods to become ready: 3
2026-09-05T03:19:26.651676271Z Gathering data for ns/openshift-ingress-canary...
2026-09-05T03:19:27.014122132Z Gathering data for ns/openshift-insights...
2026-09-05T03:19:27.489932324Z Waiting for performance profile collector pods to become ready: 4
2026-09-05T03:19:28.599500618Z Waiting for performance profile collector pods to become ready: 5
2026-09-05T03:19:28.802832403Z Gathering data for ns/openshift-jobset-operator...
2026-09-05T03:19:29.583214958Z Gathering data for ns/openshift-config...
2026-09-05T03:19:29.720555609Z Waiting for performance profile collector pods to become ready: 6
2026-09-05T03:19:29.848464781Z Gathering data for ns/openshift-config-managed...
2026-09-05T03:19:30.082635253Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:30.289926338Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-05T03:19:30.537571836Z Gathering data for ns/openshift-kube-apiserver...
2026-09-05T03:19:30.790291975Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-05T03:19:30.850223903Z Waiting for performance profile collector pods to become ready: 7
2026-09-05T03:19:31.070507328Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-05T03:19:31.381323640Z Gathering data for ns/openshift-kube-scheduler...
2026-09-05T03:19:31.618635934Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-05T03:19:31.883626220Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-05T03:19:31.962511450Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-05T03:19:32.213586948Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-05T03:19:32.221258094Z Collecting performance related data for node ip-10-0-143-31.ec2.internal
2026-09-05T03:19:32.228445927Z Collecting performance related data for node ip-10-0-136-251.ec2.internal
2026-09-05T03:19:32.236578733Z Collecting performance related data for node ip-10-0-129-239.ec2.internal
2026-09-05T03:19:32.579262390Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-05T03:19:33.189025600Z Gathering data for ns/openshift-multus...
2026-09-05T03:19:34.612878932Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-05T03:19:35.087933138Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:35.809381654Z Gathering data for ns/openshift-host-network...
2026-09-05T03:19:36.132553518Z Gathering data for ns/openshift-network-diagnostics...
2026-09-05T03:19:36.687743346Z Gathering data for ns/openshift-network-node-identity...
2026-09-05T03:19:37.092953302Z Gathering data for ns/openshift-network-operator...
2026-09-05T03:19:37.468221816Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-05T03:19:37.728164109Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-05T03:19:38.201371059Z Gathering data for ns/openshift-apiserver-operator...
2026-09-05T03:19:38.471134510Z Gathering data for ns/openshift-apiserver...
2026-09-05T03:19:38.832283326Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-05T03:19:39.072131175Z Gathering data for ns/openshift-controller-manager...
2026-09-05T03:19:39.356184165Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-05T03:19:39.994266748Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-05T03:19:40.079874736Z Collecting kubelet logs for node ip-10-0-143-31.ec2.internal
2026-09-05T03:19:40.081869677Z Collecting kubelet logs for node ip-10-0-136-251.ec2.internal
2026-09-05T03:19:40.083716633Z Collecting kubelet logs for node ip-10-0-129-239.ec2.internal
2026-09-05T03:19:40.093347671Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-05T03:19:40.327069714Z Gathering data for ns/openshift-service-ca-operator...
2026-09-05T03:19:40.428441457Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-n6fsp namespace
2026-09-05T03:19:40.433329886Z INFO: Node performance data collection complete.
2026-09-05T03:19:40.665239127Z Gathering data for ns/openshift-service-ca...
2026-09-05T03:19:41.037085478Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-05T03:19:41.927851735Z Wrote inspect data to must-gather.
2026-09-05T03:19:41.927898838Z error: inspection completed with the errors occurred while gathering data:
2026-09-05T03:19:41.927898838Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-05T03:19:42.051094123Z error: the server doesn't have a resource type "clusters"
2026-09-05T03:19:42.182147838Z Caches written to disk