2026-07-14T14:37:37.766538430Z [disk usage checker] Started
2026-07-14T14:37:37.769957278Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:37:37.961029974Z Gathering data for ns/openshift-cluster-version...
2026-07-14T14:37:38.085249067Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T14:37:38.187193070Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:37:38.296600569Z Gathering data for ns/default...
2026-07-14T14:37:38.612324216Z Gathering data for ns/openshift...
2026-07-14T14:37:39.012415879Z Gathering data for ns/kube-system...
2026-07-14T14:37:39.745177923Z Gathering data for ns/openshift-etcd...
2026-07-14T14:37:40.077815444Z Wrote inspect data to must-gather.
2026-07-14T14:37:40.077848244Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T14:37:40.077848244Z     namespaces "assisted-installer" not found
2026-07-14T14:37:40.544726257Z Waiting on subprocesses to finish execution.
2026-07-14T14:37:40.547468662Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-14T14:37:40.549599221Z INFO: Gathering HAProxy config files
2026-07-14T14:37:40.554978367Z INFO: Collecting host service logs for crio
2026-07-14T14:37:40.555294715Z INFO: Collecting host service logs for kubelet
2026-07-14T14:37:40.555648027Z INFO: Collecting host service logs for rpm-ostreed
2026-07-14T14:37:40.556009073Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-14T14:37:40.556354265Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-14T14:37:40.556726036Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-14T14:37:40.557093842Z INFO: Collecting host service logs for NetworkManager
2026-07-14T14:37:40.557437085Z INFO: Collecting host service logs for openvswitch
2026-07-14T14:37:40.557829244Z INFO: Collecting host service logs for ovs-configuration
2026-07-14T14:37:40.558197333Z INFO: Collecting host service logs for ovsdb-server
2026-07-14T14:37:40.558574326Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-14T14:37:40.558983774Z INFO: Waiting for worker host service log collection to complete ...
2026-07-14T14:37:40.559038008Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-14T14:37:40.559317841Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-14T14:37:40.586249796Z INFO: Waiting for node performance related collection to complete ...
2026-07-14T14:37:40.605711452Z Executing Istio gather script
2026-07-14T14:37:41.279829367Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-4zkfm
2026-07-14T14:37:41.279829367Z insights-runtime-extractor-fvq5z
2026-07-14T14:37:41.279829367Z insights-runtime-extractor-lrl8m
2026-07-14T14:37:41.279829367Z insights-runtime-extractor-lrmh7
2026-07-14T14:37:41.393308075Z error: the server doesn't have a resource type "performanceprofile"
2026-07-14T14:37:41.479187471Z No resources found
2026-07-14T14:37:41.534988571Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-14T14:37:41.657638640Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-14T14:37:41.806053998Z No resources found in openshift-etcd namespace.
2026-07-14T14:37:41.923849626Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-14T14:37:41.957299259Z INFO: Worker host service log collection to complete.
2026-07-14T14:37:41.993917329Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-14T14:37:42.003677651Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-14T14:37:42.003745406Z INFO: on-disk MachineConfig config collection complete.
2026-07-14T14:37:42.053747970Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-14T14:37:42.053747970Z See 'oc rsync -h' for help and examples
2026-07-14T14:37:42.058868322Z Wrote inspect data to must-gather.
2026-07-14T14:37:42.202179652Z INFO: "metallb-operator" not detected. Skipping.
2026-07-14T14:37:42.205407894Z Wrote inspect data to must-gather.
2026-07-14T14:37:42.455308417Z ERROR: No running kube-apiserver pods found
2026-07-14T14:37:42.494006316Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-14T14:37:42.605189698Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-14T14:37:42.842698827Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:37:42.942390956Z Wrote inspect data to must-gather.
2026-07-14T14:37:42.957313648Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:37:43.033216940Z No resources found
2026-07-14T14:37:43.075435975Z error: the server doesn't have a resource type "machineconfigs"
2026-07-14T14:37:43.093825494Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.136927682Z Gathering data for ns/openshift-monitoring...
2026-07-14T14:37:43.138725685Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.143076246Z INFO: 'previous-logs' folder not found on ip-10-0-133-126.ec2.internal, skipping...
2026-07-14T14:37:43.149439652Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-14T14:37:43.310441642Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.450688752Z tar: Removing leading `/' from member names
2026-07-14T14:37:43.456630614Z INFO: HAProxy config collection complete.
2026-07-14T14:37:43.485000079Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.518019512Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-14T14:37:43.617661976Z INFO: 'previous-logs' folder not found on ip-10-0-134-71.ec2.internal, skipping...
2026-07-14T14:37:43.631017030Z INFO: Getting rules from prometheus-k8s-0
2026-07-14T14:37:43.800215884Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.804854367Z Wrote inspect data to must-gather.
2026-07-14T14:37:43.927030094Z INFO: 'previous-logs' folder not found on ip-10-0-142-68.ec2.internal, skipping...
2026-07-14T14:37:43.927131322Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-14T14:37:43.927182439Z INFO: Machine Config Daemon termination log collection complete.
2026-07-14T14:37:43.992368518Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-14T14:37:44.115752224Z Wrote inspect data to must-gather.
2026-07-14T14:37:44.123161261Z INFO: Getting status/config from prometheus-k8s-0
2026-07-14T14:37:44.124614106Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-14T14:37:44.252437081Z Wrote inspect data to must-gather.
2026-07-14T14:37:44.372511453Z Wrote inspect data to must-gather.
2026-07-14T14:37:44.487333513Z Wrote inspect data to must-gather.
2026-07-14T14:37:44.561962370Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-14T14:37:44.706708653Z INFO: INTERCONNECT MODE
2026-07-14T14:37:44.706746971Z INFO: Gathering ovn-kubernetes DBs
2026-07-14T14:37:44.748886989Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T14:37:44.757976119Z Wrote inspect data to must-gather.
2026-07-14T14:37:44.836193066Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-14T14:37:44.897310549Z INFO: Gathering OVN_Northbound from ovnkube-node-lthzh...
2026-07-14T14:37:44.897650504Z INFO: Gathering OVN_Northbound from ovnkube-node-qq9kk...
2026-07-14T14:37:44.897907487Z INFO: Gathering OVN_Northbound from ovnkube-node-s4w22...
2026-07-14T14:37:44.898196610Z INFO: Gathering OVN_Southbound from ovnkube-node-lthzh...
2026-07-14T14:37:44.898473972Z INFO: Gathering OVN_Southbound from ovnkube-node-qq9kk...
2026-07-14T14:37:44.898742093Z INFO: Gathering OVN_Southbound from ovnkube-node-s4w22...
2026-07-14T14:37:45.035029731Z Wrote inspect data to must-gather.
2026-07-14T14:37:45.218787032Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.227721157Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-14T14:37:45.289267617Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.290704167Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.332303376Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.342582900Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.353303483Z tar: Removing leading `/' from member names
2026-07-14T14:37:45.427049494Z Wrote inspect data to must-gather.
2026-07-14T14:37:45.428907641Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-14T14:37:45.530122361Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-14T14:37:45.590726928Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:37:45.677954283Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:45.804703549Z Wrote inspect data to must-gather.
2026-07-14T14:37:45.842924411Z Gathering data for ns/openshift-network-console...
2026-07-14T14:37:45.915966761Z INFO: Waiting for network log collection to complete ...
2026-07-14T14:37:45.916009401Z INFO: Waiting for ovnk database copies to complete ...
2026-07-14T14:37:45.916535218Z INFO: Copying ovnk databases complete.
2026-07-14T14:37:45.918740737Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-14T14:37:45.921680769Z ovnk_database_store/
2026-07-14T14:37:45.921730785Z ovnk_database_store/ovnkube-node-qq9kk_sbdb
2026-07-14T14:37:45.940258911Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:45.943595126Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-14T14:37:45.946746280Z ovnk_database_store/ovnkube-node-qq9kk_nbdb
2026-07-14T14:37:45.954870099Z ovnk_database_store/ovnkube-node-lthzh_sbdb
2026-07-14T14:37:45.974404179Z ovnk_database_store/ovnkube-node-lthzh_nbdb
2026-07-14T14:37:45.984273780Z ovnk_database_store/ovnkube-node-s4w22_sbdb
2026-07-14T14:37:46.006179998Z ovnk_database_store/ovnkube-node-s4w22_nbdb
2026-07-14T14:37:46.018609971Z INFO: Network log collection complete.
2026-07-14T14:37:46.038881682Z INFO: Getting status from alertmanager-main-0
2026-07-14T14:37:46.170257985Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:46.182195746Z Wrote inspect data to must-gather.
2026-07-14T14:37:46.354804120Z Gathering data for ns/openshift-console-operator...
2026-07-14T14:37:46.394466560Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:46.396831943Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-14T14:37:46.480890717Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-14T14:37:46.483615344Z INFO: Fallback to identify the container image from release info
2026-07-14T14:37:46.623083453Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:46.784415093Z Gathering data for ns/openshift-console...
2026-07-14T14:37:46.816564213Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:46.819205396Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-14T14:37:47.039223543Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:47.186518983Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-14T14:37:47.267988740Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:47.271046038Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-14T14:37:47.340827944Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-14T14:37:47.345516914Z daemonset.apps/perf-node-gather-daemonset created
2026-07-14T14:37:47.500965472Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:47.523372155Z Waiting for performance profile collector pods to become ready: 1
2026-07-14T14:37:47.708581091Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:47.711155366Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-14T14:37:47.926168064Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:47.987555867Z Gathering data for ns/openshift-dns-operator...
2026-07-14T14:37:48.006785335Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:37:48.171156972Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:48.173732455Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-14T14:37:48.344438285Z Gathering data for ns/openshift-dns...
2026-07-14T14:37:48.361200092Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:48.537265581Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:37:48.675781797Z Waiting for performance profile collector pods to become ready: 2
2026-07-14T14:37:48.994863175Z Gathering data for ns/openshift-image-registry...
2026-07-14T14:37:49.238615261Z Done executing Istio gather script
2026-07-14T14:37:49.465171786Z Gathering data for ns/openshift-ingress-operator...
2026-07-14T14:37:49.810233169Z Waiting for performance profile collector pods to become ready: 3
2026-07-14T14:37:49.850176118Z Gathering data for ns/openshift-ingress...
2026-07-14T14:37:50.167151830Z Gathering data for ns/openshift-ingress-canary...
2026-07-14T14:37:50.543724586Z Gathering data for ns/openshift-insights...
2026-07-14T14:37:50.935702408Z Waiting for performance profile collector pods to become ready: 4
2026-07-14T14:37:52.054015069Z Waiting for performance profile collector pods to become ready: 5
2026-07-14T14:37:52.779689584Z Gathering data for ns/kserve...
2026-07-14T14:37:53.014176115Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:37:53.191191052Z Waiting for performance profile collector pods to become ready: 6
2026-07-14T14:37:54.047573399Z Gathering data for ns/openshift-config...
2026-07-14T14:37:54.324275207Z Waiting for performance profile collector pods to become ready: 7
2026-07-14T14:37:54.353862556Z Gathering data for ns/openshift-config-managed...
2026-07-14T14:37:54.790885059Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-14T14:37:55.074599418Z Gathering data for ns/openshift-kube-apiserver...
2026-07-14T14:37:55.365536473Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-14T14:37:55.462270570Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-14T14:37:55.672553205Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-14T14:37:55.800165346Z Collecting performance related data for node ip-10-0-133-126.ec2.internal
2026-07-14T14:37:55.806685434Z Collecting performance related data for node ip-10-0-142-68.ec2.internal
2026-07-14T14:37:55.813869679Z Collecting performance related data for node ip-10-0-134-71.ec2.internal
2026-07-14T14:37:56.036199606Z Gathering data for ns/openshift-kube-scheduler...
2026-07-14T14:37:56.372981054Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-14T14:37:56.751732536Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-14T14:37:57.167998009Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-14T14:37:57.529811867Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-14T14:37:58.010834012Z Gathering data for ns/openshift-multus...
2026-07-14T14:37:58.019223957Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:37:59.545742310Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-14T14:38:01.199677662Z Gathering data for ns/openshift-host-network...
2026-07-14T14:38:01.538728859Z Gathering data for ns/openshift-network-diagnostics...
2026-07-14T14:38:02.115589483Z Gathering data for ns/openshift-network-node-identity...
2026-07-14T14:38:02.512636021Z Gathering data for ns/openshift-network-operator...
2026-07-14T14:38:02.905242329Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-14T14:38:03.023782747Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:38:03.233088413Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-14T14:38:03.752470369Z Gathering data for ns/openshift-apiserver-operator...
2026-07-14T14:38:04.058075550Z Gathering data for ns/openshift-apiserver...
2026-07-14T14:38:04.114485176Z Collecting kubelet logs for node ip-10-0-133-126.ec2.internal
2026-07-14T14:38:04.117323592Z Collecting kubelet logs for node ip-10-0-142-68.ec2.internal
2026-07-14T14:38:04.119503608Z Collecting kubelet logs for node ip-10-0-134-71.ec2.internal
2026-07-14T14:38:04.510166723Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ccn9n namespace
2026-07-14T14:38:04.515383478Z INFO: Node performance data collection complete.
2026-07-14T14:38:04.516895963Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-14T14:38:04.836413706Z Gathering data for ns/openshift-controller-manager...
2026-07-14T14:38:05.153171179Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-14T14:38:05.831051540Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-14T14:38:06.145108685Z Gathering data for ns/openshift-service-ca-operator...
2026-07-14T14:38:06.501925995Z Gathering data for ns/openshift-service-ca...
2026-07-14T14:38:06.874200085Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-14T14:38:07.787426540Z Wrote inspect data to must-gather.
2026-07-14T14:38:07.787629797Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T14:38:07.787629797Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-14T14:38:07.917879934Z error: the server doesn't have a resource type "clusters"
2026-07-14T14:38:08.028765579Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:38:08.054997976Z Caches written to disk