2026-08-05T16:47:31.500652543Z [disk usage checker] Started
2026-08-05T16:47:31.504207774Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:31.702408803Z Gathering data for ns/openshift-cluster-version...
2026-08-05T16:47:31.848648600Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T16:47:31.981262871Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:47:32.091907512Z Gathering data for ns/default...
2026-08-05T16:47:32.458499155Z Gathering data for ns/openshift...
2026-08-05T16:47:32.928796793Z Gathering data for ns/kube-system...
2026-08-05T16:47:33.963007828Z Gathering data for ns/openshift-etcd...
2026-08-05T16:47:34.015042974Z Waiting on subprocesses to finish execution.
2026-08-05T16:47:34.027754642Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-08-05T16:47:34.031576132Z INFO: Collecting host service logs for crio
2026-08-05T16:47:34.031828919Z INFO: Gathering machine config daemon's old logs from all nodes
2026-08-05T16:47:34.032272768Z INFO: Collecting host service logs for kubelet
2026-08-05T16:47:34.034083234Z INFO: Gathering HAProxy config files
2026-08-05T16:47:34.034880419Z INFO: Collecting host service logs for rpm-ostreed
2026-08-05T16:47:34.036398232Z INFO: Collecting host service logs for ostree-finalize-staged
2026-08-05T16:47:34.036745186Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-08-05T16:47:34.038532514Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-08-05T16:47:34.038888176Z INFO: Collecting host service logs for machine-config-daemon-host
2026-08-05T16:47:34.039233522Z INFO: Collecting host service logs for NetworkManager
2026-08-05T16:47:34.039556004Z INFO: Collecting host service logs for openvswitch
2026-08-05T16:47:34.040296217Z INFO: Collecting host service logs for ovs-configuration
2026-08-05T16:47:34.052101769Z Executing Istio gather script
2026-08-05T16:47:34.052887016Z INFO: Collecting host service logs for ovsdb-server
2026-08-05T16:47:34.053419251Z INFO: Collecting host service logs for ovs-vswitchd
2026-08-05T16:47:34.057940530Z INFO: Waiting for worker host service log collection to complete ...
2026-08-05T16:47:34.217418953Z INFO: Waiting for node performance related collection to complete ...
2026-08-05T16:47:34.515979822Z INFO: Found 1 replicas - prometheus-k8s-0
2026-08-05T16:47:34.616948861Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-08-05T16:47:34.835539122Z No resources found in openshift-etcd namespace.
2026-08-05T16:47:34.864708707Z INFO: Collecting Insights Archives from insights-operator-77696685b-hhqj5
2026-08-05T16:47:34.864764004Z insights-runtime-extractor-442ld
2026-08-05T16:47:34.864786423Z insights-runtime-extractor-7544g
2026-08-05T16:47:34.864806667Z insights-runtime-extractor-9d6jc
2026-08-05T16:47:34.905766559Z Wrote inspect data to must-gather.
2026-08-05T16:47:34.905892653Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T16:47:34.905892653Z     namespaces "assisted-installer" not found
2026-08-05T16:47:34.975846106Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-08-05T16:47:34.976035528Z INFO: on-disk MachineConfig config collection complete.
2026-08-05T16:47:35.011582059Z No resources found
2026-08-05T16:47:35.037652388Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-08-05T16:47:35.059224751Z Wrote inspect data to must-gather.
2026-08-05T16:47:35.127897169Z INFO: "sriov-network-operator" not detected. Skipping.
2026-08-05T16:47:35.265600201Z error: the server doesn't have a resource type "performanceprofile"
2026-08-05T16:47:35.296511394Z ERROR: No running kube-apiserver pods found
2026-08-05T16:47:35.405056869Z INFO: "metallb-operator" not detected. Skipping.
2026-08-05T16:47:35.469756918Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-08-05T16:47:35.469756918Z See 'oc rsync -h' for help and examples
2026-08-05T16:47:35.493471611Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-08-05T16:47:35.494887912Z INFO: Worker host service log collection to complete.
2026-08-05T16:47:35.760687318Z INFO: Waiting for HAProxy config collection to complete ...
2026-08-05T16:47:35.857908928Z Wrote inspect data to must-gather.
2026-08-05T16:47:35.974521026Z Wrote inspect data to must-gather.
2026-08-05T16:47:36.050046346Z INFO: 'previous-logs' folder not found on ip-10-0-134-207.ec2.internal, skipping...
2026-08-05T16:47:36.248494395Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:47:36.381423281Z Wrote inspect data to must-gather.
2026-08-05T16:47:36.494284530Z tar: Removing leading `/' from member names
2026-08-05T16:47:36.502448812Z INFO: HAProxy config collection complete.
2026-08-05T16:47:36.551071074Z error: the server doesn't have a resource type "machineconfigs"
2026-08-05T16:47:36.596437237Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:36.606935307Z Wrote inspect data to must-gather.
2026-08-05T16:47:36.611489194Z Wrote inspect data to must-gather.
2026-08-05T16:47:36.629506982Z No resources found
2026-08-05T16:47:36.632469378Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-08-05T16:47:36.639467248Z INFO: Getting rules from prometheus-k8s-0
2026-08-05T16:47:36.659150581Z Gathering data for ns/openshift-monitoring...
2026-08-05T16:47:36.685789944Z Wrote inspect data to must-gather.
2026-08-05T16:47:36.806076719Z error: the server doesn't have a resource type "machineconfigpools"
2026-08-05T16:47:36.827339213Z INFO: 'previous-logs' folder not found on ip-10-0-139-183.ec2.internal, skipping...
2026-08-05T16:47:36.938701985Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.037141657Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-08-05T16:47:37.037234026Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.117151603Z INFO: 'previous-logs' folder not found on ip-10-0-143-77.ec2.internal, skipping...
2026-08-05T16:47:37.117189949Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-08-05T16:47:37.117189949Z INFO: Machine Config Daemon termination log collection complete.
2026-08-05T16:47:37.140551252Z INFO: Getting status/config from prometheus-k8s-0
2026-08-05T16:47:37.198457914Z error: the server doesn't have a resource type "kubeletconfigs"
2026-08-05T16:47:37.203540013Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.409931960Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.416931966Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.559206952Z INFO: Getting status/flags from prometheus-k8s-0
2026-08-05T16:47:37.635372110Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.711697266Z INFO: INTERCONNECT MODE
2026-08-05T16:47:37.711734552Z INFO: Gathering ovn-kubernetes DBs
2026-08-05T16:47:37.797903765Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-08-05T16:47:37.847051726Z Wrote inspect data to must-gather.
2026-08-05T16:47:37.864539373Z INFO: Gathering OVN_Northbound from ovnkube-node-55kh7...
2026-08-05T16:47:37.864836030Z INFO: Gathering OVN_Northbound from ovnkube-node-d8t5c...
2026-08-05T16:47:37.865189196Z INFO: Gathering OVN_Northbound from ovnkube-node-tg7z8...
2026-08-05T16:47:37.865431949Z INFO: Gathering OVN_Southbound from ovnkube-node-55kh7...
2026-08-05T16:47:37.865707499Z INFO: Gathering OVN_Southbound from ovnkube-node-d8t5c...
2026-08-05T16:47:37.865980856Z INFO: Gathering OVN_Southbound from ovnkube-node-tg7z8...
2026-08-05T16:47:38.098937413Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-08-05T16:47:38.150637961Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.156502347Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.172788211Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.173186982Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.187017075Z Wrote inspect data to must-gather.
2026-08-05T16:47:38.187315302Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.201706318Z tar: Removing leading `/' from member names
2026-08-05T16:47:38.242614864Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-08-05T16:47:38.393516396Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T16:47:38.394268933Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-08-05T16:47:38.426395205Z Wrote inspect data to must-gather.
2026-08-05T16:47:38.447748083Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:38.666733827Z Wrote inspect data to must-gather.
2026-08-05T16:47:38.702018106Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:38.704883199Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-08-05T16:47:38.741517955Z INFO: Waiting for network log collection to complete ...
2026-08-05T16:47:38.741652399Z INFO: Waiting for ovnk database copies to complete ...
2026-08-05T16:47:38.742056240Z INFO: Copying ovnk databases complete.
2026-08-05T16:47:38.744014038Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-08-05T16:47:38.746265690Z ovnk_database_store/
2026-08-05T16:47:38.746265690Z ovnk_database_store/ovnkube-node-tg7z8_sbdb
2026-08-05T16:47:38.768549436Z ovnk_database_store/ovnkube-node-d8t5c_sbdb
2026-08-05T16:47:38.788560857Z ovnk_database_store/ovnkube-node-55kh7_nbdb
2026-08-05T16:47:38.796260131Z ovnk_database_store/ovnkube-node-55kh7_sbdb
2026-08-05T16:47:38.811519140Z ovnk_database_store/ovnkube-node-tg7z8_nbdb
2026-08-05T16:47:38.819660803Z ovnk_database_store/ovnkube-node-d8t5c_nbdb
2026-08-05T16:47:38.832169640Z INFO: Network log collection complete.
2026-08-05T16:47:38.859402118Z INFO: Getting status from alertmanager-main-0
2026-08-05T16:47:38.905323916Z Wrote inspect data to must-gather.
2026-08-05T16:47:38.912521104Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.055445781Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T16:47:39.122926110Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.126016530Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-08-05T16:47:39.175181727Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-08-05T16:47:39.177431509Z INFO: Fallback to identify the container image from release info
2026-08-05T16:47:39.242446108Z Gathering data for ns/openshift-network-console...
2026-08-05T16:47:39.321835179Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.501905644Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.504432676Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-08-05T16:47:39.664895932Z Gathering data for ns/openshift-console-operator...
2026-08-05T16:47:39.700305293Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675
2026-08-05T16:47:39.722892379Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.836201917Z daemonset.apps/perf-node-gather-daemonset created
2026-08-05T16:47:39.919878278Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:39.922962185Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-08-05T16:47:39.969794261Z Waiting for performance profile collector pods to become ready: 1
2026-08-05T16:47:40.080761517Z Gathering data for ns/openshift-console...
2026-08-05T16:47:40.087338512Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:40.279325414Z Wrote inspect data to /must-gather/istio.
2026-08-05T16:47:40.652925682Z Gathering data for ns/openshift-cluster-storage-operator...
2026-08-05T16:47:40.924219314Z Done executing Istio gather script
2026-08-05T16:47:41.086175832Z Waiting for performance profile collector pods to become ready: 2
2026-08-05T16:47:41.165769648Z Gathering data for ns/openshift-dns-operator...
2026-08-05T16:47:41.487142175Z Gathering data for ns/openshift-dns...
2026-08-05T16:47:41.608761846Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:42.114887300Z Gathering data for ns/openshift-image-registry...
2026-08-05T16:47:42.195296050Z Waiting for performance profile collector pods to become ready: 3
2026-08-05T16:47:42.573379283Z Gathering data for ns/openshift-ingress-operator...
2026-08-05T16:47:42.896255081Z Gathering data for ns/openshift-ingress...
2026-08-05T16:47:43.221493101Z Gathering data for ns/openshift-ingress-canary...
2026-08-05T16:47:43.295565540Z Waiting for performance profile collector pods to become ready: 4
2026-08-05T16:47:43.641558587Z Gathering data for ns/openshift-insights...
2026-08-05T16:47:44.404519322Z Waiting for performance profile collector pods to become ready: 5
2026-08-05T16:47:45.519441706Z Waiting for performance profile collector pods to become ready: 6
2026-08-05T16:47:45.551283161Z Gathering data for ns/openshift-jobset-operator...
2026-08-05T16:47:46.308305793Z Gathering data for ns/openshift-config...
2026-08-05T16:47:46.563437736Z Gathering data for ns/openshift-config-managed...
2026-08-05T16:47:46.614227852Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:46.646380538Z Waiting for performance profile collector pods to become ready: 7
2026-08-05T16:47:46.976389959Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-08-05T16:47:47.343092790Z Gathering data for ns/openshift-kube-apiserver...
2026-08-05T16:47:47.651166846Z Gathering data for ns/openshift-kube-controller-manager...
2026-08-05T16:47:47.764828934Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-08-05T16:47:47.983538419Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-08-05T16:47:48.016068426Z Collecting performance related data for node ip-10-0-143-77.ec2.internal
2026-08-05T16:47:48.029838884Z Collecting performance related data for node ip-10-0-134-207.ec2.internal
2026-08-05T16:47:48.039242531Z Collecting performance related data for node ip-10-0-139-183.ec2.internal
2026-08-05T16:47:48.392239196Z Gathering data for ns/openshift-kube-scheduler...
2026-08-05T16:47:48.774321095Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-08-05T16:47:49.164961823Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-08-05T16:47:49.608792232Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-08-05T16:47:49.964891497Z Gathering data for ns/openshift-user-workload-monitoring...
2026-08-05T16:47:50.490144465Z Gathering data for ns/openshift-multus...
2026-08-05T16:47:51.619013055Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:52.120129414Z Gathering data for ns/openshift-ovn-kubernetes...
2026-08-05T16:47:53.584609880Z Gathering data for ns/openshift-host-network...
2026-08-05T16:47:53.975126359Z Gathering data for ns/openshift-network-diagnostics...
2026-08-05T16:47:54.591658049Z Gathering data for ns/openshift-network-node-identity...
2026-08-05T16:47:55.003501790Z Gathering data for ns/openshift-network-operator...
2026-08-05T16:47:55.420518473Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-08-05T16:47:55.715681944Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-08-05T16:47:56.225990138Z Collecting kubelet logs for node ip-10-0-143-77.ec2.internal
2026-08-05T16:47:56.227704599Z Collecting kubelet logs for node ip-10-0-134-207.ec2.internal
2026-08-05T16:47:56.229479088Z Collecting kubelet logs for node ip-10-0-139-183.ec2.internal
2026-08-05T16:47:56.332296123Z Gathering data for ns/openshift-apiserver-operator...
2026-08-05T16:47:56.559036471Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mtbjn namespace
2026-08-05T16:47:56.564926746Z INFO: Node performance data collection complete.
2026-08-05T16:47:56.624051336Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T16:47:56.684356465Z Gathering data for ns/openshift-apiserver...
2026-08-05T16:47:57.115915433Z Gathering data for ns/openshift-controller-manager-operator...
2026-08-05T16:47:57.380786335Z Gathering data for ns/openshift-controller-manager...
2026-08-05T16:47:57.660024059Z Gathering data for ns/openshift-cluster-samples-operator...
2026-08-05T16:47:58.359939412Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-08-05T16:47:58.684489698Z Gathering data for ns/openshift-service-ca-operator...
2026-08-05T16:47:58.987473021Z Gathering data for ns/openshift-service-ca...
2026-08-05T16:47:59.347277814Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-08-05T16:48:00.245697236Z Wrote inspect data to must-gather.
2026-08-05T16:48:00.245738153Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T16:48:00.245738153Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-08-05T16:48:00.364758671Z error: the server doesn't have a resource type "clusters"
2026-08-05T16:48:00.482266295Z Caches written to disk