2026-07-27T16:41:35.437536036Z [disk usage checker] Started
2026-07-27T16:41:35.440980269Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:41:35.643200435Z Gathering data for ns/openshift-cluster-version...
2026-07-27T16:41:35.763802964Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T16:41:35.870184270Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T16:41:35.954121414Z Gathering data for ns/default...
2026-07-27T16:41:36.288301953Z Gathering data for ns/openshift...
2026-07-27T16:41:36.704803920Z Gathering data for ns/kube-system...
2026-07-27T16:41:37.403093114Z Gathering data for ns/openshift-etcd...
2026-07-27T16:41:37.717248188Z Wrote inspect data to must-gather.
2026-07-27T16:41:37.717274296Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T16:41:37.717274296Z     namespaces "assisted-installer" not found
2026-07-27T16:41:37.956364103Z Waiting on subprocesses to finish execution.
2026-07-27T16:41:37.964741393Z INFO: Collecting host service logs for crio
2026-07-27T16:41:37.965194100Z INFO: Collecting host service logs for kubelet
2026-07-27T16:41:37.965405780Z INFO: Gathering HAProxy config files
2026-07-27T16:41:37.965913279Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T16:41:37.966062410Z Executing Istio gather script
2026-07-27T16:41:37.966265835Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T16:41:37.966365215Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T16:41:37.966639073Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T16:41:37.966980102Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T16:41:37.967319006Z INFO: Collecting host service logs for NetworkManager
2026-07-27T16:41:37.967643198Z INFO: Collecting host service logs for openvswitch
2026-07-27T16:41:37.967972099Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T16:41:37.968396641Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T16:41:37.968744722Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T16:41:37.969132093Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T16:41:37.979078470Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T16:41:37.986532651Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T16:41:38.349213805Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T16:41:38.427449051Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-qc2ql
2026-07-27T16:41:38.427512874Z insights-runtime-extractor-2ff8c
2026-07-27T16:41:38.427535071Z insights-runtime-extractor-vhvk6
2026-07-27T16:41:38.427554913Z insights-runtime-extractor-xkmsq
2026-07-27T16:41:38.606267917Z No resources found in openshift-etcd namespace.
2026-07-27T16:41:38.694886648Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T16:41:38.803887282Z Wrote inspect data to must-gather.
2026-07-27T16:41:38.826857237Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T16:41:38.826857237Z See 'oc rsync -h' for help and examples
2026-07-27T16:41:38.859332664Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T16:41:39.123528756Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T16:41:39.145737524Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T16:41:39.174691956Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T16:41:39.212575862Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T16:41:39.212606857Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T16:41:39.328950788Z No resources found
2026-07-27T16:41:39.447818636Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T16:41:39.787649065Z INFO: 'previous-logs' folder not found on ip-10-0-128-172.ec2.internal, skipping...
2026-07-27T16:41:39.795950696Z INFO: Worker host service log collection to complete.
2026-07-27T16:41:39.885790368Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T16:41:39.986309268Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.081240294Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T16:41:40.114481124Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.169504023Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T16:41:40.182045605Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T16:41:40.201676539Z ERROR: No running kube-apiserver pods found
2026-07-27T16:41:40.289778357Z No resources found
2026-07-27T16:41:40.308006836Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.339214849Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.485338545Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.495073506Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:41:40.552511938Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T16:41:40.575483978Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.646469480Z INFO: 'previous-logs' folder not found on ip-10-0-141-52.ec2.internal, skipping...
2026-07-27T16:41:40.647438536Z tar: Removing leading `/' from member names
2026-07-27T16:41:40.654500520Z INFO: HAProxy config collection complete.
2026-07-27T16:41:40.690693785Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T16:41:40.737398616Z Wrote inspect data to must-gather.
2026-07-27T16:41:40.773939378Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T16:41:40.963040382Z INFO: 'previous-logs' folder not found on ip-10-0-143-40.ec2.internal, skipping...
2026-07-27T16:41:40.963078344Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T16:41:40.963078344Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T16:41:40.988446401Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T16:41:41.027331915Z Wrote inspect data to must-gather.
2026-07-27T16:41:41.030016353Z Wrote inspect data to must-gather.
2026-07-27T16:41:41.109350295Z Gathering data for ns/openshift-monitoring...
2026-07-27T16:41:41.175169763Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T16:41:41.208653468Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T16:41:41.299189388Z Wrote inspect data to must-gather.
2026-07-27T16:41:41.398060679Z Wrote inspect data to must-gather.
2026-07-27T16:41:41.588502578Z INFO: INTERCONNECT MODE
2026-07-27T16:41:41.588623893Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T16:41:41.600160833Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T16:41:41.619580854Z Wrote inspect data to must-gather.
2026-07-27T16:41:41.739606973Z INFO: Gathering OVN_Northbound from ovnkube-node-fdtjp...
2026-07-27T16:41:41.739799100Z INFO: Gathering OVN_Northbound from ovnkube-node-knsrr...
2026-07-27T16:41:41.740013119Z INFO: Gathering OVN_Northbound from ovnkube-node-pzzjg...
2026-07-27T16:41:41.740315229Z INFO: Gathering OVN_Southbound from ovnkube-node-fdtjp...
2026-07-27T16:41:41.740568818Z INFO: Gathering OVN_Southbound from ovnkube-node-knsrr...
2026-07-27T16:41:41.740856892Z INFO: Gathering OVN_Southbound from ovnkube-node-pzzjg...
2026-07-27T16:41:41.880956522Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T16:41:41.907805704Z Wrote inspect data to must-gather.
2026-07-27T16:41:42.022947721Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.043982233Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T16:41:42.052380438Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.052411634Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.060302904Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.062439393Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.102874663Z tar: Removing leading `/' from member names
2026-07-27T16:41:42.171481432Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T16:41:42.174446367Z Wrote inspect data to must-gather.
2026-07-27T16:41:42.238730080Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:42.409212644Z Wrote inspect data to must-gather.
2026-07-27T16:41:42.437350693Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T16:41:42.477049485Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:42.479573336Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T16:41:42.538847806Z INFO: Waiting for network log collection to complete ...
2026-07-27T16:41:42.538885503Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T16:41:42.539281540Z INFO: Copying ovnk databases complete.
2026-07-27T16:41:42.540851919Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-27T16:41:42.543037763Z ovnk_database_store/
2026-07-27T16:41:42.543037763Z ovnk_database_store/ovnkube-node-knsrr_sbdb
2026-07-27T16:41:42.564158904Z ovnk_database_store/ovnkube-node-fdtjp_sbdb
2026-07-27T16:41:42.581819256Z ovnk_database_store/ovnkube-node-fdtjp_nbdb
2026-07-27T16:41:42.589484354Z ovnk_database_store/ovnkube-node-pzzjg_nbdb
2026-07-27T16:41:42.596855057Z ovnk_database_store/ovnkube-node-knsrr_nbdb
2026-07-27T16:41:42.604357802Z ovnk_database_store/ovnkube-node-pzzjg_sbdb
2026-07-27T16:41:42.625752396Z INFO: Network log collection complete.
2026-07-27T16:41:42.641988214Z Wrote inspect data to must-gather.
2026-07-27T16:41:42.692691908Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:42.797524613Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T16:41:42.819667858Z INFO: Getting status from alertmanager-main-0
2026-07-27T16:41:42.858254863Z Wrote inspect data to must-gather.
2026-07-27T16:41:42.904141207Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:42.907012869Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T16:41:43.115825657Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T16:41:43.118933053Z INFO: Fallback to identify the container image from release info
2026-07-27T16:41:43.122119152Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:43.303952716Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:43.306420312Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T16:41:43.366086850Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T16:41:43.516680545Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:43.541298146Z Gathering data for ns/openshift-network-console...
2026-07-27T16:41:43.710749810Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:43.712993512Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T16:41:43.718066082Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T16:41:43.836079043Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T16:41:43.887187892Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:43.985725320Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T16:41:44.038306316Z Gathering data for ns/openshift-console-operator...
2026-07-27T16:41:44.091906002Z Wrote inspect data to /must-gather/istio.
2026-07-27T16:41:44.468881054Z Gathering data for ns/openshift-console...
2026-07-27T16:41:44.744702735Z Done executing Istio gather script
2026-07-27T16:41:44.979408924Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T16:41:45.108976337Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T16:41:45.481975413Z Gathering data for ns/openshift-dns-operator...
2026-07-27T16:41:45.501241051Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:41:45.772823025Z Gathering data for ns/openshift-dns...
2026-07-27T16:41:46.214401326Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T16:41:46.406511834Z Gathering data for ns/openshift-image-registry...
2026-07-27T16:41:46.898943658Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T16:41:47.241357074Z Gathering data for ns/openshift-ingress...
2026-07-27T16:41:47.334003584Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T16:41:47.583652916Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T16:41:47.979553723Z Gathering data for ns/openshift-insights...
2026-07-27T16:41:48.433637676Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T16:41:49.539499045Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T16:41:49.908029204Z Gathering data for ns/openshift-jobset-operator...
2026-07-27T16:41:50.505897227Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:41:50.664065936Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T16:41:50.715059080Z Gathering data for ns/openshift-config...
2026-07-27T16:41:50.990572442Z Gathering data for ns/openshift-config-managed...
2026-07-27T16:41:51.401031075Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T16:41:51.677981846Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T16:41:51.788477076Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T16:41:51.993377378Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T16:41:52.031645165Z Collecting performance related data for node ip-10-0-128-172.ec2.internal
2026-07-27T16:41:52.042311717Z Collecting performance related data for node ip-10-0-143-40.ec2.internal
2026-07-27T16:41:52.050035561Z Collecting performance related data for node ip-10-0-141-52.ec2.internal
2026-07-27T16:41:52.337269793Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T16:41:52.740394614Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T16:41:53.087697872Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T16:41:53.482549378Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T16:41:53.852596165Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T16:41:54.164772230Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T16:41:54.734312491Z Gathering data for ns/openshift-multus...
2026-07-27T16:41:55.510558392Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:41:56.321104233Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T16:41:57.830892632Z Gathering data for ns/openshift-host-network...
2026-07-27T16:41:58.138763623Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T16:41:58.705798281Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T16:41:59.107836241Z Gathering data for ns/openshift-network-operator...
2026-07-27T16:41:59.510640791Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T16:41:59.792942825Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T16:42:00.161689234Z Collecting kubelet logs for node ip-10-0-128-172.ec2.internal
2026-07-27T16:42:00.163655358Z Collecting kubelet logs for node ip-10-0-143-40.ec2.internal
2026-07-27T16:42:00.165752544Z Collecting kubelet logs for node ip-10-0-141-52.ec2.internal
2026-07-27T16:42:00.330961367Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T16:42:00.505390065Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wgg7p namespace
2026-07-27T16:42:00.509927125Z INFO: Node performance data collection complete.
2026-07-27T16:42:00.517081795Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-27T16:42:00.691634356Z Gathering data for ns/openshift-apiserver...
2026-07-27T16:42:01.160031641Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T16:42:01.510576008Z Gathering data for ns/openshift-controller-manager...
2026-07-27T16:42:01.835403359Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T16:42:02.460675692Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T16:42:02.737838337Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T16:42:03.065798046Z Gathering data for ns/openshift-service-ca...
2026-07-27T16:42:03.444462649Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T16:42:04.347946502Z Wrote inspect data to must-gather.
2026-07-27T16:42:04.347990251Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T16:42:04.347990251Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T16:42:04.471037917Z error: the server doesn't have a resource type "clusters"
2026-07-27T16:42:04.588892681Z Caches written to disk