2026-04-24T22:31:12.563620022Z [disk usage checker] Started
2026-04-24T22:31:12.567053517Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:12.885455372Z Gathering data for ns/openshift-cluster-version...
2026-04-24T22:31:13.018915790Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T22:31:13.127884773Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T22:31:13.211625938Z Gathering data for ns/default...
2026-04-24T22:31:13.508299220Z Gathering data for ns/openshift...
2026-04-24T22:31:13.914514438Z Gathering data for ns/kube-system...
2026-04-24T22:31:14.628269446Z Gathering data for ns/openshift-etcd...
2026-04-24T22:31:14.943277830Z Wrote inspect data to must-gather.
2026-04-24T22:31:14.943315189Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T22:31:14.943315189Z     namespaces "assisted-installer" not found
2026-04-24T22:31:15.150549039Z Waiting on subprocesses to finish execution.
2026-04-24T22:31:15.158540994Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-24T22:31:15.166699828Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-24T22:31:15.167127140Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-24T22:31:15.180816298Z INFO: Collecting host service logs for crio
2026-04-24T22:31:15.181106693Z INFO: Collecting host service logs for kubelet
2026-04-24T22:31:15.181436886Z INFO: Collecting host service logs for rpm-ostreed
2026-04-24T22:31:15.181749175Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-24T22:31:15.182073118Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-24T22:31:15.182415442Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-24T22:31:15.182910466Z INFO: Collecting host service logs for NetworkManager
2026-04-24T22:31:15.183260122Z INFO: Collecting host service logs for openvswitch
2026-04-24T22:31:15.183599346Z INFO: Collecting host service logs for ovs-configuration
2026-04-24T22:31:15.183962761Z INFO: Collecting host service logs for ovsdb-server
2026-04-24T22:31:15.184314920Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-24T22:31:15.184740461Z INFO: Waiting for worker host service log collection to complete ...
2026-04-24T22:31:15.187595635Z INFO: Gathering HAProxy config files
2026-04-24T22:31:15.209246309Z INFO: Waiting for node performance related collection to complete ...
2026-04-24T22:31:15.699899244Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-24T22:31:15.820066782Z No resources found in openshift-etcd namespace.
2026-04-24T22:31:15.873733329Z No resources found
2026-04-24T22:31:16.037194698Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-l5zd6
2026-04-24T22:31:16.038228331Z insights-runtime-extractor-fcwpj
2026-04-24T22:31:16.041628604Z insights-runtime-extractor-lrbvd
2026-04-24T22:31:16.041666460Z insights-runtime-extractor-pd6d8
2026-04-24T22:31:16.107565857Z INFO: "metallb-operator" not detected. Skipping.
2026-04-24T22:31:16.112947623Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-24T22:31:16.122158880Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-24T22:31:16.155901445Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-24T22:31:16.155901445Z INFO: on-disk MachineConfig config collection complete.
2026-04-24T22:31:16.356815864Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-24T22:31:16.463669995Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-24T22:31:16.463669995Z See 'oc rsync -h' for help and examples
2026-04-24T22:31:16.618208445Z error: the server doesn't have a resource type "performanceprofile"
2026-04-24T22:31:16.720860065Z INFO: Worker host service log collection to complete.
2026-04-24T22:31:16.788551236Z Wrote inspect data to must-gather.
2026-04-24T22:31:16.913202720Z Wrote inspect data to must-gather.
2026-04-24T22:31:16.976086196Z ERROR: No running kube-apiserver pods found
2026-04-24T22:31:17.019208609Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-24T22:31:17.180459211Z No resources found
2026-04-24T22:31:17.249381041Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-24T22:31:17.260609799Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T22:31:17.270164192Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.323533039Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.379793846Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.401527164Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.436102136Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-24T22:31:17.486561068Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.538449742Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.577609836Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:17.587303901Z tar: Removing leading `/' from member names
2026-04-24T22:31:17.594039398Z INFO: HAProxy config collection complete.
2026-04-24T22:31:17.624785712Z INFO: 'previous-logs' folder not found on ip-10-0-129-230.ec2.internal, skipping...
2026-04-24T22:31:17.659874677Z Gathering data for ns/openshift-monitoring...
2026-04-24T22:31:17.669672221Z error: the server doesn't have a resource type "machineconfigs"
2026-04-24T22:31:17.785691231Z INFO: Getting rules from prometheus-k8s-0
2026-04-24T22:31:17.815365946Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.902232996Z Wrote inspect data to must-gather.
2026-04-24T22:31:17.908226106Z INFO: 'previous-logs' folder not found on ip-10-0-129-36.ec2.internal, skipping...
2026-04-24T22:31:18.070654090Z Wrote inspect data to must-gather.
2026-04-24T22:31:18.156209983Z Wrote inspect data to must-gather.
2026-04-24T22:31:18.183952889Z INFO: 'previous-logs' folder not found on ip-10-0-136-160.ec2.internal, skipping...
2026-04-24T22:31:18.184145863Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-24T22:31:18.184222717Z INFO: Machine Config Daemon termination log collection complete.
2026-04-24T22:31:18.210493884Z INFO: Getting status/config from prometheus-k8s-0
2026-04-24T22:31:18.313004510Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-24T22:31:18.386330866Z INFO: INTERCONNECT MODE
2026-04-24T22:31:18.386413222Z INFO: Gathering ovn-kubernetes DBs
2026-04-24T22:31:18.523969296Z Wrote inspect data to must-gather.
2026-04-24T22:31:18.539369025Z INFO: Gathering OVN_Northbound from ovnkube-node-6c6jh...
2026-04-24T22:31:18.539585144Z INFO: Gathering OVN_Northbound from ovnkube-node-csxqq...
2026-04-24T22:31:18.539769745Z INFO: Gathering OVN_Northbound from ovnkube-node-zglkz...
2026-04-24T22:31:18.540055378Z INFO: Gathering OVN_Southbound from ovnkube-node-6c6jh...
2026-04-24T22:31:18.540270807Z INFO: Gathering OVN_Southbound from ovnkube-node-csxqq...
2026-04-24T22:31:18.540523614Z INFO: Gathering OVN_Southbound from ovnkube-node-zglkz...
2026-04-24T22:31:18.695682673Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-24T22:31:18.849441856Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.856886577Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.862547680Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.863159309Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.863532224Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.878281346Z Wrote inspect data to must-gather.
2026-04-24T22:31:18.908479764Z tar: Removing leading `/' from member names
2026-04-24T22:31:18.990206793Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-24T22:31:19.143569260Z Wrote inspect data to must-gather.
2026-04-24T22:31:19.172295233Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T22:31:19.254647084Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-24T22:31:19.388850814Z Wrote inspect data to must-gather.
2026-04-24T22:31:19.410854062Z INFO: Waiting for network log collection to complete ...
2026-04-24T22:31:19.410980523Z INFO: Waiting for ovnk database copies to complete ...
2026-04-24T22:31:19.411473262Z INFO: Copying ovnk databases complete.
2026-04-24T22:31:19.415947702Z 4.4M	must-gather/network_logs/ovnk_database_store
2026-04-24T22:31:19.419424056Z ovnk_database_store/
2026-04-24T22:31:19.419476348Z ovnk_database_store/ovnkube-node-csxqq_nbdb
2026-04-24T22:31:19.432307655Z ovnk_database_store/ovnkube-node-6c6jh_nbdb
2026-04-24T22:31:19.446640073Z ovnk_database_store/ovnkube-node-zglkz_sbdb
2026-04-24T22:31:19.472859210Z ovnk_database_store/ovnkube-node-zglkz_nbdb
2026-04-24T22:31:19.483989689Z ovnk_database_store/ovnkube-node-csxqq_sbdb
2026-04-24T22:31:19.504564201Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-24T22:31:19.505855106Z ovnk_database_store/ovnkube-node-6c6jh_sbdb
2026-04-24T22:31:19.535855848Z INFO: Network log collection complete.
2026-04-24T22:31:19.628603141Z Wrote inspect data to must-gather.
2026-04-24T22:31:19.783878914Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T22:31:19.901587879Z INFO: Getting status from alertmanager-main-0
2026-04-24T22:31:19.929515389Z Wrote inspect data to must-gather.
2026-04-24T22:31:19.966811036Z Gathering data for ns/openshift-network-console...
2026-04-24T22:31:20.217851972Z Wrote inspect data to must-gather.
2026-04-24T22:31:20.391418023Z Gathering data for ns/openshift-console-operator...
2026-04-24T22:31:20.502332916Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-24T22:31:20.505644798Z INFO: Fallback to identify the container image from release info
2026-04-24T22:31:20.751950224Z Gathering data for ns/openshift-console...
2026-04-24T22:31:20.989854259Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-24T22:31:21.128134494Z daemonset.apps/perf-node-gather-daemonset created
2026-04-24T22:31:21.221204125Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-24T22:31:21.275682422Z Waiting for performance profile collector pods to become ready: 1
2026-04-24T22:31:21.598817720Z Gathering data for ns/openshift-dns-operator...
2026-04-24T22:31:21.920859533Z Gathering data for ns/openshift-dns...
2026-04-24T22:31:22.416001841Z Waiting for performance profile collector pods to become ready: 2
2026-04-24T22:31:22.506938331Z Gathering data for ns/openshift-image-registry...
2026-04-24T22:31:22.582534960Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:22.963882134Z Gathering data for ns/openshift-ingress-operator...
2026-04-24T22:31:23.286536383Z Gathering data for ns/openshift-ingress...
2026-04-24T22:31:23.541821813Z Waiting for performance profile collector pods to become ready: 3
2026-04-24T22:31:23.613936491Z Gathering data for ns/openshift-ingress-canary...
2026-04-24T22:31:23.995341128Z Gathering data for ns/openshift-insights...
2026-04-24T22:31:24.679413180Z Waiting for performance profile collector pods to become ready: 4
2026-04-24T22:31:25.796154906Z Waiting for performance profile collector pods to become ready: 5
2026-04-24T22:31:26.114509653Z Gathering data for ns/kserve...
2026-04-24T22:31:26.917793155Z Waiting for performance profile collector pods to become ready: 6
2026-04-24T22:31:27.595929017Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:27.946688679Z Gathering data for ns/openshift-config...
2026-04-24T22:31:28.067120730Z Waiting for performance profile collector pods to become ready: 7
2026-04-24T22:31:28.325415696Z Gathering data for ns/openshift-config-managed...
2026-04-24T22:31:28.762049248Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-24T22:31:29.017334804Z Gathering data for ns/openshift-kube-apiserver...
2026-04-24T22:31:29.191098775Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-24T22:31:29.292223755Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-24T22:31:29.475723105Z Collecting performance related data for node ip-10-0-136-160.ec2.internal
2026-04-24T22:31:29.483956245Z Collecting performance related data for node ip-10-0-129-230.ec2.internal
2026-04-24T22:31:29.492182757Z Collecting performance related data for node ip-10-0-129-36.ec2.internal
2026-04-24T22:31:29.644617547Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-24T22:31:29.976975293Z Gathering data for ns/openshift-kube-scheduler...
2026-04-24T22:31:30.280801879Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-24T22:31:30.661299612Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-24T22:31:31.052971202Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-24T22:31:31.378072010Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-24T22:31:31.907898679Z Gathering data for ns/openshift-multus...
2026-04-24T22:31:32.600985482Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:33.378749358Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-24T22:31:34.957916470Z Gathering data for ns/openshift-host-network...
2026-04-24T22:31:35.302383277Z Gathering data for ns/openshift-network-diagnostics...
2026-04-24T22:31:35.908971058Z Gathering data for ns/openshift-network-node-identity...
2026-04-24T22:31:36.316646630Z Gathering data for ns/openshift-network-operator...
2026-04-24T22:31:36.686143740Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-24T22:31:36.984826041Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-24T22:31:37.424348043Z Gathering data for ns/openshift-apiserver-operator...
2026-04-24T22:31:37.607588362Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70
2026-04-24T22:31:37.725836107Z Gathering data for ns/openshift-apiserver...
2026-04-24T22:31:38.097259360Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-24T22:31:38.348505984Z Gathering data for ns/openshift-controller-manager...
2026-04-24T22:31:38.643613033Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-24T22:31:39.281467049Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-24T22:31:39.555536060Z Gathering data for ns/openshift-service-ca-operator...
2026-04-24T22:31:39.852561435Z Gathering data for ns/openshift-service-ca...
2026-04-24T22:31:40.207676132Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-24T22:31:41.088380980Z Wrote inspect data to must-gather.
2026-04-24T22:31:41.088420361Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T22:31:41.088420361Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-24T22:31:42.612688156Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-24T22:31:47.617639020Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-24T22:31:47.929057081Z Collecting kubelet logs for node ip-10-0-136-160.ec2.internal
2026-04-24T22:31:47.930828381Z Collecting kubelet logs for node ip-10-0-129-230.ec2.internal
2026-04-24T22:31:47.932559959Z Collecting kubelet logs for node ip-10-0-129-36.ec2.internal
2026-04-24T22:31:48.315296615Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-24T22:31:48.321780573Z INFO: Node performance data collection complete.
2026-04-24T22:31:48.455361997Z error: the server doesn't have a resource type "clusters"
2026-04-24T22:31:48.604637776Z Caches written to disk