2026-05-21T03:53:26.008195522Z [disk usage checker] Started
2026-05-21T03:53:26.011657042Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:26.276756641Z Gathering data for ns/openshift-cluster-version...
2026-05-21T03:53:26.442481561Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T03:53:26.540752671Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T03:53:26.642234946Z Gathering data for ns/default...
2026-05-21T03:53:27.020685707Z Gathering data for ns/openshift...
2026-05-21T03:53:27.498841255Z Gathering data for ns/kube-system...
2026-05-21T03:53:28.245634844Z Gathering data for ns/openshift-etcd...
2026-05-21T03:53:28.589146616Z Wrote inspect data to must-gather.
2026-05-21T03:53:28.589183351Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T03:53:28.589183351Z     namespaces "assisted-installer" not found
2026-05-21T03:53:28.630850950Z Waiting on subprocesses to finish execution.
2026-05-21T03:53:28.636847499Z INFO: Gathering HAProxy config files
2026-05-21T03:53:28.637698887Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-21T03:53:28.646621167Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-21T03:53:28.652221853Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-21T03:53:28.663941911Z INFO: Collecting host service logs for crio
2026-05-21T03:53:28.664508428Z INFO: Collecting host service logs for kubelet
2026-05-21T03:53:28.664842664Z INFO: Collecting host service logs for rpm-ostreed
2026-05-21T03:53:28.665146219Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-21T03:53:28.665466369Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-21T03:53:28.665777993Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-21T03:53:28.666079357Z INFO: Collecting host service logs for NetworkManager
2026-05-21T03:53:28.666418315Z INFO: Collecting host service logs for openvswitch
2026-05-21T03:53:28.666724055Z INFO: Collecting host service logs for ovs-configuration
2026-05-21T03:53:28.667024863Z INFO: Collecting host service logs for ovsdb-server
2026-05-21T03:53:28.667332005Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-21T03:53:28.667716082Z INFO: Waiting for worker host service log collection to complete ...
2026-05-21T03:53:28.688402932Z INFO: Waiting for node performance related collection to complete ...
2026-05-21T03:53:29.114521696Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-21T03:53:29.249385225Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-21T03:53:29.465040638Z No resources found in openshift-etcd namespace.
2026-05-21T03:53:29.600102944Z error: the server doesn't have a resource type "performanceprofile"
2026-05-21T03:53:29.645190010Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-7hp2b
2026-05-21T03:53:29.645190010Z insights-runtime-extractor-l92vd
2026-05-21T03:53:29.645190010Z insights-runtime-extractor-vczrs
2026-05-21T03:53:29.645190010Z insights-runtime-extractor-ztkkq
2026-05-21T03:53:29.837622294Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-21T03:53:29.854773978Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-21T03:53:29.909700648Z Wrote inspect data to must-gather.
2026-05-21T03:53:29.917000963Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-21T03:53:29.917000963Z See 'oc rsync -h' for help and examples
2026-05-21T03:53:29.933015661Z INFO: "metallb-operator" not detected. Skipping.
2026-05-21T03:53:29.954066587Z No resources found
2026-05-21T03:53:30.318189965Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-21T03:53:30.360238733Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-21T03:53:30.360324041Z INFO: on-disk MachineConfig config collection complete.
2026-05-21T03:53:30.418208357Z INFO: Worker host service log collection to complete.
2026-05-21T03:53:30.607048402Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-21T03:53:30.622615568Z Wrote inspect data to must-gather.
2026-05-21T03:53:30.765395253Z Wrote inspect data to must-gather.
2026-05-21T03:53:30.828574887Z ERROR: No running kube-apiserver pods found
2026-05-21T03:53:30.859408550Z No resources found
2026-05-21T03:53:30.894641091Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T03:53:31.112113590Z Wrote inspect data to must-gather.
2026-05-21T03:53:31.119161455Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:31.196106516Z Wrote inspect data to must-gather.
2026-05-21T03:53:31.215294499Z error: the server doesn't have a resource type "machineconfigs"
2026-05-21T03:53:31.265038403Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-21T03:53:31.281082827Z Wrote inspect data to must-gather.
2026-05-21T03:53:31.462799905Z INFO: 'previous-logs' folder not found on ip-10-0-128-161.ec2.internal, skipping...
2026-05-21T03:53:31.598692375Z Wrote inspect data to must-gather.
2026-05-21T03:53:31.612036474Z tar: Removing leading `/' from member names
2026-05-21T03:53:31.621631432Z INFO: Getting rules from prometheus-k8s-0
2026-05-21T03:53:31.635065525Z INFO: HAProxy config collection complete.
2026-05-21T03:53:31.689219795Z Wrote inspect data to must-gather.
2026-05-21T03:53:31.795037316Z Gathering data for ns/kuadrant-system...
2026-05-21T03:53:31.871442520Z INFO: 'previous-logs' folder not found on ip-10-0-131-76.ec2.internal, skipping...
2026-05-21T03:53:32.013501428Z Wrote inspect data to must-gather.
2026-05-21T03:53:32.055560975Z Wrote inspect data to must-gather.
2026-05-21T03:53:32.187237158Z INFO: Getting status/config from prometheus-k8s-0
2026-05-21T03:53:32.238624847Z INFO: 'previous-logs' folder not found on ip-10-0-137-251.ec2.internal, skipping...
2026-05-21T03:53:32.238733600Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-21T03:53:32.238804962Z INFO: Machine Config Daemon termination log collection complete.
2026-05-21T03:53:32.290802819Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-21T03:53:32.480914895Z Wrote inspect data to must-gather.
2026-05-21T03:53:32.655293205Z Wrote inspect data to must-gather.
2026-05-21T03:53:32.786777309Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-21T03:53:32.868770214Z INFO: INTERCONNECT MODE
2026-05-21T03:53:32.868863901Z INFO: Gathering ovn-kubernetes DBs
2026-05-21T03:53:33.000615748Z Wrote inspect data to must-gather.
2026-05-21T03:53:33.075038193Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-21T03:53:33.095210710Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T03:53:33.120334965Z INFO: Gathering OVN_Northbound from ovnkube-node-h8c7x...
2026-05-21T03:53:33.120698177Z INFO: Gathering OVN_Northbound from ovnkube-node-hkgwf...
2026-05-21T03:53:33.120977566Z INFO: Gathering OVN_Northbound from ovnkube-node-tqqk8...
2026-05-21T03:53:33.121299988Z INFO: Gathering OVN_Southbound from ovnkube-node-h8c7x...
2026-05-21T03:53:33.121549181Z INFO: Gathering OVN_Southbound from ovnkube-node-hkgwf...
2026-05-21T03:53:33.121835553Z INFO: Gathering OVN_Southbound from ovnkube-node-tqqk8...
2026-05-21T03:53:33.383759903Z Wrote inspect data to must-gather.
2026-05-21T03:53:33.399754602Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-21T03:53:33.530960248Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.546580189Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.620819425Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.679641794Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.710927565Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.712667295Z tar: Removing leading `/' from member names
2026-05-21T03:53:33.787823783Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-21T03:53:33.876884727Z W0521 03:53:33.876836    1295 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-21T03:53:33.905339407Z Wrote inspect data to must-gather.
2026-05-21T03:53:33.914972018Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T03:53:34.217119502Z Gathering data for ns/openshift-monitoring...
2026-05-21T03:53:34.333824223Z Wrote inspect data to must-gather.
2026-05-21T03:53:34.338998473Z INFO: Waiting for network log collection to complete ...
2026-05-21T03:53:34.339060482Z INFO: Waiting for ovnk database copies to complete ...
2026-05-21T03:53:34.339603402Z INFO: Copying ovnk databases complete.
2026-05-21T03:53:34.341836259Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-21T03:53:34.347921308Z ovnk_database_store/
2026-05-21T03:53:34.347970901Z ovnk_database_store/ovnkube-node-h8c7x_sbdb
2026-05-21T03:53:34.375087966Z ovnk_database_store/ovnkube-node-tqqk8_sbdb
2026-05-21T03:53:34.392989754Z INFO: Getting status from alertmanager-main-0
2026-05-21T03:53:34.398994500Z ovnk_database_store/ovnkube-node-h8c7x_nbdb
2026-05-21T03:53:34.408878522Z ovnk_database_store/ovnkube-node-tqqk8_nbdb
2026-05-21T03:53:34.421573151Z ovnk_database_store/ovnkube-node-hkgwf_nbdb
2026-05-21T03:53:34.437149621Z ovnk_database_store/ovnkube-node-hkgwf_sbdb
2026-05-21T03:53:34.475863011Z INFO: Network log collection complete.
2026-05-21T03:53:34.667367272Z Wrote inspect data to must-gather.
2026-05-21T03:53:34.983188253Z Wrote inspect data to must-gather.
2026-05-21T03:53:35.229614419Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-21T03:53:35.232700965Z INFO: Fallback to identify the container image from release info
2026-05-21T03:53:35.874481283Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-21T03:53:36.004381066Z daemonset.apps/perf-node-gather-daemonset created
2026-05-21T03:53:36.129302794Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:36.159558413Z Waiting for performance profile collector pods to become ready: 1
2026-05-21T03:53:36.320938586Z Gathering data for ns/openshift-network-console...
2026-05-21T03:53:36.859682847Z Gathering data for ns/openshift-console-operator...
2026-05-21T03:53:37.322222063Z Gathering data for ns/openshift-console...
2026-05-21T03:53:37.323497274Z Waiting for performance profile collector pods to become ready: 2
2026-05-21T03:53:37.821617345Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-21T03:53:38.248454698Z Gathering data for ns/openshift-dns-operator...
2026-05-21T03:53:38.380120925Z Wrote inspect data to must-gather.
2026-05-21T03:53:38.448746903Z Waiting for performance profile collector pods to become ready: 3
2026-05-21T03:53:38.575038975Z Gathering data for ns/openshift-dns...
2026-05-21T03:53:39.174594958Z Gathering data for ns/openshift-image-registry...
2026-05-21T03:53:39.582579606Z Waiting for performance profile collector pods to become ready: 4
2026-05-21T03:53:39.664705849Z Gathering data for ns/openshift-ingress-operator...
2026-05-21T03:53:40.033759197Z Gathering data for ns/openshift-ingress...
2026-05-21T03:53:40.710703980Z Waiting for performance profile collector pods to become ready: 5
2026-05-21T03:53:40.966263469Z Gathering data for ns/openshift-ingress-canary...
2026-05-21T03:53:41.134276524Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:41.417105592Z Gathering data for ns/openshift-insights...
2026-05-21T03:53:41.828130402Z Waiting for performance profile collector pods to become ready: 6
2026-05-21T03:53:42.957077172Z Waiting for performance profile collector pods to become ready: 7
2026-05-21T03:53:43.532125586Z Gathering data for ns/opendatahub...
2026-05-21T03:53:44.093715940Z Waiting for performance profile collector pods to become ready: 8
2026-05-21T03:53:45.138377632Z Gathering data for ns/openshift-lws-operator...
2026-05-21T03:53:45.216226090Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-21T03:53:45.502229079Z Collecting performance related data for node ip-10-0-131-76.ec2.internal
2026-05-21T03:53:45.511872949Z Collecting performance related data for node ip-10-0-128-161.ec2.internal
2026-05-21T03:53:45.519533376Z Collecting performance related data for node ip-10-0-137-251.ec2.internal
2026-05-21T03:53:46.139243818Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:46.644242461Z Gathering data for ns/openshift-config...
2026-05-21T03:53:47.040102627Z Gathering data for ns/openshift-config-managed...
2026-05-21T03:53:47.562110348Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-21T03:53:47.892039607Z Gathering data for ns/openshift-kube-apiserver...
2026-05-21T03:53:48.273713519Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-21T03:53:48.681368566Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-21T03:53:49.024931866Z Gathering data for ns/openshift-kube-scheduler...
2026-05-21T03:53:49.366247059Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-21T03:53:49.706432473Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-21T03:53:50.072661490Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-21T03:53:50.434720256Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-21T03:53:51.040082248Z Gathering data for ns/openshift-multus...
2026-05-21T03:53:51.144115956Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:52.465637542Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-21T03:53:54.095160175Z Gathering data for ns/openshift-host-network...
2026-05-21T03:53:54.437619370Z Gathering data for ns/openshift-network-diagnostics...
2026-05-21T03:53:54.445821878Z Collecting kubelet logs for node ip-10-0-131-76.ec2.internal
2026-05-21T03:53:54.447834136Z Collecting kubelet logs for node ip-10-0-128-161.ec2.internal
2026-05-21T03:53:54.450128762Z Collecting kubelet logs for node ip-10-0-137-251.ec2.internal
2026-05-21T03:53:54.841572141Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-21T03:53:54.847325685Z INFO: Node performance data collection complete.
2026-05-21T03:53:55.115475235Z Gathering data for ns/openshift-network-node-identity...
2026-05-21T03:53:55.584666332Z Gathering data for ns/openshift-network-operator...
2026-05-21T03:53:55.985452388Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-21T03:53:56.151415048Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:53:56.282659248Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-21T03:53:56.771136915Z Gathering data for ns/openshift-apiserver-operator...
2026-05-21T03:53:57.113113267Z Gathering data for ns/openshift-apiserver...
2026-05-21T03:53:57.585414204Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-21T03:53:57.877572707Z Gathering data for ns/openshift-controller-manager...
2026-05-21T03:53:58.204382787Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-21T03:53:58.892567489Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-21T03:53:59.306622919Z Gathering data for ns/openshift-service-ca-operator...
2026-05-21T03:53:59.640501492Z Gathering data for ns/openshift-service-ca...
2026-05-21T03:54:00.055045555Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-21T03:54:01.048077661Z Wrote inspect data to must-gather.
2026-05-21T03:54:01.048127299Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T03:54:01.048127299Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-21T03:54:01.156053061Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-21T03:54:01.188605251Z error: the server doesn't have a resource type "clusters"
2026-05-21T03:54:01.356820449Z Caches written to disk