2026-05-26T00:37:39.326261947Z [disk usage checker] Started
2026-05-26T00:37:39.329921834Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:37:39.532207051Z Gathering data for ns/openshift-cluster-version...
2026-05-26T00:37:39.679492686Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-26T00:37:39.755151226Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T00:37:39.836379063Z Gathering data for ns/default...
2026-05-26T00:37:40.145209620Z Gathering data for ns/openshift...
2026-05-26T00:37:40.537958885Z Gathering data for ns/kube-system...
2026-05-26T00:37:41.200355792Z Gathering data for ns/openshift-etcd...
2026-05-26T00:37:41.486288384Z Wrote inspect data to must-gather.
2026-05-26T00:37:41.486323977Z error: inspection completed with the errors occurred while gathering data:
2026-05-26T00:37:41.486323977Z     namespaces "assisted-installer" not found
2026-05-26T00:37:41.563284351Z Waiting on subprocesses to finish execution.
2026-05-26T00:37:41.570814270Z INFO: Gathering HAProxy config files
2026-05-26T00:37:41.571430773Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-26T00:37:41.574879844Z INFO: Collecting host service logs for crio
2026-05-26T00:37:41.575202585Z INFO: Collecting host service logs for kubelet
2026-05-26T00:37:41.575556770Z INFO: Collecting host service logs for rpm-ostreed
2026-05-26T00:37:41.575933453Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-26T00:37:41.576279187Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-26T00:37:41.576828783Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-26T00:37:41.577166205Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-26T00:37:41.577187405Z INFO: Collecting host service logs for NetworkManager
2026-05-26T00:37:41.577487200Z INFO: Collecting host service logs for openvswitch
2026-05-26T00:37:41.577868650Z INFO: Collecting host service logs for ovs-configuration
2026-05-26T00:37:41.578215502Z INFO: Collecting host service logs for ovsdb-server
2026-05-26T00:37:41.578554031Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-26T00:37:41.578977486Z INFO: Waiting for worker host service log collection to complete ...
2026-05-26T00:37:41.582679876Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-26T00:37:41.692153739Z INFO: Waiting for node performance related collection to complete ...
2026-05-26T00:37:41.888919796Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-26T00:37:42.373015026Z No resources found in openshift-etcd namespace.
2026-05-26T00:37:42.435230003Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-26T00:37:42.555538543Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-627zg
2026-05-26T00:37:42.555585967Z insights-runtime-extractor-rhdnf
2026-05-26T00:37:42.555606353Z insights-runtime-extractor-szkxc
2026-05-26T00:37:42.555626211Z insights-runtime-extractor-wcqc6
2026-05-26T00:37:42.565503774Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-26T00:37:42.565535363Z INFO: on-disk MachineConfig config collection complete.
2026-05-26T00:37:42.618566658Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-26T00:37:42.678106899Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-26T00:37:42.682099071Z INFO: Worker host service log collection to complete.
2026-05-26T00:37:42.779776469Z No resources found
2026-05-26T00:37:42.788844202Z INFO: "metallb-operator" not detected. Skipping.
2026-05-26T00:37:42.865549005Z error: the server doesn't have a resource type "performanceprofile"
2026-05-26T00:37:42.892183553Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-26T00:37:42.935190397Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.212260043Z No resources found
2026-05-26T00:37:43.312654607Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.378627560Z ERROR: No running kube-apiserver pods found
2026-05-26T00:37:43.392138479Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.495046302Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-26T00:37:43.527026276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T00:37:43.534041594Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-26T00:37:43.546163114Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-26T00:37:43.546163114Z See 'oc rsync -h' for help and examples
2026-05-26T00:37:43.564522366Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.602559033Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.606742604Z INFO: 'previous-logs' folder not found on ip-10-0-131-165.ec2.internal, skipping...
2026-05-26T00:37:43.731009663Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.773974808Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.820806984Z error: the server doesn't have a resource type "machineconfigs"
2026-05-26T00:37:43.846953565Z tar: Removing leading `/' from member names
2026-05-26T00:37:43.854176013Z INFO: HAProxy config collection complete.
2026-05-26T00:37:43.906034821Z Wrote inspect data to must-gather.
2026-05-26T00:37:43.945791797Z INFO: 'previous-logs' folder not found on ip-10-0-134-90.ec2.internal, skipping...
2026-05-26T00:37:43.982752729Z Gathering data for ns/openshift-monitoring...
2026-05-26T00:37:44.055788478Z INFO: Getting rules from prometheus-k8s-0
2026-05-26T00:37:44.092010555Z Wrote inspect data to must-gather.
2026-05-26T00:37:44.188890166Z Wrote inspect data to must-gather.
2026-05-26T00:37:44.207140882Z INFO: 'previous-logs' folder not found on ip-10-0-139-51.ec2.internal, skipping...
2026-05-26T00:37:44.207255498Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-26T00:37:44.207332272Z INFO: Machine Config Daemon termination log collection complete.
2026-05-26T00:37:44.290064216Z Wrote inspect data to must-gather.
2026-05-26T00:37:44.338344170Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:37:44.411668575Z Wrote inspect data to must-gather.
2026-05-26T00:37:44.431082889Z INFO: Getting status/config from prometheus-k8s-0
2026-05-26T00:37:44.451613463Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-26T00:37:44.643880159Z Wrote inspect data to must-gather.
2026-05-26T00:37:44.676420509Z INFO: INTERCONNECT MODE
2026-05-26T00:37:44.676454148Z INFO: Gathering ovn-kubernetes DBs
2026-05-26T00:37:44.796269071Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-26T00:37:44.827227731Z INFO: Gathering OVN_Northbound from ovnkube-node-hcsmb...
2026-05-26T00:37:44.827491041Z INFO: Gathering OVN_Northbound from ovnkube-node-j7xf4...
2026-05-26T00:37:44.827779283Z INFO: Gathering OVN_Northbound from ovnkube-node-r6gb6...
2026-05-26T00:37:44.828031306Z INFO: Gathering OVN_Southbound from ovnkube-node-hcsmb...
2026-05-26T00:37:44.828281491Z INFO: Gathering OVN_Southbound from ovnkube-node-j7xf4...
2026-05-26T00:37:44.828565438Z INFO: Gathering OVN_Southbound from ovnkube-node-r6gb6...
2026-05-26T00:37:44.844509639Z Wrote inspect data to must-gather.
2026-05-26T00:37:45.104543136Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-26T00:37:45.106224898Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.135364149Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.137850254Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.142110573Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.150002357Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.157719199Z tar: Removing leading `/' from member names
2026-05-26T00:37:45.180224837Z Wrote inspect data to must-gather.
2026-05-26T00:37:45.321649286Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-26T00:37:45.364209810Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-26T00:37:45.409521407Z Wrote inspect data to must-gather.
2026-05-26T00:37:45.583757726Z INFO: Waiting for network log collection to complete ...
2026-05-26T00:37:45.583790709Z INFO: Waiting for ovnk database copies to complete ...
2026-05-26T00:37:45.584160980Z INFO: Copying ovnk databases complete.
2026-05-26T00:37:45.585821234Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-05-26T00:37:45.588109605Z ovnk_database_store/
2026-05-26T00:37:45.588109605Z ovnk_database_store/ovnkube-node-j7xf4_nbdb
2026-05-26T00:37:45.598413044Z ovnk_database_store/ovnkube-node-hcsmb_sbdb
2026-05-26T00:37:45.610194889Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-26T00:37:45.619984574Z ovnk_database_store/ovnkube-node-r6gb6_sbdb
2026-05-26T00:37:45.622990266Z Wrote inspect data to must-gather.
2026-05-26T00:37:45.643745307Z ovnk_database_store/ovnkube-node-r6gb6_nbdb
2026-05-26T00:37:45.654550233Z ovnk_database_store/ovnkube-node-hcsmb_nbdb
2026-05-26T00:37:45.666119134Z ovnk_database_store/ovnkube-node-j7xf4_sbdb
2026-05-26T00:37:45.692167958Z INFO: Network log collection complete.
2026-05-26T00:37:45.811674322Z W0526 00:37:45.811634    1380 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-26T00:37:45.816592336Z Wrote inspect data to must-gather.
2026-05-26T00:37:45.904094783Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T00:37:45.984029455Z INFO: Getting status from alertmanager-main-0
2026-05-26T00:37:46.090323355Z Gathering data for ns/openshift-network-console...
2026-05-26T00:37:46.090648534Z Wrote inspect data to must-gather.
2026-05-26T00:37:46.341544344Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-26T00:37:46.344195869Z INFO: Fallback to identify the container image from release info
2026-05-26T00:37:46.457592763Z Gathering data for ns/openshift-console-operator...
2026-05-26T00:37:46.763896485Z Gathering data for ns/openshift-console...
2026-05-26T00:37:46.908239522Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-26T00:37:47.019624679Z daemonset.apps/perf-node-gather-daemonset created
2026-05-26T00:37:47.145235144Z Waiting for performance profile collector pods to become ready: 1
2026-05-26T00:37:47.187133551Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-26T00:37:47.516593707Z Gathering data for ns/openshift-dns-operator...
2026-05-26T00:37:47.768396185Z Gathering data for ns/openshift-dns...
2026-05-26T00:37:48.269231036Z Waiting for performance profile collector pods to become ready: 2
2026-05-26T00:37:48.286719677Z Gathering data for ns/openshift-image-registry...
2026-05-26T00:37:48.686723713Z Gathering data for ns/openshift-ingress-operator...
2026-05-26T00:37:48.999180894Z Gathering data for ns/openshift-ingress...
2026-05-26T00:37:49.306188374Z Gathering data for ns/openshift-ingress-canary...
2026-05-26T00:37:49.344340294Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:37:49.376357689Z Waiting for performance profile collector pods to become ready: 3
2026-05-26T00:37:49.655099521Z Gathering data for ns/openshift-insights...
2026-05-26T00:37:50.484012204Z Waiting for performance profile collector pods to become ready: 4
2026-05-26T00:37:51.493225078Z Gathering data for ns/openshift-config...
2026-05-26T00:37:51.603183432Z Waiting for performance profile collector pods to become ready: 5
2026-05-26T00:37:51.774875124Z Gathering data for ns/openshift-config-managed...
2026-05-26T00:37:52.154043385Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-26T00:37:52.380908570Z Gathering data for ns/openshift-kube-apiserver...
2026-05-26T00:37:52.610586441Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-26T00:37:52.722002880Z Waiting for performance profile collector pods to become ready: 6
2026-05-26T00:37:52.883469468Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-26T00:37:53.139817944Z Gathering data for ns/openshift-kube-scheduler...
2026-05-26T00:37:53.369535368Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-26T00:37:53.613825203Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-26T00:37:53.837931673Z Waiting for performance profile collector pods to become ready: 7
2026-05-26T00:37:53.900368723Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-26T00:37:54.167134736Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-26T00:37:54.350236606Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:37:54.751536599Z Gathering data for ns/openshift-multus...
2026-05-26T00:37:54.941571642Z Waiting for performance profile collector pods to become ready: 8
2026-05-26T00:37:55.959525736Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-26T00:37:56.046104030Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-26T00:37:56.274940352Z Collecting performance related data for node ip-10-0-139-51.ec2.internal
2026-05-26T00:37:56.282363285Z Collecting performance related data for node ip-10-0-131-165.ec2.internal
2026-05-26T00:37:56.289772022Z Collecting performance related data for node ip-10-0-134-90.ec2.internal
2026-05-26T00:37:57.503173195Z Gathering data for ns/openshift-host-network...
2026-05-26T00:37:57.829133036Z Gathering data for ns/openshift-network-diagnostics...
2026-05-26T00:37:58.285777606Z Gathering data for ns/openshift-network-node-identity...
2026-05-26T00:37:58.715076806Z Gathering data for ns/openshift-network-operator...
2026-05-26T00:37:59.084871044Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-26T00:37:59.351995603Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-26T00:37:59.355573320Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:37:59.750875111Z Gathering data for ns/openshift-apiserver-operator...
2026-05-26T00:38:00.021233051Z Gathering data for ns/openshift-apiserver...
2026-05-26T00:38:00.388902381Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-26T00:38:00.637405244Z Gathering data for ns/openshift-controller-manager...
2026-05-26T00:38:00.888827630Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-26T00:38:01.529013671Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-26T00:38:01.800122168Z Gathering data for ns/openshift-service-ca-operator...
2026-05-26T00:38:02.112304134Z Gathering data for ns/openshift-service-ca...
2026-05-26T00:38:02.446534113Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-26T00:38:03.266114705Z Wrote inspect data to must-gather.
2026-05-26T00:38:03.266166457Z error: inspection completed with the errors occurred while gathering data:
2026-05-26T00:38:03.266166457Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-26T00:38:04.022868943Z Collecting kubelet logs for node ip-10-0-139-51.ec2.internal
2026-05-26T00:38:04.024805211Z Collecting kubelet logs for node ip-10-0-131-165.ec2.internal
2026-05-26T00:38:04.026624581Z Collecting kubelet logs for node ip-10-0-134-90.ec2.internal
2026-05-26T00:38:04.336630145Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-26T00:38:04.342108044Z INFO: Node performance data collection complete.
2026-05-26T00:38:04.362874967Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-26T00:38:04.474017180Z error: the server doesn't have a resource type "clusters"
2026-05-26T00:38:04.642170607Z Caches written to disk