2026-05-20T16:23:08.203243891Z [disk usage checker] Started
2026-05-20T16:23:08.206678141Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-20T16:23:08.411624668Z Gathering data for ns/openshift-cluster-version...
2026-05-20T16:23:08.555200812Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T16:23:08.653241304Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:23:08.735540373Z Gathering data for ns/default...
2026-05-20T16:23:09.057272436Z Gathering data for ns/openshift...
2026-05-20T16:23:09.486456876Z Gathering data for ns/kube-system...
2026-05-20T16:23:10.197910282Z Gathering data for ns/openshift-etcd...
2026-05-20T16:23:10.511094302Z Wrote inspect data to must-gather.
2026-05-20T16:23:10.511128551Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T16:23:10.511128551Z     namespaces "assisted-installer" not found
2026-05-20T16:23:10.627640106Z Waiting on subprocesses to finish execution.
2026-05-20T16:23:10.635919817Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-20T16:23:10.642385202Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-20T16:23:10.647461704Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-20T16:23:10.651755411Z INFO: Gathering HAProxy config files
2026-05-20T16:23:10.659106213Z INFO: Collecting host service logs for crio
2026-05-20T16:23:10.659437642Z INFO: Collecting host service logs for kubelet
2026-05-20T16:23:10.659758891Z INFO: Collecting host service logs for rpm-ostreed
2026-05-20T16:23:10.660090115Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-20T16:23:10.660377181Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-20T16:23:10.660677396Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-20T16:23:10.660983820Z INFO: Collecting host service logs for NetworkManager
2026-05-20T16:23:10.661306519Z INFO: Collecting host service logs for openvswitch
2026-05-20T16:23:10.661565065Z INFO: Collecting host service logs for ovs-configuration
2026-05-20T16:23:10.661820235Z INFO: Collecting host service logs for ovsdb-server
2026-05-20T16:23:10.662089058Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-20T16:23:10.662395364Z INFO: Waiting for worker host service log collection to complete ...
2026-05-20T16:23:10.686739972Z INFO: Waiting for node performance related collection to complete ...
2026-05-20T16:23:11.015835766Z No resources found in openshift-etcd namespace.
2026-05-20T16:23:11.242356940Z No resources found
2026-05-20T16:23:11.358265217Z INFO: "metallb-operator" not detected. Skipping.
2026-05-20T16:23:11.360553271Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-20T16:23:11.538995342Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-20T16:23:11.765806201Z Wrote inspect data to must-gather.
2026-05-20T16:23:11.768913239Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-8ltwr
2026-05-20T16:23:11.768952677Z insights-runtime-extractor-6jm9g
2026-05-20T16:23:11.768971801Z insights-runtime-extractor-9dswv
2026-05-20T16:23:11.768989747Z insights-runtime-extractor-w8wdt
2026-05-20T16:23:11.831162404Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-20T16:23:11.859220244Z INFO: Worker host service log collection to complete.
2026-05-20T16:23:11.870483204Z error: the server doesn't have a resource type "performanceprofile"
2026-05-20T16:23:11.890867007Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-20T16:23:11.920314570Z Wrote inspect data to must-gather.
2026-05-20T16:23:11.969776420Z ERROR: No running kube-apiserver pods found
2026-05-20T16:23:11.984066851Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-20T16:23:11.984144385Z INFO: on-disk MachineConfig config collection complete.
2026-05-20T16:23:12.067532437Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-20T16:23:12.067532437Z See 'oc rsync -h' for help and examples
2026-05-20T16:23:12.075619153Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-20T16:23:12.242785194Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-20T16:23:12.355322526Z Wrote inspect data to must-gather.
2026-05-20T16:23:12.584149419Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-20T16:23:12.742733278Z Wrote inspect data to must-gather.
2026-05-20T16:23:12.791982431Z INFO: 'previous-logs' folder not found on ip-10-0-129-117.ec2.internal, skipping...
2026-05-20T16:23:12.917528349Z No resources found
2026-05-20T16:23:12.917714889Z Gathering data for ns/openshift-monitoring...
2026-05-20T16:23:12.933676571Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:23:12.937319695Z Wrote inspect data to must-gather.
2026-05-20T16:23:12.955211887Z Wrote inspect data to must-gather.
2026-05-20T16:23:12.972792372Z error: the server doesn't have a resource type "machineconfigs"
2026-05-20T16:23:13.057593081Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.126154945Z INFO: Getting rules from prometheus-k8s-0
2026-05-20T16:23:13.149577816Z tar: Removing leading `/' from member names
2026-05-20T16:23:13.155362676Z INFO: HAProxy config collection complete.
2026-05-20T16:23:13.202774439Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.213327298Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-20T16:23:13.218163522Z INFO: 'previous-logs' folder not found on ip-10-0-135-174.ec2.internal, skipping...
2026-05-20T16:23:13.219715930Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.467891385Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.482392035Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.488195119Z INFO: 'previous-logs' folder not found on ip-10-0-136-42.ec2.internal, skipping...
2026-05-20T16:23:13.488236133Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-20T16:23:13.488268601Z INFO: Machine Config Daemon termination log collection complete.
2026-05-20T16:23:13.575193966Z INFO: Getting status/config from prometheus-k8s-0
2026-05-20T16:23:13.638874326Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-20T16:23:13.739850308Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.855291064Z Wrote inspect data to must-gather.
2026-05-20T16:23:13.982578068Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-20T16:23:14.023206074Z INFO: INTERCONNECT MODE
2026-05-20T16:23:14.023300878Z INFO: Gathering ovn-kubernetes DBs
2026-05-20T16:23:14.071865838Z Wrote inspect data to must-gather.
2026-05-20T16:23:14.183260780Z INFO: Gathering OVN_Northbound from ovnkube-node-gqhp4...
2026-05-20T16:23:14.183538253Z INFO: Gathering OVN_Northbound from ovnkube-node-hgrgj...
2026-05-20T16:23:14.183828122Z INFO: Gathering OVN_Northbound from ovnkube-node-vdghx...
2026-05-20T16:23:14.184128807Z INFO: Gathering OVN_Southbound from ovnkube-node-gqhp4...
2026-05-20T16:23:14.184415450Z INFO: Gathering OVN_Southbound from ovnkube-node-hgrgj...
2026-05-20T16:23:14.184671976Z INFO: Gathering OVN_Southbound from ovnkube-node-vdghx...
2026-05-20T16:23:14.235357280Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-20T16:23:14.346147077Z Wrote inspect data to must-gather.
2026-05-20T16:23:14.472561812Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.494114310Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.528963034Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.530745948Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.553631287Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.553631287Z tar: Removing leading `/' from member names
2026-05-20T16:23:14.581288750Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T16:23:14.583890329Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-20T16:23:14.628929361Z Wrote inspect data to must-gather.
2026-05-20T16:23:14.859083303Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-20T16:23:14.886618274Z Wrote inspect data to must-gather.
2026-05-20T16:23:15.036805945Z INFO: Waiting for network log collection to complete ...
2026-05-20T16:23:15.036861897Z INFO: Waiting for ovnk database copies to complete ...
2026-05-20T16:23:15.037396703Z INFO: Copying ovnk databases complete.
2026-05-20T16:23:15.039492031Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-05-20T16:23:15.042601756Z ovnk_database_store/
2026-05-20T16:23:15.042601756Z ovnk_database_store/ovnkube-node-vdghx_sbdb
2026-05-20T16:23:15.093598274Z ovnk_database_store/ovnkube-node-vdghx_nbdb
2026-05-20T16:23:15.101821091Z ovnk_database_store/ovnkube-node-gqhp4_nbdb
2026-05-20T16:23:15.111560200Z ovnk_database_store/ovnkube-node-gqhp4_sbdb
2026-05-20T16:23:15.113877253Z W0520 16:23:15.113844    1380 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-20T16:23:15.118903362Z Wrote inspect data to must-gather.
2026-05-20T16:23:15.137800161Z ovnk_database_store/ovnkube-node-hgrgj_nbdb
2026-05-20T16:23:15.147504495Z ovnk_database_store/ovnkube-node-hgrgj_sbdb
2026-05-20T16:23:15.171200377Z INFO: Network log collection complete.
2026-05-20T16:23:15.211374626Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:23:15.285184230Z INFO: Getting status from alertmanager-main-0
2026-05-20T16:23:15.394469492Z Gathering data for ns/openshift-network-console...
2026-05-20T16:23:15.424808722Z Wrote inspect data to must-gather.
2026-05-20T16:23:15.692572601Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-20T16:23:15.695351614Z INFO: Fallback to identify the container image from release info
2026-05-20T16:23:15.813277731Z Gathering data for ns/openshift-console-operator...
2026-05-20T16:23:16.195703655Z Gathering data for ns/openshift-console...
2026-05-20T16:23:16.241192394Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-20T16:23:16.378059048Z daemonset.apps/perf-node-gather-daemonset created
2026-05-20T16:23:16.512167597Z Waiting for performance profile collector pods to become ready: 1
2026-05-20T16:23:16.652470015Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-20T16:23:17.022329600Z Gathering data for ns/openshift-dns-operator...
2026-05-20T16:23:17.329235738Z Gathering data for ns/openshift-dns...
2026-05-20T16:23:17.631679515Z Waiting for performance profile collector pods to become ready: 2
2026-05-20T16:23:17.993775246Z Gathering data for ns/openshift-image-registry...
2026-05-20T16:23:18.218302385Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-20T16:23:18.485744722Z Gathering data for ns/openshift-ingress-operator...
2026-05-20T16:23:18.779487216Z Waiting for performance profile collector pods to become ready: 3
2026-05-20T16:23:18.867414630Z Gathering data for ns/openshift-ingress...
2026-05-20T16:23:19.213880144Z Gathering data for ns/openshift-ingress-canary...
2026-05-20T16:23:19.607801243Z Gathering data for ns/openshift-insights...
2026-05-20T16:23:19.895518275Z Waiting for performance profile collector pods to become ready: 4
2026-05-20T16:23:21.022251910Z Waiting for performance profile collector pods to become ready: 5
2026-05-20T16:23:21.759469742Z Gathering data for ns/kserve...
2026-05-20T16:23:22.153146046Z Waiting for performance profile collector pods to become ready: 6
2026-05-20T16:23:23.223057460Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-20T16:23:23.278266267Z Waiting for performance profile collector pods to become ready: 7
2026-05-20T16:23:23.350990436Z Gathering data for ns/openshift-config...
2026-05-20T16:23:23.636466247Z Gathering data for ns/openshift-config-managed...
2026-05-20T16:23:24.052530013Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-20T16:23:24.361885539Z Gathering data for ns/openshift-kube-apiserver...
2026-05-20T16:23:24.414593915Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-20T16:23:24.638653739Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-20T16:23:24.691796426Z Collecting performance related data for node ip-10-0-129-117.ec2.internal
2026-05-20T16:23:24.701458118Z Collecting performance related data for node ip-10-0-136-42.ec2.internal
2026-05-20T16:23:24.709044829Z Collecting performance related data for node ip-10-0-135-174.ec2.internal
2026-05-20T16:23:24.980183116Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-20T16:23:25.388614937Z Gathering data for ns/openshift-kube-scheduler...
2026-05-20T16:23:25.737748264Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-20T16:23:26.123913042Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-20T16:23:26.495696514Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-20T16:23:26.819137540Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-20T16:23:27.382721360Z Gathering data for ns/openshift-multus...
2026-05-20T16:23:28.230148204Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-20T16:23:28.848211075Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-20T16:23:30.414459800Z Gathering data for ns/openshift-host-network...
2026-05-20T16:23:30.791672284Z Gathering data for ns/openshift-network-diagnostics...
2026-05-20T16:23:31.317159770Z Gathering data for ns/openshift-network-node-identity...
2026-05-20T16:23:31.746783679Z Gathering data for ns/openshift-network-operator...
2026-05-20T16:23:32.170375895Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-20T16:23:32.465779276Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-20T16:23:32.932155970Z Gathering data for ns/openshift-apiserver-operator...
2026-05-20T16:23:32.978124355Z Collecting kubelet logs for node ip-10-0-129-117.ec2.internal
2026-05-20T16:23:32.980336249Z Collecting kubelet logs for node ip-10-0-136-42.ec2.internal
2026-05-20T16:23:32.985207109Z Collecting kubelet logs for node ip-10-0-135-174.ec2.internal
2026-05-20T16:23:33.238934967Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-20T16:23:33.285544858Z Gathering data for ns/openshift-apiserver...
2026-05-20T16:23:33.399482306Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-20T16:23:33.404390179Z INFO: Node performance data collection complete.
2026-05-20T16:23:33.729236694Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-20T16:23:34.004407415Z Gathering data for ns/openshift-controller-manager...
2026-05-20T16:23:34.338751243Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-20T16:23:34.967344553Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-20T16:23:35.246355228Z Gathering data for ns/openshift-service-ca-operator...
2026-05-20T16:23:35.549970152Z Gathering data for ns/openshift-service-ca...
2026-05-20T16:23:35.906518657Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-20T16:23:36.774296466Z Wrote inspect data to must-gather.
2026-05-20T16:23:36.774339311Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T16:23:36.774339311Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-20T16:23:36.907376500Z error: the server doesn't have a resource type "clusters"
2026-05-20T16:23:37.049879954Z Caches written to disk