2026-04-16T19:14:05.834339442Z [disk usage checker] Started
2026-04-16T19:14:05.837721113Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:06.033572013Z Gathering data for ns/openshift-cluster-version...
2026-04-16T19:14:06.169095379Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:14:06.258821609Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:14:06.341158495Z Gathering data for ns/default...
2026-04-16T19:14:06.647490001Z Gathering data for ns/openshift...
2026-04-16T19:14:07.080073918Z Gathering data for ns/kube-system...
2026-04-16T19:14:07.793555883Z Gathering data for ns/openshift-etcd...
2026-04-16T19:14:08.084910083Z Wrote inspect data to must-gather.
2026-04-16T19:14:08.084935105Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:14:08.084935105Z     namespaces "assisted-installer" not found
2026-04-16T19:14:08.209445419Z Waiting on subprocesses to finish execution.
2026-04-16T19:14:08.216265603Z INFO: Collecting host service logs for crio
2026-04-16T19:14:08.216589348Z INFO: Collecting host service logs for kubelet
2026-04-16T19:14:08.216928243Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T19:14:08.217262058Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T19:14:08.217610153Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T19:14:08.218092941Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T19:14:08.218434959Z INFO: Collecting host service logs for NetworkManager
2026-04-16T19:14:08.218780577Z INFO: Collecting host service logs for openvswitch
2026-04-16T19:14:08.219095685Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T19:14:08.219507654Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T19:14:08.219879822Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T19:14:08.220284400Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T19:14:08.225147553Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T19:14:08.231719861Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T19:14:08.233149983Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T19:14:08.234199889Z INFO: Gathering HAProxy config files
2026-04-16T19:14:08.517083235Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T19:14:08.738420752Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T19:14:08.960550800Z No resources found in openshift-etcd namespace.
2026-04-16T19:14:09.021445574Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T19:14:09.066288415Z Wrote inspect data to must-gather.
2026-04-16T19:14:09.127305130Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T19:14:09.173384475Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T19:14:09.225930176Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T19:14:09.263968450Z No resources found
2026-04-16T19:14:09.374517515Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T19:14:09.390121864Z INFO: Collecting Insights Archives from insights-operator-5785d4fcdd-f6kdh
2026-04-16T19:14:09.390218963Z insights-runtime-extractor-2z8gk
2026-04-16T19:14:09.390262618Z insights-runtime-extractor-48x9c
2026-04-16T19:14:09.390283209Z insights-runtime-extractor-78gqq
2026-04-16T19:14:09.469357065Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T19:14:09.469388089Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T19:14:09.751266290Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T19:14:09.974821638Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T19:14:09.974821638Z See 'oc rsync -h' for help and examples
2026-04-16T19:14:09.998610479Z INFO: Worker host service log collection to complete.
2026-04-16T19:14:10.239672059Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.254006975Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T19:14:10.278183731Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.281982322Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T19:14:10.283786271Z No resources found
2026-04-16T19:14:10.337827902Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.366440144Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.418727134Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:14:10.424520473Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.552859602Z tar: Removing leading `/' from member names
2026-04-16T19:14:10.561170538Z INFO: HAProxy config collection complete.
2026-04-16T19:14:10.561623531Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.683615369Z ERROR: No running kube-apiserver pods found
2026-04-16T19:14:10.698054647Z INFO: 'previous-logs' folder not found on ip-10-0-139-96.ec2.internal, skipping...
2026-04-16T19:14:10.699410431Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T19:14:10.793136300Z Gathering data for ns/openshift-monitoring...
2026-04-16T19:14:10.848285959Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:10.888830912Z Wrote inspect data to must-gather.
2026-04-16T19:14:10.959771242Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.016941061Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T19:14:11.020838862Z INFO: 'previous-logs' folder not found on ip-10-0-141-192.ec2.internal, skipping...
2026-04-16T19:14:11.166299683Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.180918199Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.283035107Z INFO: 'previous-logs' folder not found on ip-10-0-143-48.ec2.internal, skipping...
2026-04-16T19:14:11.283084174Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T19:14:11.283115838Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T19:14:11.340310466Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T19:14:11.420731602Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.430304384Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T19:14:11.551448684Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.702627679Z INFO: INTERCONNECT MODE
2026-04-16T19:14:11.702660494Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T19:14:11.755676872Z Wrote inspect data to must-gather.
2026-04-16T19:14:11.839485194Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T19:14:11.872814811Z INFO: Gathering OVN_Northbound from ovnkube-node-b6sfk...
2026-04-16T19:14:11.873151176Z INFO: Gathering OVN_Northbound from ovnkube-node-fk24r...
2026-04-16T19:14:11.873448365Z INFO: Gathering OVN_Northbound from ovnkube-node-mbm6k...
2026-04-16T19:14:11.873778203Z INFO: Gathering OVN_Southbound from ovnkube-node-b6sfk...
2026-04-16T19:14:11.874068542Z INFO: Gathering OVN_Southbound from ovnkube-node-fk24r...
2026-04-16T19:14:11.874404182Z INFO: Gathering OVN_Southbound from ovnkube-node-mbm6k...
2026-04-16T19:14:12.087926429Z Wrote inspect data to must-gather.
2026-04-16T19:14:12.180526590Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.189261146Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.189874536Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.192133306Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.203855771Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.207325036Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T19:14:12.238913640Z tar: Removing leading `/' from member names
2026-04-16T19:14:12.284663943Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:14:12.353636798Z Wrote inspect data to must-gather.
2026-04-16T19:14:12.463577916Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T19:14:12.598489527Z Wrote inspect data to must-gather.
2026-04-16T19:14:12.662792732Z INFO: Waiting for network log collection to complete ...
2026-04-16T19:14:12.662821680Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T19:14:12.663195312Z INFO: Copying ovnk databases complete.
2026-04-16T19:14:12.664714607Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-04-16T19:14:12.666664053Z ovnk_database_store/
2026-04-16T19:14:12.666664053Z ovnk_database_store/ovnkube-node-b6sfk_sbdb
2026-04-16T19:14:12.692541595Z ovnk_database_store/ovnkube-node-mbm6k_nbdb
2026-04-16T19:14:12.704681996Z ovnk_database_store/ovnkube-node-b6sfk_nbdb
2026-04-16T19:14:12.715879953Z ovnk_database_store/ovnkube-node-mbm6k_sbdb
2026-04-16T19:14:12.736359667Z ovnk_database_store/ovnkube-node-fk24r_sbdb
2026-04-16T19:14:12.740652331Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T19:14:12.767583243Z ovnk_database_store/ovnkube-node-fk24r_nbdb
2026-04-16T19:14:12.781728128Z INFO: Network log collection complete.
2026-04-16T19:14:12.834102381Z W0416 19:14:12.834053    1379 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T19:14:12.841571619Z Wrote inspect data to must-gather.
2026-04-16T19:14:12.905374551Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:14:13.079988163Z Gathering data for ns/openshift-network-console...
2026-04-16T19:14:13.118750082Z INFO: Getting status from alertmanager-main-0
2026-04-16T19:14:13.138716243Z Wrote inspect data to must-gather.
2026-04-16T19:14:13.409103804Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T19:14:13.411078320Z INFO: Fallback to identify the container image from release info
2026-04-16T19:14:13.480153609Z Gathering data for ns/openshift-console-operator...
2026-04-16T19:14:13.813950081Z Gathering data for ns/openshift-console...
2026-04-16T19:14:13.924670438Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc77fc5f1546189d6b57967ccc684e794bd846ea36c3e3c1e61d83a684f0f9b9
2026-04-16T19:14:14.045726635Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T19:14:14.177291794Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T19:14:14.255450463Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T19:14:14.620535437Z Gathering data for ns/openshift-dns-operator...
2026-04-16T19:14:14.917172047Z Gathering data for ns/openshift-dns...
2026-04-16T19:14:15.298730615Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T19:14:15.474296778Z Gathering data for ns/openshift-image-registry...
2026-04-16T19:14:15.854434080Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:15.908049772Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T19:14:16.201686665Z Gathering data for ns/openshift-ingress...
2026-04-16T19:14:16.420917285Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T19:14:16.518026424Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T19:14:16.875266242Z Gathering data for ns/openshift-insights...
2026-04-16T19:14:17.597736884Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T19:14:18.711325864Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T19:14:18.931442229Z Gathering data for ns/kserve...
2026-04-16T19:14:19.845536933Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T19:14:20.723808882Z Gathering data for ns/openshift-config...
2026-04-16T19:14:20.859636329Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:20.986991855Z Gathering data for ns/openshift-config-managed...
2026-04-16T19:14:20.989492732Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T19:14:21.387863068Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T19:14:21.640218403Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T19:14:21.889097247Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T19:14:22.115600477Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T19:14:22.193339924Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T19:14:22.391220396Z Collecting performance related data for node ip-10-0-141-192.ec2.internal
2026-04-16T19:14:22.400545492Z Collecting performance related data for node ip-10-0-143-48.ec2.internal
2026-04-16T19:14:22.410388796Z Collecting performance related data for node ip-10-0-139-96.ec2.internal
2026-04-16T19:14:22.531915492Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T19:14:22.829472520Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T19:14:23.199637143Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T19:14:23.592760736Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T19:14:23.969156227Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T19:14:24.469073565Z Gathering data for ns/openshift-multus...
2026-04-16T19:14:25.865365670Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:25.904771374Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T19:14:27.332343968Z Gathering data for ns/openshift-host-network...
2026-04-16T19:14:27.675363466Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T19:14:28.189891800Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T19:14:28.604322904Z Gathering data for ns/openshift-network-operator...
2026-04-16T19:14:28.956545892Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T19:14:29.225275976Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T19:14:29.650553184Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T19:14:29.946560812Z Gathering data for ns/openshift-apiserver...
2026-04-16T19:14:30.368575812Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T19:14:30.544220223Z Collecting kubelet logs for node ip-10-0-141-192.ec2.internal
2026-04-16T19:14:30.546778120Z Collecting kubelet logs for node ip-10-0-143-48.ec2.internal
2026-04-16T19:14:30.548659981Z Collecting kubelet logs for node ip-10-0-139-96.ec2.internal
2026-04-16T19:14:30.642715131Z Gathering data for ns/openshift-controller-manager...
2026-04-16T19:14:30.872673276Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T19:14:30.956850434Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T19:14:30.961870325Z INFO: Node performance data collection complete.
2026-04-16T19:14:30.965552500Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T19:14:31.596791359Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T19:14:31.861750636Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T19:14:32.160774802Z Gathering data for ns/openshift-service-ca...
2026-04-16T19:14:32.496519794Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T19:14:33.333029877Z Wrote inspect data to must-gather.
2026-04-16T19:14:33.333069304Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:14:33.333069304Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T19:14:33.458867253Z error: the server doesn't have a resource type "clusters"
2026-04-16T19:14:33.825741973Z Caches written to disk