2026-04-22T19:46:24.982181272Z [disk usage checker] Started
2026-04-22T19:46:24.985411363Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:25.201415911Z Gathering data for ns/openshift-cluster-version...
2026-04-22T19:46:25.340717213Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T19:46:25.419934964Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:46:25.501771445Z Gathering data for ns/default...
2026-04-22T19:46:25.794414138Z Gathering data for ns/openshift...
2026-04-22T19:46:26.199826002Z Gathering data for ns/kube-system...
2026-04-22T19:46:26.874665563Z Gathering data for ns/openshift-etcd...
2026-04-22T19:46:27.186655384Z Wrote inspect data to must-gather.
2026-04-22T19:46:27.186688177Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T19:46:27.186688177Z     namespaces "assisted-installer" not found
2026-04-22T19:46:27.420268442Z Waiting on subprocesses to finish execution.
2026-04-22T19:46:27.430843995Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-22T19:46:27.431681067Z INFO: Gathering HAProxy config files
2026-04-22T19:46:27.433873242Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-22T19:46:27.438604879Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-22T19:46:27.446746038Z INFO: Collecting host service logs for crio
2026-04-22T19:46:27.447141788Z INFO: Collecting host service logs for kubelet
2026-04-22T19:46:27.447753171Z INFO: Collecting host service logs for rpm-ostreed
2026-04-22T19:46:27.448110938Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-22T19:46:27.448464794Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-22T19:46:27.448790784Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-22T19:46:27.449206081Z INFO: Collecting host service logs for NetworkManager
2026-04-22T19:46:27.449548607Z INFO: Collecting host service logs for openvswitch
2026-04-22T19:46:27.449879377Z INFO: Collecting host service logs for ovs-configuration
2026-04-22T19:46:27.450242864Z INFO: Collecting host service logs for ovsdb-server
2026-04-22T19:46:27.450586060Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-22T19:46:27.450973362Z INFO: Waiting for worker host service log collection to complete ...
2026-04-22T19:46:27.814066739Z INFO: Waiting for node performance related collection to complete ...
2026-04-22T19:46:27.857249335Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-22T19:46:28.085229215Z No resources found in openshift-etcd namespace.
2026-04-22T19:46:28.186637342Z No resources found
2026-04-22T19:46:28.257645787Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-22T19:46:28.331541741Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-22T19:46:28.331637115Z INFO: on-disk MachineConfig config collection complete.
2026-04-22T19:46:28.431407971Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-22T19:46:28.598177768Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-gs4mb
2026-04-22T19:46:28.598251242Z insights-runtime-extractor-6lmd2
2026-04-22T19:46:28.598273382Z insights-runtime-extractor-bkxxm
2026-04-22T19:46:28.598295009Z insights-runtime-extractor-j9zwr
2026-04-22T19:46:28.680188428Z error: the server doesn't have a resource type "performanceprofile"
2026-04-22T19:46:28.711739621Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-22T19:46:28.747229131Z INFO: "metallb-operator" not detected. Skipping.
2026-04-22T19:46:28.836880244Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-22T19:46:28.837496987Z Wrote inspect data to must-gather.
2026-04-22T19:46:28.940156937Z INFO: Worker host service log collection to complete.
2026-04-22T19:46:29.067744474Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-22T19:46:29.067744474Z See 'oc rsync -h' for help and examples
2026-04-22T19:46:29.141337040Z tar: Removing leading `/' from member names
2026-04-22T19:46:29.142953691Z ERROR: No running kube-apiserver pods found
2026-04-22T19:46:29.152560587Z INFO: HAProxy config collection complete.
2026-04-22T19:46:29.399363438Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-22T19:46:29.400906359Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.453702419Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.588836867Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-22T19:46:29.606175377Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.687387911Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.688889736Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:46:29.733586048Z No resources found
2026-04-22T19:46:29.737658671Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.803109008Z Wrote inspect data to must-gather.
2026-04-22T19:46:29.843642846Z error: the server doesn't have a resource type "machineconfigs"
2026-04-22T19:46:29.853293067Z INFO: 'previous-logs' folder not found on ip-10-0-131-5.ec2.internal, skipping...
2026-04-22T19:46:29.947621545Z Gathering data for ns/openshift-monitoring...
2026-04-22T19:46:30.000030272Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:30.042939402Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.090453205Z INFO: Getting rules from prometheus-k8s-0
2026-04-22T19:46:30.106936668Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.134692548Z INFO: 'previous-logs' folder not found on ip-10-0-131-85.ec2.internal, skipping...
2026-04-22T19:46:30.325314874Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.343511477Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.414343917Z INFO: 'previous-logs' folder not found on ip-10-0-134-126.ec2.internal, skipping...
2026-04-22T19:46:30.414440360Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-22T19:46:30.414475335Z INFO: Machine Config Daemon termination log collection complete.
2026-04-22T19:46:30.512977776Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-22T19:46:30.519262098Z INFO: Getting status/config from prometheus-k8s-0
2026-04-22T19:46:30.575598723Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.732935342Z Wrote inspect data to must-gather.
2026-04-22T19:46:30.877154075Z INFO: INTERCONNECT MODE
2026-04-22T19:46:30.877185402Z INFO: Gathering ovn-kubernetes DBs
2026-04-22T19:46:30.932477627Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-22T19:46:30.935887956Z Wrote inspect data to must-gather.
2026-04-22T19:46:31.036821418Z INFO: Gathering OVN_Northbound from ovnkube-node-ncvvr...
2026-04-22T19:46:31.037217038Z INFO: Gathering OVN_Northbound from ovnkube-node-rx4st...
2026-04-22T19:46:31.037529341Z INFO: Gathering OVN_Northbound from ovnkube-node-vjgnk...
2026-04-22T19:46:31.037873584Z INFO: Gathering OVN_Southbound from ovnkube-node-ncvvr...
2026-04-22T19:46:31.038185569Z INFO: Gathering OVN_Southbound from ovnkube-node-rx4st...
2026-04-22T19:46:31.038491336Z INFO: Gathering OVN_Southbound from ovnkube-node-vjgnk...
2026-04-22T19:46:31.249342417Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-22T19:46:31.281203348Z Wrote inspect data to must-gather.
2026-04-22T19:46:31.335777824Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.342905385Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.348770395Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.357608967Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.364730304Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.379686142Z tar: Removing leading `/' from member names
2026-04-22T19:46:31.411908223Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T19:46:31.524662377Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-22T19:46:31.528264452Z Wrote inspect data to must-gather.
2026-04-22T19:46:31.767128991Z Wrote inspect data to must-gather.
2026-04-22T19:46:31.803502889Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-22T19:46:31.921545364Z INFO: Waiting for network log collection to complete ...
2026-04-22T19:46:31.921545364Z INFO: Waiting for ovnk database copies to complete ...
2026-04-22T19:46:31.922046412Z INFO: Copying ovnk databases complete.
2026-04-22T19:46:31.923678417Z 3.1M	must-gather/network_logs/ovnk_database_store
2026-04-22T19:46:31.926528858Z ovnk_database_store/
2026-04-22T19:46:31.926573705Z ovnk_database_store/ovnkube-node-ncvvr_nbdb
2026-04-22T19:46:31.934977760Z ovnk_database_store/ovnkube-node-vjgnk_sbdb
2026-04-22T19:46:31.953400449Z ovnk_database_store/ovnkube-node-ncvvr_sbdb
2026-04-22T19:46:31.973876528Z ovnk_database_store/ovnkube-node-rx4st_nbdb
2026-04-22T19:46:31.983821900Z ovnk_database_store/ovnkube-node-vjgnk_nbdb
2026-04-22T19:46:31.986961812Z W0422 19:46:31.986917    1364 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-22T19:46:31.992053162Z ovnk_database_store/ovnkube-node-rx4st_sbdb
2026-04-22T19:46:31.992945602Z Wrote inspect data to must-gather.
2026-04-22T19:46:32.013156822Z INFO: Network log collection complete.
2026-04-22T19:46:32.040719133Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:46:32.215867432Z Gathering data for ns/openshift-network-console...
2026-04-22T19:46:32.239202049Z INFO: Getting status from alertmanager-main-0
2026-04-22T19:46:32.321032179Z Wrote inspect data to must-gather.
2026-04-22T19:46:32.604760478Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-22T19:46:32.607021526Z INFO: Fallback to identify the container image from release info
2026-04-22T19:46:32.626873296Z Gathering data for ns/openshift-console-operator...
2026-04-22T19:46:32.983022700Z Gathering data for ns/openshift-console...
2026-04-22T19:46:33.177409095Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-22T19:46:33.302439947Z daemonset.apps/perf-node-gather-daemonset created
2026-04-22T19:46:33.445333545Z Waiting for performance profile collector pods to become ready: 1
2026-04-22T19:46:33.446893924Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-22T19:46:33.833005434Z Gathering data for ns/openshift-dns-operator...
2026-04-22T19:46:34.137547399Z Gathering data for ns/openshift-dns...
2026-04-22T19:46:34.567472492Z Waiting for performance profile collector pods to become ready: 2
2026-04-22T19:46:34.739322200Z Gathering data for ns/openshift-image-registry...
2026-04-22T19:46:35.005107821Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:35.195264030Z Gathering data for ns/openshift-ingress-operator...
2026-04-22T19:46:35.495935585Z Gathering data for ns/openshift-ingress...
2026-04-22T19:46:35.681782187Z Waiting for performance profile collector pods to become ready: 3
2026-04-22T19:46:35.827033620Z Gathering data for ns/openshift-ingress-canary...
2026-04-22T19:46:36.198212936Z Gathering data for ns/openshift-insights...
2026-04-22T19:46:36.823961668Z Waiting for performance profile collector pods to become ready: 4
2026-04-22T19:46:37.950902535Z Waiting for performance profile collector pods to become ready: 5
2026-04-22T19:46:38.252646258Z Gathering data for ns/kserve...
2026-04-22T19:46:39.070325811Z Waiting for performance profile collector pods to become ready: 6
2026-04-22T19:46:40.010204653Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:40.012160233Z Gathering data for ns/openshift-config...
2026-04-22T19:46:40.194010319Z Waiting for performance profile collector pods to become ready: 7
2026-04-22T19:46:40.261184938Z Gathering data for ns/openshift-config-managed...
2026-04-22T19:46:40.667044319Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-22T19:46:40.921186953Z Gathering data for ns/openshift-kube-apiserver...
2026-04-22T19:46:41.158882335Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-22T19:46:41.325776664Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-22T19:46:41.451353279Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-22T19:46:41.605004389Z Collecting performance related data for node ip-10-0-134-126.ec2.internal
2026-04-22T19:46:41.614806390Z Collecting performance related data for node ip-10-0-131-5.ec2.internal
2026-04-22T19:46:41.624654998Z Collecting performance related data for node ip-10-0-131-85.ec2.internal
2026-04-22T19:46:41.784798996Z Gathering data for ns/openshift-kube-scheduler...
2026-04-22T19:46:42.087049817Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-22T19:46:42.423432744Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-22T19:46:42.822730210Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-22T19:46:43.184536283Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-22T19:46:43.678900570Z Gathering data for ns/openshift-multus...
2026-04-22T19:46:45.014596371Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:45.043826015Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-22T19:46:46.613632675Z Gathering data for ns/openshift-host-network...
2026-04-22T19:46:46.940859779Z Gathering data for ns/openshift-network-diagnostics...
2026-04-22T19:46:47.567494081Z Gathering data for ns/openshift-network-node-identity...
2026-04-22T19:46:47.960849069Z Gathering data for ns/openshift-network-operator...
2026-04-22T19:46:48.311641327Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-22T19:46:48.558317950Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-22T19:46:49.016274545Z Gathering data for ns/openshift-apiserver-operator...
2026-04-22T19:46:49.294287920Z Gathering data for ns/openshift-apiserver...
2026-04-22T19:46:49.699357120Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-22T19:46:49.851414306Z Collecting kubelet logs for node ip-10-0-134-126.ec2.internal
2026-04-22T19:46:49.853683571Z Collecting kubelet logs for node ip-10-0-131-5.ec2.internal
2026-04-22T19:46:49.856569990Z Collecting kubelet logs for node ip-10-0-131-85.ec2.internal
2026-04-22T19:46:49.996496397Z Gathering data for ns/openshift-controller-manager...
2026-04-22T19:46:50.021610838Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-22T19:46:50.247978955Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-22T19:46:50.255189035Z INFO: Node performance data collection complete.
2026-04-22T19:46:50.305346683Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-22T19:46:50.932617038Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-22T19:46:51.195007576Z Gathering data for ns/openshift-service-ca-operator...
2026-04-22T19:46:51.476286688Z Gathering data for ns/openshift-service-ca...
2026-04-22T19:46:51.819289910Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-22T19:46:52.653284737Z Wrote inspect data to must-gather.
2026-04-22T19:46:52.653330202Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T19:46:52.653330202Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-22T19:46:52.781895341Z error: the server doesn't have a resource type "clusters"
2026-04-22T19:46:52.921938960Z Caches written to disk