2026-04-20T14:01:48.872823837Z [disk usage checker] Started
2026-04-20T14:01:48.875952897Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:01:49.097769808Z Gathering data for ns/openshift-cluster-version...
2026-04-20T14:01:49.320271784Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T14:01:49.431801926Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T14:01:49.529164926Z Gathering data for ns/default...
2026-04-20T14:01:49.884503601Z Gathering data for ns/openshift...
2026-04-20T14:01:50.380027920Z Gathering data for ns/kube-system...
2026-04-20T14:01:51.114108716Z Gathering data for ns/openshift-etcd...
2026-04-20T14:01:51.301593419Z Waiting on subprocesses to finish execution.
2026-04-20T14:01:51.316376409Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-20T14:01:51.316831861Z INFO: Gathering HAProxy config files
2026-04-20T14:01:51.317734103Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-20T14:01:51.320954525Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-20T14:01:51.338322245Z INFO: Collecting host service logs for crio
2026-04-20T14:01:51.338825102Z INFO: Collecting host service logs for kubelet
2026-04-20T14:01:51.339197452Z INFO: Collecting host service logs for rpm-ostreed
2026-04-20T14:01:51.339552491Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-20T14:01:51.339956842Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-20T14:01:51.340306343Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-20T14:01:51.340663448Z INFO: Collecting host service logs for NetworkManager
2026-04-20T14:01:51.349455288Z INFO: Collecting host service logs for openvswitch
2026-04-20T14:01:51.349876343Z INFO: Collecting host service logs for ovs-configuration
2026-04-20T14:01:51.350273726Z INFO: Collecting host service logs for ovsdb-server
2026-04-20T14:01:51.350646756Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-20T14:01:51.351321783Z INFO: Waiting for worker host service log collection to complete ...
2026-04-20T14:01:51.375997265Z INFO: Waiting for node performance related collection to complete ...
2026-04-20T14:01:51.954200519Z Wrote inspect data to must-gather.
2026-04-20T14:01:51.954287663Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T14:01:51.954287663Z     namespaces "assisted-installer" not found
2026-04-20T14:01:52.101685295Z No resources found in openshift-etcd namespace.
2026-04-20T14:01:52.218222708Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-qrddw
2026-04-20T14:01:52.218274301Z insights-runtime-extractor-h754j
2026-04-20T14:01:52.218294452Z insights-runtime-extractor-l2cjz
2026-04-20T14:01:52.218314029Z insights-runtime-extractor-qdwz4
2026-04-20T14:01:52.262950213Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-20T14:01:52.498131228Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-20T14:01:52.542834551Z error: the server doesn't have a resource type "performanceprofile"
2026-04-20T14:01:52.680504959Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-20T14:01:52.686205493Z No resources found
2026-04-20T14:01:52.697316608Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-20T14:01:52.784548427Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-20T14:01:52.784644216Z INFO: on-disk MachineConfig config collection complete.
2026-04-20T14:01:53.102943150Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-20T14:01:53.102943150Z See 'oc rsync -h' for help and examples
2026-04-20T14:01:53.142254120Z INFO: Worker host service log collection to complete.
2026-04-20T14:01:53.158421885Z INFO: "metallb-operator" not detected. Skipping.
2026-04-20T14:01:53.198661701Z Wrote inspect data to must-gather.
2026-04-20T14:01:53.325128470Z Wrote inspect data to must-gather.
2026-04-20T14:01:53.445914419Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T14:01:53.615741827Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-20T14:01:53.638838960Z ERROR: No running kube-apiserver pods found
2026-04-20T14:01:53.641855993Z Wrote inspect data to must-gather.
2026-04-20T14:01:53.800220631Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-20T14:01:53.914546303Z Wrote inspect data to must-gather.
2026-04-20T14:01:53.938597792Z No resources found
2026-04-20T14:01:53.942678684Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:01:54.010111308Z Wrote inspect data to must-gather.
2026-04-20T14:01:54.066706155Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-20T14:01:54.117390388Z Wrote inspect data to must-gather.
2026-04-20T14:01:54.126083329Z INFO: 'previous-logs' folder not found on ip-10-0-132-232.ec2.internal, skipping...
2026-04-20T14:01:54.187258064Z tar: Removing leading `/' from member names
2026-04-20T14:01:54.193099436Z INFO: HAProxy config collection complete.
2026-04-20T14:01:54.252206076Z error: the server doesn't have a resource type "machineconfigs"
2026-04-20T14:01:54.391306487Z Gathering data for ns/kuadrant-system...
2026-04-20T14:01:54.474466956Z Wrote inspect data to must-gather.
2026-04-20T14:01:54.509299632Z INFO: 'previous-logs' folder not found on ip-10-0-133-1.ec2.internal, skipping...
2026-04-20T14:01:54.569607926Z INFO: Getting rules from prometheus-k8s-0
2026-04-20T14:01:54.587093512Z Wrote inspect data to must-gather.
2026-04-20T14:01:54.834555403Z INFO: 'previous-logs' folder not found on ip-10-0-142-144.ec2.internal, skipping...
2026-04-20T14:01:54.834688311Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-20T14:01:54.834778161Z INFO: Machine Config Daemon termination log collection complete.
2026-04-20T14:01:54.857568142Z Wrote inspect data to must-gather.
2026-04-20T14:01:54.890886515Z Wrote inspect data to must-gather.
2026-04-20T14:01:55.067423379Z INFO: Getting status/config from prometheus-k8s-0
2026-04-20T14:01:55.106470012Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-20T14:01:55.179778949Z Wrote inspect data to must-gather.
2026-04-20T14:01:55.450175796Z Wrote inspect data to must-gather.
2026-04-20T14:01:55.570920862Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T14:01:55.578252309Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-20T14:01:55.590647799Z INFO: INTERCONNECT MODE
2026-04-20T14:01:55.590763780Z INFO: Gathering ovn-kubernetes DBs
2026-04-20T14:01:55.761446496Z Wrote inspect data to must-gather.
2026-04-20T14:01:55.786028868Z INFO: Gathering OVN_Northbound from ovnkube-node-drksq...
2026-04-20T14:01:55.786399932Z INFO: Gathering OVN_Northbound from ovnkube-node-mp62t...
2026-04-20T14:01:55.786699410Z INFO: Gathering OVN_Northbound from ovnkube-node-qtnxb...
2026-04-20T14:01:55.787067419Z INFO: Gathering OVN_Southbound from ovnkube-node-drksq...
2026-04-20T14:01:55.787357901Z INFO: Gathering OVN_Southbound from ovnkube-node-mp62t...
2026-04-20T14:01:55.787639797Z INFO: Gathering OVN_Southbound from ovnkube-node-qtnxb...
2026-04-20T14:01:55.904830235Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-20T14:01:56.186022148Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.191642979Z Wrote inspect data to must-gather.
2026-04-20T14:01:56.259414043Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.337128589Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.348936424Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.368811133Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.392927068Z tar: Removing leading `/' from member names
2026-04-20T14:01:56.393969583Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T14:01:56.463179530Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-20T14:01:56.657938172Z Gathering data for ns/openshift-monitoring...
2026-04-20T14:01:56.671441870Z Wrote inspect data to must-gather.
2026-04-20T14:01:56.789049002Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-20T14:01:57.002621152Z INFO: Waiting for network log collection to complete ...
2026-04-20T14:01:57.002650625Z INFO: Waiting for ovnk database copies to complete ...
2026-04-20T14:01:57.003046107Z INFO: Copying ovnk databases complete.
2026-04-20T14:01:57.004589050Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-20T14:01:57.006739346Z ovnk_database_store/
2026-04-20T14:01:57.006739346Z ovnk_database_store/ovnkube-node-drksq_sbdb
2026-04-20T14:01:57.033811481Z ovnk_database_store/ovnkube-node-mp62t_nbdb
2026-04-20T14:01:57.044206965Z ovnk_database_store/ovnkube-node-qtnxb_sbdb
2026-04-20T14:01:57.065487020Z Wrote inspect data to must-gather.
2026-04-20T14:01:57.066453346Z ovnk_database_store/ovnkube-node-drksq_nbdb
2026-04-20T14:01:57.078931924Z ovnk_database_store/ovnkube-node-mp62t_sbdb
2026-04-20T14:01:57.104817483Z ovnk_database_store/ovnkube-node-qtnxb_nbdb
2026-04-20T14:01:57.119882240Z INFO: Network log collection complete.
2026-04-20T14:01:57.271859676Z INFO: Getting status from alertmanager-main-0
2026-04-20T14:01:57.438863100Z Wrote inspect data to must-gather.
2026-04-20T14:01:57.757643713Z Wrote inspect data to must-gather.
2026-04-20T14:01:58.023112641Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-20T14:01:58.025457725Z INFO: Fallback to identify the container image from release info
2026-04-20T14:01:58.623976601Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-20T14:01:58.761279445Z daemonset.apps/perf-node-gather-daemonset created
2026-04-20T14:01:58.786356276Z Gathering data for ns/openshift-network-console...
2026-04-20T14:01:58.931868642Z Waiting for performance profile collector pods to become ready: 1
2026-04-20T14:01:58.947685249Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:01:59.358272699Z Gathering data for ns/openshift-console-operator...
2026-04-20T14:01:59.836083882Z Gathering data for ns/openshift-console...
2026-04-20T14:02:00.067362190Z Waiting for performance profile collector pods to become ready: 2
2026-04-20T14:02:00.383380509Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-20T14:02:00.729240269Z Wrote inspect data to must-gather.
2026-04-20T14:02:00.820043085Z Gathering data for ns/openshift-dns-operator...
2026-04-20T14:02:01.181787678Z Gathering data for ns/openshift-dns...
2026-04-20T14:02:01.203706030Z Waiting for performance profile collector pods to become ready: 3
2026-04-20T14:02:01.804387406Z Gathering data for ns/openshift-image-registry...
2026-04-20T14:02:02.308986537Z Gathering data for ns/openshift-ingress-operator...
2026-04-20T14:02:02.338926884Z Waiting for performance profile collector pods to become ready: 4
2026-04-20T14:02:02.683974252Z Gathering data for ns/openshift-ingress...
2026-04-20T14:02:03.384939394Z Gathering data for ns/openshift-ingress-canary...
2026-04-20T14:02:03.457221009Z Waiting for performance profile collector pods to become ready: 5
2026-04-20T14:02:03.839451554Z Gathering data for ns/openshift-insights...
2026-04-20T14:02:03.952155870Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:02:04.586147895Z Waiting for performance profile collector pods to become ready: 6
2026-04-20T14:02:05.698686042Z Waiting for performance profile collector pods to become ready: 7
2026-04-20T14:02:05.859717985Z Gathering data for ns/opendatahub...
2026-04-20T14:02:06.829039575Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-20T14:02:07.077821340Z Collecting performance related data for node ip-10-0-142-144.ec2.internal
2026-04-20T14:02:07.084498149Z Collecting performance related data for node ip-10-0-132-232.ec2.internal
2026-04-20T14:02:07.091660481Z Collecting performance related data for node ip-10-0-133-1.ec2.internal
2026-04-20T14:02:07.281774318Z Gathering data for ns/openshift-lws-operator...
2026-04-20T14:02:08.763062835Z Gathering data for ns/openshift-config...
2026-04-20T14:02:08.957371845Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:02:09.075152271Z Gathering data for ns/openshift-config-managed...
2026-04-20T14:02:09.617852348Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-20T14:02:09.996487038Z Gathering data for ns/openshift-kube-apiserver...
2026-04-20T14:02:10.341110991Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-20T14:02:10.692505123Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-20T14:02:11.048614652Z Gathering data for ns/openshift-kube-scheduler...
2026-04-20T14:02:11.370366882Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-20T14:02:11.721169377Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-20T14:02:12.098066848Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-20T14:02:12.455098234Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-20T14:02:13.033630147Z Gathering data for ns/openshift-multus...
2026-04-20T14:02:13.964219279Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:02:14.484609987Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-20T14:02:15.346387169Z Collecting kubelet logs for node ip-10-0-142-144.ec2.internal
2026-04-20T14:02:15.348842900Z Collecting kubelet logs for node ip-10-0-132-232.ec2.internal
2026-04-20T14:02:15.350947106Z Collecting kubelet logs for node ip-10-0-133-1.ec2.internal
2026-04-20T14:02:15.719865435Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-20T14:02:15.725208674Z INFO: Node performance data collection complete.
2026-04-20T14:02:16.146593510Z Gathering data for ns/openshift-host-network...
2026-04-20T14:02:16.533425646Z Gathering data for ns/openshift-network-diagnostics...
2026-04-20T14:02:17.123916954Z Gathering data for ns/openshift-network-node-identity...
2026-04-20T14:02:17.614719452Z Gathering data for ns/openshift-network-operator...
2026-04-20T14:02:18.018524357Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-20T14:02:18.334642439Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-20T14:02:18.813102312Z Gathering data for ns/openshift-apiserver-operator...
2026-04-20T14:02:18.968908892Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T14:02:19.155645012Z Gathering data for ns/openshift-apiserver...
2026-04-20T14:02:19.593703649Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-20T14:02:19.940733789Z Gathering data for ns/openshift-controller-manager...
2026-04-20T14:02:20.260742766Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-20T14:02:20.978151125Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-20T14:02:21.320018876Z Gathering data for ns/openshift-service-ca-operator...
2026-04-20T14:02:21.667089466Z Gathering data for ns/openshift-service-ca...
2026-04-20T14:02:22.087467140Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-20T14:02:23.212143633Z Wrote inspect data to must-gather.
2026-04-20T14:02:23.212188043Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T14:02:23.212188043Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-20T14:02:23.347408807Z error: the server doesn't have a resource type "clusters"
2026-04-20T14:02:23.530790094Z Caches written to disk