2026-04-17T18:20:58.950942683Z [disk usage checker] Started
2026-04-17T18:20:58.990250924Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:20:59.145035928Z Gathering data for ns/openshift-cluster-version...
2026-04-17T18:20:59.294476319Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-17T18:20:59.384317724Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-17T18:20:59.458838124Z Gathering data for ns/default...
2026-04-17T18:20:59.763820728Z Gathering data for ns/openshift...
2026-04-17T18:21:00.163354017Z Gathering data for ns/kube-system...
2026-04-17T18:21:00.843095566Z Gathering data for ns/openshift-etcd...
2026-04-17T18:21:01.120863236Z Waiting on subprocesses to finish execution.
2026-04-17T18:21:01.127597281Z INFO: Gathering HAProxy config files
2026-04-17T18:21:01.128282086Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-17T18:21:01.136527614Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-17T18:21:01.146672326Z INFO: Collecting host service logs for crio
2026-04-17T18:21:01.147004871Z INFO: Collecting host service logs for kubelet
2026-04-17T18:21:01.147308002Z Wrote inspect data to must-gather.
2026-04-17T18:21:01.147362962Z INFO: Collecting host service logs for rpm-ostreed
2026-04-17T18:21:01.147381418Z error: inspection completed with the errors occurred while gathering data:
2026-04-17T18:21:01.147381418Z     namespaces "assisted-installer" not found
2026-04-17T18:21:01.147743590Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-17T18:21:01.149284811Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-17T18:21:01.149610472Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-17T18:21:01.149978993Z INFO: Collecting host service logs for NetworkManager
2026-04-17T18:21:01.150310823Z INFO: Collecting host service logs for openvswitch
2026-04-17T18:21:01.150608615Z INFO: Collecting host service logs for ovs-configuration
2026-04-17T18:21:01.150945021Z INFO: Collecting host service logs for ovsdb-server
2026-04-17T18:21:01.151270049Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-17T18:21:01.151661147Z INFO: Waiting for worker host service log collection to complete ...
2026-04-17T18:21:01.153219048Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-17T18:21:01.205399744Z INFO: Waiting for node performance related collection to complete ...
2026-04-17T18:21:01.860559435Z No resources found in openshift-etcd namespace.
2026-04-17T18:21:01.919140765Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-17T18:21:01.919179301Z INFO: on-disk MachineConfig config collection complete.
2026-04-17T18:21:01.949262101Z No resources found
2026-04-17T18:21:01.988253049Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-17T18:21:02.134077722Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-17T18:21:02.161245557Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-17T18:21:02.188343250Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-tlzlj
2026-04-17T18:21:02.188397409Z insights-runtime-extractor-bmqm9
2026-04-17T18:21:02.188417543Z insights-runtime-extractor-l65jn
2026-04-17T18:21:02.188436451Z insights-runtime-extractor-vsbmf
2026-04-17T18:21:02.202138838Z Wrote inspect data to must-gather.
2026-04-17T18:21:02.245138239Z INFO: "metallb-operator" not detected. Skipping.
2026-04-17T18:21:02.288584527Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-17T18:21:02.291658249Z error: the server doesn't have a resource type "performanceprofile"
2026-04-17T18:21:02.341786461Z INFO: Worker host service log collection to complete.
2026-04-17T18:21:02.394918318Z ERROR: No running kube-apiserver pods found
2026-04-17T18:21:02.413983038Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-17T18:21:02.453114326Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-17T18:21:02.453114326Z See 'oc rsync -h' for help and examples
2026-04-17T18:21:02.641777134Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-17T18:21:02.930026000Z Gathering data for ns/openshift-monitoring...
2026-04-17T18:21:02.936221317Z No resources found
2026-04-17T18:21:03.065064542Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-17T18:21:03.071451823Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.119137106Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.267049626Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.274030154Z INFO: 'previous-logs' folder not found on ip-10-0-133-142.ec2.internal, skipping...
2026-04-17T18:21:03.285700393Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.297662971Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-17T18:21:03.334575472Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.386772757Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.390338187Z error: the server doesn't have a resource type "machineconfigs"
2026-04-17T18:21:03.391046329Z tar: Removing leading `/' from member names
2026-04-17T18:21:03.395831084Z INFO: HAProxy config collection complete.
2026-04-17T18:21:03.535074651Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.550076817Z INFO: 'previous-logs' folder not found on ip-10-0-134-83.ec2.internal, skipping...
2026-04-17T18:21:03.559262188Z INFO: Getting rules from prometheus-k8s-0
2026-04-17T18:21:03.611560101Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.780615955Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.811526852Z INFO: 'previous-logs' folder not found on ip-10-0-139-47.ec2.internal, skipping...
2026-04-17T18:21:03.811561806Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-17T18:21:03.811561806Z INFO: Machine Config Daemon termination log collection complete.
2026-04-17T18:21:03.840789270Z Wrote inspect data to must-gather.
2026-04-17T18:21:03.957864558Z INFO: Getting status/config from prometheus-k8s-0
2026-04-17T18:21:03.981130194Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-17T18:21:03.995703489Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:21:04.006582469Z Wrote inspect data to must-gather.
2026-04-17T18:21:04.170230303Z Wrote inspect data to must-gather.
2026-04-17T18:21:04.261574211Z INFO: INTERCONNECT MODE
2026-04-17T18:21:04.261604882Z INFO: Gathering ovn-kubernetes DBs
2026-04-17T18:21:04.337153754Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-17T18:21:04.376127458Z Wrote inspect data to must-gather.
2026-04-17T18:21:04.448917707Z INFO: Gathering OVN_Northbound from ovnkube-node-5w6tv...
2026-04-17T18:21:04.449190832Z INFO: Gathering OVN_Northbound from ovnkube-node-ql5pl...
2026-04-17T18:21:04.449431454Z INFO: Gathering OVN_Northbound from ovnkube-node-sk45l...
2026-04-17T18:21:04.449789892Z INFO: Gathering OVN_Southbound from ovnkube-node-5w6tv...
2026-04-17T18:21:04.450035068Z INFO: Gathering OVN_Southbound from ovnkube-node-ql5pl...
2026-04-17T18:21:04.450281180Z INFO: Gathering OVN_Southbound from ovnkube-node-sk45l...
2026-04-17T18:21:04.586419456Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-17T18:21:04.671963583Z Wrote inspect data to must-gather.
2026-04-17T18:21:04.765830795Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.765862565Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.766327989Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.772422339Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.812690616Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.819180236Z tar: Removing leading `/' from member names
2026-04-17T18:21:04.835379020Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-17T18:21:04.890647424Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-17T18:21:04.900786603Z Wrote inspect data to must-gather.
2026-04-17T18:21:05.115790155Z Wrote inspect data to must-gather.
2026-04-17T18:21:05.145623322Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-17T18:21:05.201246898Z INFO: Waiting for network log collection to complete ...
2026-04-17T18:21:05.201276302Z INFO: Waiting for ovnk database copies to complete ...
2026-04-17T18:21:05.201677192Z INFO: Copying ovnk databases complete.
2026-04-17T18:21:05.203184528Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-04-17T18:21:05.205316269Z ovnk_database_store/
2026-04-17T18:21:05.205365759Z ovnk_database_store/ovnkube-node-sk45l_nbdb
2026-04-17T18:21:05.214890217Z ovnk_database_store/ovnkube-node-5w6tv_nbdb
2026-04-17T18:21:05.220436404Z ovnk_database_store/ovnkube-node-5w6tv_sbdb
2026-04-17T18:21:05.237731733Z ovnk_database_store/ovnkube-node-ql5pl_nbdb
2026-04-17T18:21:05.244559609Z ovnk_database_store/ovnkube-node-ql5pl_sbdb
2026-04-17T18:21:05.265291107Z ovnk_database_store/ovnkube-node-sk45l_sbdb
2026-04-17T18:21:05.283422583Z INFO: Network log collection complete.
2026-04-17T18:21:05.318896935Z W0417 18:21:05.318851    1350 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-17T18:21:05.323364522Z Wrote inspect data to must-gather.
2026-04-17T18:21:05.435910245Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-17T18:21:05.503065729Z INFO: Getting status from alertmanager-main-0
2026-04-17T18:21:05.603426810Z Wrote inspect data to must-gather.
2026-04-17T18:21:05.606183517Z Gathering data for ns/openshift-network-console...
2026-04-17T18:21:05.832378599Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-17T18:21:05.834723374Z INFO: Fallback to identify the container image from release info
2026-04-17T18:21:06.042219220Z Gathering data for ns/openshift-console-operator...
2026-04-17T18:21:06.386878670Z Gathering data for ns/openshift-console...
2026-04-17T18:21:06.436483016Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-17T18:21:06.566211136Z daemonset.apps/perf-node-gather-daemonset created
2026-04-17T18:21:06.683492243Z Waiting for performance profile collector pods to become ready: 1
2026-04-17T18:21:06.841613875Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-17T18:21:07.175400532Z Gathering data for ns/openshift-dns-operator...
2026-04-17T18:21:07.471432665Z Gathering data for ns/openshift-dns...
2026-04-17T18:21:07.803182234Z Waiting for performance profile collector pods to become ready: 2
2026-04-17T18:21:08.080721758Z Gathering data for ns/openshift-image-registry...
2026-04-17T18:21:08.526545530Z Gathering data for ns/openshift-ingress-operator...
2026-04-17T18:21:08.832087822Z Gathering data for ns/openshift-ingress...
2026-04-17T18:21:08.918373667Z Waiting for performance profile collector pods to become ready: 3
2026-04-17T18:21:09.001700864Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:21:09.156075259Z Gathering data for ns/openshift-ingress-canary...
2026-04-17T18:21:09.532575844Z Gathering data for ns/openshift-insights...
2026-04-17T18:21:10.035037694Z Waiting for performance profile collector pods to become ready: 4
2026-04-17T18:21:11.143095513Z Waiting for performance profile collector pods to become ready: 5
2026-04-17T18:21:11.326346793Z Gathering data for ns/openshift-jobset-operator...
2026-04-17T18:21:12.059845453Z Gathering data for ns/openshift-config...
2026-04-17T18:21:12.251309689Z Waiting for performance profile collector pods to become ready: 6
2026-04-17T18:21:12.314107735Z Gathering data for ns/openshift-config-managed...
2026-04-17T18:21:12.777018477Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-17T18:21:13.063825997Z Gathering data for ns/openshift-kube-apiserver...
2026-04-17T18:21:13.367169773Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-17T18:21:13.369651525Z Waiting for performance profile collector pods to become ready: 7
2026-04-17T18:21:13.662887572Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-17T18:21:14.006476557Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:21:14.024013758Z Gathering data for ns/openshift-kube-scheduler...
2026-04-17T18:21:14.303199012Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-17T18:21:14.467117229Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-17T18:21:14.651806444Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-17T18:21:14.713428280Z Collecting performance related data for node ip-10-0-134-83.ec2.internal
2026-04-17T18:21:14.721204259Z Collecting performance related data for node ip-10-0-139-47.ec2.internal
2026-04-17T18:21:14.737903306Z Collecting performance related data for node ip-10-0-133-142.ec2.internal
2026-04-17T18:21:15.025729929Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-17T18:21:15.462744705Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-17T18:21:16.073791019Z Gathering data for ns/openshift-multus...
2026-04-17T18:21:17.671732778Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-17T18:21:19.011847999Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:21:19.340539297Z Gathering data for ns/openshift-host-network...
2026-04-17T18:21:19.654155915Z Gathering data for ns/openshift-network-diagnostics...
2026-04-17T18:21:20.234947162Z Gathering data for ns/openshift-network-node-identity...
2026-04-17T18:21:20.716495349Z Gathering data for ns/openshift-network-operator...
2026-04-17T18:21:21.122503300Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-17T18:21:21.427559757Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-17T18:21:21.918064639Z Gathering data for ns/openshift-apiserver-operator...
2026-04-17T18:21:22.251047931Z Gathering data for ns/openshift-apiserver...
2026-04-17T18:21:22.392203074Z Collecting kubelet logs for node ip-10-0-134-83.ec2.internal
2026-04-17T18:21:22.394701802Z Collecting kubelet logs for node ip-10-0-139-47.ec2.internal
2026-04-17T18:21:22.397048054Z Collecting kubelet logs for node ip-10-0-133-142.ec2.internal
2026-04-17T18:21:22.702325284Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-17T18:21:22.755935131Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-17T18:21:22.762619253Z INFO: Node performance data collection complete.
2026-04-17T18:21:22.994307401Z Gathering data for ns/openshift-controller-manager...
2026-04-17T18:21:23.257428013Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-17T18:21:23.885882646Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-17T18:21:24.016914782Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-17T18:21:24.165369306Z Gathering data for ns/openshift-service-ca-operator...
2026-04-17T18:21:24.489055084Z Gathering data for ns/openshift-service-ca...
2026-04-17T18:21:24.840678616Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-17T18:21:25.744005645Z Wrote inspect data to must-gather.
2026-04-17T18:21:25.744041728Z error: inspection completed with the errors occurred while gathering data:
2026-04-17T18:21:25.744041728Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-17T18:21:25.861768732Z error: the server doesn't have a resource type "clusters"
2026-04-17T18:21:25.997168388Z Caches written to disk