2026-06-03T16:54:18.745098264Z [disk usage checker] Started
2026-06-03T16:54:18.748725974Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:18.970233749Z Gathering data for ns/openshift-cluster-version...
2026-06-03T16:54:19.133583209Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-03T16:54:19.226225230Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:54:19.327778220Z Gathering data for ns/default...
2026-06-03T16:54:19.806217010Z Gathering data for ns/openshift...
2026-06-03T16:54:20.289889040Z Gathering data for ns/kube-system...
2026-06-03T16:54:20.984848797Z Gathering data for ns/openshift-etcd...
2026-06-03T16:54:21.219226299Z Waiting on subprocesses to finish execution.
2026-06-03T16:54:21.231874637Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-03T16:54:21.233176635Z INFO: Gathering HAProxy config files
2026-06-03T16:54:21.250576891Z INFO: Collecting host service logs for crio
2026-06-03T16:54:21.250989407Z INFO: Collecting host service logs for kubelet
2026-06-03T16:54:21.251381243Z INFO: Collecting host service logs for rpm-ostreed
2026-06-03T16:54:21.251728331Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-03T16:54:21.252067867Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-03T16:54:21.252419016Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-03T16:54:21.252467089Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-03T16:54:21.252752626Z INFO: Collecting host service logs for NetworkManager
2026-06-03T16:54:21.253097464Z INFO: Collecting host service logs for openvswitch
2026-06-03T16:54:21.253433574Z INFO: Collecting host service logs for ovs-configuration
2026-06-03T16:54:21.253776834Z INFO: Collecting host service logs for ovsdb-server
2026-06-03T16:54:21.254129078Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-03T16:54:21.254551060Z INFO: Waiting for worker host service log collection to complete ...
2026-06-03T16:54:21.258658225Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-03T16:54:21.380429830Z INFO: Waiting for node performance related collection to complete ...
2026-06-03T16:54:21.770348492Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-03T16:54:21.845066350Z No resources found
2026-06-03T16:54:22.013132408Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-03T16:54:22.317634924Z No resources found in openshift-etcd namespace.
2026-06-03T16:54:22.399488760Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-03T16:54:22.399488760Z INFO: on-disk MachineConfig config collection complete.
2026-06-03T16:54:22.458507761Z Wrote inspect data to must-gather.
2026-06-03T16:54:22.458594224Z error: inspection completed with the errors occurred while gathering data:
2026-06-03T16:54:22.458594224Z     namespaces "assisted-installer" not found
2026-06-03T16:54:22.520737050Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-03T16:54:22.631396330Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-03T16:54:22.730630735Z error: the server doesn't have a resource type "performanceprofile"
2026-06-03T16:54:22.811029549Z INFO: "metallb-operator" not detected. Skipping.
2026-06-03T16:54:22.838224249Z INFO: Worker host service log collection to complete.
2026-06-03T16:54:22.923581064Z INFO: Collecting Insights Archives from insights-operator-788bf58df5-ng9qq
2026-06-03T16:54:22.923630090Z insights-runtime-extractor-5xm9n
2026-06-03T16:54:22.923649866Z insights-runtime-extractor-kwjkx
2026-06-03T16:54:22.923668932Z insights-runtime-extractor-zx2cx
2026-06-03T16:54:22.946163598Z Wrote inspect data to must-gather.
2026-06-03T16:54:23.022511309Z ERROR: No running kube-apiserver pods found
2026-06-03T16:54:23.537313099Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-03T16:54:23.602723964Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-03T16:54:23.619077503Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:54:23.677764992Z No resources found
2026-06-03T16:54:23.697124087Z Wrote inspect data to must-gather.
2026-06-03T16:54:23.781589718Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-03T16:54:23.825335521Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:23.890964548Z Wrote inspect data to must-gather.
2026-06-03T16:54:23.973045947Z INFO: 'previous-logs' folder not found on ip-10-0-135-67.ec2.internal, skipping...
2026-06-03T16:54:24.002039182Z tar: Removing leading `/' from member names
2026-06-03T16:54:24.010681385Z INFO: HAProxy config collection complete.
2026-06-03T16:54:24.072512089Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-03T16:54:24.072512089Z See 'oc rsync -h' for help and examples
2026-06-03T16:54:24.255236925Z Wrote inspect data to must-gather.
2026-06-03T16:54:24.277211299Z Wrote inspect data to must-gather.
2026-06-03T16:54:24.321707643Z error: the server doesn't have a resource type "machineconfigs"
2026-06-03T16:54:24.345606954Z Gathering data for ns/kuadrant-system...
2026-06-03T16:54:24.387782137Z Wrote inspect data to must-gather.
2026-06-03T16:54:24.491892247Z INFO: 'previous-logs' folder not found on ip-10-0-140-123.ec2.internal, skipping...
2026-06-03T16:54:24.523885329Z INFO: Getting rules from prometheus-k8s-0
2026-06-03T16:54:24.780532470Z Wrote inspect data to must-gather.
2026-06-03T16:54:24.790780605Z Wrote inspect data to must-gather.
2026-06-03T16:54:24.837405018Z INFO: 'previous-logs' folder not found on ip-10-0-142-172.ec2.internal, skipping...
2026-06-03T16:54:24.837448892Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-03T16:54:24.837461851Z INFO: Machine Config Daemon termination log collection complete.
2026-06-03T16:54:25.022080370Z INFO: Getting status/config from prometheus-k8s-0
2026-06-03T16:54:25.210358317Z Wrote inspect data to must-gather.
2026-06-03T16:54:25.247782402Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-03T16:54:25.248731504Z Wrote inspect data to must-gather.
2026-06-03T16:54:25.414105898Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-03T16:54:25.563582475Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-03T16:54:25.684199520Z Wrote inspect data to must-gather.
2026-06-03T16:54:25.773948549Z W0603 16:54:25.773905    1170 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-03T16:54:25.801235446Z Wrote inspect data to must-gather.
2026-06-03T16:54:25.838930796Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-03T16:54:25.856369262Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:54:26.105277026Z INFO: INTERCONNECT MODE
2026-06-03T16:54:26.105357617Z INFO: Gathering ovn-kubernetes DBs
2026-06-03T16:54:26.136299425Z Gathering data for ns/openshift-monitoring...
2026-06-03T16:54:26.156615149Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-03T16:54:26.316044402Z Wrote inspect data to must-gather.
2026-06-03T16:54:26.331868056Z INFO: Gathering OVN_Northbound from ovnkube-node-94fdt...
2026-06-03T16:54:26.332225811Z INFO: Gathering OVN_Northbound from ovnkube-node-9mbgc...
2026-06-03T16:54:26.332515282Z INFO: Gathering OVN_Northbound from ovnkube-node-xh2c4...
2026-06-03T16:54:26.332843787Z INFO: Gathering OVN_Southbound from ovnkube-node-94fdt...
2026-06-03T16:54:26.333226821Z INFO: Gathering OVN_Southbound from ovnkube-node-9mbgc...
2026-06-03T16:54:26.333507842Z INFO: Gathering OVN_Southbound from ovnkube-node-xh2c4...
2026-06-03T16:54:26.541980487Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-03T16:54:26.673196595Z tar: Removing leading `/' from member names
2026-06-03T16:54:26.684332581Z tar: Removing leading `/' from member names
2026-06-03T16:54:26.710167338Z tar: Removing leading `/' from member names
2026-06-03T16:54:26.723219748Z tar: Removing leading `/' from member names
2026-06-03T16:54:26.771755116Z tar: Removing leading `/' from member names
2026-06-03T16:54:26.783320692Z tar: Removing leading `/' from member names
2026-06-03T16:54:27.023266432Z Wrote inspect data to must-gather.
2026-06-03T16:54:27.086866069Z INFO: Getting status from alertmanager-main-0
2026-06-03T16:54:27.314642215Z INFO: Waiting for network log collection to complete ...
2026-06-03T16:54:27.314698985Z INFO: Waiting for ovnk database copies to complete ...
2026-06-03T16:54:27.315200920Z INFO: Copying ovnk databases complete.
2026-06-03T16:54:27.316898509Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-06-03T16:54:27.319412553Z ovnk_database_store/
2026-06-03T16:54:27.319412553Z ovnk_database_store/ovnkube-node-xh2c4_sbdb
2026-06-03T16:54:27.343115624Z ovnk_database_store/ovnkube-node-94fdt_sbdb
2026-06-03T16:54:27.397815065Z ovnk_database_store/ovnkube-node-xh2c4_nbdb
2026-06-03T16:54:27.410042426Z ovnk_database_store/ovnkube-node-9mbgc_sbdb
2026-06-03T16:54:27.437886423Z ovnk_database_store/ovnkube-node-9mbgc_nbdb
2026-06-03T16:54:27.449714591Z ovnk_database_store/ovnkube-node-94fdt_nbdb
2026-06-03T16:54:27.465823584Z INFO: Network log collection complete.
2026-06-03T16:54:27.480881280Z Wrote inspect data to must-gather.
2026-06-03T16:54:27.912396217Z Wrote inspect data to must-gather.
2026-06-03T16:54:28.243390756Z W0603 16:54:28.243346    1451 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-03T16:54:28.286729646Z Wrote inspect data to must-gather.
2026-06-03T16:54:28.501416418Z Gathering data for ns/openshift-network-console...
2026-06-03T16:54:28.725348212Z Wrote inspect data to must-gather.
2026-06-03T16:54:28.849888776Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:29.047772019Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-03T16:54:29.051125831Z INFO: Fallback to identify the container image from release info
2026-06-03T16:54:29.080922307Z Gathering data for ns/openshift-console-operator...
2026-06-03T16:54:29.527139452Z Gathering data for ns/openshift-console...
2026-06-03T16:54:29.799095941Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8381cb85e479760bd6b601e111b39e6c56a632bfa298867f4197ec4421e9f5
2026-06-03T16:54:29.938522722Z daemonset.apps/perf-node-gather-daemonset created
2026-06-03T16:54:30.096335870Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-03T16:54:30.109894168Z Waiting for performance profile collector pods to become ready: 1
2026-06-03T16:54:30.576948353Z Gathering data for ns/openshift-dns-operator...
2026-06-03T16:54:30.905417313Z Wrote inspect data to must-gather.
2026-06-03T16:54:30.958123061Z Gathering data for ns/openshift-dns...
2026-06-03T16:54:31.218687231Z Waiting for performance profile collector pods to become ready: 2
2026-06-03T16:54:31.613854965Z Gathering data for ns/openshift-image-registry...
2026-06-03T16:54:32.142369777Z Gathering data for ns/openshift-ingress-operator...
2026-06-03T16:54:32.331176947Z Waiting for performance profile collector pods to become ready: 3
2026-06-03T16:54:32.517167804Z Gathering data for ns/openshift-ingress...
2026-06-03T16:54:33.211714010Z Gathering data for ns/openshift-ingress-canary...
2026-06-03T16:54:33.449574859Z Waiting for performance profile collector pods to become ready: 4
2026-06-03T16:54:33.688928877Z Gathering data for ns/openshift-insights...
2026-06-03T16:54:33.854672989Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:34.562150922Z Waiting for performance profile collector pods to become ready: 5
2026-06-03T16:54:35.676210492Z Waiting for performance profile collector pods to become ready: 6
2026-06-03T16:54:35.773199020Z Gathering data for ns/opendatahub...
2026-06-03T16:54:36.805991377Z Waiting for performance profile collector pods to become ready: 7
2026-06-03T16:54:37.455291618Z Gathering data for ns/openshift-lws-operator...
2026-06-03T16:54:37.936890754Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-03T16:54:38.200914008Z Collecting performance related data for node ip-10-0-142-172.ec2.internal
2026-06-03T16:54:38.207388888Z Collecting performance related data for node ip-10-0-135-67.ec2.internal
2026-06-03T16:54:38.214692323Z Collecting performance related data for node ip-10-0-140-123.ec2.internal
2026-06-03T16:54:38.860104565Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:39.155131905Z Gathering data for ns/openshift-config...
2026-06-03T16:54:39.575577743Z Gathering data for ns/openshift-config-managed...
2026-06-03T16:54:40.077187591Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-03T16:54:40.395893616Z Gathering data for ns/openshift-kube-apiserver...
2026-06-03T16:54:40.728264147Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-03T16:54:41.127775523Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-03T16:54:41.504465621Z Gathering data for ns/openshift-kube-scheduler...
2026-06-03T16:54:41.826706892Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-03T16:54:42.191423188Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-03T16:54:42.576081844Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-03T16:54:42.935659137Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-03T16:54:43.602272926Z Gathering data for ns/openshift-multus...
2026-06-03T16:54:43.866847319Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:45.065914479Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-03T16:54:46.660495271Z Gathering data for ns/openshift-host-network...
2026-06-03T16:54:46.678100048Z Collecting kubelet logs for node ip-10-0-142-172.ec2.internal
2026-06-03T16:54:46.680311798Z Collecting kubelet logs for node ip-10-0-135-67.ec2.internal
2026-06-03T16:54:46.683520685Z Collecting kubelet logs for node ip-10-0-140-123.ec2.internal
2026-06-03T16:54:47.088756554Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-06-03T16:54:47.095383171Z INFO: Node performance data collection complete.
2026-06-03T16:54:47.113300118Z Gathering data for ns/openshift-network-diagnostics...
2026-06-03T16:54:47.749597602Z Gathering data for ns/openshift-network-node-identity...
2026-06-03T16:54:48.210278229Z Gathering data for ns/openshift-network-operator...
2026-06-03T16:54:48.631118218Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-03T16:54:48.872786484Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:48.928684244Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-03T16:54:49.420220505Z Gathering data for ns/openshift-apiserver-operator...
2026-06-03T16:54:49.770615597Z Gathering data for ns/openshift-apiserver...
2026-06-03T16:54:50.224503095Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-03T16:54:50.541941699Z Gathering data for ns/openshift-controller-manager...
2026-06-03T16:54:50.867202319Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-03T16:54:51.535727878Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-03T16:54:51.862995975Z Gathering data for ns/openshift-service-ca-operator...
2026-06-03T16:54:52.203484656Z Gathering data for ns/openshift-service-ca...
2026-06-03T16:54:52.598640931Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-03T16:54:53.644097511Z Wrote inspect data to must-gather.
2026-06-03T16:54:53.644144125Z error: inspection completed with the errors occurred while gathering data:
2026-06-03T16:54:53.644144125Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-03T16:54:53.779624086Z error: the server doesn't have a resource type "clusters"
2026-06-03T16:54:53.877472456Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-06-03T16:54:54.044256272Z Caches written to disk