2026-04-20T19:57:19.196853381Z [disk usage checker] Started
2026-04-20T19:57:19.200930571Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-20T19:57:19.419169902Z Gathering data for ns/openshift-cluster-version...
2026-04-20T19:57:19.555119190Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T19:57:19.683215946Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T19:57:19.782902038Z Gathering data for ns/default...
2026-04-20T19:57:20.135849686Z Gathering data for ns/openshift...
2026-04-20T19:57:20.626661273Z Gathering data for ns/kube-system...
2026-04-20T19:57:21.341635179Z Gathering data for ns/openshift-etcd...
2026-04-20T19:57:21.708443433Z Wrote inspect data to must-gather.
2026-04-20T19:57:21.708477222Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T19:57:21.708477222Z     namespaces "assisted-installer" not found
2026-04-20T19:57:21.813415914Z Waiting on subprocesses to finish execution.
2026-04-20T19:57:21.817035752Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-20T19:57:21.823155927Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-20T19:57:21.824193047Z INFO: Gathering HAProxy config files
2026-04-20T19:57:21.834213963Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-20T19:57:21.834936169Z INFO: Collecting host service logs for crio
2026-04-20T19:57:21.835501802Z INFO: Collecting host service logs for kubelet
2026-04-20T19:57:21.835855527Z INFO: Collecting host service logs for rpm-ostreed
2026-04-20T19:57:21.836187353Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-20T19:57:21.836573873Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-20T19:57:21.836921542Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-20T19:57:21.837280996Z INFO: Collecting host service logs for NetworkManager
2026-04-20T19:57:21.837682181Z INFO: Collecting host service logs for openvswitch
2026-04-20T19:57:21.838095792Z INFO: Collecting host service logs for ovs-configuration
2026-04-20T19:57:21.838515838Z INFO: Collecting host service logs for ovsdb-server
2026-04-20T19:57:21.838834786Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-20T19:57:21.839200218Z INFO: Waiting for worker host service log collection to complete ...
2026-04-20T19:57:21.863363977Z INFO: Waiting for node performance related collection to complete ...
2026-04-20T19:57:22.465955554Z error: the server doesn't have a resource type "performanceprofile"
2026-04-20T19:57:22.743042166Z No resources found
2026-04-20T19:57:22.751524970Z No resources found in openshift-etcd namespace.
2026-04-20T19:57:22.848888475Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-20T19:57:22.958663780Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-20T19:57:23.060557529Z INFO: "metallb-operator" not detected. Skipping.
2026-04-20T19:57:23.169914642Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-20T19:57:23.169947549Z INFO: on-disk MachineConfig config collection complete.
2026-04-20T19:57:23.290396596Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-20T19:57:23.517493928Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-tjv47
2026-04-20T19:57:23.517549224Z insights-runtime-extractor-52pdc
2026-04-20T19:57:23.517583873Z insights-runtime-extractor-6xz5p
2026-04-20T19:57:23.517604586Z insights-runtime-extractor-8pzkw
2026-04-20T19:57:23.517912705Z Wrote inspect data to must-gather.
2026-04-20T19:57:23.555762089Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-20T19:57:23.587104637Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T19:57:23.740082763Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-20T19:57:23.858394143Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.002397307Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-20T19:57:24.025210182Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-20T19:57:24.025210182Z See 'oc rsync -h' for help and examples
2026-04-20T19:57:24.107447639Z INFO: Worker host service log collection to complete.
2026-04-20T19:57:24.167600475Z ERROR: No running kube-apiserver pods found
2026-04-20T19:57:24.215450203Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.235595767Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-20T19:57:24.409183505Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.474070078Z No resources found
2026-04-20T19:57:24.504589165Z INFO: 'previous-logs' folder not found on ip-10-0-129-98.ec2.internal, skipping...
2026-04-20T19:57:24.555733351Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-20T19:57:24.621776253Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.659817147Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.700327036Z Wrote inspect data to must-gather.
2026-04-20T19:57:24.912020659Z error: the server doesn't have a resource type "machineconfigs"
2026-04-20T19:57:24.944195065Z tar: Removing leading `/' from member names
2026-04-20T19:57:24.950547762Z INFO: HAProxy config collection complete.
2026-04-20T19:57:25.034095495Z INFO: Getting rules from prometheus-k8s-0
2026-04-20T19:57:25.062089516Z INFO: 'previous-logs' folder not found on ip-10-0-133-149.ec2.internal, skipping...
2026-04-20T19:57:25.140887779Z Wrote inspect data to must-gather.
2026-04-20T19:57:25.287409505Z Wrote inspect data to must-gather.
2026-04-20T19:57:25.328956988Z Gathering data for ns/kuadrant-system...
2026-04-20T19:57:25.438931384Z INFO: 'previous-logs' folder not found on ip-10-0-134-118.ec2.internal, skipping...
2026-04-20T19:57:25.439044533Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-20T19:57:25.439097421Z INFO: Machine Config Daemon termination log collection complete.
2026-04-20T19:57:25.540002432Z Wrote inspect data to must-gather.
2026-04-20T19:57:25.549818153Z INFO: Getting status/config from prometheus-k8s-0
2026-04-20T19:57:25.621128432Z Wrote inspect data to must-gather.
2026-04-20T19:57:25.866529574Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-20T19:57:25.953267769Z INFO: INTERCONNECT MODE
2026-04-20T19:57:25.953356221Z INFO: Gathering ovn-kubernetes DBs
2026-04-20T19:57:26.059719219Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-20T19:57:26.180243456Z INFO: Gathering OVN_Northbound from ovnkube-node-d9tnf...
2026-04-20T19:57:26.180539317Z INFO: Gathering OVN_Northbound from ovnkube-node-f5z9f...
2026-04-20T19:57:26.180786518Z INFO: Gathering OVN_Northbound from ovnkube-node-rhxmj...
2026-04-20T19:57:26.181325703Z INFO: Gathering OVN_Southbound from ovnkube-node-d9tnf...
2026-04-20T19:57:26.181711703Z INFO: Gathering OVN_Southbound from ovnkube-node-f5z9f...
2026-04-20T19:57:26.182003852Z INFO: Gathering OVN_Southbound from ovnkube-node-rhxmj...
2026-04-20T19:57:26.186647508Z Wrote inspect data to must-gather.
2026-04-20T19:57:26.437521916Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-20T19:57:26.523577241Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.607585957Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.615943816Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.697690203Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.698503895Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.705280215Z Wrote inspect data to must-gather.
2026-04-20T19:57:26.818891893Z tar: Removing leading `/' from member names
2026-04-20T19:57:26.837487445Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T19:57:26.932737772Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-20T19:57:27.091149395Z Wrote inspect data to must-gather.
2026-04-20T19:57:27.236895430Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-20T19:57:27.416604827Z W0420 19:57:27.416552    1318 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-20T19:57:27.445346238Z INFO: Waiting for network log collection to complete ...
2026-04-20T19:57:27.445386189Z INFO: Waiting for ovnk database copies to complete ...
2026-04-20T19:57:27.445908088Z INFO: Copying ovnk databases complete.
2026-04-20T19:57:27.448208159Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-20T19:57:27.452185852Z ovnk_database_store/
2026-04-20T19:57:27.452239695Z ovnk_database_store/ovnkube-node-rhxmj_nbdb
2026-04-20T19:57:27.452741712Z Wrote inspect data to must-gather.
2026-04-20T19:57:27.463019618Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T19:57:27.469788473Z ovnk_database_store/ovnkube-node-f5z9f_nbdb
2026-04-20T19:57:27.483971139Z ovnk_database_store/ovnkube-node-f5z9f_sbdb
2026-04-20T19:57:27.522468895Z ovnk_database_store/ovnkube-node-rhxmj_sbdb
2026-04-20T19:57:27.552698863Z ovnk_database_store/ovnkube-node-d9tnf_nbdb
2026-04-20T19:57:27.568731264Z ovnk_database_store/ovnkube-node-d9tnf_sbdb
2026-04-20T19:57:27.604307993Z INFO: Network log collection complete.
2026-04-20T19:57:27.708556581Z Gathering data for ns/openshift-monitoring...
2026-04-20T19:57:27.749295883Z INFO: Getting status from alertmanager-main-0
2026-04-20T19:57:27.865236033Z Wrote inspect data to must-gather.
2026-04-20T19:57:28.178959939Z Wrote inspect data to must-gather.
2026-04-20T19:57:28.452533829Z Wrote inspect data to must-gather.
2026-04-20T19:57:28.744962448Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-20T19:57:28.749834019Z INFO: Fallback to identify the container image from release info
2026-04-20T19:57:29.244557047Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-20T19:57:29.407305728Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-20T19:57:29.535484406Z daemonset.apps/perf-node-gather-daemonset created
2026-04-20T19:57:29.695821184Z Waiting for performance profile collector pods to become ready: 1
2026-04-20T19:57:29.786635093Z Gathering data for ns/openshift-network-console...
2026-04-20T19:57:30.328430547Z Gathering data for ns/openshift-console-operator...
2026-04-20T19:57:30.799866207Z Gathering data for ns/openshift-console...
2026-04-20T19:57:30.854409388Z Waiting for performance profile collector pods to become ready: 2
2026-04-20T19:57:31.305124136Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-20T19:57:31.483885078Z Wrote inspect data to must-gather.
2026-04-20T19:57:31.677574071Z Gathering data for ns/openshift-dns-operator...
2026-04-20T19:57:31.997070953Z Gathering data for ns/openshift-dns...
2026-04-20T19:57:32.003035671Z Waiting for performance profile collector pods to become ready: 3
2026-04-20T19:57:32.598016885Z Gathering data for ns/openshift-image-registry...
2026-04-20T19:57:33.078870661Z Gathering data for ns/openshift-ingress-operator...
2026-04-20T19:57:33.132142751Z Waiting for performance profile collector pods to become ready: 4
2026-04-20T19:57:33.468932171Z Gathering data for ns/openshift-ingress...
2026-04-20T19:57:34.251182376Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-20T19:57:34.258852718Z Gathering data for ns/openshift-ingress-canary...
2026-04-20T19:57:34.274112960Z Waiting for performance profile collector pods to become ready: 5
2026-04-20T19:57:34.759086064Z Gathering data for ns/openshift-insights...
2026-04-20T19:57:35.421324408Z Waiting for performance profile collector pods to become ready: 6
2026-04-20T19:57:36.545165326Z Waiting for performance profile collector pods to become ready: 7
2026-04-20T19:57:36.778604403Z Gathering data for ns/opendatahub...
2026-04-20T19:57:37.681653039Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-20T19:57:37.933331123Z Collecting performance related data for node ip-10-0-134-118.ec2.internal
2026-04-20T19:57:37.940988354Z Collecting performance related data for node ip-10-0-129-98.ec2.internal
2026-04-20T19:57:37.949943130Z Collecting performance related data for node ip-10-0-133-149.ec2.internal
2026-04-20T19:57:38.315589767Z Gathering data for ns/openshift-lws-operator...
2026-04-20T19:57:39.256646141Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-20T19:57:39.810009409Z Gathering data for ns/openshift-config...
2026-04-20T19:57:40.131937301Z Gathering data for ns/openshift-config-managed...
2026-04-20T19:57:40.708502634Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-20T19:57:41.062941014Z Gathering data for ns/openshift-kube-apiserver...
2026-04-20T19:57:41.383566464Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-20T19:57:41.740668490Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-20T19:57:42.083903959Z Gathering data for ns/openshift-kube-scheduler...
2026-04-20T19:57:42.408770402Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-20T19:57:42.856174002Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-20T19:57:43.239921427Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-20T19:57:43.606657586Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-20T19:57:44.219760796Z Gathering data for ns/openshift-multus...
2026-04-20T19:57:44.261819588Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-20T19:57:45.637494820Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-20T19:57:46.699426835Z Collecting kubelet logs for node ip-10-0-134-118.ec2.internal
2026-04-20T19:57:46.701852513Z Collecting kubelet logs for node ip-10-0-129-98.ec2.internal
2026-04-20T19:57:46.704116271Z Collecting kubelet logs for node ip-10-0-133-149.ec2.internal
2026-04-20T19:57:47.136876886Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-20T19:57:47.142202922Z INFO: Node performance data collection complete.
2026-04-20T19:57:47.287458491Z Gathering data for ns/openshift-host-network...
2026-04-20T19:57:47.654607413Z Gathering data for ns/openshift-network-diagnostics...
2026-04-20T19:57:48.269991302Z Gathering data for ns/openshift-network-node-identity...
2026-04-20T19:57:48.732424761Z Gathering data for ns/openshift-network-operator...
2026-04-20T19:57:49.135487032Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-20T19:57:49.266562681Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-20T19:57:49.442346410Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-20T19:57:49.916546400Z Gathering data for ns/openshift-apiserver-operator...
2026-04-20T19:57:50.257715138Z Gathering data for ns/openshift-apiserver...
2026-04-20T19:57:50.744196210Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-20T19:57:51.054171046Z Gathering data for ns/openshift-controller-manager...
2026-04-20T19:57:51.369740597Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-20T19:57:52.061977014Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-20T19:57:52.400278085Z Gathering data for ns/openshift-service-ca-operator...
2026-04-20T19:57:52.739406373Z Gathering data for ns/openshift-service-ca...
2026-04-20T19:57:53.135351717Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-20T19:57:54.162133009Z Wrote inspect data to must-gather.
2026-04-20T19:57:54.162179644Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T19:57:54.162179644Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-20T19:57:54.271614180Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-20T19:57:54.303038065Z error: the server doesn't have a resource type "clusters"
2026-04-20T19:57:54.480592615Z Caches written to disk