2026-05-21T13:12:22.829885513Z [disk usage checker] Started
2026-05-21T13:12:22.833338710Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:23.084238585Z Gathering data for ns/openshift-cluster-version...
2026-05-21T13:12:23.311201735Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T13:12:23.413997622Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T13:12:23.514535960Z Gathering data for ns/default...
2026-05-21T13:12:23.889447557Z Gathering data for ns/openshift...
2026-05-21T13:12:24.377179230Z Gathering data for ns/kube-system...
2026-05-21T13:12:25.273820245Z Waiting on subprocesses to finish execution.
2026-05-21T13:12:25.279750529Z INFO: Gathering HAProxy config files
2026-05-21T13:12:25.283662365Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-21T13:12:25.284495592Z INFO: Collecting host service logs for crio
2026-05-21T13:12:25.284845096Z INFO: Collecting host service logs for kubelet
2026-05-21T13:12:25.285194419Z INFO: Collecting host service logs for rpm-ostreed
2026-05-21T13:12:25.285496806Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-21T13:12:25.285795943Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-21T13:12:25.286134523Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-21T13:12:25.286447536Z INFO: Collecting host service logs for NetworkManager
2026-05-21T13:12:25.286740725Z INFO: Collecting host service logs for openvswitch
2026-05-21T13:12:25.287173340Z INFO: Collecting host service logs for ovs-configuration
2026-05-21T13:12:25.287498090Z INFO: Collecting host service logs for ovsdb-server
2026-05-21T13:12:25.287806439Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-21T13:12:25.288209565Z INFO: Waiting for worker host service log collection to complete ...
2026-05-21T13:12:25.341093079Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-21T13:12:25.356570177Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-21T13:12:25.385660163Z Gathering data for ns/openshift-etcd...
2026-05-21T13:12:25.563538811Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-21T13:12:25.684071033Z INFO: Waiting for node performance related collection to complete ...
2026-05-21T13:12:25.856594566Z No resources found
2026-05-21T13:12:26.022011335Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-c4t5s
2026-05-21T13:12:26.022076229Z insights-runtime-extractor-bznzd
2026-05-21T13:12:26.022096349Z insights-runtime-extractor-gvgpq
2026-05-21T13:12:26.022115662Z insights-runtime-extractor-sz9tc
2026-05-21T13:12:26.060544024Z No resources found in openshift-etcd namespace.
2026-05-21T13:12:26.354438889Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-21T13:12:26.514479391Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-21T13:12:26.585005668Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-21T13:12:26.585218540Z INFO: on-disk MachineConfig config collection complete.
2026-05-21T13:12:26.606073181Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-21T13:12:26.606073181Z See 'oc rsync -h' for help and examples
2026-05-21T13:12:26.683386817Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-21T13:12:26.771974295Z ERROR: No running kube-apiserver pods found
2026-05-21T13:12:26.802698868Z INFO: Worker host service log collection to complete.
2026-05-21T13:12:26.832196241Z INFO: "metallb-operator" not detected. Skipping.
2026-05-21T13:12:26.882122217Z Wrote inspect data to must-gather.
2026-05-21T13:12:26.882159262Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T13:12:26.882159262Z     namespaces "assisted-installer" not found
2026-05-21T13:12:26.906032488Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-21T13:12:27.069779282Z error: the server doesn't have a resource type "performanceprofile"
2026-05-21T13:12:27.161052149Z Wrote inspect data to must-gather.
2026-05-21T13:12:27.452318008Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-21T13:12:27.494475202Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-21T13:12:27.526223728Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T13:12:27.718471356Z Wrote inspect data to must-gather.
2026-05-21T13:12:27.794957418Z Wrote inspect data to must-gather.
2026-05-21T13:12:27.915646937Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:27.982620230Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.051253072Z INFO: 'previous-logs' folder not found on ip-10-0-133-193.ec2.internal, skipping...
2026-05-21T13:12:28.066049646Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.261183098Z No resources found
2026-05-21T13:12:28.327820571Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.366785119Z Gathering data for ns/kuadrant-system...
2026-05-21T13:12:28.371471878Z INFO: Getting rules from prometheus-k8s-0
2026-05-21T13:12:28.391028005Z tar: Removing leading `/' from member names
2026-05-21T13:12:28.398097523Z INFO: HAProxy config collection complete.
2026-05-21T13:12:28.424372997Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.493016741Z INFO: 'previous-logs' folder not found on ip-10-0-142-58.ec2.internal, skipping...
2026-05-21T13:12:28.559946225Z error: the server doesn't have a resource type "machineconfigs"
2026-05-21T13:12:28.798208884Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.871526542Z INFO: 'previous-logs' folder not found on ip-10-0-143-244.ec2.internal, skipping...
2026-05-21T13:12:28.871564040Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-21T13:12:28.871600196Z INFO: Machine Config Daemon termination log collection complete.
2026-05-21T13:12:28.896692632Z Wrote inspect data to must-gather.
2026-05-21T13:12:28.947039463Z INFO: Getting status/config from prometheus-k8s-0
2026-05-21T13:12:29.184768096Z Wrote inspect data to must-gather.
2026-05-21T13:12:29.188384900Z Wrote inspect data to must-gather.
2026-05-21T13:12:29.364751423Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-21T13:12:29.541053185Z INFO: INTERCONNECT MODE
2026-05-21T13:12:29.541138692Z INFO: Gathering ovn-kubernetes DBs
2026-05-21T13:12:29.548057705Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-21T13:12:29.687049813Z Wrote inspect data to must-gather.
2026-05-21T13:12:29.753192671Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T13:12:29.767930505Z INFO: Gathering OVN_Northbound from ovnkube-node-27mmh...
2026-05-21T13:12:29.768175464Z INFO: Gathering OVN_Northbound from ovnkube-node-gn2dt...
2026-05-21T13:12:29.768436830Z INFO: Gathering OVN_Northbound from ovnkube-node-s65jl...
2026-05-21T13:12:29.768734151Z INFO: Gathering OVN_Southbound from ovnkube-node-27mmh...
2026-05-21T13:12:29.769011100Z INFO: Gathering OVN_Southbound from ovnkube-node-gn2dt...
2026-05-21T13:12:29.769295651Z INFO: Gathering OVN_Southbound from ovnkube-node-s65jl...
2026-05-21T13:12:29.917041797Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-21T13:12:30.087461102Z Wrote inspect data to must-gather.
2026-05-21T13:12:30.152056985Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.270052291Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.323178338Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.349089527Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.352262299Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.354220114Z tar: Removing leading `/' from member names
2026-05-21T13:12:30.413927786Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-21T13:12:30.479194265Z W0521 13:12:30.479141    1306 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-21T13:12:30.510959793Z Wrote inspect data to must-gather.
2026-05-21T13:12:30.536566436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T13:12:30.853118625Z Gathering data for ns/openshift-monitoring...
2026-05-21T13:12:30.892536503Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-21T13:12:30.898899672Z INFO: Waiting for network log collection to complete ...
2026-05-21T13:12:30.899041782Z INFO: Waiting for ovnk database copies to complete ...
2026-05-21T13:12:30.899704366Z INFO: Copying ovnk databases complete.
2026-05-21T13:12:30.901599910Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-21T13:12:30.904419420Z ovnk_database_store/
2026-05-21T13:12:30.904463115Z ovnk_database_store/ovnkube-node-gn2dt_nbdb
2026-05-21T13:12:30.918905468Z ovnk_database_store/ovnkube-node-gn2dt_sbdb
2026-05-21T13:12:30.942279489Z ovnk_database_store/ovnkube-node-27mmh_sbdb
2026-05-21T13:12:30.950151260Z Wrote inspect data to must-gather.
2026-05-21T13:12:30.965922502Z ovnk_database_store/ovnkube-node-27mmh_nbdb
2026-05-21T13:12:30.976037978Z ovnk_database_store/ovnkube-node-s65jl_sbdb
2026-05-21T13:12:31.002193784Z ovnk_database_store/ovnkube-node-s65jl_nbdb
2026-05-21T13:12:31.019460731Z INFO: Network log collection complete.
2026-05-21T13:12:31.318602916Z INFO: Getting status from alertmanager-main-0
2026-05-21T13:12:31.321508864Z Wrote inspect data to must-gather.
2026-05-21T13:12:31.609692617Z Wrote inspect data to must-gather.
2026-05-21T13:12:31.893174629Z Wrote inspect data to must-gather.
2026-05-21T13:12:32.146119873Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-21T13:12:32.148266514Z INFO: Fallback to identify the container image from release info
2026-05-21T13:12:32.834511830Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-21T13:12:32.931513397Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:32.968142831Z daemonset.apps/perf-node-gather-daemonset created
2026-05-21T13:12:33.118880901Z Waiting for performance profile collector pods to become ready: 1
2026-05-21T13:12:33.485421346Z Gathering data for ns/openshift-network-console...
2026-05-21T13:12:34.108552744Z Gathering data for ns/openshift-console-operator...
2026-05-21T13:12:34.273993125Z Waiting for performance profile collector pods to become ready: 2
2026-05-21T13:12:34.583598145Z Gathering data for ns/openshift-console...
2026-05-21T13:12:34.896221148Z Wrote inspect data to must-gather.
2026-05-21T13:12:35.077779369Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-21T13:12:35.401414889Z Waiting for performance profile collector pods to become ready: 3
2026-05-21T13:12:35.473167674Z Gathering data for ns/openshift-dns-operator...
2026-05-21T13:12:35.843764517Z Gathering data for ns/openshift-dns...
2026-05-21T13:12:36.543369540Z Waiting for performance profile collector pods to become ready: 4
2026-05-21T13:12:36.601227865Z Gathering data for ns/openshift-image-registry...
2026-05-21T13:12:37.204158303Z Gathering data for ns/openshift-ingress-operator...
2026-05-21T13:12:37.616668785Z Gathering data for ns/openshift-ingress...
2026-05-21T13:12:37.663997980Z Waiting for performance profile collector pods to become ready: 5
2026-05-21T13:12:37.935991198Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:38.499671092Z Gathering data for ns/openshift-ingress-canary...
2026-05-21T13:12:38.771951364Z Waiting for performance profile collector pods to become ready: 6
2026-05-21T13:12:39.022677376Z Gathering data for ns/openshift-insights...
2026-05-21T13:12:39.896623804Z Waiting for performance profile collector pods to become ready: 7
2026-05-21T13:12:41.021870392Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-21T13:12:41.277261853Z Collecting performance related data for node ip-10-0-133-193.ec2.internal
2026-05-21T13:12:41.284149147Z Collecting performance related data for node ip-10-0-142-58.ec2.internal
2026-05-21T13:12:41.291366585Z Collecting performance related data for node ip-10-0-143-244.ec2.internal
2026-05-21T13:12:41.388191585Z Gathering data for ns/opendatahub...
2026-05-21T13:12:42.942313624Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:43.264774475Z Gathering data for ns/openshift-lws-operator...
2026-05-21T13:12:44.673613409Z Gathering data for ns/openshift-config...
2026-05-21T13:12:45.010218774Z Gathering data for ns/openshift-config-managed...
2026-05-21T13:12:45.563170462Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-21T13:12:45.922212356Z Gathering data for ns/openshift-kube-apiserver...
2026-05-21T13:12:46.286762423Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-21T13:12:46.659998817Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-21T13:12:47.053400866Z Gathering data for ns/openshift-kube-scheduler...
2026-05-21T13:12:47.427924697Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-21T13:12:47.883379520Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-21T13:12:47.948974876Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:48.300498000Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-21T13:12:48.679564723Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-21T13:12:49.367752308Z Gathering data for ns/openshift-multus...
2026-05-21T13:12:49.902842567Z Collecting kubelet logs for node ip-10-0-133-193.ec2.internal
2026-05-21T13:12:49.904483314Z Collecting kubelet logs for node ip-10-0-142-58.ec2.internal
2026-05-21T13:12:49.906172232Z Collecting kubelet logs for node ip-10-0-143-244.ec2.internal
2026-05-21T13:12:50.234341546Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-21T13:12:50.239766836Z INFO: Node performance data collection complete.
2026-05-21T13:12:51.262841807Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-21T13:12:52.958355155Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:53.055910721Z Gathering data for ns/openshift-host-network...
2026-05-21T13:12:53.576505716Z Gathering data for ns/openshift-network-diagnostics...
2026-05-21T13:12:54.333684934Z Gathering data for ns/openshift-network-node-identity...
2026-05-21T13:12:54.856983423Z Gathering data for ns/openshift-network-operator...
2026-05-21T13:12:55.329791334Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-21T13:12:55.687593929Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-21T13:12:56.285571207Z Gathering data for ns/openshift-apiserver-operator...
2026-05-21T13:12:56.641136713Z Gathering data for ns/openshift-apiserver...
2026-05-21T13:12:57.134818049Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-21T13:12:57.451469173Z Gathering data for ns/openshift-controller-manager...
2026-05-21T13:12:57.778996950Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-21T13:12:57.968172453Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-21T13:12:58.492045273Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-21T13:12:58.878764437Z Gathering data for ns/openshift-service-ca-operator...
2026-05-21T13:12:59.343993245Z Gathering data for ns/openshift-service-ca...
2026-05-21T13:12:59.878960811Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-21T13:13:01.307453170Z Wrote inspect data to must-gather.
2026-05-21T13:13:01.307500500Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T13:13:01.307500500Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-21T13:13:01.489939322Z error: the server doesn't have a resource type "clusters"
2026-05-21T13:13:01.704583480Z Caches written to disk