2026-05-28T10:59:49.693908618Z [disk usage checker] Started
2026-05-28T10:59:49.697778015Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T10:59:49.924664328Z Gathering data for ns/openshift-cluster-version...
2026-05-28T10:59:50.098983633Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T10:59:50.202309500Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T10:59:50.307078955Z Gathering data for ns/default...
2026-05-28T10:59:50.707600362Z Gathering data for ns/openshift...
2026-05-28T10:59:51.220770431Z Gathering data for ns/kube-system...
2026-05-28T10:59:52.039171472Z Gathering data for ns/openshift-etcd...
2026-05-28T10:59:52.266716288Z Waiting on subprocesses to finish execution.
2026-05-28T10:59:52.282537458Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T10:59:52.289533432Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T10:59:52.295632387Z INFO: Gathering HAProxy config files
2026-05-28T10:59:52.300325275Z INFO: Collecting host service logs for crio
2026-05-28T10:59:52.300969490Z INFO: Collecting host service logs for kubelet
2026-05-28T10:59:52.301343370Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T10:59:52.301675097Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T10:59:52.302023362Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T10:59:52.302344033Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T10:59:52.302673372Z INFO: Collecting host service logs for NetworkManager
2026-05-28T10:59:52.303010521Z INFO: Collecting host service logs for openvswitch
2026-05-28T10:59:52.303332975Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T10:59:52.303665409Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T10:59:52.304050869Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T10:59:52.304429118Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T10:59:52.346588672Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T10:59:52.573705430Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T10:59:53.020487887Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T10:59:53.056534335Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T10:59:53.065754260Z No resources found in openshift-etcd namespace.
2026-05-28T10:59:53.331001034Z Wrote inspect data to must-gather.
2026-05-28T10:59:53.337602376Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T10:59:53.338355081Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T10:59:53.400583423Z Wrote inspect data to must-gather.
2026-05-28T10:59:53.400627091Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T10:59:53.400627091Z     namespaces "assisted-installer" not found
2026-05-28T10:59:53.549209755Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T10:59:53.708451846Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T10:59:53.738600629Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T10:59:53.766654991Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-gfcfd
2026-05-28T10:59:53.766654991Z insights-runtime-extractor-42b49
2026-05-28T10:59:53.766654991Z insights-runtime-extractor-cdj8x
2026-05-28T10:59:53.766654991Z insights-runtime-extractor-grdb5
2026-05-28T10:59:53.793289805Z No resources found
2026-05-28T10:59:53.812810546Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T10:59:53.818897066Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T10:59:53.841863060Z INFO: Worker host service log collection to complete.
2026-05-28T10:59:53.946538970Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T10:59:54.185160960Z tar: Removing leading `/' from member names
2026-05-28T10:59:54.189831294Z INFO: HAProxy config collection complete.
2026-05-28T10:59:54.351533686Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T10:59:54.351533686Z See 'oc rsync -h' for help and examples
2026-05-28T10:59:54.412098308Z ERROR: No running kube-apiserver pods found
2026-05-28T10:59:54.650158960Z Wrote inspect data to must-gather.
2026-05-28T10:59:54.657300262Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T10:59:54.744117144Z Wrote inspect data to must-gather.
2026-05-28T10:59:54.759432651Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T10:59:55.092223618Z No resources found
2026-05-28T10:59:55.156654095Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T10:59:55.170946381Z Wrote inspect data to must-gather.
2026-05-28T10:59:55.196439451Z Wrote inspect data to must-gather.
2026-05-28T10:59:55.232004670Z INFO: 'previous-logs' folder not found on ip-10-0-137-205.ec2.internal, skipping...
2026-05-28T10:59:55.255935067Z Wrote inspect data to must-gather.
2026-05-28T10:59:55.329046708Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T10:59:55.436057527Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T10:59:55.577563771Z Wrote inspect data to must-gather.
2026-05-28T10:59:55.638381927Z INFO: 'previous-logs' folder not found on ip-10-0-137-232.ec2.internal, skipping...
2026-05-28T10:59:55.689347116Z Gathering data for ns/kuadrant-system...
2026-05-28T10:59:55.791593000Z Wrote inspect data to must-gather.
2026-05-28T10:59:55.900356475Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T10:59:55.960213979Z Wrote inspect data to must-gather.
2026-05-28T10:59:56.024488475Z INFO: 'previous-logs' folder not found on ip-10-0-141-178.ec2.internal, skipping...
2026-05-28T10:59:56.024525621Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T10:59:56.024538514Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T10:59:56.126382874Z Wrote inspect data to must-gather.
2026-05-28T10:59:56.364265160Z Wrote inspect data to must-gather.
2026-05-28T10:59:56.372158063Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T10:59:56.462343729Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T10:59:56.752482117Z Wrote inspect data to must-gather.
2026-05-28T10:59:56.771898638Z INFO: INTERCONNECT MODE
2026-05-28T10:59:56.772017572Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T10:59:56.865039541Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T10:59:57.012732144Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T10:59:57.049478222Z Wrote inspect data to must-gather.
2026-05-28T10:59:57.062314031Z INFO: Gathering OVN_Northbound from ovnkube-node-25trz...
2026-05-28T10:59:57.062626117Z INFO: Gathering OVN_Northbound from ovnkube-node-bbq6f...
2026-05-28T10:59:57.062888779Z INFO: Gathering OVN_Northbound from ovnkube-node-dmrl8...
2026-05-28T10:59:57.063397084Z INFO: Gathering OVN_Southbound from ovnkube-node-25trz...
2026-05-28T10:59:57.063683314Z INFO: Gathering OVN_Southbound from ovnkube-node-bbq6f...
2026-05-28T10:59:57.063939810Z INFO: Gathering OVN_Southbound from ovnkube-node-dmrl8...
2026-05-28T10:59:57.148295343Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T10:59:57.533395743Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.570685633Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.591199114Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.596359734Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.657664177Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T10:59:57.670250011Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.670359109Z tar: Removing leading `/' from member names
2026-05-28T10:59:57.680048829Z W0528 10:59:57.680000    1205 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T10:59:57.704619755Z Wrote inspect data to must-gather.
2026-05-28T10:59:57.829444961Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T10:59:58.108616386Z Gathering data for ns/openshift-monitoring...
2026-05-28T10:59:58.154186505Z Wrote inspect data to must-gather.
2026-05-28T10:59:58.219408171Z INFO: Getting status from alertmanager-main-0
2026-05-28T10:59:58.287194265Z INFO: Waiting for network log collection to complete ...
2026-05-28T10:59:58.287311044Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T10:59:58.287819768Z INFO: Copying ovnk databases complete.
2026-05-28T10:59:58.289987987Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-28T10:59:58.293236240Z ovnk_database_store/
2026-05-28T10:59:58.293236240Z ovnk_database_store/ovnkube-node-bbq6f_sbdb
2026-05-28T10:59:58.357238949Z ovnk_database_store/ovnkube-node-bbq6f_nbdb
2026-05-28T10:59:58.369620879Z ovnk_database_store/ovnkube-node-25trz_sbdb
2026-05-28T10:59:58.396178544Z ovnk_database_store/ovnkube-node-25trz_nbdb
2026-05-28T10:59:58.405890626Z ovnk_database_store/ovnkube-node-dmrl8_sbdb
2026-05-28T10:59:58.429117820Z ovnk_database_store/ovnkube-node-dmrl8_nbdb
2026-05-28T10:59:58.443837508Z INFO: Network log collection complete.
2026-05-28T10:59:58.541088396Z Wrote inspect data to must-gather.
2026-05-28T10:59:58.852097768Z Wrote inspect data to must-gather.
2026-05-28T10:59:59.117743005Z Wrote inspect data to must-gather.
2026-05-28T10:59:59.386724037Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T10:59:59.389199708Z INFO: Fallback to identify the container image from release info
2026-05-28T10:59:59.796468620Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T10:59:59.968725204Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T11:00:00.119415793Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T11:00:00.297027550Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T11:00:00.440061180Z Gathering data for ns/openshift-network-console...
2026-05-28T11:00:01.031231311Z Gathering data for ns/openshift-console-operator...
2026-05-28T11:00:01.434271411Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T11:00:01.453838999Z Wrote inspect data to must-gather.
2026-05-28T11:00:01.562268246Z Gathering data for ns/openshift-console...
2026-05-28T11:00:02.061784326Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T11:00:02.484223258Z Gathering data for ns/openshift-dns-operator...
2026-05-28T11:00:02.582580802Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T11:00:02.841864967Z Gathering data for ns/openshift-dns...
2026-05-28T11:00:03.529781181Z Gathering data for ns/openshift-image-registry...
2026-05-28T11:00:03.710822670Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T11:00:04.095310397Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T11:00:04.506313701Z Gathering data for ns/openshift-ingress...
2026-05-28T11:00:04.802914455Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T11:00:04.832426461Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T11:00:05.470749871Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T11:00:05.983346364Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T11:00:06.010934290Z Gathering data for ns/openshift-insights...
2026-05-28T11:00:07.098739446Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T11:00:08.221524382Z Gathering data for ns/opendatahub...
2026-05-28T11:00:08.224158507Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T11:00:08.466177867Z Collecting performance related data for node ip-10-0-137-205.ec2.internal
2026-05-28T11:00:08.473805379Z Collecting performance related data for node ip-10-0-141-178.ec2.internal
2026-05-28T11:00:08.481936955Z Collecting performance related data for node ip-10-0-137-232.ec2.internal
2026-05-28T11:00:09.810895458Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T11:00:10.199579196Z Gathering data for ns/openshift-lws-operator...
2026-05-28T11:00:11.645382007Z Gathering data for ns/openshift-config...
2026-05-28T11:00:11.991413973Z Gathering data for ns/openshift-config-managed...
2026-05-28T11:00:12.488212593Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T11:00:12.840933860Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T11:00:13.206738283Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T11:00:13.685668877Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T11:00:14.070092184Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T11:00:14.402259717Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T11:00:14.790203012Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T11:00:14.815784747Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T11:00:15.203985128Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T11:00:15.614501406Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T11:00:16.253076292Z Gathering data for ns/openshift-multus...
2026-05-28T11:00:17.143557804Z Collecting kubelet logs for node ip-10-0-137-205.ec2.internal
2026-05-28T11:00:17.145397707Z Collecting kubelet logs for node ip-10-0-141-178.ec2.internal
2026-05-28T11:00:17.147087361Z Collecting kubelet logs for node ip-10-0-137-232.ec2.internal
2026-05-28T11:00:17.507821900Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T11:00:17.512559265Z INFO: Node performance data collection complete.
2026-05-28T11:00:18.177662095Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T11:00:19.820682813Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T11:00:19.952699632Z Gathering data for ns/openshift-host-network...
2026-05-28T11:00:20.369613760Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T11:00:21.008199357Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T11:00:21.518598303Z Gathering data for ns/openshift-network-operator...
2026-05-28T11:00:21.992425365Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T11:00:22.333817086Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T11:00:22.847544159Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T11:00:23.211713842Z Gathering data for ns/openshift-apiserver...
2026-05-28T11:00:23.707686268Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T11:00:24.012616810Z Gathering data for ns/openshift-controller-manager...
2026-05-28T11:00:24.364776376Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T11:00:24.825671332Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T11:00:25.084980685Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T11:00:25.413206692Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T11:00:25.784046161Z Gathering data for ns/openshift-service-ca...
2026-05-28T11:00:26.202855747Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T11:00:27.264635099Z Wrote inspect data to must-gather.
2026-05-28T11:00:27.264688196Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T11:00:27.264688196Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T11:00:27.404273157Z error: the server doesn't have a resource type "clusters"
2026-05-28T11:00:27.613068749Z Caches written to disk