2026-04-21T02:55:53.423285029Z [disk usage checker] Started
2026-04-21T02:55:53.460104596Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:55:53.662313043Z Gathering data for ns/openshift-cluster-version...
2026-04-21T02:55:53.820619207Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T02:55:53.918642360Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:55:54.026250222Z Gathering data for ns/default...
2026-04-21T02:55:54.372139806Z Gathering data for ns/openshift...
2026-04-21T02:55:54.849082026Z Gathering data for ns/kube-system...
2026-04-21T02:55:55.563641840Z Gathering data for ns/openshift-etcd...
2026-04-21T02:55:55.921648612Z Waiting on subprocesses to finish execution.
2026-04-21T02:55:55.924956328Z Wrote inspect data to must-gather.
2026-04-21T02:55:55.925039789Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T02:55:55.925039789Z     namespaces "assisted-installer" not found
2026-04-21T02:55:55.925180311Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T02:55:55.927454804Z INFO: Gathering HAProxy config files
2026-04-21T02:55:55.933116698Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T02:55:55.941430338Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T02:55:55.962889253Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T02:55:55.964660853Z INFO: Collecting host service logs for crio
2026-04-21T02:55:55.965002912Z INFO: Collecting host service logs for kubelet
2026-04-21T02:55:55.965335636Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T02:55:55.965642362Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T02:55:55.965929713Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T02:55:55.966252133Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T02:55:55.966576956Z INFO: Collecting host service logs for NetworkManager
2026-04-21T02:55:55.966872133Z INFO: Collecting host service logs for openvswitch
2026-04-21T02:55:55.967176734Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T02:55:55.967478945Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T02:55:55.967789491Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T02:55:55.968152290Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T02:55:56.394576473Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T02:55:56.433800685Z No resources found in openshift-etcd namespace.
2026-04-21T02:55:56.706120042Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T02:55:56.865928555Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-8rwzw
2026-04-21T02:55:56.866226776Z insights-runtime-extractor-2652q
2026-04-21T02:55:56.866254039Z insights-runtime-extractor-9khlr
2026-04-21T02:55:56.866274816Z insights-runtime-extractor-z6lqg
2026-04-21T02:55:56.882662559Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T02:55:56.882731081Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T02:55:57.011783824Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T02:55:57.050770218Z No resources found
2026-04-21T02:55:57.152995751Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T02:55:57.153317689Z Wrote inspect data to must-gather.
2026-04-21T02:55:57.339766668Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T02:55:57.339766668Z See 'oc rsync -h' for help and examples
2026-04-21T02:55:57.382724723Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T02:55:57.498878850Z ERROR: No running kube-apiserver pods found
2026-04-21T02:55:57.595012147Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T02:55:57.611187362Z INFO: Worker host service log collection to complete.
2026-04-21T02:55:58.155772506Z Wrote inspect data to must-gather.
2026-04-21T02:55:58.173246439Z No resources found
2026-04-21T02:55:58.316139048Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T02:55:58.474531345Z INFO: 'previous-logs' folder not found on ip-10-0-131-170.ec2.internal, skipping...
2026-04-21T02:55:58.475746763Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:55:58.523217938Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T02:55:58.549153880Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T02:55:58.563829886Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:55:58.594791830Z Wrote inspect data to must-gather.
2026-04-21T02:55:58.727691931Z Wrote inspect data to must-gather.
2026-04-21T02:55:58.731241561Z Wrote inspect data to must-gather.
2026-04-21T02:55:58.740428444Z Wrote inspect data to must-gather.
2026-04-21T02:55:58.795235120Z Gathering data for ns/kuadrant-system...
2026-04-21T02:55:58.944705034Z tar: Removing leading `/' from member names
2026-04-21T02:55:58.951634931Z INFO: HAProxy config collection complete.
2026-04-21T02:55:58.963238883Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T02:55:59.009362787Z INFO: 'previous-logs' folder not found on ip-10-0-134-66.ec2.internal, skipping...
2026-04-21T02:55:59.118796942Z Wrote inspect data to must-gather.
2026-04-21T02:55:59.175478846Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T02:55:59.270571909Z Wrote inspect data to must-gather.
2026-04-21T02:55:59.383339025Z INFO: 'previous-logs' folder not found on ip-10-0-137-147.ec2.internal, skipping...
2026-04-21T02:55:59.383440241Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T02:55:59.383488681Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T02:55:59.487906341Z Wrote inspect data to must-gather.
2026-04-21T02:55:59.602526889Z Wrote inspect data to must-gather.
2026-04-21T02:55:59.736773083Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T02:55:59.815567605Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T02:55:59.822046673Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T02:55:59.852086505Z Wrote inspect data to must-gather.
2026-04-21T02:56:00.104097647Z Wrote inspect data to must-gather.
2026-04-21T02:56:00.254523508Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T02:56:00.258804151Z INFO: INTERCONNECT MODE
2026-04-21T02:56:00.258907596Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T02:56:00.437768565Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:56:00.473136862Z INFO: Gathering OVN_Northbound from ovnkube-node-ckmv8...
2026-04-21T02:56:00.473475124Z INFO: Gathering OVN_Northbound from ovnkube-node-cpdzb...
2026-04-21T02:56:00.473771972Z INFO: Gathering OVN_Northbound from ovnkube-node-p8b6t...
2026-04-21T02:56:00.474103809Z INFO: Gathering OVN_Southbound from ovnkube-node-ckmv8...
2026-04-21T02:56:00.474390258Z INFO: Gathering OVN_Southbound from ovnkube-node-cpdzb...
2026-04-21T02:56:00.474721712Z INFO: Gathering OVN_Southbound from ovnkube-node-p8b6t...
2026-04-21T02:56:00.649260128Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T02:56:00.653944256Z Wrote inspect data to must-gather.
2026-04-21T02:56:00.798764963Z Gathering data for ns/openshift-monitoring...
2026-04-21T02:56:00.813452545Z tar: Removing leading `/' from member names
2026-04-21T02:56:00.845480293Z tar: Removing leading `/' from member names
2026-04-21T02:56:00.897480978Z tar: Removing leading `/' from member names
2026-04-21T02:56:00.930042221Z tar: Removing leading `/' from member names
2026-04-21T02:56:00.939003105Z tar: Removing leading `/' from member names
2026-04-21T02:56:00.990019385Z tar: Removing leading `/' from member names
2026-04-21T02:56:01.050677526Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T02:56:01.217422307Z Wrote inspect data to must-gather.
2026-04-21T02:56:01.351212588Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T02:56:01.571646998Z INFO: Waiting for network log collection to complete ...
2026-04-21T02:56:01.571685197Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T02:56:01.572168333Z INFO: Copying ovnk databases complete.
2026-04-21T02:56:01.573836404Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-21T02:56:01.576025983Z ovnk_database_store/
2026-04-21T02:56:01.576025983Z ovnk_database_store/ovnkube-node-ckmv8_sbdb
2026-04-21T02:56:01.590790369Z Wrote inspect data to must-gather.
2026-04-21T02:56:01.599022326Z ovnk_database_store/ovnkube-node-cpdzb_nbdb
2026-04-21T02:56:01.610080833Z ovnk_database_store/ovnkube-node-p8b6t_sbdb
2026-04-21T02:56:01.643828973Z ovnk_database_store/ovnkube-node-ckmv8_nbdb
2026-04-21T02:56:01.656646492Z ovnk_database_store/ovnkube-node-p8b6t_nbdb
2026-04-21T02:56:01.667893199Z ovnk_database_store/ovnkube-node-cpdzb_sbdb
2026-04-21T02:56:01.696786080Z INFO: Network log collection complete.
2026-04-21T02:56:01.833661081Z INFO: Getting status from alertmanager-main-0
2026-04-21T02:56:01.930483033Z Wrote inspect data to must-gather.
2026-04-21T02:56:02.253671003Z Wrote inspect data to must-gather.
2026-04-21T02:56:02.553187677Z Wrote inspect data to must-gather.
2026-04-21T02:56:02.804734521Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T02:56:02.807681205Z INFO: Fallback to identify the container image from release info
2026-04-21T02:56:03.227933508Z Gathering data for ns/openshift-network-console...
2026-04-21T02:56:03.404248547Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T02:56:03.483138299Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:56:03.541091154Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T02:56:03.696800766Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T02:56:03.793890143Z Gathering data for ns/openshift-console-operator...
2026-04-21T02:56:04.266402043Z Gathering data for ns/openshift-console...
2026-04-21T02:56:04.813819650Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T02:56:04.856335502Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T02:56:05.230760440Z Gathering data for ns/openshift-dns-operator...
2026-04-21T02:56:05.317368358Z Wrote inspect data to must-gather.
2026-04-21T02:56:05.555464471Z Gathering data for ns/openshift-dns...
2026-04-21T02:56:05.968324900Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T02:56:06.171704596Z Gathering data for ns/openshift-image-registry...
2026-04-21T02:56:06.651619797Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T02:56:07.075103389Z Gathering data for ns/openshift-ingress...
2026-04-21T02:56:07.095583009Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T02:56:07.710893067Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T02:56:08.219564721Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T02:56:08.249582653Z Gathering data for ns/openshift-insights...
2026-04-21T02:56:08.488773782Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:56:09.325643675Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T02:56:10.254790789Z Gathering data for ns/opendatahub...
2026-04-21T02:56:10.444674921Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T02:56:11.551430440Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T02:56:11.684380250Z Gathering data for ns/openshift-lws-operator...
2026-04-21T02:56:11.804853324Z Collecting performance related data for node ip-10-0-137-147.ec2.internal
2026-04-21T02:56:11.814577262Z Collecting performance related data for node ip-10-0-134-66.ec2.internal
2026-04-21T02:56:11.823676336Z Collecting performance related data for node ip-10-0-131-170.ec2.internal
2026-04-21T02:56:13.172854418Z Gathering data for ns/openshift-config...
2026-04-21T02:56:13.493863223Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:56:13.532543189Z Gathering data for ns/openshift-config-managed...
2026-04-21T02:56:14.016818402Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T02:56:14.393780905Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T02:56:14.809371945Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T02:56:15.182783734Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T02:56:15.521765983Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T02:56:15.849414437Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T02:56:16.194590063Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T02:56:16.585925981Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T02:56:16.949302495Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T02:56:17.563758063Z Gathering data for ns/openshift-multus...
2026-04-21T02:56:18.501642224Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:56:19.073548511Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T02:56:20.340463634Z Collecting kubelet logs for node ip-10-0-137-147.ec2.internal
2026-04-21T02:56:20.342133757Z Collecting kubelet logs for node ip-10-0-134-66.ec2.internal
2026-04-21T02:56:20.343886029Z Collecting kubelet logs for node ip-10-0-131-170.ec2.internal
2026-04-21T02:56:20.604995038Z Gathering data for ns/openshift-host-network...
2026-04-21T02:56:20.746885098Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T02:56:20.751604813Z INFO: Node performance data collection complete.
2026-04-21T02:56:20.980837634Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T02:56:21.551641529Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T02:56:22.027281438Z Gathering data for ns/openshift-network-operator...
2026-04-21T02:56:22.429335198Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T02:56:22.734525632Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T02:56:23.213614567Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T02:56:23.506086738Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-21T02:56:23.535064270Z Gathering data for ns/openshift-apiserver...
2026-04-21T02:56:23.986633629Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T02:56:24.288694918Z Gathering data for ns/openshift-controller-manager...
2026-04-21T02:56:24.616793862Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T02:56:25.283448987Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T02:56:25.612446317Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T02:56:25.947478558Z Gathering data for ns/openshift-service-ca...
2026-04-21T02:56:26.344643380Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T02:56:27.368343810Z Wrote inspect data to must-gather.
2026-04-21T02:56:27.368387014Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T02:56:27.368387014Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T02:56:27.506576694Z error: the server doesn't have a resource type "clusters"
2026-04-21T02:56:27.676517141Z Caches written to disk