2026-04-21T04:18:28.524740860Z [disk usage checker] Started
2026-04-21T04:18:28.587491163Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:28.738705975Z Gathering data for ns/openshift-cluster-version...
2026-04-21T04:18:28.872238847Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T04:18:28.957063785Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:18:29.043910247Z Gathering data for ns/default...
2026-04-21T04:18:29.354812666Z Gathering data for ns/openshift...
2026-04-21T04:18:29.787398102Z Gathering data for ns/kube-system...
2026-04-21T04:18:30.410611927Z Gathering data for ns/openshift-etcd...
2026-04-21T04:18:30.738241314Z Wrote inspect data to must-gather.
2026-04-21T04:18:30.738305817Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T04:18:30.738305817Z     namespaces "assisted-installer" not found
2026-04-21T04:18:31.068856083Z Waiting on subprocesses to finish execution.
2026-04-21T04:18:31.076797438Z INFO: Gathering HAProxy config files
2026-04-21T04:18:31.086263829Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T04:18:31.100766215Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T04:18:31.103984816Z INFO: Collecting host service logs for crio
2026-04-21T04:18:31.104346205Z INFO: Collecting host service logs for kubelet
2026-04-21T04:18:31.104664187Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T04:18:31.104985523Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T04:18:31.105378895Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T04:18:31.105611950Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T04:18:31.105908476Z INFO: Collecting host service logs for NetworkManager
2026-04-21T04:18:31.106205677Z INFO: Collecting host service logs for openvswitch
2026-04-21T04:18:31.106541449Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T04:18:31.106832167Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T04:18:31.107120309Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T04:18:31.107581189Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T04:18:31.114891323Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T04:18:31.161199553Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T04:18:31.788365538Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T04:18:31.954351096Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T04:18:32.020569641Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T04:18:32.031671402Z No resources found in openshift-etcd namespace.
2026-04-21T04:18:32.164557406Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T04:18:32.164639969Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T04:18:32.230541591Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-657t5
2026-04-21T04:18:32.230591971Z insights-runtime-extractor-7nh4z
2026-04-21T04:18:32.230612188Z insights-runtime-extractor-dhkjb
2026-04-21T04:18:32.230632010Z insights-runtime-extractor-qsn72
2026-04-21T04:18:32.265055536Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T04:18:32.633524465Z No resources found
2026-04-21T04:18:32.765459211Z INFO: Worker host service log collection to complete.
2026-04-21T04:18:32.807066151Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T04:18:33.104674102Z ERROR: No running kube-apiserver pods found
2026-04-21T04:18:33.189098724Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T04:18:33.216025857Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T04:18:33.258518800Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T04:18:33.328848164Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:18:33.330029319Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.331611970Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T04:18:33.335731470Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T04:18:33.335731470Z See 'oc rsync -h' for help and examples
2026-04-21T04:18:33.475498135Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.607762718Z No resources found
2026-04-21T04:18:33.632444281Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:33.635197065Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.660133913Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.793077300Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.844698958Z INFO: HAProxy config collection complete.
2026-04-21T04:18:33.844726381Z INFO: 'previous-logs' folder not found on ip-10-0-131-93.ec2.internal, skipping...
2026-04-21T04:18:33.911870309Z Wrote inspect data to must-gather.
2026-04-21T04:18:33.924952532Z Gathering data for ns/kuadrant-system...
2026-04-21T04:18:34.068211447Z Wrote inspect data to must-gather.
2026-04-21T04:18:34.163089612Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T04:18:34.205778845Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T04:18:34.232708382Z INFO: 'previous-logs' folder not found on ip-10-0-134-136.ec2.internal, skipping...
2026-04-21T04:18:34.437135388Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T04:18:34.470931503Z Wrote inspect data to must-gather.
2026-04-21T04:18:34.524838797Z Wrote inspect data to must-gather.
2026-04-21T04:18:34.592601937Z INFO: 'previous-logs' folder not found on ip-10-0-138-120.ec2.internal, skipping...
2026-04-21T04:18:34.592741836Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T04:18:34.592810239Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T04:18:34.733464633Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:18:34.786721965Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T04:18:34.879308371Z Wrote inspect data to must-gather.
2026-04-21T04:18:34.881541770Z Wrote inspect data to must-gather.
2026-04-21T04:18:34.939316017Z Gathering data for ns/openshift-monitoring...
2026-04-21T04:18:35.095268461Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T04:18:35.279837584Z INFO: INTERCONNECT MODE
2026-04-21T04:18:35.279867667Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T04:18:35.308206110Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T04:18:35.402177904Z Wrote inspect data to must-gather.
2026-04-21T04:18:35.483432171Z INFO: Gathering OVN_Northbound from ovnkube-node-6p6fc...
2026-04-21T04:18:35.483646206Z INFO: Gathering OVN_Northbound from ovnkube-node-82wml...
2026-04-21T04:18:35.483856161Z INFO: Gathering OVN_Northbound from ovnkube-node-bknrd...
2026-04-21T04:18:35.484123077Z INFO: Gathering OVN_Southbound from ovnkube-node-6p6fc...
2026-04-21T04:18:35.484381105Z INFO: Gathering OVN_Southbound from ovnkube-node-82wml...
2026-04-21T04:18:35.484681747Z INFO: Gathering OVN_Southbound from ovnkube-node-bknrd...
2026-04-21T04:18:35.602231494Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T04:18:35.769457327Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.782239934Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.803501837Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.804974935Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.867021997Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.896898404Z tar: Removing leading `/' from member names
2026-04-21T04:18:35.946763247Z Wrote inspect data to must-gather.
2026-04-21T04:18:35.989807968Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T04:18:36.280150176Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T04:18:36.322546017Z Wrote inspect data to must-gather.
2026-04-21T04:18:36.532409546Z INFO: Waiting for network log collection to complete ...
2026-04-21T04:18:36.532524857Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T04:18:36.533019888Z INFO: Copying ovnk databases complete.
2026-04-21T04:18:36.535291462Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-04-21T04:18:36.538472953Z ovnk_database_store/
2026-04-21T04:18:36.538518651Z ovnk_database_store/ovnkube-node-82wml_nbdb
2026-04-21T04:18:36.551577363Z ovnk_database_store/ovnkube-node-bknrd_sbdb
2026-04-21T04:18:36.570882530Z ovnk_database_store/ovnkube-node-82wml_sbdb
2026-04-21T04:18:36.595905298Z ovnk_database_store/ovnkube-node-6p6fc_sbdb
2026-04-21T04:18:36.615206503Z ovnk_database_store/ovnkube-node-bknrd_nbdb
2026-04-21T04:18:36.622438462Z ovnk_database_store/ovnkube-node-6p6fc_nbdb
2026-04-21T04:18:36.634270241Z Wrote inspect data to must-gather.
2026-04-21T04:18:36.634620982Z INFO: Network log collection complete.
2026-04-21T04:18:36.742883157Z INFO: Getting status from alertmanager-main-0
2026-04-21T04:18:36.918579490Z Wrote inspect data to must-gather.
2026-04-21T04:18:37.254955359Z Gathering data for ns/openshift-network-console...
2026-04-21T04:18:37.265685053Z Wrote inspect data to must-gather.
2026-04-21T04:18:37.633838152Z Wrote inspect data to must-gather.
2026-04-21T04:18:37.740690216Z Gathering data for ns/openshift-console-operator...
2026-04-21T04:18:37.949150783Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T04:18:37.951609170Z INFO: Fallback to identify the container image from release info
2026-04-21T04:18:38.192326032Z Gathering data for ns/openshift-console...
2026-04-21T04:18:38.407628196Z Wrote inspect data to must-gather.
2026-04-21T04:18:38.546480816Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T04:18:38.637819336Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:38.663618442Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T04:18:38.686799251Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T04:18:38.836598009Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T04:18:39.048332398Z Gathering data for ns/openshift-dns-operator...
2026-04-21T04:18:39.368580741Z Gathering data for ns/openshift-dns...
2026-04-21T04:18:39.968773907Z Gathering data for ns/openshift-image-registry...
2026-04-21T04:18:39.980170702Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T04:18:40.421359363Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T04:18:40.792664339Z Gathering data for ns/openshift-ingress...
2026-04-21T04:18:41.109096696Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T04:18:41.284097978Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T04:18:41.700088066Z Gathering data for ns/openshift-insights...
2026-04-21T04:18:42.240238588Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T04:18:43.362032929Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T04:18:43.642455303Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:43.960029833Z Gathering data for ns/openshift-lws-operator...
2026-04-21T04:18:44.498955349Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T04:18:44.904306295Z Gathering data for ns/openshift-config...
2026-04-21T04:18:45.175798147Z Gathering data for ns/openshift-config-managed...
2026-04-21T04:18:45.616456889Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T04:18:45.648636125Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T04:18:45.917729549Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T04:18:46.212947589Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T04:18:46.497074568Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T04:18:46.788236487Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T04:18:46.809727065Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T04:18:47.067260138Z Collecting performance related data for node ip-10-0-138-120.ec2.internal
2026-04-21T04:18:47.076700058Z Collecting performance related data for node ip-10-0-134-136.ec2.internal
2026-04-21T04:18:47.084495452Z Collecting performance related data for node ip-10-0-131-93.ec2.internal
2026-04-21T04:18:47.143429860Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T04:18:47.536457877Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T04:18:47.964954382Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T04:18:48.370187662Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T04:18:48.651883491Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:49.068716666Z Gathering data for ns/openshift-multus...
2026-04-21T04:18:50.525985924Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T04:18:52.126964281Z Gathering data for ns/openshift-host-network...
2026-04-21T04:18:52.466824390Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T04:18:53.024650591Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T04:18:53.496885617Z Gathering data for ns/openshift-network-operator...
2026-04-21T04:18:53.657865390Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:53.897260187Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T04:18:54.216571794Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T04:18:54.690326398Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T04:18:54.998422162Z Gathering data for ns/openshift-apiserver...
2026-04-21T04:18:55.452315271Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T04:18:55.754865148Z Gathering data for ns/openshift-controller-manager...
2026-04-21T04:18:55.935846830Z Collecting kubelet logs for node ip-10-0-138-120.ec2.internal
2026-04-21T04:18:55.938097946Z Collecting kubelet logs for node ip-10-0-134-136.ec2.internal
2026-04-21T04:18:55.940421281Z Collecting kubelet logs for node ip-10-0-131-93.ec2.internal
2026-04-21T04:18:56.129831009Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T04:18:56.359710363Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T04:18:56.364081137Z INFO: Node performance data collection complete.
2026-04-21T04:18:56.854120346Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T04:18:57.184664892Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T04:18:57.501431732Z Gathering data for ns/openshift-service-ca...
2026-04-21T04:18:57.879427154Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T04:18:58.662380963Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T04:18:58.783962590Z Wrote inspect data to must-gather.
2026-04-21T04:18:58.784010022Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T04:18:58.784010022Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T04:18:58.922657400Z error: the server doesn't have a resource type "clusters"
2026-04-21T04:18:59.083653799Z Caches written to disk