2026-07-07T14:34:40.592624017Z [disk usage checker] Started
2026-07-07T14:34:40.596474570Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:34:40.791176129Z Gathering data for ns/openshift-cluster-version...
2026-07-07T14:34:40.944711508Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:34:41.059537084Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:34:41.158537446Z Gathering data for ns/default...
2026-07-07T14:34:41.521349917Z Gathering data for ns/openshift...
2026-07-07T14:34:42.014176172Z Gathering data for ns/kube-system...
2026-07-07T14:34:42.850972570Z Gathering data for ns/openshift-etcd...
2026-07-07T14:34:43.251086528Z Wrote inspect data to must-gather.
2026-07-07T14:34:43.251277533Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:34:43.251277533Z     namespaces "assisted-installer" not found
2026-07-07T14:34:43.515528133Z Waiting on subprocesses to finish execution.
2026-07-07T14:34:43.523248227Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T14:34:43.524810173Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T14:34:43.535539033Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T14:34:43.538090695Z Executing Istio gather script
2026-07-07T14:34:43.540128477Z INFO: Gathering HAProxy config files
2026-07-07T14:34:43.560021421Z INFO: Collecting host service logs for crio
2026-07-07T14:34:43.560494386Z INFO: Collecting host service logs for kubelet
2026-07-07T14:34:43.560928923Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T14:34:43.561332458Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T14:34:43.561734097Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T14:34:43.562099709Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T14:34:43.562448348Z INFO: Collecting host service logs for NetworkManager
2026-07-07T14:34:43.562736705Z INFO: Collecting host service logs for openvswitch
2026-07-07T14:34:43.563048623Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T14:34:43.563364651Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T14:34:43.563673333Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T14:34:43.564033773Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T14:34:43.602898379Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T14:34:44.148016682Z No resources found in openshift-etcd namespace.
2026-07-07T14:34:44.164068877Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-pjmct
2026-07-07T14:34:44.164124064Z insights-runtime-extractor-7ln9c
2026-07-07T14:34:44.164147705Z insights-runtime-extractor-f4hzs
2026-07-07T14:34:44.164169947Z insights-runtime-extractor-rtgw8
2026-07-07T14:34:44.326280980Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T14:34:44.413407231Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T14:34:44.510063883Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T14:34:44.647863272Z No resources found
2026-07-07T14:34:44.702982608Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T14:34:44.769493014Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T14:34:44.852127169Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T14:34:44.852164622Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T14:34:45.063314831Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T14:34:45.064968113Z Wrote inspect data to must-gather.
2026-07-07T14:34:45.215057144Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T14:34:45.215057144Z See 'oc rsync -h' for help and examples
2026-07-07T14:34:45.385408310Z Wrote inspect data to must-gather.
2026-07-07T14:34:45.543639766Z INFO: Worker host service log collection to complete.
2026-07-07T14:34:45.549630330Z Wrote inspect data to must-gather.
2026-07-07T14:34:45.567447324Z ERROR: No running kube-apiserver pods found
2026-07-07T14:34:45.598508686Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T14:34:45.655057489Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:34:45.656560487Z Wrote inspect data to must-gather.
2026-07-07T14:34:45.765457561Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:34:45.847672819Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T14:34:45.956758503Z Wrote inspect data to must-gather.
2026-07-07T14:34:46.014793307Z No resources found
2026-07-07T14:34:46.028896744Z Wrote inspect data to must-gather.
2026-07-07T14:34:46.084118323Z INFO: 'previous-logs' folder not found on ip-10-0-139-136.ec2.internal, skipping...
2026-07-07T14:34:46.190405265Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T14:34:46.223310934Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T14:34:46.280878736Z Wrote inspect data to must-gather.
2026-07-07T14:34:46.419621823Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T14:34:46.464683440Z Gathering data for ns/openshift-monitoring...
2026-07-07T14:34:46.525567467Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T14:34:46.542908903Z INFO: HAProxy config collection complete.
2026-07-07T14:34:46.553361655Z INFO: 'previous-logs' folder not found on ip-10-0-140-100.ec2.internal, skipping...
2026-07-07T14:34:46.659153381Z Wrote inspect data to must-gather.
2026-07-07T14:34:46.730308162Z Wrote inspect data to must-gather.
2026-07-07T14:34:46.851949581Z INFO: 'previous-logs' folder not found on ip-10-0-142-186.ec2.internal, skipping...
2026-07-07T14:34:46.852050505Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T14:34:46.852104839Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T14:34:46.947651249Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T14:34:46.971066157Z Wrote inspect data to must-gather.
2026-07-07T14:34:47.032512970Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T14:34:47.124450759Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T14:34:47.212061768Z Wrote inspect data to must-gather.
2026-07-07T14:34:47.271097943Z Wrote inspect data to must-gather.
2026-07-07T14:34:47.449340150Z Wrote inspect data to must-gather.
2026-07-07T14:34:47.596149519Z INFO: INTERCONNECT MODE
2026-07-07T14:34:47.596184130Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T14:34:47.638962427Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T14:34:47.698662629Z Wrote inspect data to must-gather.
2026-07-07T14:34:47.781675647Z INFO: Gathering OVN_Northbound from ovnkube-node-9dnd7...
2026-07-07T14:34:47.781966509Z INFO: Gathering OVN_Northbound from ovnkube-node-njsw2...
2026-07-07T14:34:47.782298393Z INFO: Gathering OVN_Northbound from ovnkube-node-wbqsv...
2026-07-07T14:34:47.782576976Z INFO: Gathering OVN_Southbound from ovnkube-node-9dnd7...
2026-07-07T14:34:47.782837734Z INFO: Gathering OVN_Southbound from ovnkube-node-njsw2...
2026-07-07T14:34:47.783136141Z INFO: Gathering OVN_Southbound from ovnkube-node-wbqsv...
2026-07-07T14:34:47.907628443Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T14:34:48.028454799Z Wrote inspect data to must-gather.
2026-07-07T14:34:48.066596684Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.073785809Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.099119203Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.118838337Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.119466942Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.138752084Z tar: Removing leading `/' from member names
2026-07-07T14:34:48.214249410Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T14:34:48.261306703Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T14:34:48.295596524Z Wrote inspect data to must-gather.
2026-07-07T14:34:48.483085046Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:48.582754370Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:34:48.583711939Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T14:34:48.585050357Z Wrote inspect data to must-gather.
2026-07-07T14:34:48.699257068Z INFO: Waiting for network log collection to complete ...
2026-07-07T14:34:48.699377578Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T14:34:48.699884164Z INFO: Copying ovnk databases complete.
2026-07-07T14:34:48.701634822Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-07T14:34:48.704163993Z ovnk_database_store/
2026-07-07T14:34:48.704163993Z ovnk_database_store/ovnkube-node-njsw2_sbdb
2026-07-07T14:34:48.727006597Z ovnk_database_store/ovnkube-node-9dnd7_sbdb
2026-07-07T14:34:48.743417417Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:48.746973996Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T14:34:48.748873571Z ovnk_database_store/ovnkube-node-njsw2_nbdb
2026-07-07T14:34:48.757536868Z ovnk_database_store/ovnkube-node-wbqsv_sbdb
2026-07-07T14:34:48.778752294Z ovnk_database_store/ovnkube-node-9dnd7_nbdb
2026-07-07T14:34:48.789313863Z ovnk_database_store/ovnkube-node-wbqsv_nbdb
2026-07-07T14:34:48.803016024Z INFO: Network log collection complete.
2026-07-07T14:34:48.833755504Z Wrote inspect data to must-gather.
2026-07-07T14:34:48.983763771Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:49.027566013Z INFO: Getting status from alertmanager-main-0
2026-07-07T14:34:49.141138661Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T14:34:49.143054574Z INFO: Fallback to identify the container image from release info
2026-07-07T14:34:49.235043127Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:49.237475072Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T14:34:49.238657674Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:34:49.413542963Z Gathering data for ns/openshift-network-console...
2026-07-07T14:34:49.447348084Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:49.653555933Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:49.657892508Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T14:34:49.791914280Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T14:34:49.910474483Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:49.934080728Z Gathering data for ns/openshift-console-operator...
2026-07-07T14:34:49.943548649Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T14:34:50.104430218Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T14:34:50.148737912Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:50.151299067Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T14:34:50.318995590Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:50.535377439Z Gathering data for ns/openshift-console...
2026-07-07T14:34:50.561586609Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:50.564421075Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T14:34:50.661073506Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:34:50.750013362Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:50.957054697Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:50.960053596Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T14:34:51.108005620Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T14:34:51.152009440Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:51.259317703Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T14:34:51.364877579Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:51.368092754Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T14:34:51.556343198Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:51.731306455Z Gathering data for ns/openshift-dns-operator...
2026-07-07T14:34:51.774122428Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:51.778307862Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T14:34:51.953974377Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:52.167817188Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:52.170164548Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T14:34:52.171443485Z Gathering data for ns/openshift-dns...
2026-07-07T14:34:52.340944039Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:52.411447021Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T14:34:52.554551421Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:34:52.972359995Z Gathering data for ns/openshift-image-registry...
2026-07-07T14:34:53.295805708Z Done executing Istio gather script
2026-07-07T14:34:53.549532313Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T14:34:53.555410578Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T14:34:53.957294811Z Gathering data for ns/openshift-ingress...
2026-07-07T14:34:54.378999804Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T14:34:54.671657729Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T14:34:54.866162890Z Gathering data for ns/openshift-insights...
2026-07-07T14:34:55.665663001Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:34:55.792388746Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T14:34:56.914505698Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T14:34:57.265157929Z Gathering data for ns/kserve...
2026-07-07T14:34:58.048816069Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T14:34:58.317051770Z Collecting performance related data for node ip-10-0-142-186.ec2.internal
2026-07-07T14:34:58.324276282Z Collecting performance related data for node ip-10-0-139-136.ec2.internal
2026-07-07T14:34:58.332602876Z Collecting performance related data for node ip-10-0-140-100.ec2.internal
2026-07-07T14:34:59.036038408Z Gathering data for ns/openshift-config...
2026-07-07T14:34:59.405206033Z Gathering data for ns/openshift-config-managed...
2026-07-07T14:34:59.999925960Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T14:35:00.416586183Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T14:35:00.670385141Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:35:00.736409240Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T14:35:01.125716825Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T14:35:01.519168971Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T14:35:01.915291244Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T14:35:02.288758825Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T14:35:02.692168087Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T14:35:03.174447813Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T14:35:03.863938652Z Gathering data for ns/openshift-multus...
2026-07-07T14:35:05.675618888Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:35:06.378183340Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T14:35:06.977805563Z Collecting kubelet logs for node ip-10-0-142-186.ec2.internal
2026-07-07T14:35:06.979616631Z Collecting kubelet logs for node ip-10-0-139-136.ec2.internal
2026-07-07T14:35:06.981339526Z Collecting kubelet logs for node ip-10-0-140-100.ec2.internal
2026-07-07T14:35:07.365624772Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x5hr6 namespace
2026-07-07T14:35:07.374079521Z INFO: Node performance data collection complete.
2026-07-07T14:35:08.479905772Z Gathering data for ns/openshift-host-network...
2026-07-07T14:35:08.892106186Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T14:35:09.525299228Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T14:35:10.035178231Z Gathering data for ns/openshift-network-operator...
2026-07-07T14:35:10.521325678Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T14:35:10.680284217Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:35:10.848688355Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T14:35:11.403824126Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T14:35:11.762025675Z Gathering data for ns/openshift-apiserver...
2026-07-07T14:35:12.237513648Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T14:35:12.614693600Z Gathering data for ns/openshift-controller-manager...
2026-07-07T14:35:12.929145347Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T14:35:13.668073179Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T14:35:14.062900740Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T14:35:14.447216007Z Gathering data for ns/openshift-service-ca...
2026-07-07T14:35:14.851643134Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T14:35:15.684698855Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T14:35:15.829561506Z Wrote inspect data to must-gather.
2026-07-07T14:35:15.829605830Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:35:15.829605830Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T14:35:15.968024691Z error: the server doesn't have a resource type "clusters"
2026-07-07T14:35:16.033510202Z Caches written to disk