2026-07-16T18:01:33.395050662Z [disk usage checker] Started
2026-07-16T18:01:33.399080166Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:33.629086020Z Gathering data for ns/openshift-cluster-version...
2026-07-16T18:01:33.801912969Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T18:01:33.918429517Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:01:34.010654150Z Gathering data for ns/default...
2026-07-16T18:01:34.373866347Z Gathering data for ns/openshift...
2026-07-16T18:01:34.871768174Z Gathering data for ns/kube-system...
2026-07-16T18:01:35.659184845Z Gathering data for ns/openshift-etcd...
2026-07-16T18:01:36.041134064Z Wrote inspect data to must-gather.
2026-07-16T18:01:36.041173922Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T18:01:36.041173922Z     namespaces "assisted-installer" not found
2026-07-16T18:01:36.450070654Z Waiting on subprocesses to finish execution.
2026-07-16T18:01:36.459761355Z INFO: Gathering HAProxy config files
2026-07-16T18:01:36.460493457Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T18:01:36.461930645Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T18:01:36.462605139Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T18:01:36.472408314Z Executing Istio gather script
2026-07-16T18:01:36.480326643Z INFO: Collecting host service logs for crio
2026-07-16T18:01:36.480661372Z INFO: Collecting host service logs for kubelet
2026-07-16T18:01:36.481262246Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T18:01:36.481636043Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T18:01:36.481961273Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T18:01:36.482307974Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T18:01:36.482656488Z INFO: Collecting host service logs for NetworkManager
2026-07-16T18:01:36.482996872Z INFO: Collecting host service logs for openvswitch
2026-07-16T18:01:36.483402611Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T18:01:36.483737600Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T18:01:36.484059661Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T18:01:36.484456348Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T18:01:36.840886112Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T18:01:36.904979255Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T18:01:37.068937203Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T18:01:37.196974218Z No resources found
2026-07-16T18:01:37.372007653Z No resources found in openshift-etcd namespace.
2026-07-16T18:01:37.538586491Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T18:01:37.683525453Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T18:01:37.765019544Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-7mght
2026-07-16T18:01:37.765068318Z insights-runtime-extractor-cmk6s
2026-07-16T18:01:37.765087910Z insights-runtime-extractor-ts6vm
2026-07-16T18:01:37.765107324Z insights-runtime-extractor-zdv22
2026-07-16T18:01:37.856933707Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T18:01:38.066832103Z INFO: Worker host service log collection to complete.
2026-07-16T18:01:38.104860258Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T18:01:38.104860258Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T18:01:38.108473890Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T18:01:38.135470762Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T18:01:38.141536769Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T18:01:38.180293455Z Wrote inspect data to must-gather.
2026-07-16T18:01:38.446139659Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T18:01:38.446139659Z See 'oc rsync -h' for help and examples
2026-07-16T18:01:38.562760659Z Wrote inspect data to must-gather.
2026-07-16T18:01:38.622695295Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:38.790662195Z No resources found
2026-07-16T18:01:38.929429888Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:01:38.950850867Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T18:01:39.022861958Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.044937138Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.064195277Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.093066179Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.149207660Z ERROR: No running kube-apiserver pods found
2026-07-16T18:01:39.169191721Z INFO: 'previous-logs' folder not found on ip-10-0-133-178.ec2.internal, skipping...
2026-07-16T18:01:39.169567146Z Gathering data for ns/openshift-monitoring...
2026-07-16T18:01:39.228522974Z tar: Removing leading `/' from member names
2026-07-16T18:01:39.234187693Z INFO: HAProxy config collection complete.
2026-07-16T18:01:39.302361440Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.381713166Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T18:01:39.467008814Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.525560464Z INFO: 'previous-logs' folder not found on ip-10-0-139-0.ec2.internal, skipping...
2026-07-16T18:01:39.541461725Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T18:01:39.760676329Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T18:01:39.810204226Z Wrote inspect data to must-gather.
2026-07-16T18:01:39.863592619Z INFO: 'previous-logs' folder not found on ip-10-0-143-77.ec2.internal, skipping...
2026-07-16T18:01:39.863706733Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T18:01:39.863765229Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T18:01:39.906588784Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T18:01:40.043634131Z Wrote inspect data to must-gather.
2026-07-16T18:01:40.099035134Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T18:01:40.149357497Z Wrote inspect data to must-gather.
2026-07-16T18:01:40.224746997Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T18:01:40.380940714Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T18:01:40.469858433Z Wrote inspect data to must-gather.
2026-07-16T18:01:40.487750313Z INFO: INTERCONNECT MODE
2026-07-16T18:01:40.487795275Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T18:01:40.653386330Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T18:01:40.683927882Z INFO: Gathering OVN_Northbound from ovnkube-node-f2tdf...
2026-07-16T18:01:40.684133709Z INFO: Gathering OVN_Northbound from ovnkube-node-ktrzs...
2026-07-16T18:01:40.684367284Z INFO: Gathering OVN_Northbound from ovnkube-node-nzkrn...
2026-07-16T18:01:40.684692528Z INFO: Gathering OVN_Southbound from ovnkube-node-f2tdf...
2026-07-16T18:01:40.685085708Z INFO: Gathering OVN_Southbound from ovnkube-node-ktrzs...
2026-07-16T18:01:40.685391232Z INFO: Gathering OVN_Southbound from ovnkube-node-nzkrn...
2026-07-16T18:01:40.736693759Z Wrote inspect data to must-gather.
2026-07-16T18:01:40.969004592Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.011433908Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.066195807Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.085730863Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T18:01:41.107695520Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.112060463Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.128335784Z tar: Removing leading `/' from member names
2026-07-16T18:01:41.202929693Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T18:01:41.215787606Z Wrote inspect data to must-gather.
2026-07-16T18:01:41.266310260Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T18:01:41.372824982Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T18:01:41.487164845Z Wrote inspect data to must-gather.
2026-07-16T18:01:41.521378205Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:41.641965735Z INFO: Waiting for network log collection to complete ...
2026-07-16T18:01:41.642124660Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T18:01:41.642428438Z INFO: Copying ovnk databases complete.
2026-07-16T18:01:41.644151028Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-16T18:01:41.646700133Z ovnk_database_store/
2026-07-16T18:01:41.646700133Z ovnk_database_store/ovnkube-node-ktrzs_sbdb
2026-07-16T18:01:41.669352349Z ovnk_database_store/ovnkube-node-f2tdf_nbdb
2026-07-16T18:01:41.680017468Z ovnk_database_store/ovnkube-node-nzkrn_nbdb
2026-07-16T18:01:41.691673416Z ovnk_database_store/ovnkube-node-ktrzs_nbdb
2026-07-16T18:01:41.702738172Z ovnk_database_store/ovnkube-node-nzkrn_sbdb
2026-07-16T18:01:41.724907440Z ovnk_database_store/ovnkube-node-f2tdf_sbdb
2026-07-16T18:01:41.742860818Z Wrote inspect data to must-gather.
2026-07-16T18:01:41.752396599Z INFO: Network log collection complete.
2026-07-16T18:01:41.801555729Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:41.803938510Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T18:01:41.847553952Z INFO: Getting status from alertmanager-main-0
2026-07-16T18:01:41.929051206Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T18:01:42.063293488Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:42.085414880Z Wrote inspect data to must-gather.
2026-07-16T18:01:42.140792491Z Gathering data for ns/openshift-network-console...
2026-07-16T18:01:42.310367058Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:42.312931215Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T18:01:42.400020549Z Wrote inspect data to must-gather.
2026-07-16T18:01:42.519155085Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:42.696676213Z Gathering data for ns/openshift-console-operator...
2026-07-16T18:01:42.717822108Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T18:01:42.720203988Z INFO: Fallback to identify the container image from release info
2026-07-16T18:01:42.760382622Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:42.765276582Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T18:01:43.013466416Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:43.159603259Z Gathering data for ns/openshift-console...
2026-07-16T18:01:43.234282032Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:43.237051566Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-16T18:01:43.422271715Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:43.487114381Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-16T18:01:43.646143906Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T18:01:43.671040422Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:43.674514401Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-16T18:01:43.730279423Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:43.811746979Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T18:01:43.814581262Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T18:01:43.880809497Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.124594187Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.127076746Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-16T18:01:44.340516802Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.460870301Z Gathering data for ns/openshift-dns-operator...
2026-07-16T18:01:44.572758798Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.575262224Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-16T18:01:44.757867543Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.817069941Z Gathering data for ns/openshift-dns...
2026-07-16T18:01:44.948011800Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T18:01:44.970637350Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:44.973221269Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-16T18:01:45.158945299Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:45.368449570Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:45.371928138Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T18:01:45.555208026Z Gathering data for ns/openshift-image-registry...
2026-07-16T18:01:45.568297563Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:45.779271001Z Wrote inspect data to /must-gather/istio.
2026-07-16T18:01:46.107593884Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T18:01:46.162004035Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T18:01:46.538772414Z Done executing Istio gather script
2026-07-16T18:01:46.566026141Z Gathering data for ns/openshift-ingress...
2026-07-16T18:01:46.950794330Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T18:01:47.229153477Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T18:01:47.413948029Z Gathering data for ns/openshift-insights...
2026-07-16T18:01:48.353147211Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T18:01:48.735182462Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:49.485594337Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T18:01:49.762638378Z Gathering data for ns/kserve...
2026-07-16T18:01:50.618083993Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T18:01:51.586950399Z Gathering data for ns/openshift-config...
2026-07-16T18:01:51.750216344Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T18:01:51.957593760Z Gathering data for ns/openshift-config-managed...
2026-07-16T18:01:52.086444916Z Collecting performance related data for node ip-10-0-143-77.ec2.internal
2026-07-16T18:01:52.094337735Z Collecting performance related data for node ip-10-0-139-0.ec2.internal
2026-07-16T18:01:52.103513688Z Collecting performance related data for node ip-10-0-133-178.ec2.internal
2026-07-16T18:01:52.524544660Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T18:01:52.932273948Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T18:01:53.364200574Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T18:01:53.745730050Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:53.801739948Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T18:01:54.161354865Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T18:01:54.524772051Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T18:01:54.914406925Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T18:01:55.356150218Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T18:01:55.778154824Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T18:01:56.373103202Z Gathering data for ns/openshift-multus...
2026-07-16T18:01:58.051109266Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T18:01:58.752251523Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:01:59.436734109Z Gathering data for ns/openshift-host-network...
2026-07-16T18:01:59.813696953Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T18:02:00.462457848Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T18:02:00.960329748Z Gathering data for ns/openshift-network-operator...
2026-07-16T18:02:00.999025779Z Collecting kubelet logs for node ip-10-0-143-77.ec2.internal
2026-07-16T18:02:01.001071503Z Collecting kubelet logs for node ip-10-0-139-0.ec2.internal
2026-07-16T18:02:01.003568064Z Collecting kubelet logs for node ip-10-0-133-178.ec2.internal
2026-07-16T18:02:01.401331591Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x48rf namespace
2026-07-16T18:02:01.409164310Z INFO: Node performance data collection complete.
2026-07-16T18:02:01.466636583Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T18:02:01.793560486Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T18:02:02.342788494Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T18:02:02.736059494Z Gathering data for ns/openshift-apiserver...
2026-07-16T18:02:03.188320792Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T18:02:03.531535545Z Gathering data for ns/openshift-controller-manager...
2026-07-16T18:02:03.758512009Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T18:02:03.851710992Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T18:02:04.585032293Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T18:02:04.984482083Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T18:02:05.348583718Z Gathering data for ns/openshift-service-ca...
2026-07-16T18:02:05.776385490Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T18:02:06.774732344Z Wrote inspect data to must-gather.
2026-07-16T18:02:06.774782466Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T18:02:06.774782466Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T18:02:06.916681281Z error: the server doesn't have a resource type "clusters"
2026-07-16T18:02:07.593798477Z Caches written to disk