2026-07-16T22:16:30.412366024Z [disk usage checker] Started
2026-07-16T22:16:30.415675962Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:30.618222669Z Gathering data for ns/openshift-cluster-version...
2026-07-16T22:16:30.959266070Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T22:16:31.051322715Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T22:16:31.138735011Z Gathering data for ns/default...
2026-07-16T22:16:31.443277998Z Gathering data for ns/openshift...
2026-07-16T22:16:31.916197937Z Gathering data for ns/kube-system...
2026-07-16T22:16:32.637417642Z Gathering data for ns/openshift-etcd...
2026-07-16T22:16:32.941992786Z Wrote inspect data to must-gather.
2026-07-16T22:16:32.942030480Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T22:16:32.942030480Z     namespaces "assisted-installer" not found
2026-07-16T22:16:33.140185287Z Waiting on subprocesses to finish execution.
2026-07-16T22:16:33.146051002Z Executing Istio gather script
2026-07-16T22:16:33.155143535Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T22:16:33.166380965Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T22:16:33.185673140Z INFO: Collecting host service logs for crio
2026-07-16T22:16:33.186057448Z INFO: Collecting host service logs for kubelet
2026-07-16T22:16:33.186512963Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T22:16:33.186638682Z INFO: Gathering HAProxy config files
2026-07-16T22:16:33.186892038Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T22:16:33.187204815Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T22:16:33.187501518Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T22:16:33.187799485Z INFO: Collecting host service logs for NetworkManager
2026-07-16T22:16:33.188217995Z INFO: Collecting host service logs for openvswitch
2026-07-16T22:16:33.188551350Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T22:16:33.188848567Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T22:16:33.189185629Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T22:16:33.189562541Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T22:16:33.192565437Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T22:16:33.197555062Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T22:16:33.860550656Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T22:16:34.060496359Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-khtbj
2026-07-16T22:16:34.060496359Z insights-runtime-extractor-bklrd
2026-07-16T22:16:34.060496359Z insights-runtime-extractor-dmh8v
2026-07-16T22:16:34.060496359Z insights-runtime-extractor-r8jdg
2026-07-16T22:16:34.240742588Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T22:16:34.258320998Z No resources found in openshift-etcd namespace.
2026-07-16T22:16:34.474405943Z Wrote inspect data to must-gather.
2026-07-16T22:16:34.548651131Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T22:16:34.637779346Z No resources found
2026-07-16T22:16:34.688988502Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T22:16:34.840673507Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T22:16:34.856443843Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T22:16:34.856523033Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T22:16:34.881554129Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T22:16:35.068349820Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T22:16:35.176559336Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T22:16:35.195845162Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.252165062Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.319606820Z No resources found
2026-07-16T22:16:35.325116148Z INFO: Worker host service log collection to complete.
2026-07-16T22:16:35.440057523Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.440648361Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T22:16:35.463468850Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:35.476392626Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T22:16:35.476392626Z See 'oc rsync -h' for help and examples
2026-07-16T22:16:35.548201510Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.549183139Z INFO: 'previous-logs' folder not found on ip-10-0-137-177.ec2.internal, skipping...
2026-07-16T22:16:35.633358654Z ERROR: No running kube-apiserver pods found
2026-07-16T22:16:35.654877086Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T22:16:35.755145055Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.817490046Z tar: Removing leading `/' from member names
2026-07-16T22:16:35.821826455Z Wrote inspect data to must-gather.
2026-07-16T22:16:35.826694461Z INFO: HAProxy config collection complete.
2026-07-16T22:16:35.987486923Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T22:16:36.030908227Z Wrote inspect data to must-gather.
2026-07-16T22:16:36.034601869Z INFO: 'previous-logs' folder not found on ip-10-0-137-85.ec2.internal, skipping...
2026-07-16T22:16:36.062767195Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T22:16:36.292259039Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T22:16:36.335967170Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T22:16:36.340072896Z Gathering data for ns/openshift-monitoring...
2026-07-16T22:16:36.362383184Z INFO: 'previous-logs' folder not found on ip-10-0-139-247.ec2.internal, skipping...
2026-07-16T22:16:36.362417247Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T22:16:36.362417247Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T22:16:36.365644622Z Wrote inspect data to must-gather.
2026-07-16T22:16:36.464778883Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T22:16:36.547823551Z Wrote inspect data to must-gather.
2026-07-16T22:16:36.617701396Z Wrote inspect data to must-gather.
2026-07-16T22:16:36.695367157Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T22:16:36.905832118Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T22:16:36.919058425Z INFO: INTERCONNECT MODE
2026-07-16T22:16:36.919104175Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T22:16:36.926855427Z Wrote inspect data to must-gather.
2026-07-16T22:16:37.088301362Z INFO: Gathering OVN_Northbound from ovnkube-node-24qjh...
2026-07-16T22:16:37.088526002Z INFO: Gathering OVN_Northbound from ovnkube-node-hz2pt...
2026-07-16T22:16:37.088723762Z INFO: Gathering OVN_Northbound from ovnkube-node-r6hvb...
2026-07-16T22:16:37.089004237Z INFO: Gathering OVN_Southbound from ovnkube-node-24qjh...
2026-07-16T22:16:37.089267361Z INFO: Gathering OVN_Southbound from ovnkube-node-hz2pt...
2026-07-16T22:16:37.089555152Z INFO: Gathering OVN_Southbound from ovnkube-node-r6hvb...
2026-07-16T22:16:37.188776740Z Wrote inspect data to must-gather.
2026-07-16T22:16:37.215215210Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T22:16:37.395834456Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.420631324Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.429878078Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.434594398Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.442078170Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.469955410Z tar: Removing leading `/' from member names
2026-07-16T22:16:37.501412912Z Wrote inspect data to must-gather.
2026-07-16T22:16:37.538646513Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T22:16:37.735377528Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T22:16:37.752954495Z Wrote inspect data to must-gather.
2026-07-16T22:16:37.801704014Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T22:16:37.928691988Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:37.956797795Z INFO: Waiting for network log collection to complete ...
2026-07-16T22:16:37.956832846Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T22:16:37.957363271Z INFO: Copying ovnk databases complete.
2026-07-16T22:16:37.959489760Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-07-16T22:16:37.963401105Z ovnk_database_store/
2026-07-16T22:16:37.963401105Z ovnk_database_store/ovnkube-node-hz2pt_sbdb
2026-07-16T22:16:37.987753792Z ovnk_database_store/ovnkube-node-r6hvb_sbdb
2026-07-16T22:16:37.996744491Z Wrote inspect data to must-gather.
2026-07-16T22:16:38.009369542Z ovnk_database_store/ovnkube-node-24qjh_nbdb
2026-07-16T22:16:38.016885676Z ovnk_database_store/ovnkube-node-hz2pt_nbdb
2026-07-16T22:16:38.024643391Z ovnk_database_store/ovnkube-node-r6hvb_nbdb
2026-07-16T22:16:38.032416673Z ovnk_database_store/ovnkube-node-24qjh_sbdb
2026-07-16T22:16:38.054522359Z INFO: Network log collection complete.
2026-07-16T22:16:38.132064773Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:38.135205054Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T22:16:38.201115865Z Wrote inspect data to must-gather.
2026-07-16T22:16:38.225326845Z INFO: Getting status from alertmanager-main-0
2026-07-16T22:16:38.330129196Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:38.432143693Z Wrote inspect data to must-gather.
2026-07-16T22:16:38.550965719Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:38.553253764Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T22:16:38.554000454Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T22:16:38.698474207Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T22:16:38.700955664Z INFO: Fallback to identify the container image from release info
2026-07-16T22:16:38.757834479Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:38.953554523Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:38.955674746Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T22:16:39.094361050Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T22:16:39.168828651Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:39.256195517Z Gathering data for ns/openshift-network-console...
2026-07-16T22:16:39.320357664Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-16T22:16:39.399831338Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:39.403560070Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-16T22:16:39.462936860Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T22:16:39.598741856Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:39.626554192Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T22:16:39.749288150Z Gathering data for ns/openshift-console-operator...
2026-07-16T22:16:39.808800253Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:39.811159630Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-16T22:16:40.016804752Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.212837934Z Gathering data for ns/openshift-console...
2026-07-16T22:16:40.223052706Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.225523799Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-16T22:16:40.393735469Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.471338028Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:40.579476115Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.581752316Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-16T22:16:40.774485735Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T22:16:40.780854812Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.784061770Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T22:16:40.972770541Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:40.975921566Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-16T22:16:41.156952922Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:41.328886163Z Gathering data for ns/openshift-dns-operator...
2026-07-16T22:16:41.370633060Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:41.373759583Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T22:16:41.530310477Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:41.658118417Z Gathering data for ns/openshift-dns...
2026-07-16T22:16:41.723746651Z Wrote inspect data to /must-gather/istio.
2026-07-16T22:16:41.909397859Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T22:16:42.300737125Z Gathering data for ns/openshift-image-registry...
2026-07-16T22:16:42.391585845Z Done executing Istio gather script
2026-07-16T22:16:42.757021120Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T22:16:43.038620696Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T22:16:43.087801181Z Gathering data for ns/openshift-ingress...
2026-07-16T22:16:43.422465808Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T22:16:43.809170653Z Gathering data for ns/openshift-insights...
2026-07-16T22:16:44.173971494Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T22:16:45.289805728Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T22:16:45.476123856Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:46.129279304Z Gathering data for ns/kserve...
2026-07-16T22:16:46.399732510Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T22:16:47.523390421Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T22:16:47.680934715Z Gathering data for ns/openshift-config...
2026-07-16T22:16:47.793996487Z Collecting performance related data for node ip-10-0-137-177.ec2.internal
2026-07-16T22:16:47.800663096Z Collecting performance related data for node ip-10-0-137-85.ec2.internal
2026-07-16T22:16:47.807699539Z Collecting performance related data for node ip-10-0-139-247.ec2.internal
2026-07-16T22:16:48.006504161Z Gathering data for ns/openshift-config-managed...
2026-07-16T22:16:48.535027120Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T22:16:49.022122623Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T22:16:49.436001934Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T22:16:49.773664853Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T22:16:50.119051392Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T22:16:50.473716363Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T22:16:50.481149300Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:50.826384637Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T22:16:51.185028193Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T22:16:51.572928955Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T22:16:52.114989358Z Gathering data for ns/openshift-multus...
2026-07-16T22:16:53.573506413Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T22:16:54.828885273Z Gathering data for ns/openshift-host-network...
2026-07-16T22:16:55.164682782Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T22:16:55.485560773Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:16:55.717116231Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T22:16:56.035552389Z Collecting kubelet logs for node ip-10-0-137-177.ec2.internal
2026-07-16T22:16:56.038178486Z Collecting kubelet logs for node ip-10-0-137-85.ec2.internal
2026-07-16T22:16:56.040214723Z Collecting kubelet logs for node ip-10-0-139-247.ec2.internal
2026-07-16T22:16:56.219598523Z Gathering data for ns/openshift-network-operator...
2026-07-16T22:16:56.418424623Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-k8wws namespace
2026-07-16T22:16:56.423400450Z INFO: Node performance data collection complete.
2026-07-16T22:16:56.647172176Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T22:16:57.010200429Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T22:16:57.497220181Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T22:16:57.782093110Z Gathering data for ns/openshift-apiserver...
2026-07-16T22:16:58.182860293Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T22:16:58.492823925Z Gathering data for ns/openshift-controller-manager...
2026-07-16T22:16:58.795242670Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T22:16:59.448768598Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T22:16:59.735517080Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T22:17:00.058518282Z Gathering data for ns/openshift-service-ca...
2026-07-16T22:17:00.491010403Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-16T22:17:00.569093803Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T22:17:01.460604487Z Wrote inspect data to must-gather.
2026-07-16T22:17:01.460647385Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T22:17:01.460647385Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T22:17:01.590457285Z error: the server doesn't have a resource type "clusters"
2026-07-16T22:17:01.729990560Z Caches written to disk