2026-07-16T17:23:28.886158842Z [disk usage checker] Started
2026-07-16T17:23:28.889672691Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:29.095146277Z Gathering data for ns/openshift-cluster-version...
2026-07-16T17:23:29.240157912Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T17:23:29.347823165Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T17:23:29.442302030Z Gathering data for ns/default...
2026-07-16T17:23:29.775773617Z Gathering data for ns/openshift...
2026-07-16T17:23:30.246813461Z Gathering data for ns/kube-system...
2026-07-16T17:23:31.197371170Z Gathering data for ns/openshift-etcd...
2026-07-16T17:23:31.667541916Z Wrote inspect data to must-gather.
2026-07-16T17:23:31.667585554Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T17:23:31.667585554Z     namespaces "assisted-installer" not found
2026-07-16T17:23:31.782918643Z Waiting on subprocesses to finish execution.
2026-07-16T17:23:31.786162493Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T17:23:31.789815293Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T17:23:31.798256222Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T17:23:31.800931033Z INFO: Gathering HAProxy config files
2026-07-16T17:23:31.809122896Z Executing Istio gather script
2026-07-16T17:23:31.817770001Z INFO: Collecting host service logs for crio
2026-07-16T17:23:31.818524197Z INFO: Collecting host service logs for kubelet
2026-07-16T17:23:31.818927944Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T17:23:31.819367854Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T17:23:31.819787176Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T17:23:31.828760294Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T17:23:31.829250556Z INFO: Collecting host service logs for NetworkManager
2026-07-16T17:23:31.829700689Z INFO: Collecting host service logs for openvswitch
2026-07-16T17:23:31.830141109Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T17:23:31.831580073Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T17:23:31.831991344Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T17:23:31.832489647Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T17:23:32.162020913Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T17:23:32.570696655Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T17:23:32.570740451Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T17:23:32.579567597Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T17:23:32.658807353Z No resources found in openshift-etcd namespace.
2026-07-16T17:23:32.819038680Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T17:23:33.055339676Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T17:23:33.147532689Z Wrote inspect data to must-gather.
2026-07-16T17:23:33.175419059Z No resources found
2026-07-16T17:23:33.198863221Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T17:23:33.358333056Z INFO: Worker host service log collection to complete.
2026-07-16T17:23:33.375214440Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-d9cd6
2026-07-16T17:23:33.375273420Z insights-runtime-extractor-5rkmf
2026-07-16T17:23:33.375292164Z insights-runtime-extractor-j5r8s
2026-07-16T17:23:33.375310589Z insights-runtime-extractor-vdp8w
2026-07-16T17:23:33.440349014Z Wrote inspect data to must-gather.
2026-07-16T17:23:33.514988035Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T17:23:33.582702123Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T17:23:33.797837809Z ERROR: No running kube-apiserver pods found
2026-07-16T17:23:33.938899393Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:33.955651664Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T17:23:33.955651664Z See 'oc rsync -h' for help and examples
2026-07-16T17:23:33.984665605Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T17:23:34.037555384Z Gathering data for ns/openshift-monitoring...
2026-07-16T17:23:34.077795778Z INFO: 'previous-logs' folder not found on ip-10-0-128-136.ec2.internal, skipping...
2026-07-16T17:23:34.093165654Z No resources found
2026-07-16T17:23:34.159696627Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T17:23:34.166175917Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.210984723Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T17:23:34.263434982Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.320560220Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.378888622Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T17:23:34.506730495Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.573375278Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.615003348Z INFO: 'previous-logs' folder not found on ip-10-0-129-41.ec2.internal, skipping...
2026-07-16T17:23:34.725611283Z Wrote inspect data to must-gather.
2026-07-16T17:23:34.728297868Z tar: Removing leading `/' from member names
2026-07-16T17:23:34.738675261Z INFO: HAProxy config collection complete.
2026-07-16T17:23:34.740499192Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T17:23:34.788516859Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T17:23:34.947349888Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T17:23:34.956652317Z INFO: 'previous-logs' folder not found on ip-10-0-133-190.ec2.internal, skipping...
2026-07-16T17:23:34.956685492Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T17:23:34.956696721Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T17:23:35.028076337Z Wrote inspect data to must-gather.
2026-07-16T17:23:35.189855829Z Wrote inspect data to must-gather.
2026-07-16T17:23:35.286209572Z Wrote inspect data to must-gather.
2026-07-16T17:23:35.333177553Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T17:23:35.334459597Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T17:23:35.369144064Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T17:23:35.591243372Z Wrote inspect data to must-gather.
2026-07-16T17:23:35.613460231Z INFO: INTERCONNECT MODE
2026-07-16T17:23:35.613504411Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T17:23:35.789673381Z INFO: Gathering OVN_Northbound from ovnkube-node-7ck8p...
2026-07-16T17:23:35.789910404Z INFO: Gathering OVN_Northbound from ovnkube-node-pp7pb...
2026-07-16T17:23:35.790189657Z INFO: Gathering OVN_Northbound from ovnkube-node-ssrxz...
2026-07-16T17:23:35.790467062Z INFO: Gathering OVN_Southbound from ovnkube-node-7ck8p...
2026-07-16T17:23:35.790720212Z INFO: Gathering OVN_Southbound from ovnkube-node-pp7pb...
2026-07-16T17:23:35.791010226Z INFO: Gathering OVN_Southbound from ovnkube-node-ssrxz...
2026-07-16T17:23:35.852493900Z Wrote inspect data to must-gather.
2026-07-16T17:23:35.935117694Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T17:23:36.123205849Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.169717472Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.188648964Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.192551278Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.196387935Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.219559217Z tar: Removing leading `/' from member names
2026-07-16T17:23:36.315734841Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T17:23:36.318717476Z Wrote inspect data to must-gather.
2026-07-16T17:23:36.358603015Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T17:23:36.589195454Z Wrote inspect data to must-gather.
2026-07-16T17:23:36.694734036Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T17:23:36.756743355Z INFO: Waiting for network log collection to complete ...
2026-07-16T17:23:36.756777329Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T17:23:36.757151650Z INFO: Copying ovnk databases complete.
2026-07-16T17:23:36.758667191Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-16T17:23:36.760635169Z ovnk_database_store/
2026-07-16T17:23:36.760635169Z ovnk_database_store/ovnkube-node-ssrxz_nbdb
2026-07-16T17:23:36.770699133Z ovnk_database_store/ovnkube-node-7ck8p_nbdb
2026-07-16T17:23:36.778502351Z ovnk_database_store/ovnkube-node-pp7pb_nbdb
2026-07-16T17:23:36.788093863Z ovnk_database_store/ovnkube-node-ssrxz_sbdb
2026-07-16T17:23:36.811094109Z ovnk_database_store/ovnkube-node-7ck8p_sbdb
2026-07-16T17:23:36.833192039Z ovnk_database_store/ovnkube-node-pp7pb_sbdb
2026-07-16T17:23:36.855191182Z INFO: Network log collection complete.
2026-07-16T17:23:36.865364940Z Wrote inspect data to must-gather.
2026-07-16T17:23:36.877711475Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T17:23:37.018678414Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T17:23:37.044231269Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T17:23:37.075069652Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:37.189356380Z Wrote inspect data to must-gather.
2026-07-16T17:23:37.264208061Z Gathering data for ns/openshift-network-console...
2026-07-16T17:23:37.319476132Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:37.322110436Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T17:23:37.495279763Z INFO: Getting status from alertmanager-main-0
2026-07-16T17:23:37.511925668Z Wrote inspect data to must-gather.
2026-07-16T17:23:37.561532812Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:37.807310050Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:37.810190264Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T17:23:37.842857774Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T17:23:37.845989608Z INFO: Fallback to identify the container image from release info
2026-07-16T17:23:37.863539330Z Gathering data for ns/openshift-console-operator...
2026-07-16T17:23:38.034322662Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:38.226025872Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:38.229264291Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T17:23:38.297221756Z Gathering data for ns/openshift-console...
2026-07-16T17:23:38.471685232Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:38.507264987Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-16T17:23:38.650196821Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T17:23:38.679163648Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:38.681651913Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-16T17:23:38.810312361Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T17:23:38.888266439Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:38.904406488Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T17:23:38.969110831Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:39.100576035Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:39.103070399Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-16T17:23:39.307466271Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:39.496053966Z Gathering data for ns/openshift-dns-operator...
2026-07-16T17:23:39.524346955Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:39.527956820Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T17:23:39.711219132Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:39.824248143Z Gathering data for ns/openshift-dns...
2026-07-16T17:23:39.907685376Z Wrote inspect data to /must-gather/istio.
2026-07-16T17:23:39.947987053Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T17:23:40.550760647Z Gathering data for ns/openshift-image-registry...
2026-07-16T17:23:40.623068214Z Done executing Istio gather script
2026-07-16T17:23:41.079167588Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T17:23:41.160998319Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T17:23:41.594454706Z Gathering data for ns/openshift-ingress...
2026-07-16T17:23:41.977033976Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T17:23:42.213449478Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T17:23:42.450922157Z Gathering data for ns/openshift-insights...
2026-07-16T17:23:43.339558908Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T17:23:43.974023498Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:44.466620841Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T17:23:45.139173024Z Gathering data for ns/kserve...
2026-07-16T17:23:45.594680016Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T17:23:46.594263735Z Gathering data for ns/openshift-config...
2026-07-16T17:23:46.731465049Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T17:23:47.007444861Z Collecting performance related data for node ip-10-0-133-190.ec2.internal
2026-07-16T17:23:47.009413159Z Gathering data for ns/openshift-config-managed...
2026-07-16T17:23:47.014511130Z Collecting performance related data for node ip-10-0-128-136.ec2.internal
2026-07-16T17:23:47.021507454Z Collecting performance related data for node ip-10-0-129-41.ec2.internal
2026-07-16T17:23:48.033423976Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T17:23:48.512539953Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T17:23:48.978914682Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:49.007425633Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T17:23:49.475416731Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T17:23:49.864745991Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T17:23:50.292783177Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T17:23:50.734771425Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T17:23:51.235398829Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T17:23:51.643484052Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T17:23:52.292519645Z Gathering data for ns/openshift-multus...
2026-07-16T17:23:53.936559224Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T17:23:53.983559909Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:55.481971538Z Gathering data for ns/openshift-host-network...
2026-07-16T17:23:55.873855913Z Collecting kubelet logs for node ip-10-0-133-190.ec2.internal
2026-07-16T17:23:55.875747461Z Collecting kubelet logs for node ip-10-0-128-136.ec2.internal
2026-07-16T17:23:55.877907787Z Collecting kubelet logs for node ip-10-0-129-41.ec2.internal
2026-07-16T17:23:55.883619698Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T17:23:56.232067596Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sc4rz namespace
2026-07-16T17:23:56.237338708Z INFO: Node performance data collection complete.
2026-07-16T17:23:56.600927769Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T17:23:57.058192922Z Gathering data for ns/openshift-network-operator...
2026-07-16T17:23:57.517167263Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T17:23:57.927836739Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T17:23:58.469652105Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T17:23:58.788310310Z Gathering data for ns/openshift-apiserver...
2026-07-16T17:23:58.988227998Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T17:23:59.207875155Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T17:23:59.532035194Z Gathering data for ns/openshift-controller-manager...
2026-07-16T17:23:59.820833632Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T17:24:00.505182213Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T17:24:00.918224388Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T17:24:01.270848313Z Gathering data for ns/openshift-service-ca...
2026-07-16T17:24:01.668895927Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T17:24:02.772878107Z Wrote inspect data to must-gather.
2026-07-16T17:24:02.772928992Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T17:24:02.772928992Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T17:24:02.911745552Z error: the server doesn't have a resource type "clusters"
2026-07-16T17:24:03.032580521Z Caches written to disk