2026-07-09T18:06:57.523915778Z [disk usage checker] Started
2026-07-09T18:06:57.527528236Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:06:57.721557323Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:06:57.859264505Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:06:57.950091091Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:06:58.030390735Z Gathering data for ns/default...
2026-07-09T18:06:58.331189231Z Gathering data for ns/openshift...
2026-07-09T18:06:58.746376892Z Gathering data for ns/kube-system...
2026-07-09T18:06:59.438200094Z Gathering data for ns/openshift-etcd...
2026-07-09T18:06:59.749118858Z Wrote inspect data to must-gather.
2026-07-09T18:06:59.749152223Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:06:59.749152223Z     namespaces "assisted-installer" not found
2026-07-09T18:07:00.235317354Z Waiting on subprocesses to finish execution.
2026-07-09T18:07:00.246506722Z Executing Istio gather script
2026-07-09T18:07:00.252121526Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:07:00.252156298Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:07:00.263305267Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:07:00.265660823Z INFO: Gathering HAProxy config files
2026-07-09T18:07:00.265937416Z INFO: Collecting host service logs for crio
2026-07-09T18:07:00.266311655Z INFO: Collecting host service logs for kubelet
2026-07-09T18:07:00.266644523Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:07:00.267037037Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:07:00.267336878Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:07:00.267624943Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:07:00.267930467Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:07:00.268270066Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:07:00.268588601Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:07:00.268914295Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:07:00.269214421Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:07:00.269539913Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:07:00.396589710Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:07:00.607819186Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:07:01.035325536Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:07:01.209008307Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:07:01.230099382Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:07:01.239558375Z No resources found
2026-07-09T18:07:01.349101726Z No resources found in openshift-etcd namespace.
2026-07-09T18:07:01.355595743Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-dcfxb
2026-07-09T18:07:01.355637440Z insights-runtime-extractor-64dct
2026-07-09T18:07:01.355656899Z insights-runtime-extractor-6zqcz
2026-07-09T18:07:01.355675751Z insights-runtime-extractor-zmvq9
2026-07-09T18:07:01.462793452Z Wrote inspect data to must-gather.
2026-07-09T18:07:01.735105312Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:07:01.735105312Z See 'oc rsync -h' for help and examples
2026-07-09T18:07:01.770472420Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:07:01.987050906Z Wrote inspect data to must-gather.
2026-07-09T18:07:02.073211296Z INFO: Worker host service log collection to complete.
2026-07-09T18:07:02.122278364Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:07:02.148936308Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:07:02.149023869Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:07:02.320529585Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:07:02.430140235Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:07:02.502209054Z Wrote inspect data to must-gather.
2026-07-09T18:07:02.576582954Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:07:02.583532233Z ERROR: No running kube-apiserver pods found
2026-07-09T18:07:02.619111490Z Wrote inspect data to must-gather.
2026-07-09T18:07:02.632196649Z Wrote inspect data to must-gather.
2026-07-09T18:07:02.726143385Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:07:02.741499889Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:07:02.777414515Z No resources found
2026-07-09T18:07:02.802540951Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:07:02.807809177Z Wrote inspect data to must-gather.
2026-07-09T18:07:02.821548630Z INFO: 'previous-logs' folder not found on ip-10-0-132-123.ec2.internal, skipping...
2026-07-09T18:07:02.904198541Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:07:02.938943187Z Wrote inspect data to must-gather.
2026-07-09T18:07:03.104154228Z tar: Removing leading `/' from member names
2026-07-09T18:07:03.109863458Z INFO: HAProxy config collection complete.
2026-07-09T18:07:03.164550156Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:07:03.247867715Z INFO: 'previous-logs' folder not found on ip-10-0-133-47.ec2.internal, skipping...
2026-07-09T18:07:03.251687850Z Wrote inspect data to must-gather.
2026-07-09T18:07:03.315774983Z Wrote inspect data to must-gather.
2026-07-09T18:07:03.372169021Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:07:03.539216258Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:07:03.574021088Z INFO: 'previous-logs' folder not found on ip-10-0-136-153.ec2.internal, skipping...
2026-07-09T18:07:03.574101541Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:07:03.574135779Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:07:03.583926806Z Wrote inspect data to must-gather.
2026-07-09T18:07:03.800397687Z Wrote inspect data to must-gather.
2026-07-09T18:07:03.853936499Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:07:03.895831494Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:07:03.899809503Z Wrote inspect data to must-gather.
2026-07-09T18:07:04.043497162Z Wrote inspect data to must-gather.
2026-07-09T18:07:04.209471236Z INFO: INTERCONNECT MODE
2026-07-09T18:07:04.209506474Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:07:04.280661876Z Wrote inspect data to must-gather.
2026-07-09T18:07:04.285980403Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:07:04.408679499Z INFO: Gathering OVN_Northbound from ovnkube-node-4z9mb...
2026-07-09T18:07:04.408962055Z INFO: Gathering OVN_Northbound from ovnkube-node-6mrd6...
2026-07-09T18:07:04.409291467Z INFO: Gathering OVN_Northbound from ovnkube-node-rpllh...
2026-07-09T18:07:04.409806567Z INFO: Gathering OVN_Southbound from ovnkube-node-4z9mb...
2026-07-09T18:07:04.410134894Z INFO: Gathering OVN_Southbound from ovnkube-node-6mrd6...
2026-07-09T18:07:04.410562276Z INFO: Gathering OVN_Southbound from ovnkube-node-rpllh...
2026-07-09T18:07:04.633070084Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:07:04.636459931Z Wrote inspect data to must-gather.
2026-07-09T18:07:04.707871048Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.731582433Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.737858427Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.746876948Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.774626394Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.783086829Z tar: Removing leading `/' from member names
2026-07-09T18:07:04.857560957Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:07:04.936157801Z Wrote inspect data to must-gather.
2026-07-09T18:07:04.945661611Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:07:04.978648387Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:07:05.212726046Z Wrote inspect data to must-gather.
2026-07-09T18:07:05.220875903Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:05.242390301Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:07:05.338071456Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:07:05.338128911Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:07:05.338617792Z INFO: Copying ovnk databases complete.
2026-07-09T18:07:05.341567107Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:07:05.346141517Z ovnk_database_store/
2026-07-09T18:07:05.346194919Z ovnk_database_store/ovnkube-node-6mrd6_sbdb
2026-07-09T18:07:05.367706278Z ovnk_database_store/ovnkube-node-6mrd6_nbdb
2026-07-09T18:07:05.378878171Z ovnk_database_store/ovnkube-node-rpllh_nbdb
2026-07-09T18:07:05.387305228Z ovnk_database_store/ovnkube-node-4z9mb_nbdb
2026-07-09T18:07:05.396860137Z ovnk_database_store/ovnkube-node-rpllh_sbdb
2026-07-09T18:07:05.416928933Z ovnk_database_store/ovnkube-node-4z9mb_sbdb
2026-07-09T18:07:05.438074129Z INFO: Network log collection complete.
2026-07-09T18:07:05.455208256Z W0709 18:07:05.455166    1485 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-09T18:07:05.460305698Z Wrote inspect data to must-gather.
2026-07-09T18:07:05.475475030Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:05.478802594Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:07:05.565151726Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:07:05.690454867Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:07:05.722397124Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:05.770708065Z Gathering data for ns/openshift-network-console...
2026-07-09T18:07:05.780320619Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:07:05.783086373Z INFO: Fallback to identify the container image from release info
2026-07-09T18:07:05.951926347Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:05.953882793Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:07:06.166167782Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:06.228149631Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:07:06.383028918Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:06.386036068Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:07:06.403345132Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:07:06.541975278Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:07:06.630789113Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:06.702091545Z Gathering data for ns/openshift-console...
2026-07-09T18:07:06.702997201Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:07:06.833418623Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:06.836990962Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:07:07.016641471Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:07.237856057Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:07.241396941Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:07:07.290713332Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:07:07.424969682Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:07.603262482Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:07:07.625647347Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:07.628742959Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:07:07.817518558Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:07:07.822645746Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:07.865013852Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:07:08.030191358Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:07:08.145246953Z Gathering data for ns/openshift-dns...
2026-07-09T18:07:08.731475160Z Done executing Istio gather script
2026-07-09T18:07:08.776231222Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:07:09.020870198Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:07:09.272542431Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:07:09.593534470Z Gathering data for ns/openshift-ingress...
2026-07-09T18:07:09.916770702Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:07:10.143382221Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:07:10.286656533Z Gathering data for ns/openshift-insights...
2026-07-09T18:07:11.256305712Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:07:12.388733263Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:07:12.495016935Z Gathering data for ns/kserve...
2026-07-09T18:07:12.614139412Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:07:13.518087049Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:07:13.728036103Z Gathering data for ns/openshift-config...
2026-07-09T18:07:13.979348711Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:07:14.428613374Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:07:14.640823200Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:07:14.701294883Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:07:14.944299143Z Collecting performance related data for node ip-10-0-136-153.ec2.internal
2026-07-09T18:07:14.952371739Z Collecting performance related data for node ip-10-0-132-123.ec2.internal
2026-07-09T18:07:14.963563515Z Collecting performance related data for node ip-10-0-133-47.ec2.internal
2026-07-09T18:07:15.020341880Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:07:15.438986372Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:07:15.782850573Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:07:16.110505096Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:07:16.456504920Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:07:16.797528567Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:07:17.097358744Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:07:17.619332020Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:07:17.707776854Z Gathering data for ns/openshift-multus...
2026-07-09T18:07:19.145836096Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:07:20.615011180Z Gathering data for ns/openshift-host-network...
2026-07-09T18:07:20.963865650Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:07:21.483137948Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:07:21.899857018Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:07:22.285444023Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:07:22.586456455Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:07:22.624335383Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-09T18:07:22.951742782Z Collecting kubelet logs for node ip-10-0-136-153.ec2.internal
2026-07-09T18:07:22.953860700Z Collecting kubelet logs for node ip-10-0-132-123.ec2.internal
2026-07-09T18:07:22.955850779Z Collecting kubelet logs for node ip-10-0-133-47.ec2.internal
2026-07-09T18:07:23.098840090Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:07:23.333706544Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sng9t namespace
2026-07-09T18:07:23.339324718Z INFO: Node performance data collection complete.
2026-07-09T18:07:23.444397239Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:07:23.858474235Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:07:24.154063208Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:07:24.404767657Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:07:25.017721322Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:07:25.286872989Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:07:25.602142084Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:07:25.939651677Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:07:26.782627498Z Wrote inspect data to must-gather.
2026-07-09T18:07:26.782669642Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:07:26.782669642Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:07:26.906610663Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:07:27.045713509Z Caches written to disk