2026-07-23T10:52:07.746040292Z [disk usage checker] Started
2026-07-23T10:52:07.749259463Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:07.933217573Z Gathering data for ns/openshift-cluster-version...
2026-07-23T10:52:08.056143633Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-23T10:52:08.148823336Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T10:52:08.228684114Z Gathering data for ns/default...
2026-07-23T10:52:08.497454531Z Gathering data for ns/openshift...
2026-07-23T10:52:08.881634868Z Gathering data for ns/kube-system...
2026-07-23T10:52:09.568031420Z Gathering data for ns/openshift-etcd...
2026-07-23T10:52:09.916466471Z Wrote inspect data to must-gather.
2026-07-23T10:52:09.916502803Z error: inspection completed with the errors occurred while gathering data:
2026-07-23T10:52:09.916502803Z     namespaces "assisted-installer" not found
2026-07-23T10:52:10.197465978Z Waiting on subprocesses to finish execution.
2026-07-23T10:52:10.199226588Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-23T10:52:10.211991971Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-23T10:52:10.213416492Z INFO: Gathering HAProxy config files
2026-07-23T10:52:10.218201492Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-23T10:52:10.222892033Z Executing Istio gather script
2026-07-23T10:52:10.233518824Z INFO: Waiting for node performance related collection to complete ...
2026-07-23T10:52:10.263106745Z INFO: Collecting host service logs for crio
2026-07-23T10:52:10.263494773Z INFO: Collecting host service logs for kubelet
2026-07-23T10:52:10.263866445Z INFO: Collecting host service logs for rpm-ostreed
2026-07-23T10:52:10.264186139Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-23T10:52:10.264473466Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-23T10:52:10.264759438Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-23T10:52:10.265035153Z INFO: Collecting host service logs for NetworkManager
2026-07-23T10:52:10.265307175Z INFO: Collecting host service logs for openvswitch
2026-07-23T10:52:10.265573027Z INFO: Collecting host service logs for ovs-configuration
2026-07-23T10:52:10.265841211Z INFO: Collecting host service logs for ovsdb-server
2026-07-23T10:52:10.266112658Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-23T10:52:10.266429052Z INFO: Waiting for worker host service log collection to complete ...
2026-07-23T10:52:10.586589129Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-23T10:52:10.667345907Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-23T10:52:10.667435808Z INFO: on-disk MachineConfig config collection complete.
2026-07-23T10:52:10.857483079Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-23T10:52:10.871160952Z Wrote inspect data to must-gather.
2026-07-23T10:52:10.931978270Z No resources found
2026-07-23T10:52:11.398904816Z ERROR: No running kube-apiserver pods found
2026-07-23T10:52:11.406747197Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-23T10:52:11.639993164Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-23T10:52:11.679664764Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-zmc64
2026-07-23T10:52:11.679720886Z insights-runtime-extractor-d8p86
2026-07-23T10:52:11.679741435Z insights-runtime-extractor-rzvhb
2026-07-23T10:52:11.679762303Z insights-runtime-extractor-zzlrf
2026-07-23T10:52:11.725647657Z Wrote inspect data to must-gather.
2026-07-23T10:52:11.895817512Z INFO: "metallb-operator" not detected. Skipping.
2026-07-23T10:52:11.986872155Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-23T10:52:12.078827769Z error: the server doesn't have a resource type "performanceprofile"
2026-07-23T10:52:12.083054261Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.092404970Z No resources found in openshift-etcd namespace.
2026-07-23T10:52:12.097620437Z INFO: Worker host service log collection to complete.
2026-07-23T10:52:12.168455521Z No resources found
2026-07-23T10:52:12.194141375Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-23T10:52:12.346567475Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T10:52:12.350510807Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-23T10:52:12.350510807Z See 'oc rsync -h' for help and examples
2026-07-23T10:52:12.383268934Z tar: Removing leading `/' from member names
2026-07-23T10:52:12.395102798Z INFO: HAProxy config collection complete.
2026-07-23T10:52:12.483775188Z INFO: 'previous-logs' folder not found on ip-10-0-135-80.ec2.internal, skipping...
2026-07-23T10:52:12.509809418Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.591366061Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-23T10:52:12.596389930Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.663951299Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.732849975Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.781283515Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:12.870603342Z INFO: 'previous-logs' folder not found on ip-10-0-139-238.ec2.internal, skipping...
2026-07-23T10:52:12.905531593Z Wrote inspect data to must-gather.
2026-07-23T10:52:12.910421226Z error: the server doesn't have a resource type "machineconfigs"
2026-07-23T10:52:12.919463799Z Gathering data for ns/openshift-monitoring...
2026-07-23T10:52:13.097745910Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-23T10:52:13.119153991Z INFO: Getting rules from prometheus-k8s-0
2026-07-23T10:52:13.150792307Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-23T10:52:13.179716199Z INFO: 'previous-logs' folder not found on ip-10-0-140-84.ec2.internal, skipping...
2026-07-23T10:52:13.179753288Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-23T10:52:13.179753288Z INFO: Machine Config Daemon termination log collection complete.
2026-07-23T10:52:13.218610370Z Wrote inspect data to must-gather.
2026-07-23T10:52:13.333314040Z Wrote inspect data to must-gather.
2026-07-23T10:52:13.449538744Z Wrote inspect data to must-gather.
2026-07-23T10:52:13.504110542Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-23T10:52:13.512300569Z INFO: Getting status/config from prometheus-k8s-0
2026-07-23T10:52:13.714024496Z Wrote inspect data to must-gather.
2026-07-23T10:52:13.739321240Z INFO: INTERCONNECT MODE
2026-07-23T10:52:13.739356014Z INFO: Gathering ovn-kubernetes DBs
2026-07-23T10:52:13.892785306Z INFO: Gathering OVN_Northbound from ovnkube-node-4rs4f...
2026-07-23T10:52:13.893047005Z INFO: Gathering OVN_Northbound from ovnkube-node-8cmgc...
2026-07-23T10:52:13.893352786Z INFO: Gathering OVN_Northbound from ovnkube-node-hs2nx...
2026-07-23T10:52:13.893671960Z INFO: Gathering OVN_Southbound from ovnkube-node-4rs4f...
2026-07-23T10:52:13.893927483Z INFO: Gathering OVN_Southbound from ovnkube-node-8cmgc...
2026-07-23T10:52:13.894220544Z INFO: Gathering OVN_Southbound from ovnkube-node-hs2nx...
2026-07-23T10:52:13.933998811Z Wrote inspect data to must-gather.
2026-07-23T10:52:13.997987198Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-23T10:52:14.157606413Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.164095949Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.195233837Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.196770585Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.249660222Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-23T10:52:14.289550039Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.306842922Z tar: Removing leading `/' from member names
2026-07-23T10:52:14.333940698Z Wrote inspect data to must-gather.
2026-07-23T10:52:14.370304415Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-23T10:52:14.395466565Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-23T10:52:14.458137028Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:14.595804647Z Wrote inspect data to must-gather.
2026-07-23T10:52:14.628459861Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-23T10:52:14.725029780Z INFO: Waiting for network log collection to complete ...
2026-07-23T10:52:14.725200880Z INFO: Waiting for ovnk database copies to complete ...
2026-07-23T10:52:14.726007391Z INFO: Copying ovnk databases complete.
2026-07-23T10:52:14.727473814Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:14.727928139Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-23T10:52:14.729828918Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-23T10:52:14.730585050Z ovnk_database_store/
2026-07-23T10:52:14.730634022Z ovnk_database_store/ovnkube-node-8cmgc_sbdb
2026-07-23T10:52:14.751696776Z ovnk_database_store/ovnkube-node-4rs4f_nbdb
2026-07-23T10:52:14.758919628Z ovnk_database_store/ovnkube-node-hs2nx_sbdb
2026-07-23T10:52:14.775740460Z ovnk_database_store/ovnkube-node-8cmgc_nbdb
2026-07-23T10:52:14.782745405Z ovnk_database_store/ovnkube-node-4rs4f_sbdb
2026-07-23T10:52:14.800987470Z ovnk_database_store/ovnkube-node-hs2nx_nbdb
2026-07-23T10:52:14.810589557Z INFO: Network log collection complete.
2026-07-23T10:52:14.848183341Z Wrote inspect data to must-gather.
2026-07-23T10:52:14.891059770Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-23T10:52:14.933965498Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.042203334Z W0723 10:52:15.042161    1511 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-23T10:52:15.048557533Z Wrote inspect data to must-gather.
2026-07-23T10:52:15.059938801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T10:52:15.154360567Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.157323534Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-23T10:52:15.253161969Z Gathering data for ns/openshift-network-console...
2026-07-23T10:52:15.292939563Z INFO: Getting status from alertmanager-main-0
2026-07-23T10:52:15.363593819Z Wrote inspect data to must-gather.
2026-07-23T10:52:15.380113165Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.571971805Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.574731974Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-23T10:52:15.625379809Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-23T10:52:15.627847831Z INFO: Fallback to identify the container image from release info
2026-07-23T10:52:15.714296397Z Gathering data for ns/openshift-console-operator...
2026-07-23T10:52:15.800623526Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.977317969Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:15.980701909Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-23T10:52:16.099489991Z Gathering data for ns/openshift-console...
2026-07-23T10:52:16.144651115Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:16.246537379Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-23T10:52:16.336064678Z Wrote inspect data to /must-gather/istio.
2026-07-23T10:52:16.366502256Z daemonset.apps/perf-node-gather-daemonset created
2026-07-23T10:52:16.509014445Z Waiting for performance profile collector pods to become ready: 1
2026-07-23T10:52:16.655939996Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-23T10:52:17.039017393Z Done executing Istio gather script
2026-07-23T10:52:17.155511841Z Gathering data for ns/openshift-dns-operator...
2026-07-23T10:52:17.426877338Z Gathering data for ns/openshift-dns...
2026-07-23T10:52:17.611665804Z Waiting for performance profile collector pods to become ready: 2
2026-07-23T10:52:17.786740124Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:17.961582836Z Gathering data for ns/openshift-image-registry...
2026-07-23T10:52:18.393623768Z Gathering data for ns/openshift-ingress-operator...
2026-07-23T10:52:18.696828153Z Gathering data for ns/openshift-ingress...
2026-07-23T10:52:18.721579347Z Waiting for performance profile collector pods to become ready: 3
2026-07-23T10:52:19.023623729Z Gathering data for ns/openshift-ingress-canary...
2026-07-23T10:52:19.369262479Z Gathering data for ns/openshift-insights...
2026-07-23T10:52:19.834286266Z Waiting for performance profile collector pods to become ready: 4
2026-07-23T10:52:20.951816449Z Waiting for performance profile collector pods to become ready: 5
2026-07-23T10:52:21.191351231Z Gathering data for ns/openshift-jobset-operator...
2026-07-23T10:52:21.980991985Z Gathering data for ns/openshift-config...
2026-07-23T10:52:22.070432991Z Waiting for performance profile collector pods to become ready: 6
2026-07-23T10:52:22.241743680Z Gathering data for ns/openshift-config-managed...
2026-07-23T10:52:22.690660455Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-23T10:52:22.791694213Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:22.935276128Z Gathering data for ns/openshift-kube-apiserver...
2026-07-23T10:52:23.188060945Z Waiting for performance profile collector pods to become ready: 7
2026-07-23T10:52:23.200661454Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-23T10:52:23.477539750Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-23T10:52:23.787311835Z Gathering data for ns/openshift-kube-scheduler...
2026-07-23T10:52:24.084543859Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-23T10:52:24.296020393Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-23T10:52:24.420735545Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-23T10:52:24.544564252Z Collecting performance related data for node ip-10-0-140-84.ec2.internal
2026-07-23T10:52:24.550966579Z Collecting performance related data for node ip-10-0-139-238.ec2.internal
2026-07-23T10:52:24.558044478Z Collecting performance related data for node ip-10-0-135-80.ec2.internal
2026-07-23T10:52:24.838622978Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-23T10:52:25.290513731Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-23T10:52:25.907291493Z Gathering data for ns/openshift-multus...
2026-07-23T10:52:27.797223203Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:27.823000191Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-23T10:52:29.794267614Z Gathering data for ns/openshift-host-network...
2026-07-23T10:52:30.192747967Z Gathering data for ns/openshift-network-diagnostics...
2026-07-23T10:52:30.990194933Z Gathering data for ns/openshift-network-node-identity...
2026-07-23T10:52:31.411202592Z Gathering data for ns/openshift-network-operator...
2026-07-23T10:52:31.812771588Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-23T10:52:32.106163762Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-23T10:52:32.482548583Z Collecting kubelet logs for node ip-10-0-140-84.ec2.internal
2026-07-23T10:52:32.484653020Z Collecting kubelet logs for node ip-10-0-139-238.ec2.internal
2026-07-23T10:52:32.486772497Z Collecting kubelet logs for node ip-10-0-135-80.ec2.internal
2026-07-23T10:52:32.754339660Z Gathering data for ns/openshift-apiserver-operator...
2026-07-23T10:52:32.803501137Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-23T10:52:32.845124440Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5fvb2 namespace
2026-07-23T10:52:32.850861085Z INFO: Node performance data collection complete.
2026-07-23T10:52:33.107979798Z Gathering data for ns/openshift-apiserver...
2026-07-23T10:52:33.491505883Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-23T10:52:33.799403018Z Gathering data for ns/openshift-controller-manager...
2026-07-23T10:52:34.075804284Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-23T10:52:34.718894013Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-23T10:52:35.115863114Z Gathering data for ns/openshift-service-ca-operator...
2026-07-23T10:52:35.417811219Z Gathering data for ns/openshift-service-ca...
2026-07-23T10:52:35.787427007Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-23T10:52:36.746123044Z Wrote inspect data to must-gather.
2026-07-23T10:52:36.746163902Z error: inspection completed with the errors occurred while gathering data:
2026-07-23T10:52:36.746163902Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-23T10:52:36.865653293Z error: the server doesn't have a resource type "clusters"
2026-07-23T10:52:36.991446239Z Caches written to disk