2026-07-31T09:38:49.461161268Z [disk usage checker] Started
2026-07-31T09:38:49.464489609Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:38:49.655971365Z Gathering data for ns/openshift-cluster-version...
2026-07-31T09:38:49.780442044Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-31T09:38:49.887810064Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T09:38:49.981187928Z Gathering data for ns/default...
2026-07-31T09:38:50.325334746Z Gathering data for ns/openshift...
2026-07-31T09:38:50.758073909Z Gathering data for ns/kube-system...
2026-07-31T09:38:51.501319286Z Gathering data for ns/openshift-etcd...
2026-07-31T09:38:51.848449870Z Wrote inspect data to must-gather.
2026-07-31T09:38:51.848482443Z error: inspection completed with the errors occurred while gathering data:
2026-07-31T09:38:51.848482443Z     namespaces "assisted-installer" not found
2026-07-31T09:38:52.118676212Z Waiting on subprocesses to finish execution.
2026-07-31T09:38:52.125814270Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-31T09:38:52.127856140Z INFO: Gathering HAProxy config files
2026-07-31T09:38:52.129721464Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-31T09:38:52.131345186Z INFO: Collecting host service logs for crio
2026-07-31T09:38:52.131683783Z INFO: Collecting host service logs for kubelet
2026-07-31T09:38:52.132017131Z INFO: Collecting host service logs for rpm-ostreed
2026-07-31T09:38:52.132318362Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-31T09:38:52.132687260Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-31T09:38:52.132990832Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-31T09:38:52.133309013Z INFO: Collecting host service logs for NetworkManager
2026-07-31T09:38:52.133646818Z INFO: Collecting host service logs for openvswitch
2026-07-31T09:38:52.133935555Z INFO: Collecting host service logs for ovs-configuration
2026-07-31T09:38:52.134255252Z INFO: Collecting host service logs for ovsdb-server
2026-07-31T09:38:52.134586313Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-31T09:38:52.134940092Z INFO: Waiting for worker host service log collection to complete ...
2026-07-31T09:38:52.134975582Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-31T09:38:52.136156727Z Executing Istio gather script
2026-07-31T09:38:52.403875266Z INFO: Waiting for node performance related collection to complete ...
2026-07-31T09:38:52.572558733Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-31T09:38:52.726571370Z No resources found in openshift-etcd namespace.
2026-07-31T09:38:52.762690224Z Wrote inspect data to must-gather.
2026-07-31T09:38:52.824234227Z No resources found
2026-07-31T09:38:53.081168463Z INFO: "metallb-operator" not detected. Skipping.
2026-07-31T09:38:53.187496018Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-31T09:38:53.187603577Z INFO: on-disk MachineConfig config collection complete.
2026-07-31T09:38:53.250006023Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-31T09:38:53.321963229Z INFO: Collecting Insights Archives from insights-operator-77696685b-d7jjf
2026-07-31T09:38:53.322009575Z insights-runtime-extractor-8mdx7
2026-07-31T09:38:53.322029934Z insights-runtime-extractor-8p86k
2026-07-31T09:38:53.322073883Z insights-runtime-extractor-ztxsb
2026-07-31T09:38:53.347943388Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-31T09:38:53.490648266Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-31T09:38:53.490914715Z error: the server doesn't have a resource type "performanceprofile"
2026-07-31T09:38:53.528663737Z INFO: Worker host service log collection to complete.
2026-07-31T09:38:53.569611934Z Wrote inspect data to must-gather.
2026-07-31T09:38:53.702945583Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-31T09:38:53.740095020Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-31T09:38:53.740095020Z See 'oc rsync -h' for help and examples
2026-07-31T09:38:54.120932165Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-31T09:38:54.159626067Z No resources found
2026-07-31T09:38:54.166297783Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.208018265Z ERROR: No running kube-apiserver pods found
2026-07-31T09:38:54.329438195Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-31T09:38:54.344340440Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.353459783Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.405042654Z INFO: 'previous-logs' folder not found on ip-10-0-128-204.ec2.internal, skipping...
2026-07-31T09:38:54.436217252Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T09:38:54.477721595Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.498421689Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:38:54.629924264Z tar: Removing leading `/' from member names
2026-07-31T09:38:54.640294677Z INFO: HAProxy config collection complete.
2026-07-31T09:38:54.649343247Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.672108310Z Gathering data for ns/openshift-monitoring...
2026-07-31T09:38:54.777729456Z Wrote inspect data to must-gather.
2026-07-31T09:38:54.823014402Z INFO: 'previous-logs' folder not found on ip-10-0-140-243.ec2.internal, skipping...
2026-07-31T09:38:54.833883230Z INFO: Getting rules from prometheus-k8s-0
2026-07-31T09:38:54.854344809Z error: the server doesn't have a resource type "machineconfigs"
2026-07-31T09:38:55.031310328Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-31T09:38:55.074780475Z Wrote inspect data to must-gather.
2026-07-31T09:38:55.112769405Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-31T09:38:55.120386538Z INFO: 'previous-logs' folder not found on ip-10-0-141-129.ec2.internal, skipping...
2026-07-31T09:38:55.120492027Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-31T09:38:55.120560892Z INFO: Machine Config Daemon termination log collection complete.
2026-07-31T09:38:55.277031612Z Wrote inspect data to must-gather.
2026-07-31T09:38:55.309138870Z INFO: Getting status/config from prometheus-k8s-0
2026-07-31T09:38:55.334905973Z Wrote inspect data to must-gather.
2026-07-31T09:38:55.442103360Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-31T09:38:55.618615890Z INFO: INTERCONNECT MODE
2026-07-31T09:38:55.618648407Z INFO: Gathering ovn-kubernetes DBs
2026-07-31T09:38:55.658523586Z Wrote inspect data to must-gather.
2026-07-31T09:38:55.729691796Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-31T09:38:55.792593344Z INFO: Gathering OVN_Northbound from ovnkube-node-2q8cg...
2026-07-31T09:38:55.792854363Z INFO: Gathering OVN_Northbound from ovnkube-node-76zkr...
2026-07-31T09:38:55.793169563Z INFO: Gathering OVN_Northbound from ovnkube-node-lpb49...
2026-07-31T09:38:55.793428286Z INFO: Gathering OVN_Southbound from ovnkube-node-2q8cg...
2026-07-31T09:38:55.793751583Z INFO: Gathering OVN_Southbound from ovnkube-node-76zkr...
2026-07-31T09:38:55.794829892Z INFO: Gathering OVN_Southbound from ovnkube-node-lpb49...
2026-07-31T09:38:55.910896991Z Wrote inspect data to must-gather.
2026-07-31T09:38:56.065484843Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-31T09:38:56.121347027Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-31T09:38:56.126000428Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.161355611Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.164710565Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.164765938Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.176746065Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.184775801Z tar: Removing leading `/' from member names
2026-07-31T09:38:56.250737997Z Wrote inspect data to must-gather.
2026-07-31T09:38:56.332650349Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-31T09:38:56.340585855Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:56.507211789Z Wrote inspect data to must-gather.
2026-07-31T09:38:56.593229523Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:56.595857390Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-31T09:38:56.605978625Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-31T09:38:56.651808686Z INFO: Waiting for network log collection to complete ...
2026-07-31T09:38:56.651848320Z INFO: Waiting for ovnk database copies to complete ...
2026-07-31T09:38:56.652334536Z INFO: Copying ovnk databases complete.
2026-07-31T09:38:56.654423468Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-31T09:38:56.656806076Z ovnk_database_store/
2026-07-31T09:38:56.656806076Z ovnk_database_store/ovnkube-node-2q8cg_sbdb
2026-07-31T09:38:56.662230849Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-31T09:38:56.674084380Z ovnk_database_store/ovnkube-node-lpb49_nbdb
2026-07-31T09:38:56.680247360Z ovnk_database_store/ovnkube-node-76zkr_nbdb
2026-07-31T09:38:56.688726059Z ovnk_database_store/ovnkube-node-76zkr_sbdb
2026-07-31T09:38:56.709943543Z ovnk_database_store/ovnkube-node-2q8cg_nbdb
2026-07-31T09:38:56.715903406Z ovnk_database_store/ovnkube-node-lpb49_sbdb
2026-07-31T09:38:56.737623976Z INFO: Network log collection complete.
2026-07-31T09:38:56.755240447Z Wrote inspect data to must-gather.
2026-07-31T09:38:56.809875107Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:56.992387664Z Wrote inspect data to must-gather.
2026-07-31T09:38:57.016145882Z INFO: Getting status from alertmanager-main-0
2026-07-31T09:38:57.036578002Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:57.040653225Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-31T09:38:57.201331884Z W0731 09:38:57.201289    1538 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-31T09:38:57.206269027Z Wrote inspect data to must-gather.
2026-07-31T09:38:57.235097423Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:57.297760253Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T09:38:57.436156326Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:57.438668481Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-31T09:38:57.480310982Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-31T09:38:57.482612421Z INFO: Fallback to identify the container image from release info
2026-07-31T09:38:57.490895707Z Gathering data for ns/openshift-network-console...
2026-07-31T09:38:57.660646890Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:57.854659472Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:57.857044443Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-31T09:38:57.957512562Z Gathering data for ns/openshift-console-operator...
2026-07-31T09:38:57.986635635Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675
2026-07-31T09:38:58.030381036Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:58.127524885Z daemonset.apps/perf-node-gather-daemonset created
2026-07-31T09:38:58.230346222Z Wrote inspect data to /must-gather/istio.
2026-07-31T09:38:58.258704218Z Waiting for performance profile collector pods to become ready: 1
2026-07-31T09:38:58.386820308Z Gathering data for ns/openshift-console...
2026-07-31T09:38:58.903384723Z Done executing Istio gather script
2026-07-31T09:38:58.995889628Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-31T09:38:59.377662319Z Waiting for performance profile collector pods to become ready: 2
2026-07-31T09:38:59.503299048Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:38:59.583692719Z Gathering data for ns/openshift-dns-operator...
2026-07-31T09:38:59.925593475Z Gathering data for ns/openshift-dns...
2026-07-31T09:39:00.495772106Z Waiting for performance profile collector pods to become ready: 3
2026-07-31T09:39:00.595810910Z Gathering data for ns/openshift-image-registry...
2026-07-31T09:39:01.087039950Z Gathering data for ns/openshift-ingress-operator...
2026-07-31T09:39:01.469766415Z Gathering data for ns/openshift-ingress...
2026-07-31T09:39:01.617930377Z Waiting for performance profile collector pods to become ready: 4
2026-07-31T09:39:01.850147051Z Gathering data for ns/openshift-ingress-canary...
2026-07-31T09:39:02.304958665Z Gathering data for ns/openshift-insights...
2026-07-31T09:39:02.737988131Z Waiting for performance profile collector pods to become ready: 5
2026-07-31T09:39:03.849699560Z Waiting for performance profile collector pods to become ready: 6
2026-07-31T09:39:04.286349796Z Gathering data for ns/openshift-jobset-operator...
2026-07-31T09:39:04.507639162Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:39:04.961292043Z Waiting for performance profile collector pods to become ready: 7
2026-07-31T09:39:05.136799939Z Gathering data for ns/openshift-config...
2026-07-31T09:39:05.480264555Z Gathering data for ns/openshift-config-managed...
2026-07-31T09:39:05.948268784Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-31T09:39:06.081402828Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-31T09:39:06.278880608Z Gathering data for ns/openshift-kube-apiserver...
2026-07-31T09:39:06.337202522Z Collecting performance related data for node ip-10-0-128-204.ec2.internal
2026-07-31T09:39:06.347416082Z Collecting performance related data for node ip-10-0-141-129.ec2.internal
2026-07-31T09:39:06.360713213Z Collecting performance related data for node ip-10-0-140-243.ec2.internal
2026-07-31T09:39:06.610830494Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-31T09:39:07.020102905Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-31T09:39:07.398048800Z Gathering data for ns/openshift-kube-scheduler...
2026-07-31T09:39:07.773830447Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-31T09:39:08.227824560Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-31T09:39:08.599137227Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-31T09:39:08.983025273Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-31T09:39:09.513127843Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:39:09.618196091Z Gathering data for ns/openshift-multus...
2026-07-31T09:39:11.324363426Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-31T09:39:12.756113287Z Gathering data for ns/openshift-host-network...
2026-07-31T09:39:13.108194606Z Gathering data for ns/openshift-network-diagnostics...
2026-07-31T09:39:13.691980763Z Gathering data for ns/openshift-network-node-identity...
2026-07-31T09:39:14.148446797Z Gathering data for ns/openshift-network-operator...
2026-07-31T09:39:14.517830156Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:39:14.568233306Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-31T09:39:14.588652163Z Collecting kubelet logs for node ip-10-0-128-204.ec2.internal
2026-07-31T09:39:14.591095227Z Collecting kubelet logs for node ip-10-0-141-129.ec2.internal
2026-07-31T09:39:14.593177216Z Collecting kubelet logs for node ip-10-0-140-243.ec2.internal
2026-07-31T09:39:14.896344191Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-31T09:39:14.933241993Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8vczp namespace
2026-07-31T09:39:14.938198743Z INFO: Node performance data collection complete.
2026-07-31T09:39:15.446477877Z Gathering data for ns/openshift-apiserver-operator...
2026-07-31T09:39:15.813763341Z Gathering data for ns/openshift-apiserver...
2026-07-31T09:39:16.235883887Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-31T09:39:16.550454821Z Gathering data for ns/openshift-controller-manager...
2026-07-31T09:39:16.897489722Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-31T09:39:17.610274085Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-31T09:39:17.915364253Z Gathering data for ns/openshift-service-ca-operator...
2026-07-31T09:39:18.272132849Z Gathering data for ns/openshift-service-ca...
2026-07-31T09:39:18.687215768Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-31T09:39:19.522464454Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-31T09:39:19.648284713Z Wrote inspect data to must-gather.
2026-07-31T09:39:19.648329700Z error: inspection completed with the errors occurred while gathering data:
2026-07-31T09:39:19.648329700Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-31T09:39:19.776899651Z error: the server doesn't have a resource type "clusters"
2026-07-31T09:39:19.914325990Z Caches written to disk