2026-07-28T17:12:08.808319359Z [disk usage checker] Started
2026-07-28T17:12:08.811831476Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:09.007804738Z Gathering data for ns/openshift-cluster-version...
2026-07-28T17:12:09.152996935Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T17:12:09.262230646Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:12:09.358030122Z Gathering data for ns/default...
2026-07-28T17:12:09.710174323Z Gathering data for ns/openshift...
2026-07-28T17:12:10.227269909Z Gathering data for ns/kube-system...
2026-07-28T17:12:10.989711246Z Gathering data for ns/openshift-etcd...
2026-07-28T17:12:11.358096951Z Wrote inspect data to must-gather.
2026-07-28T17:12:11.358128628Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T17:12:11.358128628Z     namespaces "assisted-installer" not found
2026-07-28T17:12:11.635772146Z Waiting on subprocesses to finish execution.
2026-07-28T17:12:11.644218158Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T17:12:11.649481932Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T17:12:11.649929045Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T17:12:11.652691397Z Executing Istio gather script
2026-07-28T17:12:11.669398606Z INFO: Collecting host service logs for crio
2026-07-28T17:12:11.669725575Z INFO: Collecting host service logs for kubelet
2026-07-28T17:12:11.670081263Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T17:12:11.670407893Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T17:12:11.670717521Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T17:12:11.671025894Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T17:12:11.671391553Z INFO: Collecting host service logs for NetworkManager
2026-07-28T17:12:11.671693803Z INFO: Collecting host service logs for openvswitch
2026-07-28T17:12:11.671987634Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T17:12:11.672429685Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T17:12:11.672788444Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T17:12:11.673195340Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T17:12:11.682617558Z INFO: Gathering HAProxy config files
2026-07-28T17:12:11.743637789Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T17:12:12.273165107Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T17:12:12.382364981Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-9hwqx
2026-07-28T17:12:12.382364981Z insights-runtime-extractor-6fjcf
2026-07-28T17:12:12.382364981Z insights-runtime-extractor-chq49
2026-07-28T17:12:12.382364981Z insights-runtime-extractor-jxrnk
2026-07-28T17:12:12.719519296Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T17:12:12.728282106Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T17:12:12.774972529Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T17:12:12.893710461Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T17:12:12.921193902Z Wrote inspect data to must-gather.
2026-07-28T17:12:13.134882106Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T17:12:13.134882106Z See 'oc rsync -h' for help and examples
2026-07-28T17:12:13.140594328Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T17:12:13.151694818Z No resources found
2026-07-28T17:12:13.170491116Z No resources found in openshift-etcd namespace.
2026-07-28T17:12:13.195190687Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T17:12:13.209166632Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T17:12:13.209246625Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T17:12:13.403472971Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T17:12:13.522297154Z ERROR: No running kube-apiserver pods found
2026-07-28T17:12:13.734795619Z Wrote inspect data to must-gather.
2026-07-28T17:12:13.813297783Z Wrote inspect data to must-gather.
2026-07-28T17:12:13.850977143Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:13.884312951Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:12:13.995487014Z Gathering data for ns/openshift-monitoring...
2026-07-28T17:12:14.002624428Z INFO: 'previous-logs' folder not found on ip-10-0-139-46.ec2.internal, skipping...
2026-07-28T17:12:14.004684906Z No resources found
2026-07-28T17:12:14.130414348Z INFO: Worker host service log collection to complete.
2026-07-28T17:12:14.231538461Z Wrote inspect data to must-gather.
2026-07-28T17:12:14.332523285Z Wrote inspect data to must-gather.
2026-07-28T17:12:14.344460115Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T17:12:14.369397388Z Wrote inspect data to must-gather.
2026-07-28T17:12:14.392499392Z tar: Removing leading `/' from member names
2026-07-28T17:12:14.398380142Z INFO: HAProxy config collection complete.
2026-07-28T17:12:14.429087744Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T17:12:14.481668283Z Wrote inspect data to must-gather.
2026-07-28T17:12:14.593345740Z INFO: 'previous-logs' folder not found on ip-10-0-140-154.ec2.internal, skipping...
2026-07-28T17:12:14.607345588Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T17:12:14.664364535Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T17:12:14.715299759Z Wrote inspect data to must-gather.
2026-07-28T17:12:14.900732863Z INFO: 'previous-logs' folder not found on ip-10-0-140-22.ec2.internal, skipping...
2026-07-28T17:12:14.900974034Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T17:12:14.901177843Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T17:12:14.941722650Z Wrote inspect data to must-gather.
2026-07-28T17:12:15.024667708Z Wrote inspect data to must-gather.
2026-07-28T17:12:15.077849552Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T17:12:15.126563790Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T17:12:15.165116511Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T17:12:15.312513221Z Wrote inspect data to must-gather.
2026-07-28T17:12:15.361828958Z Wrote inspect data to must-gather.
2026-07-28T17:12:15.527450990Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T17:12:15.617573937Z Wrote inspect data to must-gather.
2026-07-28T17:12:15.670166886Z INFO: INTERCONNECT MODE
2026-07-28T17:12:15.670280280Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T17:12:15.805304924Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T17:12:15.819403838Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T17:12:15.851939609Z INFO: Gathering OVN_Northbound from ovnkube-node-s9vmh...
2026-07-28T17:12:15.852154218Z INFO: Gathering OVN_Northbound from ovnkube-node-sbf8h...
2026-07-28T17:12:15.852368910Z INFO: Gathering OVN_Northbound from ovnkube-node-xx7xl...
2026-07-28T17:12:15.852645172Z INFO: Gathering OVN_Southbound from ovnkube-node-s9vmh...
2026-07-28T17:12:15.853122064Z INFO: Gathering OVN_Southbound from ovnkube-node-sbf8h...
2026-07-28T17:12:15.853412528Z INFO: Gathering OVN_Southbound from ovnkube-node-xx7xl...
2026-07-28T17:12:15.865137024Z Wrote inspect data to must-gather.
2026-07-28T17:12:16.216197551Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.221303222Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.243224032Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T17:12:16.271401635Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.280132089Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.290128221Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.316174367Z tar: Removing leading `/' from member names
2026-07-28T17:12:16.340236103Z Wrote inspect data to must-gather.
2026-07-28T17:12:16.436018847Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T17:12:16.589413799Z W0728 17:12:16.589358    1410 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T17:12:16.597075962Z Wrote inspect data to must-gather.
2026-07-28T17:12:16.598033678Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T17:12:16.659020545Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:12:16.667402092Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:16.816635810Z INFO: Waiting for network log collection to complete ...
2026-07-28T17:12:16.816667266Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T17:12:16.817062658Z INFO: Copying ovnk databases complete.
2026-07-28T17:12:16.819380327Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-07-28T17:12:16.821946002Z ovnk_database_store/
2026-07-28T17:12:16.821946002Z ovnk_database_store/ovnkube-node-sbf8h_nbdb
2026-07-28T17:12:16.834835305Z ovnk_database_store/ovnkube-node-sbf8h_sbdb
2026-07-28T17:12:16.860484802Z ovnk_database_store/ovnkube-node-xx7xl_sbdb
2026-07-28T17:12:16.871477838Z Gathering data for ns/openshift-network-console...
2026-07-28T17:12:16.882668536Z ovnk_database_store/ovnkube-node-s9vmh_nbdb
2026-07-28T17:12:16.890768563Z ovnk_database_store/ovnkube-node-s9vmh_sbdb
2026-07-28T17:12:16.912684382Z ovnk_database_store/ovnkube-node-xx7xl_nbdb
2026-07-28T17:12:16.923394209Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:16.923744444Z INFO: Network log collection complete.
2026-07-28T17:12:16.926578233Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T17:12:16.962130574Z Wrote inspect data to must-gather.
2026-07-28T17:12:17.046493646Z INFO: Getting status from alertmanager-main-0
2026-07-28T17:12:17.160916605Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:17.285356520Z Wrote inspect data to must-gather.
2026-07-28T17:12:17.382176606Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:17.385379826Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T17:12:17.426014164Z Gathering data for ns/openshift-console-operator...
2026-07-28T17:12:17.592992317Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T17:12:17.596218199Z INFO: Fallback to identify the container image from release info
2026-07-28T17:12:17.605958580Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:17.814989313Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:17.818779015Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T17:12:17.839918645Z Gathering data for ns/openshift-console...
2026-07-28T17:12:18.034051179Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:18.214796862Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T17:12:18.250863150Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:18.254397760Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T17:12:18.370964408Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T17:12:18.400562989Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T17:12:18.448471855Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:18.526587244Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T17:12:18.645907617Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:18.649238040Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T17:12:18.846988919Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:18.914849414Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:19.020169880Z Gathering data for ns/openshift-dns-operator...
2026-07-28T17:12:19.079884239Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:19.083554132Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T17:12:19.255366418Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:19.371475954Z Gathering data for ns/openshift-dns...
2026-07-28T17:12:19.453460226Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:12:19.662332600Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T17:12:20.074621432Z Gathering data for ns/openshift-image-registry...
2026-07-28T17:12:20.157556334Z Done executing Istio gather script
2026-07-28T17:12:20.536704211Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T17:12:20.787640438Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T17:12:20.905056486Z Gathering data for ns/openshift-ingress...
2026-07-28T17:12:21.224581039Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T17:12:21.646904391Z Gathering data for ns/openshift-insights...
2026-07-28T17:12:21.909836963Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T17:12:23.024055379Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T17:12:23.920255265Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:23.980957765Z Gathering data for ns/kserve...
2026-07-28T17:12:24.147558045Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T17:12:25.282116248Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T17:12:25.600564436Z Gathering data for ns/openshift-config...
2026-07-28T17:12:25.865836591Z Gathering data for ns/openshift-config-managed...
2026-07-28T17:12:26.274931516Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T17:12:26.414526519Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T17:12:26.577252908Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T17:12:26.683194349Z Collecting performance related data for node ip-10-0-140-154.ec2.internal
2026-07-28T17:12:26.696465299Z Collecting performance related data for node ip-10-0-139-46.ec2.internal
2026-07-28T17:12:26.703682618Z Collecting performance related data for node ip-10-0-140-22.ec2.internal
2026-07-28T17:12:26.960468737Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T17:12:27.310325021Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T17:12:27.700090824Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T17:12:28.056887693Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T17:12:28.404547788Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T17:12:28.749397702Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T17:12:28.925769742Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:29.097288225Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T17:12:29.692312010Z Gathering data for ns/openshift-multus...
2026-07-28T17:12:31.096567787Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T17:12:32.315471923Z Gathering data for ns/openshift-host-network...
2026-07-28T17:12:32.626619529Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T17:12:33.140915219Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T17:12:33.524153724Z Gathering data for ns/openshift-network-operator...
2026-07-28T17:12:33.896044389Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T17:12:33.932715727Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T17:12:34.176162839Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T17:12:34.664840060Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T17:12:34.758959558Z Collecting kubelet logs for node ip-10-0-140-154.ec2.internal
2026-07-28T17:12:34.760927841Z Collecting kubelet logs for node ip-10-0-139-46.ec2.internal
2026-07-28T17:12:34.762683989Z Collecting kubelet logs for node ip-10-0-140-22.ec2.internal
2026-07-28T17:12:35.014917076Z Gathering data for ns/openshift-apiserver...
2026-07-28T17:12:35.147978426Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-464v8 namespace
2026-07-28T17:12:35.152654001Z INFO: Node performance data collection complete.
2026-07-28T17:12:35.467179300Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T17:12:35.725219045Z Gathering data for ns/openshift-controller-manager...
2026-07-28T17:12:36.020365800Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T17:12:36.664368568Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T17:12:36.939392877Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T17:12:37.250979375Z Gathering data for ns/openshift-service-ca...
2026-07-28T17:12:37.614470199Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T17:12:38.484041461Z Wrote inspect data to must-gather.
2026-07-28T17:12:38.484087384Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T17:12:38.484087384Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T17:12:38.615767192Z error: the server doesn't have a resource type "clusters"
2026-07-28T17:12:38.755988279Z Caches written to disk