2026-07-27T17:56:03.701242164Z [disk usage checker] Started
2026-07-27T17:56:03.705227754Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:03.898732904Z Gathering data for ns/openshift-cluster-version...
2026-07-27T17:56:04.029626894Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T17:56:04.117792698Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T17:56:04.205979762Z Gathering data for ns/default...
2026-07-27T17:56:04.503863202Z Gathering data for ns/openshift...
2026-07-27T17:56:04.926151025Z Gathering data for ns/kube-system...
2026-07-27T17:56:05.599114948Z Gathering data for ns/openshift-etcd...
2026-07-27T17:56:05.883580234Z Wrote inspect data to must-gather.
2026-07-27T17:56:05.883607627Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T17:56:05.883607627Z     namespaces "assisted-installer" not found
2026-07-27T17:56:06.251788858Z Waiting on subprocesses to finish execution.
2026-07-27T17:56:06.262142520Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T17:56:06.263394543Z INFO: Collecting host service logs for crio
2026-07-27T17:56:06.263687309Z INFO: Collecting host service logs for kubelet
2026-07-27T17:56:06.264178356Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T17:56:06.264499434Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T17:56:06.264839308Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T17:56:06.265179896Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T17:56:06.265498904Z INFO: Collecting host service logs for NetworkManager
2026-07-27T17:56:06.265836728Z INFO: Collecting host service logs for openvswitch
2026-07-27T17:56:06.266172185Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T17:56:06.266595466Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T17:56:06.266931990Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T17:56:06.267194050Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T17:56:06.267312848Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T17:56:06.267744087Z Executing Istio gather script
2026-07-27T17:56:06.271761743Z INFO: Gathering HAProxy config files
2026-07-27T17:56:06.276796356Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T17:56:06.431225178Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T17:56:06.854364110Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T17:56:06.871924277Z Wrote inspect data to must-gather.
2026-07-27T17:56:07.004919784Z No resources found in openshift-etcd namespace.
2026-07-27T17:56:07.085609134Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-sc2h8
2026-07-27T17:56:07.085663431Z insights-runtime-extractor-f7b6z
2026-07-27T17:56:07.085682099Z insights-runtime-extractor-tx5sv
2026-07-27T17:56:07.085700263Z insights-runtime-extractor-vx5qb
2026-07-27T17:56:07.250132463Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T17:56:07.256572003Z No resources found
2026-07-27T17:56:07.275332119Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T17:56:07.275370574Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T17:56:07.449606976Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T17:56:07.538855832Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T17:56:07.962765202Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T17:56:07.962765202Z See 'oc rsync -h' for help and examples
2026-07-27T17:56:08.066632722Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T17:56:08.123309372Z No resources found
2026-07-27T17:56:08.195485215Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T17:56:08.228429318Z ERROR: No running kube-apiserver pods found
2026-07-27T17:56:08.280146858Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T17:56:08.294488250Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.298385800Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T17:56:08.434769313Z INFO: Worker host service log collection to complete.
2026-07-27T17:56:08.439603105Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T17:56:08.465758964Z INFO: 'previous-logs' folder not found on ip-10-0-128-74.ec2.internal, skipping...
2026-07-27T17:56:08.468029259Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T17:56:08.488728826Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.651236866Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.725269426Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:08.739907001Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.757840673Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.779285200Z Wrote inspect data to must-gather.
2026-07-27T17:56:08.780816982Z Gathering data for ns/openshift-monitoring...
2026-07-27T17:56:08.826336249Z tar: Removing leading `/' from member names
2026-07-27T17:56:08.832808599Z INFO: HAProxy config collection complete.
2026-07-27T17:56:08.895701663Z INFO: 'previous-logs' folder not found on ip-10-0-133-31.ec2.internal, skipping...
2026-07-27T17:56:08.936606652Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T17:56:08.976118003Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T17:56:09.084289926Z Wrote inspect data to must-gather.
2026-07-27T17:56:09.139382606Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T17:56:09.158833095Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T17:56:09.191144994Z INFO: 'previous-logs' folder not found on ip-10-0-133-79.ec2.internal, skipping...
2026-07-27T17:56:09.191178140Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T17:56:09.191211827Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T17:56:09.373006248Z Wrote inspect data to must-gather.
2026-07-27T17:56:09.378270864Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T17:56:09.406025669Z Wrote inspect data to must-gather.
2026-07-27T17:56:09.554186441Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T17:56:09.610025883Z Wrote inspect data to must-gather.
2026-07-27T17:56:09.768101048Z Wrote inspect data to must-gather.
2026-07-27T17:56:09.776289824Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T17:56:09.872570975Z INFO: INTERCONNECT MODE
2026-07-27T17:56:09.872604626Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T17:56:09.950979280Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T17:56:09.963938169Z Wrote inspect data to must-gather.
2026-07-27T17:56:10.004086590Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T17:56:10.025686205Z INFO: Gathering OVN_Northbound from ovnkube-node-55xcz...
2026-07-27T17:56:10.026009113Z INFO: Gathering OVN_Northbound from ovnkube-node-ltlnd...
2026-07-27T17:56:10.026301958Z INFO: Gathering OVN_Northbound from ovnkube-node-st78p...
2026-07-27T17:56:10.026796341Z INFO: Gathering OVN_Southbound from ovnkube-node-55xcz...
2026-07-27T17:56:10.027088840Z INFO: Gathering OVN_Southbound from ovnkube-node-ltlnd...
2026-07-27T17:56:10.027355331Z INFO: Gathering OVN_Southbound from ovnkube-node-st78p...
2026-07-27T17:56:10.238806905Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:10.334511747Z Wrote inspect data to must-gather.
2026-07-27T17:56:10.459520317Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.473545535Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.477057767Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.484071421Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.485212848Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.494909049Z tar: Removing leading `/' from member names
2026-07-27T17:56:10.505670112Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T17:56:10.529012087Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:10.532345875Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T17:56:10.593610715Z Wrote inspect data to must-gather.
2026-07-27T17:56:10.714761258Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T17:56:10.774901143Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:10.779851838Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T17:56:10.839931304Z Wrote inspect data to must-gather.
2026-07-27T17:56:10.954522280Z INFO: Waiting for network log collection to complete ...
2026-07-27T17:56:10.954545339Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T17:56:10.954956450Z INFO: Copying ovnk databases complete.
2026-07-27T17:56:10.957049484Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-27T17:56:10.961527806Z ovnk_database_store/
2026-07-27T17:56:10.961577429Z ovnk_database_store/ovnkube-node-st78p_nbdb
2026-07-27T17:56:10.968365245Z ovnk_database_store/ovnkube-node-ltlnd_sbdb
2026-07-27T17:56:10.985590524Z ovnk_database_store/ovnkube-node-ltlnd_nbdb
2026-07-27T17:56:10.991144777Z ovnk_database_store/ovnkube-node-55xcz_nbdb
2026-07-27T17:56:10.996714758Z ovnk_database_store/ovnkube-node-55xcz_sbdb
2026-07-27T17:56:11.002536484Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:11.006480651Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T17:56:11.015303225Z ovnk_database_store/ovnkube-node-st78p_sbdb
2026-07-27T17:56:11.032776917Z INFO: Network log collection complete.
2026-07-27T17:56:11.082477962Z Wrote inspect data to must-gather.
2026-07-27T17:56:11.177161224Z INFO: Getting status from alertmanager-main-0
2026-07-27T17:56:11.216433630Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:11.289184168Z W0727 17:56:11.289135    1560 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-27T17:56:11.295620997Z Wrote inspect data to must-gather.
2026-07-27T17:56:11.322827869Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T17:56:11.427298181Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:11.429529624Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T17:56:11.505905122Z Gathering data for ns/openshift-network-console...
2026-07-27T17:56:11.552253277Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T17:56:11.554749189Z INFO: Fallback to identify the container image from release info
2026-07-27T17:56:11.659388965Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:11.850460533Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:11.852928082Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T17:56:11.906891387Z Gathering data for ns/openshift-console-operator...
2026-07-27T17:56:12.015676356Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:12.195476586Z Wrote inspect data to /must-gather/istio.
2026-07-27T17:56:12.282829900Z Gathering data for ns/openshift-console...
2026-07-27T17:56:12.296542365Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T17:56:12.423400523Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T17:56:12.534310624Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T17:56:12.798196244Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T17:56:12.907680769Z Done executing Istio gather script
2026-07-27T17:56:13.283469793Z Gathering data for ns/openshift-dns-operator...
2026-07-27T17:56:13.536118049Z Gathering data for ns/openshift-dns...
2026-07-27T17:56:13.633802990Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T17:56:13.730886558Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:14.103270827Z Gathering data for ns/openshift-image-registry...
2026-07-27T17:56:14.581180722Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T17:56:14.731386262Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T17:56:14.934830596Z Gathering data for ns/openshift-ingress...
2026-07-27T17:56:15.251821658Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T17:56:15.625334561Z Gathering data for ns/openshift-insights...
2026-07-27T17:56:15.827918211Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T17:56:16.937785813Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T17:56:17.414257678Z Gathering data for ns/openshift-jobset-operator...
2026-07-27T17:56:18.047734918Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T17:56:18.193055044Z Gathering data for ns/openshift-config...
2026-07-27T17:56:18.439097200Z Gathering data for ns/openshift-config-managed...
2026-07-27T17:56:18.735550802Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:18.832278586Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T17:56:19.087263191Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T17:56:19.186136953Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T17:56:19.405339521Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T17:56:19.691908343Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T17:56:20.025277515Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T17:56:20.307236143Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T17:56:20.332148223Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T17:56:20.580059865Z Collecting performance related data for node ip-10-0-133-79.ec2.internal
2026-07-27T17:56:20.589537589Z Collecting performance related data for node ip-10-0-133-31.ec2.internal
2026-07-27T17:56:20.597876266Z Collecting performance related data for node ip-10-0-128-74.ec2.internal
2026-07-27T17:56:20.664238600Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T17:56:21.044217429Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T17:56:21.433269586Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T17:56:22.104097040Z Gathering data for ns/openshift-multus...
2026-07-27T17:56:23.579763673Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T17:56:23.740517919Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:24.803253901Z Gathering data for ns/openshift-host-network...
2026-07-27T17:56:25.115472505Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T17:56:25.734195340Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T17:56:26.171862295Z Gathering data for ns/openshift-network-operator...
2026-07-27T17:56:26.559215156Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T17:56:26.811570075Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T17:56:27.279718322Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T17:56:27.588089817Z Gathering data for ns/openshift-apiserver...
2026-07-27T17:56:27.998164856Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T17:56:28.270694134Z Gathering data for ns/openshift-controller-manager...
2026-07-27T17:56:28.540109508Z Collecting kubelet logs for node ip-10-0-133-79.ec2.internal
2026-07-27T17:56:28.540142568Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T17:56:28.541986992Z Collecting kubelet logs for node ip-10-0-133-31.ec2.internal
2026-07-27T17:56:28.543785484Z Collecting kubelet logs for node ip-10-0-128-74.ec2.internal
2026-07-27T17:56:28.745997710Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-27T17:56:28.878398964Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-fg5h7 namespace
2026-07-27T17:56:28.884620511Z INFO: Node performance data collection complete.
2026-07-27T17:56:29.239415498Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T17:56:29.515457326Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T17:56:29.815240452Z Gathering data for ns/openshift-service-ca...
2026-07-27T17:56:30.191209272Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T17:56:31.056151531Z Wrote inspect data to must-gather.
2026-07-27T17:56:31.056199644Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T17:56:31.056199644Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T17:56:31.178870679Z error: the server doesn't have a resource type "clusters"
2026-07-27T17:56:31.353107311Z Caches written to disk