2026-07-28T15:03:28.901819648Z [disk usage checker] Started 2026-07-28T15:03:28.905245376Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:29.096196645Z Gathering data for ns/openshift-cluster-version... 2026-07-28T15:03:29.223060175Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-28T15:03:29.310116596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T15:03:29.392695413Z Gathering data for ns/default... 2026-07-28T15:03:29.671075778Z Gathering data for ns/openshift... 2026-07-28T15:03:30.073435822Z Gathering data for ns/kube-system... 2026-07-28T15:03:30.743500904Z Gathering data for ns/openshift-etcd... 2026-07-28T15:03:31.038061502Z Wrote inspect data to must-gather. 2026-07-28T15:03:31.038092966Z error: inspection completed with the errors occurred while gathering data: 2026-07-28T15:03:31.038092966Z namespaces "assisted-installer" not found 2026-07-28T15:03:31.427829469Z Waiting on subprocesses to finish execution. 2026-07-28T15:03:31.440301870Z Executing Istio gather script 2026-07-28T15:03:31.447800177Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-28T15:03:31.448368419Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-28T15:03:31.457898513Z INFO: Collecting host service logs for crio 2026-07-28T15:03:31.458179289Z INFO: Collecting host service logs for kubelet 2026-07-28T15:03:31.458474268Z INFO: Collecting host service logs for rpm-ostreed 2026-07-28T15:03:31.458772599Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-28T15:03:31.459081597Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-28T15:03:31.459382657Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-28T15:03:31.459700098Z INFO: Collecting host service logs for NetworkManager 2026-07-28T15:03:31.460015257Z INFO: Collecting host service logs for openvswitch 2026-07-28T15:03:31.460354771Z INFO: Collecting host service logs for ovs-configuration 2026-07-28T15:03:31.460721983Z INFO: Collecting host service logs for ovsdb-server 2026-07-28T15:03:31.461050932Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-28T15:03:31.461613380Z INFO: Waiting for worker host service log collection to complete ... 2026-07-28T15:03:31.477015392Z INFO: Gathering HAProxy config files 2026-07-28T15:03:31.483895487Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-28T15:03:31.565369528Z INFO: Waiting for node performance related collection to complete ... 2026-07-28T15:03:31.845129845Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-569lv 2026-07-28T15:03:31.845190093Z insights-runtime-extractor-8wh6z 2026-07-28T15:03:31.845210766Z insights-runtime-extractor-nvmct 2026-07-28T15:03:31.845230171Z insights-runtime-extractor-pb2hp 2026-07-28T15:03:31.904311181Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-28T15:03:32.090931831Z No resources found in openshift-etcd namespace. 2026-07-28T15:03:32.137821845Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-28T15:03:32.258922000Z Wrote inspect data to must-gather. 2026-07-28T15:03:32.450082819Z INFO: "metallb-operator" not detected. Skipping. 2026-07-28T15:03:32.595340259Z No resources found 2026-07-28T15:03:32.615307620Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-28T15:03:32.615307620Z See 'oc rsync -h' for help and examples 2026-07-28T15:03:32.671907445Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-28T15:03:32.739617753Z INFO: Worker host service log collection to complete. 2026-07-28T15:03:32.774096579Z error: the server doesn't have a resource type "performanceprofile" 2026-07-28T15:03:32.777952860Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-28T15:03:32.778095631Z INFO: on-disk MachineConfig config collection complete. 2026-07-28T15:03:32.844126482Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-28T15:03:33.021945366Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-28T15:03:33.039292905Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.116727441Z ERROR: No running kube-apiserver pods found 2026-07-28T15:03:33.205831813Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-28T15:03:33.343657781Z Gathering data for ns/openshift-monitoring... 2026-07-28T15:03:33.407996841Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.642366733Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-28T15:03:33.711561469Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.720098092Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.786538771Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T15:03:33.934579571Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:33.935266269Z No resources found 2026-07-28T15:03:33.935784727Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.942662967Z INFO: 'previous-logs' folder not found on ip-10-0-129-173.ec2.internal, skipping... 2026-07-28T15:03:33.980699562Z Wrote inspect data to must-gather. 2026-07-28T15:03:33.990681486Z error: the server doesn't have a resource type "machineconfigs" 2026-07-28T15:03:34.021373558Z tar: Removing leading `/' from member names 2026-07-28T15:03:34.028464587Z INFO: HAProxy config collection complete. 2026-07-28T15:03:34.206778985Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-28T15:03:34.212183558Z INFO: Getting rules from prometheus-k8s-0 2026-07-28T15:03:34.255140182Z Wrote inspect data to must-gather. 2026-07-28T15:03:34.270088555Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-28T15:03:34.280225616Z INFO: 'previous-logs' folder not found on ip-10-0-136-242.ec2.internal, skipping... 2026-07-28T15:03:34.450970033Z Wrote inspect data to must-gather. 2026-07-28T15:03:34.522657077Z Wrote inspect data to must-gather. 2026-07-28T15:03:34.568147218Z INFO: 'previous-logs' folder not found on ip-10-0-137-40.ec2.internal, skipping... 2026-07-28T15:03:34.568189121Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-28T15:03:34.568189121Z INFO: Machine Config Daemon termination log collection complete. 2026-07-28T15:03:34.626090053Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-28T15:03:34.627940182Z INFO: Getting status/config from prometheus-k8s-0 2026-07-28T15:03:34.790480340Z Wrote inspect data to must-gather. 2026-07-28T15:03:34.836201884Z Wrote inspect data to must-gather. 2026-07-28T15:03:35.075489083Z Wrote inspect data to must-gather. 2026-07-28T15:03:35.086182451Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-28T15:03:35.092708785Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-28T15:03:35.099966424Z INFO: INTERCONNECT MODE 2026-07-28T15:03:35.099998208Z INFO: Gathering ovn-kubernetes DBs 2026-07-28T15:03:35.117872446Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-28T15:03:35.271293676Z INFO: Gathering OVN_Northbound from ovnkube-node-ph5b7... 2026-07-28T15:03:35.271510980Z INFO: Gathering OVN_Northbound from ovnkube-node-t9wdd... 2026-07-28T15:03:35.271736981Z INFO: Gathering OVN_Northbound from ovnkube-node-zhdl9... 2026-07-28T15:03:35.272019023Z INFO: Gathering OVN_Southbound from ovnkube-node-ph5b7... 2026-07-28T15:03:35.272476069Z INFO: Gathering OVN_Southbound from ovnkube-node-t9wdd... 2026-07-28T15:03:35.272823610Z INFO: Gathering OVN_Southbound from ovnkube-node-zhdl9... 2026-07-28T15:03:35.275966068Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:35.337860582Z Wrote inspect data to must-gather. 2026-07-28T15:03:35.371105881Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-28T15:03:35.615533501Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.618901911Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.644082173Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.654492936Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.666437250Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.670531486Z tar: Removing leading `/' from member names 2026-07-28T15:03:35.719375542Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:35.721529151Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-28T15:03:35.744124522Z Wrote inspect data to must-gather. 2026-07-28T15:03:35.754252512Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-28T15:03:35.931725556Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T15:03:35.957536035Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:36.020919948Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-28T15:03:36.083410390Z Wrote inspect data to must-gather. 2026-07-28T15:03:36.154238653Z Gathering data for ns/openshift-network-console... 2026-07-28T15:03:36.185058591Z INFO: Waiting for network log collection to complete ... 2026-07-28T15:03:36.185110249Z INFO: Waiting for ovnk database copies to complete ... 2026-07-28T15:03:36.185592458Z INFO: Copying ovnk databases complete. 2026-07-28T15:03:36.187554062Z 2.6M must-gather/network_logs/ovnk_database_store 2026-07-28T15:03:36.190680172Z ovnk_database_store/ 2026-07-28T15:03:36.190706454Z ovnk_database_store/ovnkube-node-zhdl9_nbdb 2026-07-28T15:03:36.197962185Z ovnk_database_store/ovnkube-node-t9wdd_sbdb 2026-07-28T15:03:36.208021442Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:36.211781676Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-28T15:03:36.216212901Z ovnk_database_store/ovnkube-node-ph5b7_nbdb 2026-07-28T15:03:36.223309678Z ovnk_database_store/ovnkube-node-ph5b7_sbdb 2026-07-28T15:03:36.243716755Z ovnk_database_store/ovnkube-node-t9wdd_nbdb 2026-07-28T15:03:36.250425753Z ovnk_database_store/ovnkube-node-zhdl9_sbdb 2026-07-28T15:03:36.269894720Z INFO: Network log collection complete. 2026-07-28T15:03:36.395871609Z Wrote inspect data to must-gather. 2026-07-28T15:03:36.430814965Z INFO: Getting status from alertmanager-main-0 2026-07-28T15:03:36.432072349Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:36.651938187Z Gathering data for ns/openshift-console-operator... 2026-07-28T15:03:36.657845104Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:36.659769453Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-28T15:03:36.719163042Z Wrote inspect data to must-gather. 2026-07-28T15:03:36.891511525Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:36.968832081Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-28T15:03:36.973905669Z INFO: Fallback to identify the container image from release info 2026-07-28T15:03:37.051967616Z Gathering data for ns/openshift-console... 2026-07-28T15:03:37.094416818Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:37.096909541Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-28T15:03:37.248393380Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:37.442149355Z Wrote inspect data to /must-gather/istio. 2026-07-28T15:03:37.534424189Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-28T15:03:37.885428929Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-28T15:03:38.010735867Z daemonset.apps/perf-node-gather-daemonset created 2026-07-28T15:03:38.044283269Z Gathering data for ns/openshift-dns-operator... 2026-07-28T15:03:38.137954338Z Done executing Istio gather script 2026-07-28T15:03:38.151317522Z Waiting for performance profile collector pods to become ready: 1 2026-07-28T15:03:38.327418302Z Gathering data for ns/openshift-dns... 2026-07-28T15:03:38.891978889Z Gathering data for ns/openshift-image-registry... 2026-07-28T15:03:38.942111062Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:39.262712213Z Waiting for performance profile collector pods to become ready: 2 2026-07-28T15:03:39.327077200Z Gathering data for ns/openshift-ingress-operator... 2026-07-28T15:03:39.619024407Z Gathering data for ns/openshift-ingress... 2026-07-28T15:03:39.903995594Z Gathering data for ns/openshift-ingress-canary... 2026-07-28T15:03:40.244516314Z Gathering data for ns/openshift-insights... 2026-07-28T15:03:40.365354007Z Waiting for performance profile collector pods to become ready: 3 2026-07-28T15:03:41.475518383Z Waiting for performance profile collector pods to become ready: 4 2026-07-28T15:03:41.918753186Z Gathering data for ns/openshift-jobset-operator... 2026-07-28T15:03:42.581279089Z Waiting for performance profile collector pods to become ready: 5 2026-07-28T15:03:42.590076315Z Gathering data for ns/openshift-config... 2026-07-28T15:03:42.863746936Z Gathering data for ns/openshift-config-managed... 2026-07-28T15:03:43.238523485Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-28T15:03:43.474752531Z Gathering data for ns/openshift-kube-apiserver... 2026-07-28T15:03:43.699653287Z Waiting for performance profile collector pods to become ready: 6 2026-07-28T15:03:43.725355756Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-28T15:03:43.949096758Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:43.973845010Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-28T15:03:44.229717726Z Gathering data for ns/openshift-kube-scheduler... 2026-07-28T15:03:44.460052350Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-28T15:03:44.712583120Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-28T15:03:44.812515529Z Waiting for performance profile collector pods to become ready: 7 2026-07-28T15:03:45.010137426Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-28T15:03:45.283703202Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-28T15:03:45.822151309Z Gathering data for ns/openshift-multus... 2026-07-28T15:03:45.911972244Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-28T15:03:46.118301335Z Collecting performance related data for node ip-10-0-137-40.ec2.internal 2026-07-28T15:03:46.125800997Z Collecting performance related data for node ip-10-0-136-242.ec2.internal 2026-07-28T15:03:46.133181638Z Collecting performance related data for node ip-10-0-129-173.ec2.internal 2026-07-28T15:03:47.388089941Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-28T15:03:48.620858477Z Gathering data for ns/openshift-host-network... 2026-07-28T15:03:48.951069988Z Gathering data for ns/openshift-network-diagnostics... 2026-07-28T15:03:48.954473155Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:49.460353128Z Gathering data for ns/openshift-network-node-identity... 2026-07-28T15:03:49.840770307Z Gathering data for ns/openshift-network-operator... 2026-07-28T15:03:50.214678885Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-28T15:03:50.465873401Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-28T15:03:50.917537643Z Gathering data for ns/openshift-apiserver-operator... 2026-07-28T15:03:51.186564766Z Gathering data for ns/openshift-apiserver... 2026-07-28T15:03:51.542920107Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-28T15:03:51.786147283Z Gathering data for ns/openshift-controller-manager... 2026-07-28T15:03:52.046639535Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-28T15:03:52.623643968Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-28T15:03:52.883949964Z Gathering data for ns/openshift-service-ca-operator... 2026-07-28T15:03:53.176458735Z Gathering data for ns/openshift-service-ca... 2026-07-28T15:03:53.513299657Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-28T15:03:53.734701746Z Collecting kubelet logs for node ip-10-0-137-40.ec2.internal 2026-07-28T15:03:53.736907191Z Collecting kubelet logs for node ip-10-0-136-242.ec2.internal 2026-07-28T15:03:53.739350824Z Collecting kubelet logs for node ip-10-0-129-173.ec2.internal 2026-07-28T15:03:53.959189096Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-28T15:03:54.082984753Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-dlh4n namespace 2026-07-28T15:03:54.087661493Z INFO: Node performance data collection complete. 2026-07-28T15:03:54.445303071Z Wrote inspect data to must-gather. 2026-07-28T15:03:54.445343939Z error: inspection completed with the errors occurred while gathering data: 2026-07-28T15:03:54.445343939Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-28T15:03:54.562350976Z error: the server doesn't have a resource type "clusters" 2026-07-28T15:03:54.750781410Z Caches written to disk