2026-09-11T17:13:29.633555139Z [disk usage checker] Started 2026-09-11T17:13:29.637408441Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:29.982467992Z Gathering data for ns/openshift-cluster-version... 2026-09-11T17:13:30.157054686Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-11T17:13:30.267174307Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T17:13:30.360171082Z Gathering data for ns/default... 2026-09-11T17:13:30.693957790Z Gathering data for ns/openshift... 2026-09-11T17:13:31.161739089Z Gathering data for ns/kube-system... 2026-09-11T17:13:31.933030889Z Gathering data for ns/openshift-etcd... 2026-09-11T17:13:32.292600206Z Wrote inspect data to must-gather. 2026-09-11T17:13:32.292635556Z error: inspection completed with the errors occurred while gathering data: 2026-09-11T17:13:32.292635556Z namespaces "assisted-installer" not found 2026-09-11T17:13:32.640381951Z Waiting on subprocesses to finish execution. 2026-09-11T17:13:32.648689404Z Executing Istio gather script 2026-09-11T17:13:32.650253253Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-11T17:13:32.653758086Z INFO: Gathering HAProxy config files 2026-09-11T17:13:32.661485970Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-11T17:13:32.663716003Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-11T17:13:32.675097347Z INFO: Collecting host service logs for crio 2026-09-11T17:13:32.675598482Z INFO: Collecting host service logs for kubelet 2026-09-11T17:13:32.675979309Z INFO: Collecting host service logs for rpm-ostreed 2026-09-11T17:13:32.676264280Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-11T17:13:32.676710825Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-11T17:13:32.677147440Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-11T17:13:32.677434797Z INFO: Collecting host service logs for NetworkManager 2026-09-11T17:13:32.677893863Z INFO: Collecting host service logs for openvswitch 2026-09-11T17:13:32.678236785Z INFO: Collecting host service logs for ovs-configuration 2026-09-11T17:13:32.678550133Z INFO: Collecting host service logs for ovsdb-server 2026-09-11T17:13:32.678840942Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-11T17:13:32.679247226Z INFO: Waiting for worker host service log collection to complete ... 2026-09-11T17:13:32.843151358Z INFO: Waiting for node performance related collection to complete ... 2026-09-11T17:13:33.227533677Z No resources found in openshift-etcd namespace. 2026-09-11T17:13:33.679145763Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-11T17:13:33.738440606Z No resources found 2026-09-11T17:13:33.819105114Z INFO: Collecting Insights Archives from insights-operator-5844b856db-tfhct 2026-09-11T17:13:33.819105114Z insights-runtime-extractor-bbldx 2026-09-11T17:13:33.819105114Z insights-runtime-extractor-ttkdv 2026-09-11T17:13:33.819105114Z insights-runtime-extractor-xnvkj 2026-09-11T17:13:33.956349526Z error: the server doesn't have a resource type "performanceprofile" 2026-09-11T17:13:33.968924127Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-11T17:13:34.003912890Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-11T17:13:34.010392242Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-11T17:13:34.081757471Z Wrote inspect data to must-gather. 2026-09-11T17:13:34.252485798Z INFO: "metallb-operator" not detected. Skipping. 2026-09-11T17:13:34.264104705Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-11T17:13:34.264104705Z INFO: on-disk MachineConfig config collection complete. 2026-09-11T17:13:34.542095373Z INFO: Worker host service log collection to complete. 2026-09-11T17:13:34.789964831Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:34.835222434Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-11T17:13:34.972577911Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-11T17:13:34.976302188Z ERROR: No running kube-apiserver pods found 2026-09-11T17:13:35.010704634Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-11T17:13:35.010704634Z See 'oc rsync -h' for help and examples 2026-09-11T17:13:35.031181921Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T17:13:35.208269107Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.239306539Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.262998060Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.305727656Z No resources found 2026-09-11T17:13:35.316751601Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-11T17:13:35.318645917Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.389622420Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.443870256Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.445026552Z INFO: 'previous-logs' folder not found on ip-10-0-139-25.ec2.internal, skipping... 2026-09-11T17:13:35.483476723Z tar: Removing leading `/' from member names 2026-09-11T17:13:35.490202379Z INFO: HAProxy config collection complete. 2026-09-11T17:13:35.533161662Z error: the server doesn't have a resource type "machineconfigs" 2026-09-11T17:13:35.731095905Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-11T17:13:35.766770092Z INFO: 'previous-logs' folder not found on ip-10-0-142-248.ec2.internal, skipping... 2026-09-11T17:13:35.782710999Z Wrote inspect data to must-gather. 2026-09-11T17:13:35.785848682Z Gathering data for ns/openshift-monitoring... 2026-09-11T17:13:35.869123708Z INFO: Getting rules from prometheus-k8s-0 2026-09-11T17:13:36.037894177Z Wrote inspect data to must-gather. 2026-09-11T17:13:36.080117977Z INFO: 'previous-logs' folder not found on ip-10-0-143-197.ec2.internal, skipping... 2026-09-11T17:13:36.080165151Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-11T17:13:36.080192701Z INFO: Machine Config Daemon termination log collection complete. 2026-09-11T17:13:36.116325735Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-11T17:13:36.116535493Z Wrote inspect data to must-gather. 2026-09-11T17:13:36.214385588Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-11T17:13:36.333795683Z INFO: Getting status/config from prometheus-k8s-0 2026-09-11T17:13:36.379845892Z Wrote inspect data to must-gather. 2026-09-11T17:13:36.463619800Z Wrote inspect data to must-gather. 2026-09-11T17:13:36.705679798Z Wrote inspect data to must-gather. 2026-09-11T17:13:36.708167564Z INFO: INTERCONNECT MODE 2026-09-11T17:13:36.708198285Z INFO: Gathering ovn-kubernetes DBs 2026-09-11T17:13:36.764145316Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-11T17:13:36.885654436Z INFO: Gathering OVN_Northbound from ovnkube-node-gfkd5... 2026-09-11T17:13:36.885979202Z INFO: Gathering OVN_Northbound from ovnkube-node-gkzm7... 2026-09-11T17:13:36.886218613Z INFO: Gathering OVN_Northbound from ovnkube-node-sz5sg... 2026-09-11T17:13:36.886489139Z INFO: Gathering OVN_Southbound from ovnkube-node-gfkd5... 2026-09-11T17:13:36.886753514Z INFO: Gathering OVN_Southbound from ovnkube-node-gkzm7... 2026-09-11T17:13:36.887029842Z INFO: Gathering OVN_Southbound from ovnkube-node-sz5sg... 2026-09-11T17:13:36.967520763Z Wrote inspect data to must-gather. 2026-09-11T17:13:37.050126518Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-11T17:13:37.191189245Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.216714247Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.232866388Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.244965802Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.269103692Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.269131492Z tar: Removing leading `/' from member names 2026-09-11T17:13:37.321623906Z Wrote inspect data to must-gather. 2026-09-11T17:13:37.353472277Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-11T17:13:37.601290153Z Wrote inspect data to must-gather. 2026-09-11T17:13:37.634549094Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-11T17:13:37.660090294Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-11T17:13:37.669146822Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-11T17:13:37.784063734Z INFO: Waiting for network log collection to complete ... 2026-09-11T17:13:37.784178764Z INFO: Waiting for ovnk database copies to complete ... 2026-09-11T17:13:37.784652707Z INFO: Copying ovnk databases complete. 2026-09-11T17:13:37.786852069Z 4.8M must-gather/network_logs/ovnk_database_store 2026-09-11T17:13:37.790009253Z ovnk_database_store/ 2026-09-11T17:13:37.790057047Z ovnk_database_store/ovnkube-node-gkzm7_nbdb 2026-09-11T17:13:37.802007970Z ovnk_database_store/ovnkube-node-gfkd5_nbdb 2026-09-11T17:13:37.815143672Z ovnk_database_store/ovnkube-node-sz5sg_sbdb 2026-09-11T17:13:37.848868758Z ovnk_database_store/ovnkube-node-gfkd5_sbdb 2026-09-11T17:13:37.850537447Z Wrote inspect data to must-gather. 2026-09-11T17:13:37.858913010Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:37.877378913Z ovnk_database_store/ovnkube-node-sz5sg_nbdb 2026-09-11T17:13:37.888152229Z ovnk_database_store/ovnkube-node-gkzm7_sbdb 2026-09-11T17:13:37.915697334Z INFO: Network log collection complete. 2026-09-11T17:13:38.082427829Z Wrote inspect data to must-gather. 2026-09-11T17:13:38.102575856Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:38.104950939Z INFO: Getting status from alertmanager-main-0 2026-09-11T17:13:38.108294238Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-11T17:13:38.311241974Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T17:13:38.332976111Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:38.380812792Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-11T17:13:38.384054031Z INFO: Fallback to identify the container image from release info 2026-09-11T17:13:38.518770463Z Gathering data for ns/openshift-network-console... 2026-09-11T17:13:38.568212076Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:38.571206151Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-11T17:13:38.759358635Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:38.966696629Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a 2026-09-11T17:13:38.982449177Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:38.986505970Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-11T17:13:39.046831535Z Gathering data for ns/openshift-console-operator... 2026-09-11T17:13:39.113546639Z daemonset.apps/perf-node-gather-daemonset created 2026-09-11T17:13:39.229511623Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:39.263354047Z Waiting for performance profile collector pods to become ready: 1 2026-09-11T17:13:39.431468256Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:39.434953524Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-09-11T17:13:39.490200756Z Gathering data for ns/openshift-console... 2026-09-11T17:13:39.642688711Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:39.824861188Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:39.836750555Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:39.840695789Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-09-11T17:13:40.033196669Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:40.058713519Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-11T17:13:40.234323700Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:40.237755232Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-11T17:13:40.403112398Z Waiting for performance profile collector pods to become ready: 2 2026-09-11T17:13:40.434461965Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:40.642243170Z Gathering data for ns/openshift-dns-operator... 2026-09-11T17:13:40.663535402Z Wrote inspect data to /must-gather/istio. 2026-09-11T17:13:40.971089857Z Gathering data for ns/openshift-dns... 2026-09-11T17:13:41.373227087Z Done executing Istio gather script 2026-09-11T17:13:41.541270514Z Waiting for performance profile collector pods to become ready: 3 2026-09-11T17:13:41.661044609Z Gathering data for ns/openshift-image-registry... 2026-09-11T17:13:42.174530094Z Gathering data for ns/openshift-ingress-operator... 2026-09-11T17:13:42.534666171Z Gathering data for ns/openshift-ingress... 2026-09-11T17:13:42.689460807Z Waiting for performance profile collector pods to become ready: 4 2026-09-11T17:13:42.907963113Z Gathering data for ns/openshift-ingress-canary... 2026-09-11T17:13:43.325336010Z Gathering data for ns/openshift-insights... 2026-09-11T17:13:43.808188600Z Waiting for performance profile collector pods to become ready: 5 2026-09-11T17:13:44.829805608Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:44.927688914Z Waiting for performance profile collector pods to become ready: 6 2026-09-11T17:13:45.765494425Z Gathering data for ns/kserve... 2026-09-11T17:13:46.052094651Z Waiting for performance profile collector pods to become ready: 7 2026-09-11T17:13:47.176966342Z Waiting for performance profile collector pods to become ready: 8 2026-09-11T17:13:47.913290558Z Gathering data for ns/openshift-config... 2026-09-11T17:13:48.223392421Z Gathering data for ns/openshift-config-managed... 2026-09-11T17:13:48.319538275Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-11T17:13:48.569718784Z Collecting performance related data for node ip-10-0-142-248.ec2.internal 2026-09-11T17:13:48.577470892Z Collecting performance related data for node ip-10-0-143-197.ec2.internal 2026-09-11T17:13:48.586772030Z Collecting performance related data for node ip-10-0-139-25.ec2.internal 2026-09-11T17:13:48.759326437Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-11T17:13:49.110853591Z Gathering data for ns/openshift-kube-apiserver... 2026-09-11T17:13:49.450623190Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-11T17:13:49.835784669Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:49.864140301Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-11T17:13:50.295982591Z Gathering data for ns/openshift-kube-scheduler... 2026-09-11T17:13:50.608088172Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-11T17:13:50.978882697Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-11T17:13:51.419739884Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-11T17:13:51.790541394Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-11T17:13:52.457922995Z Gathering data for ns/openshift-multus... 2026-09-11T17:13:54.162480589Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-11T17:13:54.840841450Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:55.556227452Z Gathering data for ns/openshift-host-network... 2026-09-11T17:13:55.911856743Z Gathering data for ns/openshift-network-diagnostics... 2026-09-11T17:13:56.623058944Z Gathering data for ns/openshift-network-node-identity... 2026-09-11T17:13:56.847897928Z Collecting kubelet logs for node ip-10-0-142-248.ec2.internal 2026-09-11T17:13:56.849804263Z Collecting kubelet logs for node ip-10-0-143-197.ec2.internal 2026-09-11T17:13:56.852239547Z Collecting kubelet logs for node ip-10-0-139-25.ec2.internal 2026-09-11T17:13:57.175600688Z Gathering data for ns/openshift-network-operator... 2026-09-11T17:13:57.311980106Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-w5hdz namespace 2026-09-11T17:13:57.318627448Z INFO: Node performance data collection complete. 2026-09-11T17:13:57.645974250Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-11T17:13:57.941184569Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-11T17:13:58.523219432Z Gathering data for ns/openshift-apiserver-operator... 2026-09-11T17:13:58.864643612Z Gathering data for ns/openshift-apiserver... 2026-09-11T17:13:59.377379577Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-11T17:13:59.696386272Z Gathering data for ns/openshift-controller-manager... 2026-09-11T17:13:59.845671917Z [disk usage checker] Volume usage percentage: current = 35 ; allowed = 70 2026-09-11T17:13:59.991988636Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-11T17:14:00.672018131Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-11T17:14:00.985278978Z Gathering data for ns/openshift-service-ca-operator... 2026-09-11T17:14:01.326770984Z Gathering data for ns/openshift-service-ca... 2026-09-11T17:14:01.713196962Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-11T17:14:02.733601127Z Wrote inspect data to must-gather. 2026-09-11T17:14:02.733649051Z error: inspection completed with the errors occurred while gathering data: 2026-09-11T17:14:02.733649051Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-11T17:14:02.871625568Z error: the server doesn't have a resource type "clusters" 2026-09-11T17:14:03.107103290Z Caches written to disk