2026-07-02T13:30:23.287896875Z [disk usage checker] Started 2026-07-02T13:30:23.291778789Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:23.482553533Z Gathering data for ns/openshift-cluster-version... 2026-07-02T13:30:23.612168582Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-02T13:30:23.702697818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-02T13:30:23.797540120Z Gathering data for ns/default... 2026-07-02T13:30:24.123336734Z Gathering data for ns/openshift... 2026-07-02T13:30:24.554058167Z Gathering data for ns/kube-system... 2026-07-02T13:30:25.292359968Z Gathering data for ns/openshift-etcd... 2026-07-02T13:30:25.617586625Z Wrote inspect data to must-gather. 2026-07-02T13:30:25.617701006Z error: inspection completed with the errors occurred while gathering data: 2026-07-02T13:30:25.617701006Z namespaces "assisted-installer" not found 2026-07-02T13:30:26.074007219Z Waiting on subprocesses to finish execution. 2026-07-02T13:30:26.086554336Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-02T13:30:26.093458934Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-02T13:30:26.099765381Z INFO: Gathering HAProxy config files 2026-07-02T13:30:26.102289329Z INFO: Collecting host service logs for crio 2026-07-02T13:30:26.102503096Z INFO: Collecting host service logs for kubelet 2026-07-02T13:30:26.102565304Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-02T13:30:26.102817137Z INFO: Collecting host service logs for rpm-ostreed 2026-07-02T13:30:26.103114845Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-02T13:30:26.103438697Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-02T13:30:26.103735220Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-02T13:30:26.104041323Z INFO: Collecting host service logs for NetworkManager 2026-07-02T13:30:26.104361038Z INFO: Collecting host service logs for openvswitch 2026-07-02T13:30:26.104678756Z INFO: Collecting host service logs for ovs-configuration 2026-07-02T13:30:26.105045686Z INFO: Collecting host service logs for ovsdb-server 2026-07-02T13:30:26.105439868Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-02T13:30:26.106503525Z INFO: Waiting for worker host service log collection to complete ... 2026-07-02T13:30:26.110985585Z Executing Istio gather script 2026-07-02T13:30:26.505414141Z INFO: Waiting for node performance related collection to complete ... 2026-07-02T13:30:26.576628884Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-02T13:30:26.836370521Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-02T13:30:27.074068892Z No resources found 2026-07-02T13:30:27.116412961Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-fv6k7 2026-07-02T13:30:27.116466242Z insights-runtime-extractor-4xx9d 2026-07-02T13:30:27.116487985Z insights-runtime-extractor-fnj9w 2026-07-02T13:30:27.116509679Z insights-runtime-extractor-lwl5h 2026-07-02T13:30:27.319393225Z Wrote inspect data to must-gather. 2026-07-02T13:30:27.359701148Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-02T13:30:27.370426445Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-02T13:30:27.403792651Z No resources found in openshift-etcd namespace. 2026-07-02T13:30:27.458433608Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-02T13:30:27.458612747Z INFO: on-disk MachineConfig config collection complete. 2026-07-02T13:30:27.482370666Z INFO: "metallb-operator" not detected. Skipping. 2026-07-02T13:30:27.568037439Z error: the server doesn't have a resource type "performanceprofile" 2026-07-02T13:30:27.606189704Z ERROR: No running kube-apiserver pods found 2026-07-02T13:30:27.662682292Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-02T13:30:27.662682292Z See 'oc rsync -h' for help and examples 2026-07-02T13:30:27.740728565Z INFO: Worker host service log collection to complete. 2026-07-02T13:30:27.905068995Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-02T13:30:28.286326038Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-02T13:30:28.358644050Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-02T13:30:28.367641427Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.377876515Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.397303232Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:28.515539243Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.543883403Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-02T13:30:28.572622099Z No resources found 2026-07-02T13:30:28.617617246Z INFO: 'previous-logs' folder not found on ip-10-0-128-177.ec2.internal, skipping... 2026-07-02T13:30:28.710111254Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.714689331Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.745671318Z Gathering data for ns/openshift-monitoring... 2026-07-02T13:30:28.876382286Z Wrote inspect data to must-gather. 2026-07-02T13:30:28.933696190Z error: the server doesn't have a resource type "machineconfigs" 2026-07-02T13:30:28.933967911Z tar: Removing leading `/' from member names 2026-07-02T13:30:28.939015530Z INFO: HAProxy config collection complete. 2026-07-02T13:30:29.002363101Z Wrote inspect data to must-gather. 2026-07-02T13:30:29.016438162Z INFO: 'previous-logs' folder not found on ip-10-0-137-127.ec2.internal, skipping... 2026-07-02T13:30:29.101250172Z INFO: Getting rules from prometheus-k8s-0 2026-07-02T13:30:29.129874402Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-02T13:30:29.299041480Z Wrote inspect data to must-gather. 2026-07-02T13:30:29.330387903Z INFO: 'previous-logs' folder not found on ip-10-0-140-7.ec2.internal, skipping... 2026-07-02T13:30:29.330461836Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-02T13:30:29.330552628Z INFO: Machine Config Daemon termination log collection complete. 2026-07-02T13:30:29.409689936Z Wrote inspect data to must-gather. 2026-07-02T13:30:29.572081827Z Wrote inspect data to must-gather. 2026-07-02T13:30:29.575350467Z INFO: Getting status/config from prometheus-k8s-0 2026-07-02T13:30:29.611461607Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-02T13:30:29.719396763Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-02T13:30:29.853709917Z Wrote inspect data to must-gather. 2026-07-02T13:30:29.875736866Z INFO: INTERCONNECT MODE 2026-07-02T13:30:29.875771497Z INFO: Gathering ovn-kubernetes DBs 2026-07-02T13:30:30.027438419Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-02T13:30:30.051944462Z INFO: Gathering OVN_Northbound from ovnkube-node-95kwm... 2026-07-02T13:30:30.052239305Z INFO: Gathering OVN_Northbound from ovnkube-node-bfrv8... 2026-07-02T13:30:30.052635485Z INFO: Gathering OVN_Northbound from ovnkube-node-sv8zr... 2026-07-02T13:30:30.053005091Z INFO: Gathering OVN_Southbound from ovnkube-node-95kwm... 2026-07-02T13:30:30.053508062Z INFO: Gathering OVN_Southbound from ovnkube-node-bfrv8... 2026-07-02T13:30:30.053800102Z INFO: Gathering OVN_Southbound from ovnkube-node-sv8zr... 2026-07-02T13:30:30.124147526Z Wrote inspect data to must-gather. 2026-07-02T13:30:30.378551250Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-02T13:30:30.416138374Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.440607892Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.444798794Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.460044350Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.465079409Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.471483538Z tar: Removing leading `/' from member names 2026-07-02T13:30:30.500503692Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-02T13:30:30.568713192Z Wrote inspect data to must-gather. 2026-07-02T13:30:30.670090500Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-02T13:30:30.835132498Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-02T13:30:30.853527374Z Wrote inspect data to must-gather. 2026-07-02T13:30:30.963406680Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-02T13:30:31.028802607Z INFO: Waiting for network log collection to complete ... 2026-07-02T13:30:31.028840608Z INFO: Waiting for ovnk database copies to complete ... 2026-07-02T13:30:31.029219684Z INFO: Copying ovnk databases complete. 2026-07-02T13:30:31.030988803Z 4.0M must-gather/network_logs/ovnk_database_store 2026-07-02T13:30:31.033500459Z ovnk_database_store/ 2026-07-02T13:30:31.033555867Z ovnk_database_store/ovnkube-node-bfrv8_sbdb 2026-07-02T13:30:31.054896430Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:31.064383701Z ovnk_database_store/ovnkube-node-sv8zr_sbdb 2026-07-02T13:30:31.086820683Z ovnk_database_store/ovnkube-node-95kwm_sbdb 2026-07-02T13:30:31.112479016Z ovnk_database_store/ovnkube-node-bfrv8_nbdb 2026-07-02T13:30:31.120119214Z W0702 13:30:31.120077 1451 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-02T13:30:31.121747387Z ovnk_database_store/ovnkube-node-sv8zr_nbdb 2026-07-02T13:30:31.127848087Z Wrote inspect data to must-gather. 2026-07-02T13:30:31.131802041Z ovnk_database_store/ovnkube-node-95kwm_nbdb 2026-07-02T13:30:31.144476958Z INFO: Network log collection complete. 2026-07-02T13:30:31.199559756Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-02T13:30:31.283886465Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:31.287284019Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-02T13:30:31.408772852Z Gathering data for ns/openshift-network-console... 2026-07-02T13:30:31.412884713Z INFO: Getting status from alertmanager-main-0 2026-07-02T13:30:31.470199650Z Wrote inspect data to must-gather. 2026-07-02T13:30:31.519832338Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:31.762286342Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:31.764632841Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-02T13:30:31.805270368Z Wrote inspect data to must-gather. 2026-07-02T13:30:31.911444140Z Gathering data for ns/openshift-console-operator... 2026-07-02T13:30:31.967073195Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:32.109055141Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-02T13:30:32.111559404Z INFO: Fallback to identify the container image from release info 2026-07-02T13:30:32.200984113Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:32.204026226Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-02T13:30:32.339322490Z Gathering data for ns/openshift-console... 2026-07-02T13:30:32.436716346Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:32.640122664Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:32.643848370Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-02T13:30:32.831370727Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:32.908673028Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f 2026-07-02T13:30:32.942731903Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-02T13:30:33.063162456Z daemonset.apps/perf-node-gather-daemonset created 2026-07-02T13:30:33.068215634Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:33.070768136Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-02T13:30:33.205409150Z Waiting for performance profile collector pods to become ready: 1 2026-07-02T13:30:33.256672107Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:33.422944126Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:33.473668753Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:33.476348801Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-02T13:30:33.553661533Z Gathering data for ns/openshift-dns-operator... 2026-07-02T13:30:33.681784072Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:33.876031773Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:33.878743135Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-02T13:30:33.885028391Z Gathering data for ns/openshift-dns... 2026-07-02T13:30:34.030432410Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:34.213762915Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:34.216072622Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-02T13:30:34.341023355Z Waiting for performance profile collector pods to become ready: 2 2026-07-02T13:30:34.393294040Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:34.527164223Z Gathering data for ns/openshift-image-registry... 2026-07-02T13:30:34.586613226Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:34.588717074Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-02T13:30:34.755211575Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:34.936849794Z Wrote inspect data to /must-gather/istio. 2026-07-02T13:30:35.080380683Z Gathering data for ns/openshift-ingress-operator... 2026-07-02T13:30:35.480423489Z Waiting for performance profile collector pods to become ready: 3 2026-07-02T13:30:35.567150921Z Gathering data for ns/openshift-ingress... 2026-07-02T13:30:35.665721642Z Done executing Istio gather script 2026-07-02T13:30:35.895983062Z Gathering data for ns/openshift-ingress-canary... 2026-07-02T13:30:36.256680992Z Gathering data for ns/openshift-insights... 2026-07-02T13:30:36.613550129Z Waiting for performance profile collector pods to become ready: 4 2026-07-02T13:30:37.736421995Z Waiting for performance profile collector pods to become ready: 5 2026-07-02T13:30:38.427768876Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:38.517711455Z Gathering data for ns/kserve... 2026-07-02T13:30:38.852623121Z Waiting for performance profile collector pods to become ready: 6 2026-07-02T13:30:39.981039016Z Waiting for performance profile collector pods to become ready: 7 2026-07-02T13:30:40.094363746Z Gathering data for ns/openshift-config... 2026-07-02T13:30:40.348666285Z Gathering data for ns/openshift-config-managed... 2026-07-02T13:30:40.796775824Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-02T13:30:41.115109596Z Waiting for performance profile collector pods to become ready: 8 2026-07-02T13:30:41.133889328Z Gathering data for ns/openshift-kube-apiserver... 2026-07-02T13:30:41.392204999Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-02T13:30:41.681331120Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-02T13:30:41.972782678Z Gathering data for ns/openshift-kube-scheduler... 2026-07-02T13:30:42.242865422Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-02T13:30:42.253703004Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-02T13:30:42.533639536Z Collecting performance related data for node ip-10-0-137-127.ec2.internal 2026-07-02T13:30:42.541170152Z Collecting performance related data for node ip-10-0-140-7.ec2.internal 2026-07-02T13:30:42.552339108Z Collecting performance related data for node ip-10-0-128-177.ec2.internal 2026-07-02T13:30:42.580879004Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-02T13:30:43.026422403Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-02T13:30:43.435808020Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:43.460430352Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-02T13:30:44.091093779Z Gathering data for ns/openshift-multus... 2026-07-02T13:30:45.682932699Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-02T13:30:47.335113290Z Gathering data for ns/openshift-host-network... 2026-07-02T13:30:47.671337262Z Gathering data for ns/openshift-network-diagnostics... 2026-07-02T13:30:48.210345610Z Gathering data for ns/openshift-network-node-identity... 2026-07-02T13:30:48.444902627Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:48.665734937Z Gathering data for ns/openshift-network-operator... 2026-07-02T13:30:49.067550640Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-02T13:30:49.312761355Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-02T13:30:49.973940292Z Gathering data for ns/openshift-apiserver-operator... 2026-07-02T13:30:50.281765229Z Gathering data for ns/openshift-apiserver... 2026-07-02T13:30:50.685258919Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-02T13:30:50.945398179Z Gathering data for ns/openshift-controller-manager... 2026-07-02T13:30:51.040453345Z Collecting kubelet logs for node ip-10-0-137-127.ec2.internal 2026-07-02T13:30:51.042087717Z Collecting kubelet logs for node ip-10-0-140-7.ec2.internal 2026-07-02T13:30:51.043785978Z Collecting kubelet logs for node ip-10-0-128-177.ec2.internal 2026-07-02T13:30:51.259203019Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-02T13:30:51.443353880Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-74fjg namespace 2026-07-02T13:30:51.447576088Z INFO: Node performance data collection complete. 2026-07-02T13:30:51.925978231Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-02T13:30:52.199922087Z Gathering data for ns/openshift-service-ca-operator... 2026-07-02T13:30:52.514341092Z Gathering data for ns/openshift-service-ca... 2026-07-02T13:30:52.968484408Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-02T13:30:53.450642496Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-02T13:30:53.983377974Z Wrote inspect data to must-gather. 2026-07-02T13:30:53.983428537Z error: inspection completed with the errors occurred while gathering data: 2026-07-02T13:30:53.983428537Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-02T13:30:54.110625916Z error: the server doesn't have a resource type "clusters" 2026-07-02T13:30:54.254340839Z Caches written to disk