2026-07-27T21:30:29.717867150Z [disk usage checker] Started 2026-07-27T21:30:29.721294205Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:29.909293645Z Gathering data for ns/openshift-cluster-version... 2026-07-27T21:30:30.023914496Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-27T21:30:30.100076963Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-27T21:30:30.171025465Z Gathering data for ns/default... 2026-07-27T21:30:30.429980216Z Gathering data for ns/openshift... 2026-07-27T21:30:30.800439081Z Gathering data for ns/kube-system... 2026-07-27T21:30:31.501232531Z Gathering data for ns/openshift-etcd... 2026-07-27T21:30:31.781822229Z Wrote inspect data to must-gather. 2026-07-27T21:30:31.781855685Z error: inspection completed with the errors occurred while gathering data: 2026-07-27T21:30:31.781855685Z namespaces "assisted-installer" not found 2026-07-27T21:30:32.218316522Z Waiting on subprocesses to finish execution. 2026-07-27T21:30:32.222417874Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-27T21:30:32.223352253Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-27T21:30:32.225113013Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-27T21:30:32.230565873Z Executing Istio gather script 2026-07-27T21:30:32.233143441Z INFO: Gathering HAProxy config files 2026-07-27T21:30:32.256095607Z INFO: Collecting host service logs for crio 2026-07-27T21:30:32.256393828Z INFO: Collecting host service logs for kubelet 2026-07-27T21:30:32.256972044Z INFO: Collecting host service logs for rpm-ostreed 2026-07-27T21:30:32.257341443Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-27T21:30:32.257690042Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-27T21:30:32.258018071Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-27T21:30:32.258386328Z INFO: Collecting host service logs for NetworkManager 2026-07-27T21:30:32.258722431Z INFO: Collecting host service logs for openvswitch 2026-07-27T21:30:32.259074831Z INFO: Collecting host service logs for ovs-configuration 2026-07-27T21:30:32.259431550Z INFO: Collecting host service logs for ovsdb-server 2026-07-27T21:30:32.259785119Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-27T21:30:32.260282917Z INFO: Waiting for worker host service log collection to complete ... 2026-07-27T21:30:32.374575493Z INFO: Waiting for node performance related collection to complete ... 2026-07-27T21:30:33.036890947Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-jvz2z 2026-07-27T21:30:33.036890947Z insights-runtime-extractor-jf8xz 2026-07-27T21:30:33.036890947Z insights-runtime-extractor-pbp78 2026-07-27T21:30:33.036890947Z insights-runtime-extractor-pfrtr 2026-07-27T21:30:33.179205954Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-27T21:30:33.197334812Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-27T21:30:33.197419372Z INFO: on-disk MachineConfig config collection complete. 2026-07-27T21:30:33.215804203Z No resources found 2026-07-27T21:30:33.241014194Z No resources found in openshift-etcd namespace. 2026-07-27T21:30:33.474407210Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-27T21:30:33.587913010Z error: the server doesn't have a resource type "performanceprofile" 2026-07-27T21:30:33.621714543Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-27T21:30:33.642238454Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-27T21:30:33.642238454Z See 'oc rsync -h' for help and examples 2026-07-27T21:30:33.661757616Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-27T21:30:33.754127155Z Wrote inspect data to must-gather. 2026-07-27T21:30:33.767653112Z INFO: Worker host service log collection to complete. 2026-07-27T21:30:33.778154072Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-27T21:30:34.142728110Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-27T21:30:34.159181707Z INFO: "metallb-operator" not detected. Skipping. 2026-07-27T21:30:34.211325590Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-27T21:30:34.246810982Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.280747404Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.300726540Z No resources found 2026-07-27T21:30:34.363423876Z tar: Removing leading `/' from member names 2026-07-27T21:30:34.366608661Z INFO: 'previous-logs' folder not found on ip-10-0-128-132.ec2.internal, skipping... 2026-07-27T21:30:34.369183019Z INFO: HAProxy config collection complete. 2026-07-27T21:30:34.387272967Z ERROR: No running kube-apiserver pods found 2026-07-27T21:30:34.445025971Z Gathering data for ns/openshift-monitoring... 2026-07-27T21:30:34.499186231Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.570180407Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-27T21:30:34.645239607Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.692831761Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.748433468Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:34.788660204Z Wrote inspect data to must-gather. 2026-07-27T21:30:34.791012912Z error: the server doesn't have a resource type "machineconfigs" 2026-07-27T21:30:34.848546677Z INFO: 'previous-logs' folder not found on ip-10-0-138-82.ec2.internal, skipping... 2026-07-27T21:30:34.865560897Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-27T21:30:34.966234313Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-27T21:30:34.998536283Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.099405265Z INFO: Getting rules from prometheus-k8s-0 2026-07-27T21:30:35.109822503Z INFO: 'previous-logs' folder not found on ip-10-0-140-0.ec2.internal, skipping... 2026-07-27T21:30:35.109860935Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-27T21:30:35.109918539Z INFO: Machine Config Daemon termination log collection complete. 2026-07-27T21:30:35.197957950Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.266327652Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.357552786Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-27T21:30:35.475642406Z INFO: Getting status/config from prometheus-k8s-0 2026-07-27T21:30:35.497896204Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.571883421Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.724230956Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-27T21:30:35.752199666Z INFO: INTERCONNECT MODE 2026-07-27T21:30:35.752229114Z INFO: Gathering ovn-kubernetes DBs 2026-07-27T21:30:35.777063812Z Wrote inspect data to must-gather. 2026-07-27T21:30:35.894819149Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-27T21:30:35.934555185Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:35.939564289Z INFO: Gathering OVN_Northbound from ovnkube-node-4qmck... 2026-07-27T21:30:35.939789038Z INFO: Gathering OVN_Northbound from ovnkube-node-6ckhc... 2026-07-27T21:30:35.940107548Z INFO: Gathering OVN_Northbound from ovnkube-node-xr4p9... 2026-07-27T21:30:35.940452061Z INFO: Gathering OVN_Southbound from ovnkube-node-4qmck... 2026-07-27T21:30:35.940710303Z INFO: Gathering OVN_Southbound from ovnkube-node-6ckhc... 2026-07-27T21:30:35.941004194Z INFO: Gathering OVN_Southbound from ovnkube-node-xr4p9... 2026-07-27T21:30:35.990280423Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-27T21:30:36.049515497Z Wrote inspect data to must-gather. 2026-07-27T21:30:36.213066618Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-27T21:30:36.247495949Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.250456600Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.262222476Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.280008960Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:36.283075189Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-27T21:30:36.298726761Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.302230016Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.325189877Z tar: Removing leading `/' from member names 2026-07-27T21:30:36.410321710Z Wrote inspect data to must-gather. 2026-07-27T21:30:36.480078950Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-27T21:30:36.509572643Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:36.650878533Z W0727 21:30:36.650826 1432 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-27T21:30:36.657652401Z Wrote inspect data to must-gather. 2026-07-27T21:30:36.737122644Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-27T21:30:36.763184268Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:36.766181576Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-27T21:30:36.801386826Z INFO: Waiting for network log collection to complete ... 2026-07-27T21:30:36.801452258Z INFO: Waiting for ovnk database copies to complete ... 2026-07-27T21:30:36.801918945Z INFO: Copying ovnk databases complete. 2026-07-27T21:30:36.803665403Z 2.4M must-gather/network_logs/ovnk_database_store 2026-07-27T21:30:36.804184574Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-27T21:30:36.806072654Z ovnk_database_store/ 2026-07-27T21:30:36.806072654Z ovnk_database_store/ovnkube-node-4qmck_sbdb 2026-07-27T21:30:36.826594685Z ovnk_database_store/ovnkube-node-xr4p9_nbdb 2026-07-27T21:30:36.832816075Z ovnk_database_store/ovnkube-node-6ckhc_nbdb 2026-07-27T21:30:36.838681176Z ovnk_database_store/ovnkube-node-4qmck_nbdb 2026-07-27T21:30:36.845870343Z ovnk_database_store/ovnkube-node-6ckhc_sbdb 2026-07-27T21:30:36.864655006Z ovnk_database_store/ovnkube-node-xr4p9_sbdb 2026-07-27T21:30:36.885191239Z INFO: Network log collection complete. 2026-07-27T21:30:36.961578848Z Wrote inspect data to must-gather. 2026-07-27T21:30:36.967268573Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:36.992997231Z Gathering data for ns/openshift-network-console... 2026-07-27T21:30:37.153206489Z INFO: Getting status from alertmanager-main-0 2026-07-27T21:30:37.181268730Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:37.184252572Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-27T21:30:37.281748377Z Wrote inspect data to must-gather. 2026-07-27T21:30:37.434345959Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:37.440781747Z Gathering data for ns/openshift-console-operator... 2026-07-27T21:30:37.551895465Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-27T21:30:37.554092641Z INFO: Fallback to identify the container image from release info 2026-07-27T21:30:37.620168143Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:37.623512570Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-27T21:30:37.781939303Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:37.813198952Z Gathering data for ns/openshift-console... 2026-07-27T21:30:37.963460076Z Wrote inspect data to /must-gather/istio. 2026-07-27T21:30:38.305510096Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-27T21:30:38.318656760Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-27T21:30:38.435735779Z daemonset.apps/perf-node-gather-daemonset created 2026-07-27T21:30:38.565564940Z Waiting for performance profile collector pods to become ready: 1 2026-07-27T21:30:38.614319551Z Done executing Istio gather script 2026-07-27T21:30:38.824838819Z Gathering data for ns/openshift-dns-operator... 2026-07-27T21:30:39.113665766Z Gathering data for ns/openshift-dns... 2026-07-27T21:30:39.662931698Z Gathering data for ns/openshift-image-registry... 2026-07-27T21:30:39.683911771Z Waiting for performance profile collector pods to become ready: 2 2026-07-27T21:30:39.759162751Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:40.089568002Z Gathering data for ns/openshift-ingress-operator... 2026-07-27T21:30:40.379258358Z Gathering data for ns/openshift-ingress... 2026-07-27T21:30:40.678423935Z Gathering data for ns/openshift-ingress-canary... 2026-07-27T21:30:40.794517523Z Waiting for performance profile collector pods to become ready: 3 2026-07-27T21:30:41.024912157Z Gathering data for ns/openshift-insights... 2026-07-27T21:30:41.909242328Z Waiting for performance profile collector pods to become ready: 4 2026-07-27T21:30:42.762885545Z Gathering data for ns/openshift-config... 2026-07-27T21:30:43.025890692Z Waiting for performance profile collector pods to become ready: 5 2026-07-27T21:30:43.053098917Z Gathering data for ns/openshift-config-managed... 2026-07-27T21:30:43.428891956Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-27T21:30:43.645953582Z Gathering data for ns/openshift-kube-apiserver... 2026-07-27T21:30:43.870145856Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-27T21:30:44.122241780Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-27T21:30:44.146477308Z Waiting for performance profile collector pods to become ready: 6 2026-07-27T21:30:44.360449174Z Gathering data for ns/openshift-kube-scheduler... 2026-07-27T21:30:44.581333922Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-27T21:30:44.764614111Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:44.830584534Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-27T21:30:45.157694479Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-27T21:30:45.265465915Z Waiting for performance profile collector pods to become ready: 7 2026-07-27T21:30:45.434319535Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-27T21:30:45.861406865Z Gathering data for ns/openshift-multus... 2026-07-27T21:30:46.365553873Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-27T21:30:46.596922598Z Collecting performance related data for node ip-10-0-140-0.ec2.internal 2026-07-27T21:30:46.603789585Z Collecting performance related data for node ip-10-0-138-82.ec2.internal 2026-07-27T21:30:46.610875557Z Collecting performance related data for node ip-10-0-128-132.ec2.internal 2026-07-27T21:30:47.212525319Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-27T21:30:48.387738880Z Gathering data for ns/openshift-host-network... 2026-07-27T21:30:48.675419039Z Gathering data for ns/openshift-network-diagnostics... 2026-07-27T21:30:49.194882383Z Gathering data for ns/openshift-network-node-identity... 2026-07-27T21:30:49.567380438Z Gathering data for ns/openshift-network-operator... 2026-07-27T21:30:49.770642032Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:49.907719479Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-27T21:30:50.134971216Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-27T21:30:50.567198176Z Gathering data for ns/openshift-apiserver-operator... 2026-07-27T21:30:50.875300377Z Gathering data for ns/openshift-apiserver... 2026-07-27T21:30:51.222035662Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-27T21:30:51.464379241Z Gathering data for ns/openshift-controller-manager... 2026-07-27T21:30:51.705557837Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-27T21:30:52.284563888Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-27T21:30:52.543583464Z Gathering data for ns/openshift-service-ca-operator... 2026-07-27T21:30:52.842054166Z Gathering data for ns/openshift-service-ca... 2026-07-27T21:30:53.191287813Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-27T21:30:54.057800663Z Wrote inspect data to must-gather. 2026-07-27T21:30:54.057844181Z error: inspection completed with the errors occurred while gathering data: 2026-07-27T21:30:54.057844181Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-27T21:30:54.644500547Z Collecting kubelet logs for node ip-10-0-140-0.ec2.internal 2026-07-27T21:30:54.646304924Z Collecting kubelet logs for node ip-10-0-138-82.ec2.internal 2026-07-27T21:30:54.647942889Z Collecting kubelet logs for node ip-10-0-128-132.ec2.internal 2026-07-27T21:30:54.775316957Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-27T21:30:54.938071635Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5vzdq namespace 2026-07-27T21:30:54.943264500Z INFO: Node performance data collection complete. 2026-07-27T21:30:55.061735645Z error: the server doesn't have a resource type "clusters" 2026-07-27T21:30:55.218103308Z Caches written to disk