2026-07-03T20:56:42.725175407Z [disk usage checker] Started 2026-07-03T20:56:42.728590736Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:56:42.923999760Z Gathering data for ns/openshift-cluster-version... 2026-07-03T20:56:43.065849832Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-03T20:56:43.148145089Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-03T20:56:43.223239959Z Gathering data for ns/default... 2026-07-03T20:56:43.492255725Z Gathering data for ns/openshift... 2026-07-03T20:56:43.877513172Z Gathering data for ns/kube-system... 2026-07-03T20:56:44.597139685Z Gathering data for ns/openshift-etcd... 2026-07-03T20:56:44.927000211Z Wrote inspect data to must-gather. 2026-07-03T20:56:44.927033074Z error: inspection completed with the errors occurred while gathering data: 2026-07-03T20:56:44.927033074Z namespaces "assisted-installer" not found 2026-07-03T20:56:45.252960073Z Waiting on subprocesses to finish execution. 2026-07-03T20:56:45.259178925Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-03T20:56:45.261199429Z Executing Istio gather script 2026-07-03T20:56:45.264395631Z INFO: Gathering HAProxy config files 2026-07-03T20:56:45.266654178Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-03T20:56:45.270810148Z INFO: Waiting for node performance related collection to complete ... 2026-07-03T20:56:45.270961076Z INFO: Collecting host service logs for crio 2026-07-03T20:56:45.271386561Z INFO: Collecting host service logs for kubelet 2026-07-03T20:56:45.271697087Z INFO: Collecting host service logs for rpm-ostreed 2026-07-03T20:56:45.271992585Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-03T20:56:45.272273957Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-03T20:56:45.272539156Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-03T20:56:45.272782345Z INFO: Collecting host service logs for NetworkManager 2026-07-03T20:56:45.273060437Z INFO: Collecting host service logs for openvswitch 2026-07-03T20:56:45.273311980Z INFO: Collecting host service logs for ovs-configuration 2026-07-03T20:56:45.273575994Z INFO: Collecting host service logs for ovsdb-server 2026-07-03T20:56:45.273854377Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-03T20:56:45.274165380Z INFO: Waiting for worker host service log collection to complete ... 2026-07-03T20:56:45.281307320Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-03T20:56:45.865009632Z error: the server doesn't have a resource type "performanceprofile" 2026-07-03T20:56:46.013702415Z INFO: "metallb-operator" not detected. Skipping. 2026-07-03T20:56:46.102967839Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-03T20:56:46.119419731Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-r69kn 2026-07-03T20:56:46.119419731Z insights-runtime-extractor-2ksbc 2026-07-03T20:56:46.119419731Z insights-runtime-extractor-hqzzf 2026-07-03T20:56:46.119419731Z insights-runtime-extractor-kdntw 2026-07-03T20:56:46.231047033Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-03T20:56:46.254336293Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-03T20:56:46.265895852Z No resources found in openshift-etcd namespace. 2026-07-03T20:56:46.267994619Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-03T20:56:46.280859539Z Wrote inspect data to must-gather. 2026-07-03T20:56:46.404320406Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-03T20:56:46.404362906Z INFO: on-disk MachineConfig config collection complete. 2026-07-03T20:56:46.653565481Z No resources found 2026-07-03T20:56:46.711404442Z INFO: Worker host service log collection to complete. 2026-07-03T20:56:46.833210038Z Wrote inspect data to must-gather. 2026-07-03T20:56:46.883618065Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-03T20:56:46.931756630Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-03T20:56:47.104898667Z Wrote inspect data to must-gather. 2026-07-03T20:56:47.172514946Z Gathering data for ns/openshift-monitoring... 2026-07-03T20:56:47.379298840Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-03T20:56:47.387386013Z Wrote inspect data to must-gather. 2026-07-03T20:56:47.410654103Z Wrote inspect data to must-gather. 2026-07-03T20:56:47.493413635Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-03T20:56:47.493413635Z See 'oc rsync -h' for help and examples 2026-07-03T20:56:47.568195349Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-03T20:56:47.626122602Z INFO: 'previous-logs' folder not found on ip-10-0-131-78.ec2.internal, skipping... 2026-07-03T20:56:47.674010672Z ERROR: No running kube-apiserver pods found 2026-07-03T20:56:47.725497944Z tar: Removing leading `/' from member names 2026-07-03T20:56:47.731850405Z INFO: HAProxy config collection complete. 2026-07-03T20:56:47.752941831Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:56:47.766982574Z No resources found 2026-07-03T20:56:47.786462524Z error: the server doesn't have a resource type "machineconfigs" 2026-07-03T20:56:47.808465741Z Wrote inspect data to must-gather. 2026-07-03T20:56:47.875908947Z INFO: Getting rules from prometheus-k8s-0 2026-07-03T20:56:47.877239107Z Wrote inspect data to must-gather. 2026-07-03T20:56:47.983646804Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-03T20:56:47.983808319Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-03T20:56:48.040415286Z INFO: 'previous-logs' folder not found on ip-10-0-137-42.ec2.internal, skipping... 2026-07-03T20:56:48.147873801Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.204243466Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.294214081Z INFO: 'previous-logs' folder not found on ip-10-0-142-106.ec2.internal, skipping... 2026-07-03T20:56:48.294339418Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-03T20:56:48.294406358Z INFO: Machine Config Daemon termination log collection complete. 2026-07-03T20:56:48.352861268Z INFO: Getting status/config from prometheus-k8s-0 2026-07-03T20:56:48.369704592Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-03T20:56:48.409619459Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.582748360Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.643344104Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.777431799Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-03T20:56:48.806502044Z Wrote inspect data to must-gather. 2026-07-03T20:56:48.887637927Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-03T20:56:48.961282609Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-03T20:56:48.966397054Z INFO: INTERCONNECT MODE 2026-07-03T20:56:48.966481083Z INFO: Gathering ovn-kubernetes DBs 2026-07-03T20:56:49.026741902Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-03T20:56:49.048728357Z Wrote inspect data to must-gather. 2026-07-03T20:56:49.100445234Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:49.163119753Z INFO: Gathering OVN_Northbound from ovnkube-node-7kmf2... 2026-07-03T20:56:49.163404282Z INFO: Gathering OVN_Northbound from ovnkube-node-hlr6c... 2026-07-03T20:56:49.163697273Z INFO: Gathering OVN_Northbound from ovnkube-node-hrkvr... 2026-07-03T20:56:49.164034633Z INFO: Gathering OVN_Southbound from ovnkube-node-7kmf2... 2026-07-03T20:56:49.164318393Z INFO: Gathering OVN_Southbound from ovnkube-node-hlr6c... 2026-07-03T20:56:49.164574698Z INFO: Gathering OVN_Southbound from ovnkube-node-hrkvr... 2026-07-03T20:56:49.328162715Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-03T20:56:49.387156397Z Wrote inspect data to must-gather. 2026-07-03T20:56:49.465513995Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:49.469551036Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-03T20:56:49.499392279Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.533752173Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.542001117Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.559531385Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.570955615Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.573812975Z tar: Removing leading `/' from member names 2026-07-03T20:56:49.664800567Z W0703 20:56:49.664724 1424 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-03T20:56:49.671977209Z Wrote inspect data to must-gather. 2026-07-03T20:56:49.673243126Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-03T20:56:49.733114889Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:49.738556734Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-03T20:56:49.962422895Z Gathering data for ns/openshift-network-console... 2026-07-03T20:56:49.996914757Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:50.001130595Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-03T20:56:50.011531880Z Wrote inspect data to must-gather. 2026-07-03T20:56:50.066517410Z INFO: Waiting for network log collection to complete ... 2026-07-03T20:56:50.066651900Z INFO: Waiting for ovnk database copies to complete ... 2026-07-03T20:56:50.067191765Z INFO: Copying ovnk databases complete. 2026-07-03T20:56:50.073239570Z 2.4M must-gather/network_logs/ovnk_database_store 2026-07-03T20:56:50.076695591Z ovnk_database_store/ 2026-07-03T20:56:50.076744360Z ovnk_database_store/ovnkube-node-hrkvr_nbdb 2026-07-03T20:56:50.094343985Z ovnk_database_store/ovnkube-node-hlr6c_nbdb 2026-07-03T20:56:50.101177955Z ovnk_database_store/ovnkube-node-hlr6c_sbdb 2026-07-03T20:56:50.117525290Z INFO: Getting status from alertmanager-main-0 2026-07-03T20:56:50.117815334Z ovnk_database_store/ovnkube-node-hrkvr_sbdb 2026-07-03T20:56:50.135437233Z ovnk_database_store/ovnkube-node-7kmf2_nbdb 2026-07-03T20:56:50.143678148Z ovnk_database_store/ovnkube-node-7kmf2_sbdb 2026-07-03T20:56:50.160736735Z INFO: Network log collection complete. 2026-07-03T20:56:50.224818437Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:50.315053310Z Wrote inspect data to must-gather. 2026-07-03T20:56:50.439070925Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:50.442304740Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-03T20:56:50.449374038Z Gathering data for ns/openshift-console-operator... 2026-07-03T20:56:50.583530405Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-03T20:56:50.585385544Z INFO: Fallback to identify the container image from release info 2026-07-03T20:56:50.665640750Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:50.860897646Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:50.862210976Z Gathering data for ns/openshift-console... 2026-07-03T20:56:50.862989331Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-03T20:56:51.013976893Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:51.196445476Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-03T20:56:51.211412708Z Wrote inspect data to /must-gather/istio. 2026-07-03T20:56:51.333002861Z daemonset.apps/perf-node-gather-daemonset created 2026-07-03T20:56:51.391563898Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-03T20:56:51.483573298Z Waiting for performance profile collector pods to become ready: 1 2026-07-03T20:56:51.903914483Z Gathering data for ns/openshift-dns-operator... 2026-07-03T20:56:51.909814646Z Done executing Istio gather script 2026-07-03T20:56:52.192103088Z Gathering data for ns/openshift-dns... 2026-07-03T20:56:52.590005159Z Waiting for performance profile collector pods to become ready: 2 2026-07-03T20:56:52.748531941Z Gathering data for ns/openshift-image-registry... 2026-07-03T20:56:52.764462284Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:56:53.188335446Z Gathering data for ns/openshift-ingress-operator... 2026-07-03T20:56:53.481081060Z Gathering data for ns/openshift-ingress... 2026-07-03T20:56:53.703225848Z Waiting for performance profile collector pods to become ready: 3 2026-07-03T20:56:53.791458455Z Gathering data for ns/openshift-ingress-canary... 2026-07-03T20:56:54.138977523Z Gathering data for ns/openshift-insights... 2026-07-03T20:56:54.815034442Z Waiting for performance profile collector pods to become ready: 4 2026-07-03T20:56:55.878520232Z Gathering data for ns/openshift-config... 2026-07-03T20:56:55.930176189Z Waiting for performance profile collector pods to become ready: 5 2026-07-03T20:56:56.115955764Z Gathering data for ns/openshift-config-managed... 2026-07-03T20:56:56.512104009Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-03T20:56:56.745870325Z Gathering data for ns/openshift-kube-apiserver... 2026-07-03T20:56:56.973135376Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-03T20:56:57.051239909Z Waiting for performance profile collector pods to become ready: 6 2026-07-03T20:56:57.233946221Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-03T20:56:57.490126634Z Gathering data for ns/openshift-kube-scheduler... 2026-07-03T20:56:57.717135702Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-03T20:56:57.770033053Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:56:57.976985776Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-03T20:56:58.160741862Z Waiting for performance profile collector pods to become ready: 7 2026-07-03T20:56:58.274259053Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-03T20:56:58.549566434Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-03T20:56:59.045709681Z Gathering data for ns/openshift-multus... 2026-07-03T20:56:59.256841595Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-03T20:56:59.461901958Z Collecting performance related data for node ip-10-0-142-106.ec2.internal 2026-07-03T20:56:59.468335572Z Collecting performance related data for node ip-10-0-137-42.ec2.internal 2026-07-03T20:56:59.475160090Z Collecting performance related data for node ip-10-0-131-78.ec2.internal 2026-07-03T20:57:00.582626218Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-03T20:57:01.874727023Z Gathering data for ns/openshift-host-network... 2026-07-03T20:57:02.207849660Z Gathering data for ns/openshift-network-diagnostics... 2026-07-03T20:57:02.731311697Z Gathering data for ns/openshift-network-node-identity... 2026-07-03T20:57:02.775761313Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:57:03.097027995Z Gathering data for ns/openshift-network-operator... 2026-07-03T20:57:03.448171304Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-03T20:57:03.688125115Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-03T20:57:04.192191952Z Gathering data for ns/openshift-apiserver-operator... 2026-07-03T20:57:04.495855755Z Gathering data for ns/openshift-apiserver... 2026-07-03T20:57:04.868252663Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-03T20:57:05.114806504Z Gathering data for ns/openshift-controller-manager... 2026-07-03T20:57:05.365199102Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-03T20:57:05.975058193Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-03T20:57:06.226751649Z Gathering data for ns/openshift-service-ca-operator... 2026-07-03T20:57:06.505603871Z Gathering data for ns/openshift-service-ca... 2026-07-03T20:57:06.876347678Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-03T20:57:07.745744782Z Collecting kubelet logs for node ip-10-0-142-106.ec2.internal 2026-07-03T20:57:07.749477347Z Collecting kubelet logs for node ip-10-0-137-42.ec2.internal 2026-07-03T20:57:07.751553310Z Collecting kubelet logs for node ip-10-0-131-78.ec2.internal 2026-07-03T20:57:07.753152890Z Wrote inspect data to must-gather. 2026-07-03T20:57:07.753198601Z error: inspection completed with the errors occurred while gathering data: 2026-07-03T20:57:07.753198601Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-03T20:57:07.782768702Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-03T20:57:08.033553303Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-2hfcp namespace 2026-07-03T20:57:08.038563612Z INFO: Node performance data collection complete. 2026-07-03T20:57:08.163692122Z error: the server doesn't have a resource type "clusters" 2026-07-03T20:57:08.289397774Z Caches written to disk