2026-07-29T17:08:12.548856288Z [disk usage checker] Started 2026-07-29T17:08:12.552187813Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:12.759699350Z Gathering data for ns/openshift-cluster-version... 2026-07-29T17:08:12.879728164Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-29T17:08:12.972262873Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:08:13.055930656Z Gathering data for ns/default... 2026-07-29T17:08:13.350354715Z Gathering data for ns/openshift... 2026-07-29T17:08:13.831190517Z Gathering data for ns/kube-system... 2026-07-29T17:08:14.521867351Z Gathering data for ns/openshift-etcd... 2026-07-29T17:08:14.857031162Z Wrote inspect data to must-gather. 2026-07-29T17:08:14.857062211Z error: inspection completed with the errors occurred while gathering data: 2026-07-29T17:08:14.857062211Z namespaces "assisted-installer" not found 2026-07-29T17:08:15.344515268Z Waiting on subprocesses to finish execution. 2026-07-29T17:08:15.350228043Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-29T17:08:15.357150173Z Executing Istio gather script 2026-07-29T17:08:15.360244185Z INFO: Gathering HAProxy config files 2026-07-29T17:08:15.365515964Z INFO: Collecting host service logs for crio 2026-07-29T17:08:15.365831603Z INFO: Collecting host service logs for kubelet 2026-07-29T17:08:15.366193407Z INFO: Collecting host service logs for rpm-ostreed 2026-07-29T17:08:15.366549650Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-29T17:08:15.366914228Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-29T17:08:15.367257324Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-29T17:08:15.367708184Z INFO: Collecting host service logs for NetworkManager 2026-07-29T17:08:15.368045577Z INFO: Collecting host service logs for openvswitch 2026-07-29T17:08:15.368367730Z INFO: Collecting host service logs for ovs-configuration 2026-07-29T17:08:15.368724944Z INFO: Collecting host service logs for ovsdb-server 2026-07-29T17:08:15.369051982Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-29T17:08:15.369461561Z INFO: Waiting for worker host service log collection to complete ... 2026-07-29T17:08:15.373198816Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-29T17:08:15.381493751Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-29T17:08:15.388693826Z INFO: Waiting for node performance related collection to complete ... 2026-07-29T17:08:15.904248247Z No resources found in openshift-etcd namespace. 2026-07-29T17:08:16.043659398Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-29T17:08:16.126236139Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-29T17:08:16.181641382Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-29T17:08:16.288845441Z Wrote inspect data to must-gather. 2026-07-29T17:08:16.430901029Z error: the server doesn't have a resource type "performanceprofile" 2026-07-29T17:08:16.535471915Z Wrote inspect data to must-gather. 2026-07-29T17:08:16.539436042Z No resources found 2026-07-29T17:08:16.641766115Z INFO: "metallb-operator" not detected. Skipping. 2026-07-29T17:08:16.714232396Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-29T17:08:16.714303947Z INFO: on-disk MachineConfig config collection complete. 2026-07-29T17:08:16.742184120Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-29T17:08:16.857469180Z INFO: Worker host service log collection to complete. 2026-07-29T17:08:16.864737212Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-pk4v9 2026-07-29T17:08:16.864780288Z insights-runtime-extractor-d5bzt 2026-07-29T17:08:16.864799639Z insights-runtime-extractor-npw8n 2026-07-29T17:08:16.864818626Z insights-runtime-extractor-qn8pk 2026-07-29T17:08:17.337173861Z Wrote inspect data to must-gather. 2026-07-29T17:08:17.435705313Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-29T17:08:17.563923708Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-29T17:08:17.680135859Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:17.703754942Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-29T17:08:17.733826933Z Gathering data for ns/openshift-monitoring... 2026-07-29T17:08:17.751828469Z Wrote inspect data to must-gather. 2026-07-29T17:08:17.763713578Z ERROR: No running kube-apiserver pods found 2026-07-29T17:08:17.827812149Z No resources found 2026-07-29T17:08:17.850218222Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-29T17:08:17.850218222Z See 'oc rsync -h' for help and examples 2026-07-29T17:08:17.871892966Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:08:17.892006223Z Wrote inspect data to must-gather. 2026-07-29T17:08:17.975512819Z error: the server doesn't have a resource type "machineconfigs" 2026-07-29T17:08:17.981718863Z Wrote inspect data to must-gather. 2026-07-29T17:08:17.999841282Z Wrote inspect data to must-gather. 2026-07-29T17:08:18.060415103Z tar: Removing leading `/' from member names 2026-07-29T17:08:18.065068983Z INFO: HAProxy config collection complete. 2026-07-29T17:08:18.118935996Z INFO: 'previous-logs' folder not found on ip-10-0-132-134.ec2.internal, skipping... 2026-07-29T17:08:18.182195206Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-29T17:08:18.275999892Z Wrote inspect data to must-gather. 2026-07-29T17:08:18.403489752Z INFO: Getting rules from prometheus-k8s-0 2026-07-29T17:08:18.444289920Z INFO: 'previous-logs' folder not found on ip-10-0-140-217.ec2.internal, skipping... 2026-07-29T17:08:18.470443684Z Wrote inspect data to must-gather. 2026-07-29T17:08:18.625909032Z Wrote inspect data to must-gather. 2026-07-29T17:08:18.677302459Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-29T17:08:18.776902649Z INFO: 'previous-logs' folder not found on ip-10-0-140-8.ec2.internal, skipping... 2026-07-29T17:08:18.776934903Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-29T17:08:18.776955274Z INFO: Machine Config Daemon termination log collection complete. 2026-07-29T17:08:18.940772602Z INFO: Getting status/config from prometheus-k8s-0 2026-07-29T17:08:18.946130405Z Wrote inspect data to must-gather. 2026-07-29T17:08:18.950948496Z Wrote inspect data to must-gather. 2026-07-29T17:08:19.068123716Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-29T17:08:19.205009180Z Wrote inspect data to must-gather. 2026-07-29T17:08:19.294755451Z INFO: INTERCONNECT MODE 2026-07-29T17:08:19.294798196Z INFO: Gathering ovn-kubernetes DBs 2026-07-29T17:08:19.359371117Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-29T17:08:19.390357060Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-29T17:08:19.481363273Z Wrote inspect data to must-gather. 2026-07-29T17:08:19.496071536Z INFO: Gathering OVN_Northbound from ovnkube-node-czjhr... 2026-07-29T17:08:19.496314060Z INFO: Gathering OVN_Northbound from ovnkube-node-mq456... 2026-07-29T17:08:19.496576445Z INFO: Gathering OVN_Northbound from ovnkube-node-v9xpq... 2026-07-29T17:08:19.496845690Z INFO: Gathering OVN_Southbound from ovnkube-node-czjhr... 2026-07-29T17:08:19.497134474Z INFO: Gathering OVN_Southbound from ovnkube-node-mq456... 2026-07-29T17:08:19.497414758Z INFO: Gathering OVN_Southbound from ovnkube-node-v9xpq... 2026-07-29T17:08:19.730213471Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-29T17:08:19.838312882Z tar: Removing leading `/' from member names 2026-07-29T17:08:19.861351177Z tar: Removing leading `/' from member names 2026-07-29T17:08:19.866573443Z tar: Removing leading `/' from member names 2026-07-29T17:08:19.900425827Z tar: Removing leading `/' from member names 2026-07-29T17:08:19.902872251Z tar: Removing leading `/' from member names 2026-07-29T17:08:19.914616411Z Wrote inspect data to must-gather. 2026-07-29T17:08:19.930078992Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-29T17:08:19.952758915Z tar: Removing leading `/' from member names 2026-07-29T17:08:20.067948493Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-29T17:08:20.159914060Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:20.217772955Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:08:20.301918490Z Wrote inspect data to must-gather. 2026-07-29T17:08:20.376028811Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-29T17:08:20.420474732Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:20.423179979Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-29T17:08:20.443722603Z Gathering data for ns/openshift-network-console... 2026-07-29T17:08:20.458349762Z INFO: Waiting for network log collection to complete ... 2026-07-29T17:08:20.458484380Z INFO: Waiting for ovnk database copies to complete ... 2026-07-29T17:08:20.459093908Z INFO: Copying ovnk databases complete. 2026-07-29T17:08:20.461169936Z 2.8M must-gather/network_logs/ovnk_database_store 2026-07-29T17:08:20.464160439Z ovnk_database_store/ 2026-07-29T17:08:20.464207027Z ovnk_database_store/ovnkube-node-mq456_sbdb 2026-07-29T17:08:20.482386721Z ovnk_database_store/ovnkube-node-czjhr_sbdb 2026-07-29T17:08:20.501142126Z ovnk_database_store/ovnkube-node-mq456_nbdb 2026-07-29T17:08:20.508697575Z ovnk_database_store/ovnkube-node-v9xpq_nbdb 2026-07-29T17:08:20.517457081Z ovnk_database_store/ovnkube-node-v9xpq_sbdb 2026-07-29T17:08:20.538255941Z ovnk_database_store/ovnkube-node-czjhr_nbdb 2026-07-29T17:08:20.551270015Z INFO: Network log collection complete. 2026-07-29T17:08:20.648254406Z Wrote inspect data to must-gather. 2026-07-29T17:08:20.665796756Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:20.844065041Z INFO: Getting status from alertmanager-main-0 2026-07-29T17:08:20.927963494Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:20.932357427Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-29T17:08:20.958980781Z Gathering data for ns/openshift-console-operator... 2026-07-29T17:08:21.000857842Z Wrote inspect data to must-gather. 2026-07-29T17:08:21.149820127Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:21.289645042Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-29T17:08:21.291607289Z INFO: Fallback to identify the container image from release info 2026-07-29T17:08:21.374874021Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:21.377284929Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-29T17:08:21.401918390Z Gathering data for ns/openshift-console... 2026-07-29T17:08:21.593975500Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:21.797745906Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:21.802724037Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-29T17:08:21.909574552Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-29T17:08:21.944908143Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-29T17:08:22.005102208Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:22.083178317Z daemonset.apps/perf-node-gather-daemonset created 2026-07-29T17:08:22.225043894Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:22.227495917Z Waiting for performance profile collector pods to become ready: 1 2026-07-29T17:08:22.229130463Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-29T17:08:22.410052995Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:22.566071410Z Gathering data for ns/openshift-dns-operator... 2026-07-29T17:08:22.648115699Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:22.651658610Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-29T17:08:22.729753513Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:22.831227828Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:22.902036587Z Gathering data for ns/openshift-dns... 2026-07-29T17:08:23.012826493Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:23.015264968Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-29T17:08:23.192995781Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:23.368918205Z Waiting for performance profile collector pods to become ready: 2 2026-07-29T17:08:23.408683160Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:23.412265474Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-29T17:08:23.557177439Z Gathering data for ns/openshift-image-registry... 2026-07-29T17:08:23.589522203Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:23.780799472Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:23.783378587Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-29T17:08:23.963477473Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:24.068287381Z Gathering data for ns/openshift-ingress-operator... 2026-07-29T17:08:24.167757504Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:08:24.450131449Z Gathering data for ns/openshift-ingress... 2026-07-29T17:08:24.516797100Z Waiting for performance profile collector pods to become ready: 3 2026-07-29T17:08:24.820425443Z Gathering data for ns/openshift-ingress-canary... 2026-07-29T17:08:24.891665066Z Done executing Istio gather script 2026-07-29T17:08:25.207474106Z Gathering data for ns/openshift-insights... 2026-07-29T17:08:25.641379635Z Waiting for performance profile collector pods to become ready: 4 2026-07-29T17:08:26.768637592Z Waiting for performance profile collector pods to become ready: 5 2026-07-29T17:08:27.488276933Z Gathering data for ns/kserve... 2026-07-29T17:08:27.735099166Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:27.895221643Z Waiting for performance profile collector pods to become ready: 6 2026-07-29T17:08:28.724225983Z Gathering data for ns/openshift-config... 2026-07-29T17:08:29.001728347Z Gathering data for ns/openshift-config-managed... 2026-07-29T17:08:29.033276869Z Waiting for performance profile collector pods to become ready: 7 2026-07-29T17:08:29.416584292Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-29T17:08:29.717608197Z Gathering data for ns/openshift-kube-apiserver... 2026-07-29T17:08:30.010246057Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-29T17:08:30.168807522Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-29T17:08:30.320088541Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-29T17:08:30.451039959Z Collecting performance related data for node ip-10-0-140-217.ec2.internal 2026-07-29T17:08:30.457900279Z Collecting performance related data for node ip-10-0-132-134.ec2.internal 2026-07-29T17:08:30.468458954Z Collecting performance related data for node ip-10-0-140-8.ec2.internal 2026-07-29T17:08:30.665037347Z Gathering data for ns/openshift-kube-scheduler... 2026-07-29T17:08:30.958505076Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-29T17:08:31.339893998Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-29T17:08:31.731383435Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-29T17:08:32.123920983Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-29T17:08:32.656269520Z Gathering data for ns/openshift-multus... 2026-07-29T17:08:32.740192068Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:34.176003461Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-29T17:08:35.452700827Z Gathering data for ns/openshift-host-network... 2026-07-29T17:08:35.806513863Z Gathering data for ns/openshift-network-diagnostics... 2026-07-29T17:08:36.365071818Z Gathering data for ns/openshift-network-node-identity... 2026-07-29T17:08:36.854226642Z Gathering data for ns/openshift-network-operator... 2026-07-29T17:08:37.223984951Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-29T17:08:37.471472335Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-29T17:08:37.745794200Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:08:37.980242346Z Gathering data for ns/openshift-apiserver-operator... 2026-07-29T17:08:38.276550459Z Gathering data for ns/openshift-apiserver... 2026-07-29T17:08:38.676489208Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-29T17:08:38.979178263Z Gathering data for ns/openshift-controller-manager... 2026-07-29T17:08:39.006380033Z Collecting kubelet logs for node ip-10-0-140-217.ec2.internal 2026-07-29T17:08:39.008581488Z Collecting kubelet logs for node ip-10-0-132-134.ec2.internal 2026-07-29T17:08:39.010804468Z Collecting kubelet logs for node ip-10-0-140-8.ec2.internal 2026-07-29T17:08:39.315479174Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-29T17:08:39.365988437Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ksgvw namespace 2026-07-29T17:08:39.370181409Z INFO: Node performance data collection complete. 2026-07-29T17:08:39.979681548Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-29T17:08:40.257288277Z Gathering data for ns/openshift-service-ca-operator... 2026-07-29T17:08:40.559790918Z Gathering data for ns/openshift-service-ca... 2026-07-29T17:08:40.925017821Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-29T17:08:41.810831412Z Wrote inspect data to must-gather. 2026-07-29T17:08:41.810867585Z error: inspection completed with the errors occurred while gathering data: 2026-07-29T17:08:41.810867585Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-29T17:08:41.941913195Z error: the server doesn't have a resource type "clusters" 2026-07-29T17:08:42.080350933Z Caches written to disk