2026-07-07T12:05:44.437938085Z [disk usage checker] Started 2026-07-07T12:05:44.441323063Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:05:44.650609620Z Gathering data for ns/openshift-cluster-version... 2026-07-07T12:05:44.793121794Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-07T12:05:44.874917933Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-07T12:05:44.957878504Z Gathering data for ns/default... 2026-07-07T12:05:45.287636638Z Gathering data for ns/openshift... 2026-07-07T12:05:45.731234746Z Gathering data for ns/kube-system... 2026-07-07T12:05:46.435592731Z Gathering data for ns/openshift-etcd... 2026-07-07T12:05:46.744957767Z Wrote inspect data to must-gather. 2026-07-07T12:05:46.745049243Z error: inspection completed with the errors occurred while gathering data: 2026-07-07T12:05:46.745049243Z namespaces "assisted-installer" not found 2026-07-07T12:05:47.289005672Z Waiting on subprocesses to finish execution. 2026-07-07T12:05:47.297842571Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-07T12:05:47.300453734Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-07T12:05:47.300666054Z INFO: Gathering HAProxy config files 2026-07-07T12:05:47.302805867Z Executing Istio gather script 2026-07-07T12:05:47.316310596Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-07T12:05:47.323644580Z INFO: Waiting for node performance related collection to complete ... 2026-07-07T12:05:47.333828269Z INFO: Collecting host service logs for crio 2026-07-07T12:05:47.334140135Z INFO: Collecting host service logs for kubelet 2026-07-07T12:05:47.334432781Z INFO: Collecting host service logs for rpm-ostreed 2026-07-07T12:05:47.334751870Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-07T12:05:47.335060942Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-07T12:05:47.335374707Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-07T12:05:47.335719018Z INFO: Collecting host service logs for NetworkManager 2026-07-07T12:05:47.336014347Z INFO: Collecting host service logs for openvswitch 2026-07-07T12:05:47.336308377Z INFO: Collecting host service logs for ovs-configuration 2026-07-07T12:05:47.336654614Z INFO: Collecting host service logs for ovsdb-server 2026-07-07T12:05:47.336936469Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-07T12:05:47.337293886Z INFO: Waiting for worker host service log collection to complete ... 2026-07-07T12:05:47.846660666Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-fqgbl 2026-07-07T12:05:47.846728785Z insights-runtime-extractor-4qx5d 2026-07-07T12:05:47.846750193Z insights-runtime-extractor-fqnbv 2026-07-07T12:05:47.846770815Z insights-runtime-extractor-vlxns 2026-07-07T12:05:48.091868803Z INFO: "metallb-operator" not detected. Skipping. 2026-07-07T12:05:48.099743893Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-07T12:05:48.107642187Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-07T12:05:48.619823873Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-07T12:05:48.619823873Z See 'oc rsync -h' for help and examples 2026-07-07T12:05:48.676959072Z Wrote inspect data to must-gather. 2026-07-07T12:05:48.707374466Z ERROR: No running kube-apiserver pods found 2026-07-07T12:05:48.757759360Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-07T12:05:48.855496944Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-07T12:05:48.855543544Z INFO: on-disk MachineConfig config collection complete. 2026-07-07T12:05:48.942459521Z No resources found 2026-07-07T12:05:48.983249454Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-07T12:05:49.016627611Z INFO: 'previous-logs' folder not found on ip-10-0-130-200.ec2.internal, skipping... 2026-07-07T12:05:49.030173930Z No resources found in openshift-etcd namespace. 2026-07-07T12:05:49.067549037Z Gathering data for ns/openshift-monitoring... 2026-07-07T12:05:49.171160152Z error: the server doesn't have a resource type "performanceprofile" 2026-07-07T12:05:49.181420292Z Wrote inspect data to must-gather. 2026-07-07T12:05:49.298806148Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-07T12:05:49.341511213Z INFO: Worker host service log collection to complete. 2026-07-07T12:05:49.423138822Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-07T12:05:49.522780714Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:05:49.673374278Z Wrote inspect data to must-gather. 2026-07-07T12:05:49.871606414Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-07T12:05:49.959420485Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-07T12:05:50.128018060Z No resources found 2026-07-07T12:05:50.157470936Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.180410690Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.227301766Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.234163975Z INFO: 'previous-logs' folder not found on ip-10-0-138-138.ec2.internal, skipping... 2026-07-07T12:05:50.234362371Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.364944521Z tar: Removing leading `/' from member names 2026-07-07T12:05:50.373125571Z INFO: HAProxy config collection complete. 2026-07-07T12:05:50.449154039Z INFO: Getting rules from prometheus-k8s-0 2026-07-07T12:05:50.464133780Z error: the server doesn't have a resource type "machineconfigs" 2026-07-07T12:05:50.551650721Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.585088121Z INFO: 'previous-logs' folder not found on ip-10-0-143-227.ec2.internal, skipping... 2026-07-07T12:05:50.585132922Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-07T12:05:50.585158606Z INFO: Machine Config Daemon termination log collection complete. 2026-07-07T12:05:50.636306282Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-07T12:05:50.663043558Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-07T12:05:50.840906219Z Wrote inspect data to must-gather. 2026-07-07T12:05:50.896766148Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-07T12:05:50.937179959Z INFO: Getting status/config from prometheus-k8s-0 2026-07-07T12:05:50.946035051Z Wrote inspect data to must-gather. 2026-07-07T12:05:51.119872126Z Wrote inspect data to must-gather. 2026-07-07T12:05:51.141558331Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-07T12:05:51.410985270Z Wrote inspect data to must-gather. 2026-07-07T12:05:51.420624705Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-07T12:05:51.472400855Z INFO: INTERCONNECT MODE 2026-07-07T12:05:51.472448341Z INFO: Gathering ovn-kubernetes DBs 2026-07-07T12:05:51.559010223Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-07T12:05:51.609758469Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-07T12:05:51.662826936Z INFO: Gathering OVN_Northbound from ovnkube-node-2c7qq... 2026-07-07T12:05:51.663105805Z INFO: Gathering OVN_Northbound from ovnkube-node-bn47b... 2026-07-07T12:05:51.663340517Z INFO: Gathering OVN_Northbound from ovnkube-node-l5kdk... 2026-07-07T12:05:51.663649522Z INFO: Gathering OVN_Southbound from ovnkube-node-2c7qq... 2026-07-07T12:05:51.663908397Z INFO: Gathering OVN_Southbound from ovnkube-node-bn47b... 2026-07-07T12:05:51.664192980Z INFO: Gathering OVN_Southbound from ovnkube-node-l5kdk... 2026-07-07T12:05:51.693351371Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-07T12:05:51.806251849Z Wrote inspect data to must-gather. 2026-07-07T12:05:51.884331639Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:51.951311097Z Gathering data for ns/openshift-network-console... 2026-07-07T12:05:51.997088685Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.009394285Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.053536416Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.067628581Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.093716782Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.097350433Z tar: Removing leading `/' from member names 2026-07-07T12:05:52.144460670Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-07T12:05:52.211777394Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:52.215270791Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-07T12:05:52.342025021Z Wrote inspect data to must-gather. 2026-07-07T12:05:52.433806066Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-07T12:05:52.501215526Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:52.573954245Z Gathering data for ns/openshift-console-operator... 2026-07-07T12:05:52.726000290Z Wrote inspect data to must-gather. 2026-07-07T12:05:52.727784809Z INFO: Waiting for network log collection to complete ... 2026-07-07T12:05:52.727900693Z INFO: Waiting for ovnk database copies to complete ... 2026-07-07T12:05:52.728377298Z INFO: Copying ovnk databases complete. 2026-07-07T12:05:52.730232751Z 3.5M must-gather/network_logs/ovnk_database_store 2026-07-07T12:05:52.732867885Z ovnk_database_store/ 2026-07-07T12:05:52.732915601Z ovnk_database_store/ovnkube-node-2c7qq_sbdb 2026-07-07T12:05:52.755150856Z ovnk_database_store/ovnkube-node-bn47b_sbdb 2026-07-07T12:05:52.777042369Z ovnk_database_store/ovnkube-node-l5kdk_nbdb 2026-07-07T12:05:52.779813380Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:52.782191497Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-07T12:05:52.787657722Z ovnk_database_store/ovnkube-node-l5kdk_sbdb 2026-07-07T12:05:52.813251732Z ovnk_database_store/ovnkube-node-2c7qq_nbdb 2026-07-07T12:05:52.823800213Z ovnk_database_store/ovnkube-node-bn47b_nbdb 2026-07-07T12:05:52.836004668Z INFO: Network log collection complete. 2026-07-07T12:05:52.907175521Z INFO: Getting status from alertmanager-main-0 2026-07-07T12:05:53.019711387Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:53.083077441Z Gathering data for ns/openshift-console... 2026-07-07T12:05:53.103794899Z Wrote inspect data to must-gather. 2026-07-07T12:05:53.271298808Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:53.273661099Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-07T12:05:53.441439072Z Wrote inspect data to must-gather. 2026-07-07T12:05:53.527557799Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:53.673806738Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-07T12:05:53.777224912Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:53.781328357Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-07T12:05:53.807310832Z Wrote inspect data to must-gather. 2026-07-07T12:05:53.965633296Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:54.101796059Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-07T12:05:54.105230413Z INFO: Fallback to identify the container image from release info 2026-07-07T12:05:54.204942587Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:54.208911194Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-07T12:05:54.316593849Z Gathering data for ns/openshift-dns-operator... 2026-07-07T12:05:54.411558833Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:54.541989410Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:05:54.627754158Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:54.630177124Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-07T12:05:54.631117398Z Gathering data for ns/openshift-dns... 2026-07-07T12:05:54.785372503Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-07T12:05:54.802989621Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:54.928271736Z daemonset.apps/perf-node-gather-daemonset created 2026-07-07T12:05:55.024004727Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:55.026405239Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-07T12:05:55.085457527Z Waiting for performance profile collector pods to become ready: 1 2026-07-07T12:05:55.212717417Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:55.353718535Z Gathering data for ns/openshift-image-registry... 2026-07-07T12:05:55.438677517Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:55.440756333Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-07T12:05:55.621094885Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:55.828559577Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:55.831550808Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-07T12:05:55.857528681Z Gathering data for ns/openshift-ingress-operator... 2026-07-07T12:05:56.017800314Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:56.250030546Z Gathering data for ns/openshift-ingress... 2026-07-07T12:05:56.250541652Z Wrote inspect data to /must-gather/istio. 2026-07-07T12:05:56.252908309Z Waiting for performance profile collector pods to become ready: 2 2026-07-07T12:05:56.616121192Z Gathering data for ns/openshift-ingress-canary... 2026-07-07T12:05:56.973540395Z Done executing Istio gather script 2026-07-07T12:05:57.037176772Z Gathering data for ns/openshift-insights... 2026-07-07T12:05:57.387677043Z Waiting for performance profile collector pods to become ready: 3 2026-07-07T12:05:58.517744430Z Waiting for performance profile collector pods to become ready: 4 2026-07-07T12:05:59.254846761Z Gathering data for ns/kserve... 2026-07-07T12:05:59.549377304Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:05:59.650244606Z Waiting for performance profile collector pods to become ready: 5 2026-07-07T12:06:00.736670216Z Gathering data for ns/openshift-config... 2026-07-07T12:06:00.769898373Z Waiting for performance profile collector pods to become ready: 6 2026-07-07T12:06:01.037432348Z Gathering data for ns/openshift-config-managed... 2026-07-07T12:06:01.448692740Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-07T12:06:01.713260276Z Gathering data for ns/openshift-kube-apiserver... 2026-07-07T12:06:01.900755046Z Waiting for performance profile collector pods to become ready: 7 2026-07-07T12:06:01.980107826Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-07T12:06:02.256411484Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-07T12:06:02.534103722Z Gathering data for ns/openshift-kube-scheduler... 2026-07-07T12:06:02.818557182Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-07T12:06:03.032974651Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-07T12:06:03.116174971Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-07T12:06:03.320268275Z Collecting performance related data for node ip-10-0-138-138.ec2.internal 2026-07-07T12:06:03.328089846Z Collecting performance related data for node ip-10-0-143-227.ec2.internal 2026-07-07T12:06:03.338383968Z Collecting performance related data for node ip-10-0-130-200.ec2.internal 2026-07-07T12:06:03.498710784Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-07T12:06:03.867919866Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-07T12:06:04.485156613Z Gathering data for ns/openshift-multus... 2026-07-07T12:06:04.555075766Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:06:05.985939784Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-07T12:06:07.285928091Z Gathering data for ns/openshift-host-network... 2026-07-07T12:06:07.636636780Z Gathering data for ns/openshift-network-diagnostics... 2026-07-07T12:06:08.163851215Z Gathering data for ns/openshift-network-node-identity... 2026-07-07T12:06:08.586545060Z Gathering data for ns/openshift-network-operator... 2026-07-07T12:06:08.972900540Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-07T12:06:09.231841349Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-07T12:06:09.562785484Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-07-07T12:06:09.710844548Z Gathering data for ns/openshift-apiserver-operator... 2026-07-07T12:06:09.989865600Z Gathering data for ns/openshift-apiserver... 2026-07-07T12:06:10.375530138Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-07T12:06:10.651964392Z Gathering data for ns/openshift-controller-manager... 2026-07-07T12:06:10.927427152Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-07T12:06:11.563917373Z Collecting kubelet logs for node ip-10-0-138-138.ec2.internal 2026-07-07T12:06:11.565632660Z Collecting kubelet logs for node ip-10-0-143-227.ec2.internal 2026-07-07T12:06:11.567379427Z Collecting kubelet logs for node ip-10-0-130-200.ec2.internal 2026-07-07T12:06:11.606670567Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-07T12:06:11.958878255Z Gathering data for ns/openshift-service-ca-operator... 2026-07-07T12:06:11.961438821Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-n47qj namespace 2026-07-07T12:06:11.966163244Z INFO: Node performance data collection complete. 2026-07-07T12:06:12.288113397Z Gathering data for ns/openshift-service-ca... 2026-07-07T12:06:12.651401126Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-07T12:06:13.528173688Z Wrote inspect data to must-gather. 2026-07-07T12:06:13.528219850Z error: inspection completed with the errors occurred while gathering data: 2026-07-07T12:06:13.528219850Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-07T12:06:13.662152556Z error: the server doesn't have a resource type "clusters" 2026-07-07T12:06:13.775640485Z Caches written to disk