2026-07-15T11:11:41.943451035Z [disk usage checker] Started 2026-07-15T11:11:41.946951548Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:11:42.133810320Z Gathering data for ns/openshift-cluster-version... 2026-07-15T11:11:42.276913019Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-15T11:11:42.365223198Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-15T11:11:42.447952523Z Gathering data for ns/default... 2026-07-15T11:11:42.762595813Z Gathering data for ns/openshift... 2026-07-15T11:11:43.174011831Z Gathering data for ns/kube-system... 2026-07-15T11:11:43.911330074Z Gathering data for ns/openshift-etcd... 2026-07-15T11:11:44.245624837Z Wrote inspect data to must-gather. 2026-07-15T11:11:44.245656825Z error: inspection completed with the errors occurred while gathering data: 2026-07-15T11:11:44.245656825Z namespaces "assisted-installer" not found 2026-07-15T11:11:44.680726525Z Waiting on subprocesses to finish execution. 2026-07-15T11:11:44.686272122Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-15T11:11:44.691949037Z Executing Istio gather script 2026-07-15T11:11:44.694518546Z INFO: Gathering HAProxy config files 2026-07-15T11:11:44.697096015Z INFO: Collecting host service logs for crio 2026-07-15T11:11:44.697449054Z INFO: Collecting host service logs for kubelet 2026-07-15T11:11:44.697607225Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-15T11:11:44.697823697Z INFO: Collecting host service logs for rpm-ostreed 2026-07-15T11:11:44.698175590Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-15T11:11:44.698654146Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-15T11:11:44.699040133Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-15T11:11:44.699415045Z INFO: Collecting host service logs for NetworkManager 2026-07-15T11:11:44.699781565Z INFO: Collecting host service logs for openvswitch 2026-07-15T11:11:44.700122943Z INFO: Collecting host service logs for ovs-configuration 2026-07-15T11:11:44.700504099Z INFO: Collecting host service logs for ovsdb-server 2026-07-15T11:11:44.700899247Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-15T11:11:44.701346487Z INFO: Waiting for worker host service log collection to complete ... 2026-07-15T11:11:44.703659256Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-15T11:11:44.777229777Z INFO: Waiting for node performance related collection to complete ... 2026-07-15T11:11:45.446939911Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-15T11:11:45.446939911Z INFO: on-disk MachineConfig config collection complete. 2026-07-15T11:11:45.874510501Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-15T11:11:45.931919163Z No resources found 2026-07-15T11:11:45.984199281Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-15T11:11:46.037470376Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-g9swl 2026-07-15T11:11:46.037513478Z insights-runtime-extractor-69c6j 2026-07-15T11:11:46.037533359Z insights-runtime-extractor-6tctp 2026-07-15T11:11:46.037554063Z insights-runtime-extractor-bqw5l 2026-07-15T11:11:46.068236705Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-15T11:11:46.119604554Z error: the server doesn't have a resource type "performanceprofile" 2026-07-15T11:11:46.281833799Z INFO: "metallb-operator" not detected. Skipping. 2026-07-15T11:11:46.446783721Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-15T11:11:46.500923088Z ERROR: No running kube-apiserver pods found 2026-07-15T11:11:46.534876211Z No resources found in openshift-etcd namespace. 2026-07-15T11:11:46.539850363Z Wrote inspect data to must-gather. 2026-07-15T11:11:46.588601013Z INFO: Worker host service log collection to complete. 2026-07-15T11:11:46.664693714Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-15T11:11:46.664693714Z See 'oc rsync -h' for help and examples 2026-07-15T11:11:46.986126387Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-15T11:11:47.007804549Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.098726246Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:11:47.147260000Z No resources found 2026-07-15T11:11:47.181181107Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.195250576Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-15T11:11:47.267760538Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.400214812Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-15T11:11:47.428205764Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.449352962Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-15T11:11:47.538314939Z Gathering data for ns/openshift-monitoring... 2026-07-15T11:11:47.592544715Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.652252509Z error: the server doesn't have a resource type "machineconfigs" 2026-07-15T11:11:47.659790190Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.678707739Z INFO: 'previous-logs' folder not found on ip-10-0-134-216.ec2.internal, skipping... 2026-07-15T11:11:47.693526542Z tar: Removing leading `/' from member names 2026-07-15T11:11:47.697873373Z INFO: HAProxy config collection complete. 2026-07-15T11:11:47.877163483Z Wrote inspect data to must-gather. 2026-07-15T11:11:47.907384265Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-15T11:11:48.013680240Z INFO: 'previous-logs' folder not found on ip-10-0-138-171.ec2.internal, skipping... 2026-07-15T11:11:48.061284346Z INFO: Getting rules from prometheus-k8s-0 2026-07-15T11:11:48.190116341Z Wrote inspect data to must-gather. 2026-07-15T11:11:48.202803358Z Wrote inspect data to must-gather. 2026-07-15T11:11:48.342631232Z INFO: 'previous-logs' folder not found on ip-10-0-143-121.ec2.internal, skipping... 2026-07-15T11:11:48.342763520Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-15T11:11:48.342763520Z INFO: Machine Config Daemon termination log collection complete. 2026-07-15T11:11:48.397053033Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-15T11:11:48.485720696Z Wrote inspect data to must-gather. 2026-07-15T11:11:48.542386535Z INFO: Getting status/config from prometheus-k8s-0 2026-07-15T11:11:48.669657838Z Wrote inspect data to must-gather. 2026-07-15T11:11:48.791220472Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-15T11:11:48.833100609Z INFO: INTERCONNECT MODE 2026-07-15T11:11:48.833133241Z INFO: Gathering ovn-kubernetes DBs 2026-07-15T11:11:48.915949564Z Wrote inspect data to must-gather. 2026-07-15T11:11:49.007184741Z INFO: Gathering OVN_Northbound from ovnkube-node-btclt... 2026-07-15T11:11:49.007428374Z INFO: Gathering OVN_Northbound from ovnkube-node-c4lfw... 2026-07-15T11:11:49.007651838Z INFO: Gathering OVN_Northbound from ovnkube-node-qsch8... 2026-07-15T11:11:49.007941995Z INFO: Gathering OVN_Southbound from ovnkube-node-btclt... 2026-07-15T11:11:49.008437740Z INFO: Gathering OVN_Southbound from ovnkube-node-c4lfw... 2026-07-15T11:11:49.008725807Z INFO: Gathering OVN_Southbound from ovnkube-node-qsch8... 2026-07-15T11:11:49.051909386Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-15T11:11:49.284237109Z Wrote inspect data to must-gather. 2026-07-15T11:11:49.338377149Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.355371801Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.391144219Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.429146226Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.437140008Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.460257423Z tar: Removing leading `/' from member names 2026-07-15T11:11:49.479881579Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-15T11:11:49.515992144Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-15T11:11:49.570108729Z Wrote inspect data to must-gather. 2026-07-15T11:11:49.684231750Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-15T11:11:49.822825755Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-15T11:11:49.837157615Z Wrote inspect data to must-gather. 2026-07-15T11:11:49.917026542Z INFO: Waiting for network log collection to complete ... 2026-07-15T11:11:49.917062969Z INFO: Waiting for ovnk database copies to complete ... 2026-07-15T11:11:49.917583065Z INFO: Copying ovnk databases complete. 2026-07-15T11:11:49.919133445Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:49.919801658Z 3.4M must-gather/network_logs/ovnk_database_store 2026-07-15T11:11:49.923128756Z ovnk_database_store/ 2026-07-15T11:11:49.923128756Z ovnk_database_store/ovnkube-node-btclt_sbdb 2026-07-15T11:11:49.943597589Z ovnk_database_store/ovnkube-node-c4lfw_sbdb 2026-07-15T11:11:49.964559530Z ovnk_database_store/ovnkube-node-qsch8_sbdb 2026-07-15T11:11:49.988434088Z ovnk_database_store/ovnkube-node-c4lfw_nbdb 2026-07-15T11:11:49.997968029Z ovnk_database_store/ovnkube-node-qsch8_nbdb 2026-07-15T11:11:50.007242185Z ovnk_database_store/ovnkube-node-btclt_nbdb 2026-07-15T11:11:50.021453800Z INFO: Network log collection complete. 2026-07-15T11:11:50.086446646Z W0715 11:11:50.086387 1462 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-15T11:11:50.091989767Z Wrote inspect data to must-gather. 2026-07-15T11:11:50.109810615Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-15T11:11:50.161154592Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:50.164891062Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-15T11:11:50.192402783Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-15T11:11:50.395174018Z Gathering data for ns/openshift-network-console... 2026-07-15T11:11:50.406997697Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:50.419434656Z Wrote inspect data to must-gather. 2026-07-15T11:11:50.546766538Z INFO: Getting status from alertmanager-main-0 2026-07-15T11:11:50.647457598Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:50.650810250Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-15T11:11:50.731556332Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-15T11:11:50.734652007Z INFO: Fallback to identify the container image from release info 2026-07-15T11:11:50.884653070Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:50.932371800Z Gathering data for ns/openshift-console-operator... 2026-07-15T11:11:51.092430760Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:51.095103232Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-15T11:11:51.313503700Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:51.316619535Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d 2026-07-15T11:11:51.441358145Z Gathering data for ns/openshift-console... 2026-07-15T11:11:51.465285756Z daemonset.apps/perf-node-gather-daemonset created 2026-07-15T11:11:51.535355346Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:51.538501160Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-15T11:11:51.611406261Z Waiting for performance profile collector pods to become ready: 1 2026-07-15T11:11:51.717254747Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:51.921330059Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:51.925403426Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-15T11:11:52.080523522Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-15T11:11:52.108951700Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:52.180674605Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:11:52.300456747Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:52.303708745Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-15T11:11:52.481142473Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:52.619244802Z Gathering data for ns/openshift-dns-operator... 2026-07-15T11:11:52.704787182Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:52.707922572Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-15T11:11:52.761388241Z Waiting for performance profile collector pods to become ready: 2 2026-07-15T11:11:52.887213837Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:52.962673963Z Gathering data for ns/openshift-dns... 2026-07-15T11:11:53.070367600Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:53.073059126Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-15T11:11:53.230919028Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:53.430288147Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:53.433799250Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-15T11:11:53.609472927Z Gathering data for ns/openshift-image-registry... 2026-07-15T11:11:53.613135062Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:53.815845297Z Wrote inspect data to /must-gather/istio. 2026-07-15T11:11:53.898557481Z Waiting for performance profile collector pods to become ready: 3 2026-07-15T11:11:54.151092975Z Gathering data for ns/openshift-ingress-operator... 2026-07-15T11:11:54.507795502Z Done executing Istio gather script 2026-07-15T11:11:54.546372572Z Gathering data for ns/openshift-ingress... 2026-07-15T11:11:54.883647488Z Gathering data for ns/openshift-ingress-canary... 2026-07-15T11:11:55.020565331Z Waiting for performance profile collector pods to become ready: 4 2026-07-15T11:11:55.285719236Z Gathering data for ns/openshift-insights... 2026-07-15T11:11:56.136116135Z Waiting for performance profile collector pods to become ready: 5 2026-07-15T11:11:57.186038678Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:11:57.248898947Z Waiting for performance profile collector pods to become ready: 6 2026-07-15T11:11:57.534849358Z Gathering data for ns/kserve... 2026-07-15T11:11:58.377406563Z Waiting for performance profile collector pods to become ready: 7 2026-07-15T11:11:59.150206398Z Gathering data for ns/openshift-config... 2026-07-15T11:11:59.427427457Z Gathering data for ns/openshift-config-managed... 2026-07-15T11:11:59.515858517Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-15T11:11:59.768177198Z Collecting performance related data for node ip-10-0-134-216.ec2.internal 2026-07-15T11:11:59.777812880Z Collecting performance related data for node ip-10-0-138-171.ec2.internal 2026-07-15T11:11:59.789394205Z Collecting performance related data for node ip-10-0-143-121.ec2.internal 2026-07-15T11:11:59.928884242Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-15T11:12:00.251721456Z Gathering data for ns/openshift-kube-apiserver... 2026-07-15T11:12:00.571331898Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-15T11:12:00.979529509Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-15T11:12:01.350978737Z Gathering data for ns/openshift-kube-scheduler... 2026-07-15T11:12:01.668599784Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-15T11:12:01.954724720Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-15T11:12:02.196359271Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:12:02.338577985Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-15T11:12:02.724392829Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-15T11:12:03.418352773Z Gathering data for ns/openshift-multus... 2026-07-15T11:12:04.938479214Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-15T11:12:06.244818633Z Gathering data for ns/openshift-host-network... 2026-07-15T11:12:06.577030155Z Gathering data for ns/openshift-network-diagnostics... 2026-07-15T11:12:07.190347469Z Gathering data for ns/openshift-network-node-identity... 2026-07-15T11:12:07.203872252Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:12:07.599347488Z Gathering data for ns/openshift-network-operator... 2026-07-15T11:12:08.065355063Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-15T11:12:08.167020517Z Collecting kubelet logs for node ip-10-0-134-216.ec2.internal 2026-07-15T11:12:08.169025041Z Collecting kubelet logs for node ip-10-0-138-171.ec2.internal 2026-07-15T11:12:08.171058313Z Collecting kubelet logs for node ip-10-0-143-121.ec2.internal 2026-07-15T11:12:08.393371958Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-15T11:12:08.554711874Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-hxg7m namespace 2026-07-15T11:12:08.560529103Z INFO: Node performance data collection complete. 2026-07-15T11:12:08.910655690Z Gathering data for ns/openshift-apiserver-operator... 2026-07-15T11:12:09.208991456Z Gathering data for ns/openshift-apiserver... 2026-07-15T11:12:09.596512339Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-15T11:12:09.890412171Z Gathering data for ns/openshift-controller-manager... 2026-07-15T11:12:10.166479718Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-15T11:12:10.809021465Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-15T11:12:11.164886505Z Gathering data for ns/openshift-service-ca-operator... 2026-07-15T11:12:11.484198879Z Gathering data for ns/openshift-service-ca... 2026-07-15T11:12:11.846050799Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-15T11:12:12.209965983Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-15T11:12:12.718901345Z Wrote inspect data to must-gather. 2026-07-15T11:12:12.718943754Z error: inspection completed with the errors occurred while gathering data: 2026-07-15T11:12:12.718943754Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-15T11:12:12.850520676Z error: the server doesn't have a resource type "clusters" 2026-07-15T11:12:12.989475062Z Caches written to disk