2026-07-23T16:57:29.442363485Z [disk usage checker] Started 2026-07-23T16:57:29.445808596Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-23T16:57:29.643303373Z Gathering data for ns/openshift-cluster-version... 2026-07-23T16:57:29.769869733Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-23T16:57:29.858963976Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-23T16:57:29.940260871Z Gathering data for ns/default... 2026-07-23T16:57:30.244457834Z Gathering data for ns/openshift... 2026-07-23T16:57:30.670389881Z Gathering data for ns/kube-system... 2026-07-23T16:57:31.507873119Z Gathering data for ns/openshift-etcd... 2026-07-23T16:57:31.802024681Z Wrote inspect data to must-gather. 2026-07-23T16:57:31.802059710Z error: inspection completed with the errors occurred while gathering data: 2026-07-23T16:57:31.802059710Z namespaces "assisted-installer" not found 2026-07-23T16:57:32.001603982Z Waiting on subprocesses to finish execution. 2026-07-23T16:57:32.006026514Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-23T16:57:32.012166454Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-23T16:57:32.015194396Z INFO: Gathering HAProxy config files 2026-07-23T16:57:32.015344602Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-23T16:57:32.021202927Z Executing Istio gather script 2026-07-23T16:57:32.064235709Z INFO: Collecting host service logs for crio 2026-07-23T16:57:32.064695965Z INFO: Collecting host service logs for kubelet 2026-07-23T16:57:32.065144475Z INFO: Collecting host service logs for rpm-ostreed 2026-07-23T16:57:32.065503280Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-23T16:57:32.065861880Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-23T16:57:32.066240310Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-23T16:57:32.066736652Z INFO: Collecting host service logs for NetworkManager 2026-07-23T16:57:32.067108705Z INFO: Collecting host service logs for openvswitch 2026-07-23T16:57:32.067471861Z INFO: Collecting host service logs for ovs-configuration 2026-07-23T16:57:32.067839118Z INFO: Collecting host service logs for ovsdb-server 2026-07-23T16:57:32.068272929Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-23T16:57:32.068752047Z INFO: Waiting for worker host service log collection to complete ... 2026-07-23T16:57:32.230698040Z INFO: Waiting for node performance related collection to complete ... 2026-07-23T16:57:32.614893734Z No resources found in openshift-etcd namespace. 2026-07-23T16:57:32.690869938Z No resources found 2026-07-23T16:57:32.749530782Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-23T16:57:32.749605972Z INFO: on-disk MachineConfig config collection complete. 2026-07-23T16:57:32.834091230Z Wrote inspect data to must-gather. 2026-07-23T16:57:32.912071471Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-23T16:57:32.945980747Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-9f7f9 2026-07-23T16:57:32.946056292Z insights-runtime-extractor-cc72t 2026-07-23T16:57:32.946080239Z insights-runtime-extractor-fbvld 2026-07-23T16:57:32.946099837Z insights-runtime-extractor-sgwpw 2026-07-23T16:57:33.171158473Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-23T16:57:33.218346184Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-23T16:57:33.378387032Z INFO: "metallb-operator" not detected. Skipping. 2026-07-23T16:57:33.385169631Z ERROR: No running kube-apiserver pods found 2026-07-23T16:57:33.432722968Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-23T16:57:33.591789265Z INFO: 'previous-logs' folder not found on ip-10-0-129-201.ec2.internal, skipping... 2026-07-23T16:57:33.605462551Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-23T16:57:33.605462551Z See 'oc rsync -h' for help and examples 2026-07-23T16:57:33.668767180Z INFO: Worker host service log collection to complete. 2026-07-23T16:57:34.025924091Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.026284264Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-23T16:57:34.057929808Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-23T16:57:34.065760107Z error: the server doesn't have a resource type "performanceprofile" 2026-07-23T16:57:34.174824620Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-23T16:57:34.206095093Z Gathering data for ns/openshift-monitoring... 2026-07-23T16:57:34.239635841Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-23T16:57:34.307126645Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.452993776Z No resources found 2026-07-23T16:57:34.507358992Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-23T16:57:34.549063467Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.568847532Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.588631700Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-23T16:57:34.640480729Z INFO: 'previous-logs' folder not found on ip-10-0-132-198.ec2.internal, skipping... 2026-07-23T16:57:34.662398993Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.668238375Z tar: Removing leading `/' from member names 2026-07-23T16:57:34.674300646Z INFO: HAProxy config collection complete. 2026-07-23T16:57:34.682898893Z Wrote inspect data to must-gather. 2026-07-23T16:57:34.837961914Z INFO: Getting rules from prometheus-k8s-0 2026-07-23T16:57:34.852122827Z error: the server doesn't have a resource type "machineconfigs" 2026-07-23T16:57:34.935033410Z INFO: 'previous-logs' folder not found on ip-10-0-138-250.ec2.internal, skipping... 2026-07-23T16:57:34.935066990Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-23T16:57:34.935090854Z INFO: Machine Config Daemon termination log collection complete. 2026-07-23T16:57:34.947673471Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.019577949Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-23T16:57:35.205136939Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.229781689Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.266128775Z INFO: Getting status/config from prometheus-k8s-0 2026-07-23T16:57:35.387623315Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-23T16:57:35.450391937Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.643501745Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.719225452Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-23T16:57:35.775658696Z INFO: INTERCONNECT MODE 2026-07-23T16:57:35.775700498Z INFO: Gathering ovn-kubernetes DBs 2026-07-23T16:57:35.793303303Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-23T16:57:35.878035379Z Wrote inspect data to must-gather. 2026-07-23T16:57:35.957888788Z INFO: Gathering OVN_Northbound from ovnkube-node-67qj5... 2026-07-23T16:57:35.958170417Z INFO: Gathering OVN_Northbound from ovnkube-node-p4xsg... 2026-07-23T16:57:35.958425903Z INFO: Gathering OVN_Northbound from ovnkube-node-wqfc9... 2026-07-23T16:57:35.958705127Z INFO: Gathering OVN_Southbound from ovnkube-node-67qj5... 2026-07-23T16:57:35.958933989Z INFO: Gathering OVN_Southbound from ovnkube-node-p4xsg... 2026-07-23T16:57:35.959249248Z INFO: Gathering OVN_Southbound from ovnkube-node-wqfc9... 2026-07-23T16:57:36.010075967Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-23T16:57:36.018935757Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:36.019319327Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-23T16:57:36.220309626Z Wrote inspect data to must-gather. 2026-07-23T16:57:36.248325240Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.270665060Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.297543476Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.307746469Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.355583341Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.358104446Z tar: Removing leading `/' from member names 2026-07-23T16:57:36.409166730Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:36.411530963Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-23T16:57:36.412443467Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-23T16:57:36.502771225Z Wrote inspect data to must-gather. 2026-07-23T16:57:36.645853657Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:36.700632255Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-23T16:57:36.742318443Z W0723 16:57:36.742269 1479 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-23T16:57:36.750090856Z Wrote inspect data to must-gather. 2026-07-23T16:57:36.833604735Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-23T16:57:36.838173759Z INFO: Waiting for network log collection to complete ... 2026-07-23T16:57:36.838173759Z INFO: Waiting for ovnk database copies to complete ... 2026-07-23T16:57:36.838708766Z INFO: Copying ovnk databases complete. 2026-07-23T16:57:36.840602052Z 2.4M must-gather/network_logs/ovnk_database_store 2026-07-23T16:57:36.842687910Z ovnk_database_store/ 2026-07-23T16:57:36.842687910Z ovnk_database_store/ovnkube-node-wqfc9_nbdb 2026-07-23T16:57:36.851330615Z ovnk_database_store/ovnkube-node-wqfc9_sbdb 2026-07-23T16:57:36.867206634Z ovnk_database_store/ovnkube-node-67qj5_nbdb 2026-07-23T16:57:36.872290093Z ovnk_database_store/ovnkube-node-67qj5_sbdb 2026-07-23T16:57:36.886113499Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:36.888721142Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-23T16:57:36.889045180Z ovnk_database_store/ovnkube-node-p4xsg_nbdb 2026-07-23T16:57:36.895528498Z ovnk_database_store/ovnkube-node-p4xsg_sbdb 2026-07-23T16:57:36.914963388Z INFO: Network log collection complete. 2026-07-23T16:57:37.044291178Z Gathering data for ns/openshift-network-console... 2026-07-23T16:57:37.056183660Z Wrote inspect data to must-gather. 2026-07-23T16:57:37.094888527Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:37.106247911Z INFO: Getting status from alertmanager-main-0 2026-07-23T16:57:37.307201426Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:37.309486278Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-23T16:57:37.364969651Z Wrote inspect data to must-gather. 2026-07-23T16:57:37.524089310Z Gathering data for ns/openshift-console-operator... 2026-07-23T16:57:37.537863929Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:37.631676939Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-23T16:57:37.634784176Z INFO: Fallback to identify the container image from release info 2026-07-23T16:57:37.740835413Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:37.743518975Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-23T16:57:37.908807690Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:37.915361075Z Gathering data for ns/openshift-console... 2026-07-23T16:57:38.083481659Z Wrote inspect data to /must-gather/istio. 2026-07-23T16:57:38.157123410Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb 2026-07-23T16:57:38.283524992Z daemonset.apps/perf-node-gather-daemonset created 2026-07-23T16:57:38.418106562Z Waiting for performance profile collector pods to become ready: 1 2026-07-23T16:57:38.489970151Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-23T16:57:38.773923523Z Done executing Istio gather script 2026-07-23T16:57:39.055084437Z Gathering data for ns/openshift-dns-operator... 2026-07-23T16:57:39.349039144Z Gathering data for ns/openshift-dns... 2026-07-23T16:57:39.512485020Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-23T16:57:39.519035650Z Waiting for performance profile collector pods to become ready: 2 2026-07-23T16:57:39.973810287Z Gathering data for ns/openshift-image-registry... 2026-07-23T16:57:40.590147835Z Gathering data for ns/openshift-ingress-operator... 2026-07-23T16:57:40.634871252Z Waiting for performance profile collector pods to become ready: 3 2026-07-23T16:57:40.933624356Z Gathering data for ns/openshift-ingress... 2026-07-23T16:57:41.292759185Z Gathering data for ns/openshift-ingress-canary... 2026-07-23T16:57:41.736402354Z Gathering data for ns/openshift-insights... 2026-07-23T16:57:41.748648327Z Waiting for performance profile collector pods to become ready: 4 2026-07-23T16:57:42.852120237Z Waiting for performance profile collector pods to become ready: 5 2026-07-23T16:57:43.933228700Z Gathering data for ns/openshift-config... 2026-07-23T16:57:43.967151037Z Waiting for performance profile collector pods to become ready: 6 2026-07-23T16:57:44.256257186Z Gathering data for ns/openshift-config-managed... 2026-07-23T16:57:44.516870935Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-23T16:57:44.672697571Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-23T16:57:44.992292994Z Gathering data for ns/openshift-kube-apiserver... 2026-07-23T16:57:45.087856383Z Waiting for performance profile collector pods to become ready: 7 2026-07-23T16:57:45.266926876Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-23T16:57:45.550270043Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-23T16:57:45.881663695Z Gathering data for ns/openshift-kube-scheduler... 2026-07-23T16:57:46.185798860Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-23T16:57:46.217753703Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-23T16:57:46.455190200Z Collecting performance related data for node ip-10-0-132-198.ec2.internal 2026-07-23T16:57:46.463455196Z Collecting performance related data for node ip-10-0-138-250.ec2.internal 2026-07-23T16:57:46.479113706Z Collecting performance related data for node ip-10-0-129-201.ec2.internal 2026-07-23T16:57:46.514566107Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-23T16:57:46.918419759Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-23T16:57:47.312531505Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-23T16:57:47.964784535Z Gathering data for ns/openshift-multus... 2026-07-23T16:57:49.522229164Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-23T16:57:49.885621510Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-23T16:57:51.405497636Z Gathering data for ns/openshift-host-network... 2026-07-23T16:57:51.710739754Z Gathering data for ns/openshift-network-diagnostics... 2026-07-23T16:57:52.267580163Z Gathering data for ns/openshift-network-node-identity... 2026-07-23T16:57:52.667391778Z Gathering data for ns/openshift-network-operator... 2026-07-23T16:57:53.050781464Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-23T16:57:53.340032186Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-23T16:57:53.865158358Z Gathering data for ns/openshift-apiserver-operator... 2026-07-23T16:57:54.156979836Z Gathering data for ns/openshift-apiserver... 2026-07-23T16:57:54.527490087Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-23T16:57:54.563501403Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-23T16:57:54.764309071Z Collecting kubelet logs for node ip-10-0-132-198.ec2.internal 2026-07-23T16:57:54.766978707Z Collecting kubelet logs for node ip-10-0-138-250.ec2.internal 2026-07-23T16:57:54.769099409Z Collecting kubelet logs for node ip-10-0-129-201.ec2.internal 2026-07-23T16:57:54.852489323Z Gathering data for ns/openshift-controller-manager... 2026-07-23T16:57:55.129681144Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qp842 namespace 2026-07-23T16:57:55.135362731Z INFO: Node performance data collection complete. 2026-07-23T16:57:55.189978536Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-23T16:57:55.847937459Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-23T16:57:56.123626390Z Gathering data for ns/openshift-service-ca-operator... 2026-07-23T16:57:56.414935290Z Gathering data for ns/openshift-service-ca... 2026-07-23T16:57:56.765384786Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-23T16:57:57.672156413Z Wrote inspect data to must-gather. 2026-07-23T16:57:57.672199615Z error: inspection completed with the errors occurred while gathering data: 2026-07-23T16:57:57.672199615Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-23T16:57:57.803196722Z error: the server doesn't have a resource type "clusters" 2026-07-23T16:57:57.921938012Z Caches written to disk