2026-06-15T06:57:46.822873126Z [disk usage checker] Started 2026-06-15T06:57:46.826650819Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:57:47.177315401Z Gathering data for ns/openshift-cluster-version... 2026-06-15T06:57:47.326401206Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-06-15T06:57:47.428981411Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-15T06:57:47.523216867Z Gathering data for ns/default... 2026-06-15T06:57:47.870858331Z Gathering data for ns/openshift... 2026-06-15T06:57:48.293511405Z Gathering data for ns/kube-system... 2026-06-15T06:57:49.052659526Z Gathering data for ns/openshift-etcd... 2026-06-15T06:57:49.411648488Z Wrote inspect data to must-gather. 2026-06-15T06:57:49.411677183Z error: inspection completed with the errors occurred while gathering data: 2026-06-15T06:57:49.411677183Z namespaces "assisted-installer" not found 2026-06-15T06:57:49.798270023Z Waiting on subprocesses to finish execution. 2026-06-15T06:57:49.807385843Z INFO: Gathering HAProxy config files 2026-06-15T06:57:49.807607820Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-06-15T06:57:49.811010775Z INFO: Gathering machine config daemon's old logs from all nodes 2026-06-15T06:57:49.819517199Z Executing Istio gather script 2026-06-15T06:57:49.823065629Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-06-15T06:57:49.862450184Z INFO: Collecting host service logs for crio 2026-06-15T06:57:49.862812176Z INFO: Collecting host service logs for kubelet 2026-06-15T06:57:49.863153355Z INFO: Collecting host service logs for rpm-ostreed 2026-06-15T06:57:49.863489104Z INFO: Collecting host service logs for ostree-finalize-staged 2026-06-15T06:57:49.863805517Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-06-15T06:57:49.864118612Z INFO: Collecting host service logs for machine-config-daemon-host 2026-06-15T06:57:49.864455776Z INFO: Collecting host service logs for NetworkManager 2026-06-15T06:57:49.864762899Z INFO: Collecting host service logs for openvswitch 2026-06-15T06:57:49.865061631Z INFO: Collecting host service logs for ovs-configuration 2026-06-15T06:57:49.865371582Z INFO: Collecting host service logs for ovsdb-server 2026-06-15T06:57:49.865665170Z INFO: Collecting host service logs for ovs-vswitchd 2026-06-15T06:57:49.866003592Z INFO: Waiting for worker host service log collection to complete ... 2026-06-15T06:57:49.915978489Z INFO: Waiting for node performance related collection to complete ... 2026-06-15T06:57:50.207716262Z No resources found 2026-06-15T06:57:50.516420934Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-06-15T06:57:50.600759897Z Wrote inspect data to must-gather. 2026-06-15T06:57:50.815632032Z INFO: Collecting Insights Archives from insights-operator-5bbd86d6bd-vbmdw 2026-06-15T06:57:50.815676440Z insights-runtime-extractor-b277w 2026-06-15T06:57:50.815696388Z insights-runtime-extractor-l4gs5 2026-06-15T06:57:50.815715628Z insights-runtime-extractor-xpdg2 2026-06-15T06:57:50.904712933Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-06-15T06:57:50.927580688Z INFO: "sriov-network-operator" not detected. Skipping. 2026-06-15T06:57:50.969639638Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-06-15T06:57:50.984624746Z INFO: on-disk MachineConfig config collection complete. 2026-06-15T06:57:51.042320971Z No resources found in openshift-etcd namespace. 2026-06-15T06:57:51.181944200Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-06-15T06:57:51.181944200Z See 'oc rsync -h' for help and examples 2026-06-15T06:57:51.187954084Z INFO: Found 1 replicas - prometheus-k8s-0 2026-06-15T06:57:51.243514341Z INFO: "metallb-operator" not detected. Skipping. 2026-06-15T06:57:51.321475146Z INFO: Worker host service log collection to complete. 2026-06-15T06:57:51.394188919Z error: the server doesn't have a resource type "performanceprofile" 2026-06-15T06:57:51.420890497Z Wrote inspect data to must-gather. 2026-06-15T06:57:51.484226118Z INFO: 'previous-logs' folder not found on ip-10-0-131-82.ec2.internal, skipping... 2026-06-15T06:57:51.591097590Z ERROR: No running kube-apiserver pods found 2026-06-15T06:57:51.797352392Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-06-15T06:57:51.855281166Z INFO: Waiting for HAProxy config collection to complete ... 2026-06-15T06:57:52.084699731Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:57:52.099146741Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-15T06:57:52.207900922Z Wrote inspect data to must-gather. 2026-06-15T06:57:52.337184405Z Wrote inspect data to must-gather. 2026-06-15T06:57:52.340845234Z Wrote inspect data to must-gather. 2026-06-15T06:57:52.344982447Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-06-15T06:57:52.436654570Z tar: Removing leading `/' from member names 2026-06-15T06:57:52.447834114Z INFO: HAProxy config collection complete. 2026-06-15T06:57:52.526321560Z INFO: 'previous-logs' folder not found on ip-10-0-135-102.ec2.internal, skipping... 2026-06-15T06:57:52.559279335Z No resources found 2026-06-15T06:57:52.710096071Z error: the server doesn't have a resource type "machineconfigs" 2026-06-15T06:57:52.718564831Z Wrote inspect data to must-gather. 2026-06-15T06:57:52.788583450Z Wrote inspect data to must-gather. 2026-06-15T06:57:52.851839290Z Gathering data for ns/openshift-monitoring... 2026-06-15T06:57:52.902473057Z INFO: Getting rules from prometheus-k8s-0 2026-06-15T06:57:52.933519070Z error: the server doesn't have a resource type "machineconfigpools" 2026-06-15T06:57:52.986225183Z INFO: 'previous-logs' folder not found on ip-10-0-138-16.ec2.internal, skipping... 2026-06-15T06:57:52.986388311Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-06-15T06:57:52.986467265Z INFO: Machine Config Daemon termination log collection complete. 2026-06-15T06:57:53.105960277Z Wrote inspect data to must-gather. 2026-06-15T06:57:53.220576927Z Wrote inspect data to must-gather. 2026-06-15T06:57:53.417752172Z Wrote inspect data to must-gather. 2026-06-15T06:57:53.421603171Z error: the server doesn't have a resource type "kubeletconfigs" 2026-06-15T06:57:53.425433916Z INFO: Getting status/config from prometheus-k8s-0 2026-06-15T06:57:53.494711735Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-06-15T06:57:53.680792289Z Wrote inspect data to must-gather. 2026-06-15T06:57:53.701454758Z Wrote inspect data to must-gather. 2026-06-15T06:57:53.910711340Z INFO: Getting status/flags from prometheus-k8s-0 2026-06-15T06:57:53.911549797Z Wrote inspect data to must-gather. 2026-06-15T06:57:54.044091798Z INFO: INTERCONNECT MODE 2026-06-15T06:57:54.044184644Z INFO: Gathering ovn-kubernetes DBs 2026-06-15T06:57:54.143136592Z Wrote inspect data to must-gather. 2026-06-15T06:57:54.161498104Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-06-15T06:57:54.222613006Z INFO: Gathering OVN_Northbound from ovnkube-node-6dkxs... 2026-06-15T06:57:54.222874762Z INFO: Gathering OVN_Northbound from ovnkube-node-7qndp... 2026-06-15T06:57:54.223119963Z INFO: Gathering OVN_Northbound from ovnkube-node-98qt9... 2026-06-15T06:57:54.223421566Z INFO: Gathering OVN_Southbound from ovnkube-node-6dkxs... 2026-06-15T06:57:54.223679031Z INFO: Gathering OVN_Southbound from ovnkube-node-7qndp... 2026-06-15T06:57:54.223972814Z INFO: Gathering OVN_Southbound from ovnkube-node-98qt9... 2026-06-15T06:57:54.532946977Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-06-15T06:57:54.535251133Z Wrote inspect data to must-gather. 2026-06-15T06:57:54.587473649Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-06-15T06:57:54.633244230Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.641308700Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.671602957Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.684849042Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.689293390Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-06-15T06:57:54.700298007Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.728263943Z tar: Removing leading `/' from member names 2026-06-15T06:57:54.826233565Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:54.840013834Z Wrote inspect data to must-gather. 2026-06-15T06:57:54.897547183Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-06-15T06:57:55.127905531Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:55.130927993Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-06-15T06:57:55.152877731Z Wrote inspect data to must-gather. 2026-06-15T06:57:55.267591968Z INFO: Waiting for network log collection to complete ... 2026-06-15T06:57:55.267630296Z INFO: Waiting for ovnk database copies to complete ... 2026-06-15T06:57:55.268126895Z INFO: Copying ovnk databases complete. 2026-06-15T06:57:55.271394896Z 4.4M must-gather/network_logs/ovnk_database_store 2026-06-15T06:57:55.275749404Z ovnk_database_store/ 2026-06-15T06:57:55.275749404Z ovnk_database_store/ovnkube-node-6dkxs_nbdb 2026-06-15T06:57:55.287608107Z ovnk_database_store/ovnkube-node-7qndp_sbdb 2026-06-15T06:57:55.314724316Z ovnk_database_store/ovnkube-node-98qt9_sbdb 2026-06-15T06:57:55.349464827Z ovnk_database_store/ovnkube-node-7qndp_nbdb 2026-06-15T06:57:55.361998063Z ovnk_database_store/ovnkube-node-98qt9_nbdb 2026-06-15T06:57:55.369427120Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:55.375984922Z ovnk_database_store/ovnkube-node-6dkxs_sbdb 2026-06-15T06:57:55.397493885Z INFO: Getting status from alertmanager-main-0 2026-06-15T06:57:55.403070284Z INFO: Network log collection complete. 2026-06-15T06:57:55.413642469Z W0615 06:57:55.413592 1515 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-06-15T06:57:55.422965963Z Wrote inspect data to must-gather. 2026-06-15T06:57:55.447903224Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-15T06:57:55.619724501Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:55.623191216Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-06-15T06:57:55.663730821Z Gathering data for ns/openshift-network-console... 2026-06-15T06:57:55.745149666Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-06-15T06:57:55.749194654Z INFO: Fallback to identify the container image from release info 2026-06-15T06:57:55.837060440Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.047518533Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.050702728Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-06-15T06:57:56.135736382Z Gathering data for ns/openshift-console-operator... 2026-06-15T06:57:56.291304100Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.305762172Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d66a6b2ec52dede5d83ce04f6abfc519df1fd7dc36248741e03ecb9bf8d1762 2026-06-15T06:57:56.443747576Z daemonset.apps/perf-node-gather-daemonset created 2026-06-15T06:57:56.515581196Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.520797258Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-06-15T06:57:56.599819998Z Waiting for performance profile collector pods to become ready: 1 2026-06-15T06:57:56.622479917Z Gathering data for ns/openshift-console... 2026-06-15T06:57:56.695620431Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.909307974Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:56.912958137Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-06-15T06:57:57.096021095Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:57:57.097094824Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:57.243046901Z Gathering data for ns/openshift-cluster-storage-operator... 2026-06-15T06:57:57.316938488Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:57.321576955Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-06-15T06:57:57.486936655Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:57.710482066Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:57.715226950Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-06-15T06:57:57.758495486Z Waiting for performance profile collector pods to become ready: 2 2026-06-15T06:57:57.825225713Z Gathering data for ns/openshift-dns-operator... 2026-06-15T06:57:57.915688891Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.118202240Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.121558653Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-06-15T06:57:58.164496733Z Gathering data for ns/openshift-dns... 2026-06-15T06:57:58.282008184Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.479887190Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.483753373Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-06-15T06:57:58.642059239Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.835095864Z Gathering data for ns/openshift-image-registry... 2026-06-15T06:57:58.852872170Z Wrote inspect data to /must-gather/istio. 2026-06-15T06:57:58.907240372Z Waiting for performance profile collector pods to become ready: 3 2026-06-15T06:57:59.379605728Z Gathering data for ns/openshift-ingress-operator... 2026-06-15T06:57:59.557319447Z Done executing Istio gather script 2026-06-15T06:57:59.738484667Z Gathering data for ns/openshift-ingress... 2026-06-15T06:58:00.040412184Z Waiting for performance profile collector pods to become ready: 4 2026-06-15T06:58:00.097122147Z Gathering data for ns/openshift-ingress-canary... 2026-06-15T06:58:00.510677244Z Gathering data for ns/openshift-insights... 2026-06-15T06:58:01.171427675Z Waiting for performance profile collector pods to become ready: 5 2026-06-15T06:58:02.101212489Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:58:02.296023279Z Waiting for performance profile collector pods to become ready: 6 2026-06-15T06:58:02.782402288Z Gathering data for ns/kserve... 2026-06-15T06:58:03.420456632Z Waiting for performance profile collector pods to become ready: 7 2026-06-15T06:58:04.535808992Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-06-15T06:58:04.697221995Z Gathering data for ns/openshift-config... 2026-06-15T06:58:04.804448600Z Collecting performance related data for node ip-10-0-131-82.ec2.internal 2026-06-15T06:58:04.811019604Z Collecting performance related data for node ip-10-0-138-16.ec2.internal 2026-06-15T06:58:04.817935905Z Collecting performance related data for node ip-10-0-135-102.ec2.internal 2026-06-15T06:58:05.038381246Z Gathering data for ns/openshift-config-managed... 2026-06-15T06:58:05.542925945Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-06-15T06:58:05.915234458Z Gathering data for ns/openshift-kube-apiserver... 2026-06-15T06:58:06.275816508Z Gathering data for ns/openshift-kube-controller-manager... 2026-06-15T06:58:06.648433434Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-06-15T06:58:06.985002637Z Gathering data for ns/openshift-kube-scheduler... 2026-06-15T06:58:07.105761555Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:58:07.321133473Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-06-15T06:58:07.704484086Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-06-15T06:58:08.100710243Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-06-15T06:58:08.454387662Z Gathering data for ns/openshift-user-workload-monitoring... 2026-06-15T06:58:09.001161561Z Gathering data for ns/openshift-multus... 2026-06-15T06:58:10.598729503Z Gathering data for ns/openshift-ovn-kubernetes... 2026-06-15T06:58:11.927753881Z Gathering data for ns/openshift-host-network... 2026-06-15T06:58:12.111819909Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:58:12.296256350Z Gathering data for ns/openshift-network-diagnostics... 2026-06-15T06:58:12.902826013Z Gathering data for ns/openshift-network-node-identity... 2026-06-15T06:58:13.346444522Z Collecting kubelet logs for node ip-10-0-131-82.ec2.internal 2026-06-15T06:58:13.348288080Z Collecting kubelet logs for node ip-10-0-138-16.ec2.internal 2026-06-15T06:58:13.350479813Z Collecting kubelet logs for node ip-10-0-135-102.ec2.internal 2026-06-15T06:58:13.392895600Z Gathering data for ns/openshift-network-operator... 2026-06-15T06:58:13.781385715Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-w2k28 namespace 2026-06-15T06:58:13.787221160Z INFO: Node performance data collection complete. 2026-06-15T06:58:13.848929266Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-06-15T06:58:14.173945381Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-06-15T06:58:14.726414060Z Gathering data for ns/openshift-apiserver-operator... 2026-06-15T06:58:15.047404967Z Gathering data for ns/openshift-apiserver... 2026-06-15T06:58:15.458738171Z Gathering data for ns/openshift-controller-manager-operator... 2026-06-15T06:58:15.746312364Z Gathering data for ns/openshift-controller-manager... 2026-06-15T06:58:16.080197531Z Gathering data for ns/openshift-cluster-samples-operator... 2026-06-15T06:58:16.759742626Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-06-15T06:58:17.117388031Z [disk usage checker] Volume usage percentage: current = 27 ; allowed = 70 2026-06-15T06:58:17.137959198Z Gathering data for ns/openshift-service-ca-operator... 2026-06-15T06:58:17.588543646Z Gathering data for ns/openshift-service-ca... 2026-06-15T06:58:18.039118759Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-06-15T06:58:19.024432361Z Wrote inspect data to must-gather. 2026-06-15T06:58:19.024478946Z error: inspection completed with the errors occurred while gathering data: 2026-06-15T06:58:19.024478946Z skipping gathering secrets/support due to error: secrets "support" not found 2026-06-15T06:58:19.151519437Z error: the server doesn't have a resource type "clusters" 2026-06-15T06:58:19.318896436Z Caches written to disk