2026-06-03T22:32:14.915472732Z [disk usage checker] Started 2026-06-03T22:32:14.919048452Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:15.112016219Z Gathering data for ns/openshift-cluster-version... 2026-06-03T22:32:15.253160635Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-06-03T22:32:15.334515110Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-03T22:32:15.413852030Z Gathering data for ns/default... 2026-06-03T22:32:15.676152442Z Gathering data for ns/openshift... 2026-06-03T22:32:16.096008722Z Gathering data for ns/kube-system... 2026-06-03T22:32:16.735809759Z Gathering data for ns/openshift-etcd... 2026-06-03T22:32:17.024759064Z Wrote inspect data to must-gather. 2026-06-03T22:32:17.024796909Z error: inspection completed with the errors occurred while gathering data: 2026-06-03T22:32:17.024796909Z namespaces "assisted-installer" not found 2026-06-03T22:32:17.502295069Z Waiting on subprocesses to finish execution. 2026-06-03T22:32:17.512452107Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-06-03T22:32:17.516229899Z INFO: Gathering HAProxy config files 2026-06-03T22:32:17.518011156Z INFO: Collecting host service logs for crio 2026-06-03T22:32:17.518349701Z INFO: Collecting host service logs for kubelet 2026-06-03T22:32:17.518695306Z INFO: Collecting host service logs for rpm-ostreed 2026-06-03T22:32:17.518977384Z Executing Istio gather script 2026-06-03T22:32:17.519057078Z INFO: Collecting host service logs for ostree-finalize-staged 2026-06-03T22:32:17.519401235Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-06-03T22:32:17.519725515Z INFO: Collecting host service logs for machine-config-daemon-host 2026-06-03T22:32:17.520063455Z INFO: Collecting host service logs for NetworkManager 2026-06-03T22:32:17.520372623Z INFO: Collecting host service logs for openvswitch 2026-06-03T22:32:17.520717060Z INFO: Collecting host service logs for ovs-configuration 2026-06-03T22:32:17.521050955Z INFO: Collecting host service logs for ovsdb-server 2026-06-03T22:32:17.521437910Z INFO: Collecting host service logs for ovs-vswitchd 2026-06-03T22:32:17.521826330Z INFO: Waiting for worker host service log collection to complete ... 2026-06-03T22:32:17.522578340Z INFO: Gathering machine config daemon's old logs from all nodes 2026-06-03T22:32:17.544560151Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-06-03T22:32:17.557725459Z INFO: Waiting for node performance related collection to complete ... 2026-06-03T22:32:18.033639110Z INFO: Collecting Insights Archives from insights-operator-69684796bc-8jfg2 2026-06-03T22:32:18.033639110Z insights-runtime-extractor-544wc 2026-06-03T22:32:18.033639110Z insights-runtime-extractor-7b9j4 2026-06-03T22:32:18.033639110Z insights-runtime-extractor-nkg55 2026-06-03T22:32:18.071572909Z INFO: "sriov-network-operator" not detected. Skipping. 2026-06-03T22:32:18.093201036Z INFO: "metallb-operator" not detected. Skipping. 2026-06-03T22:32:18.168542283Z No resources found 2026-06-03T22:32:18.224136280Z INFO: Found 1 replicas - prometheus-k8s-0 2026-06-03T22:32:18.445782951Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-06-03T22:32:18.445782951Z See 'oc rsync -h' for help and examples 2026-06-03T22:32:18.497205432Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-06-03T22:32:18.532085385Z No resources found in openshift-etcd namespace. 2026-06-03T22:32:18.584199320Z Wrote inspect data to must-gather. 2026-06-03T22:32:18.637182928Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-06-03T22:32:18.788055615Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-06-03T22:32:18.788167608Z INFO: on-disk MachineConfig config collection complete. 2026-06-03T22:32:18.936772010Z error: the server doesn't have a resource type "performanceprofile" 2026-06-03T22:32:18.952833009Z INFO: Worker host service log collection to complete. 2026-06-03T22:32:19.207471085Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-06-03T22:32:19.234029772Z Wrote inspect data to must-gather. 2026-06-03T22:32:19.509947139Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-06-03T22:32:19.753350189Z INFO: 'previous-logs' folder not found on ip-10-0-136-176.ec2.internal, skipping... 2026-06-03T22:32:19.769670615Z ERROR: No running kube-apiserver pods found 2026-06-03T22:32:19.789972863Z Wrote inspect data to must-gather. 2026-06-03T22:32:19.811663260Z No resources found 2026-06-03T22:32:19.822339250Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-03T22:32:19.829682743Z INFO: Waiting for HAProxy config collection to complete ... 2026-06-03T22:32:19.833808701Z Wrote inspect data to must-gather. 2026-06-03T22:32:19.882079201Z Wrote inspect data to must-gather. 2026-06-03T22:32:19.923726304Z Wrote inspect data to must-gather. 2026-06-03T22:32:19.930468160Z Gathering data for ns/openshift-monitoring... 2026-06-03T22:32:19.931285875Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:20.117087896Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.203487256Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-06-03T22:32:20.218004927Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.233141379Z error: the server doesn't have a resource type "machineconfigs" 2026-06-03T22:32:20.246953912Z tar: Removing leading `/' from member names 2026-06-03T22:32:20.248748134Z INFO: Getting rules from prometheus-k8s-0 2026-06-03T22:32:20.251715054Z INFO: HAProxy config collection complete. 2026-06-03T22:32:20.260045761Z INFO: 'previous-logs' folder not found on ip-10-0-138-219.ec2.internal, skipping... 2026-06-03T22:32:20.404173093Z error: the server doesn't have a resource type "machineconfigpools" 2026-06-03T22:32:20.483330872Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.530630859Z INFO: 'previous-logs' folder not found on ip-10-0-143-238.ec2.internal, skipping... 2026-06-03T22:32:20.530673105Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-06-03T22:32:20.530692790Z INFO: Machine Config Daemon termination log collection complete. 2026-06-03T22:32:20.606439543Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.666052763Z INFO: Getting status/config from prometheus-k8s-0 2026-06-03T22:32:20.731024011Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.778065268Z error: the server doesn't have a resource type "kubeletconfigs" 2026-06-03T22:32:20.985524206Z Wrote inspect data to must-gather. 2026-06-03T22:32:20.996081815Z INFO: INTERCONNECT MODE 2026-06-03T22:32:20.996160717Z INFO: Gathering ovn-kubernetes DBs 2026-06-03T22:32:21.069289464Z INFO: Getting status/flags from prometheus-k8s-0 2026-06-03T22:32:21.097925692Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-06-03T22:32:21.182473616Z INFO: Gathering OVN_Northbound from ovnkube-node-6f4rf... 2026-06-03T22:32:21.182825431Z INFO: Gathering OVN_Northbound from ovnkube-node-qn4bn... 2026-06-03T22:32:21.183125085Z INFO: Gathering OVN_Northbound from ovnkube-node-x5grv... 2026-06-03T22:32:21.183459804Z INFO: Gathering OVN_Southbound from ovnkube-node-6f4rf... 2026-06-03T22:32:21.183781953Z INFO: Gathering OVN_Southbound from ovnkube-node-qn4bn... 2026-06-03T22:32:21.184073754Z INFO: Gathering OVN_Southbound from ovnkube-node-x5grv... 2026-06-03T22:32:21.204265337Z Wrote inspect data to must-gather. 2026-06-03T22:32:21.380790883Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:21.389772414Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-06-03T22:32:21.508059233Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.508101005Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-06-03T22:32:21.521047863Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.532700917Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.559614430Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.588979963Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.589564119Z tar: Removing leading `/' from member names 2026-06-03T22:32:21.619779035Z Wrote inspect data to must-gather. 2026-06-03T22:32:21.670825551Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:21.673240086Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-06-03T22:32:21.706237108Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-06-03T22:32:21.864405976Z Wrote inspect data to must-gather. 2026-06-03T22:32:21.906938974Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:21.987793737Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-06-03T22:32:22.039696631Z INFO: Waiting for network log collection to complete ... 2026-06-03T22:32:22.039733187Z INFO: Waiting for ovnk database copies to complete ... 2026-06-03T22:32:22.040113413Z INFO: Copying ovnk databases complete. 2026-06-03T22:32:22.041969331Z 2.5M must-gather/network_logs/ovnk_database_store 2026-06-03T22:32:22.044232318Z ovnk_database_store/ 2026-06-03T22:32:22.044232318Z ovnk_database_store/ovnkube-node-6f4rf_nbdb 2026-06-03T22:32:22.053762804Z ovnk_database_store/ovnkube-node-qn4bn_nbdb 2026-06-03T22:32:22.060455598Z ovnk_database_store/ovnkube-node-6f4rf_sbdb 2026-06-03T22:32:22.080729172Z ovnk_database_store/ovnkube-node-x5grv_sbdb 2026-06-03T22:32:22.097440561Z ovnk_database_store/ovnkube-node-x5grv_nbdb 2026-06-03T22:32:22.099474037Z W0603 22:32:22.099425 1471 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-06-03T22:32:22.103156042Z ovnk_database_store/ovnkube-node-qn4bn_sbdb 2026-06-03T22:32:22.105918194Z Wrote inspect data to must-gather. 2026-06-03T22:32:22.125050997Z INFO: Network log collection complete. 2026-06-03T22:32:22.141790251Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:22.146706463Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-06-03T22:32:22.210271621Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-06-03T22:32:22.343690254Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:22.389707784Z Wrote inspect data to must-gather. 2026-06-03T22:32:22.414676972Z Gathering data for ns/openshift-network-console... 2026-06-03T22:32:22.424321723Z INFO: Getting status from alertmanager-main-0 2026-06-03T22:32:22.571016021Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:22.574151853Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-06-03T22:32:22.680852036Z Wrote inspect data to must-gather. 2026-06-03T22:32:22.788752954Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:22.869348657Z Gathering data for ns/openshift-console-operator... 2026-06-03T22:32:22.966149664Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-06-03T22:32:22.969362011Z INFO: Fallback to identify the container image from release info 2026-06-03T22:32:23.004617360Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:23.006945022Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-06-03T22:32:23.170150200Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:23.244323223Z Gathering data for ns/openshift-console... 2026-06-03T22:32:23.362659664Z Wrote inspect data to /must-gather/istio. 2026-06-03T22:32:23.608172083Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6566c94ad72a237c5dec66f76fcbc92b95df90f391fc7a044bce3efe63d698e2 2026-06-03T22:32:23.741571368Z daemonset.apps/perf-node-gather-daemonset created 2026-06-03T22:32:23.741955077Z Gathering data for ns/openshift-cluster-storage-operator... 2026-06-03T22:32:23.875698752Z Waiting for performance profile collector pods to become ready: 1 2026-06-03T22:32:24.039356945Z Done executing Istio gather script 2026-06-03T22:32:24.272671806Z Gathering data for ns/openshift-dns-operator... 2026-06-03T22:32:24.571401400Z Gathering data for ns/openshift-dns... 2026-06-03T22:32:24.937764723Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:24.992155526Z Waiting for performance profile collector pods to become ready: 2 2026-06-03T22:32:25.133950301Z Gathering data for ns/openshift-image-registry... 2026-06-03T22:32:25.535830663Z Gathering data for ns/openshift-ingress-operator... 2026-06-03T22:32:25.821721378Z Gathering data for ns/openshift-ingress... 2026-06-03T22:32:26.115927271Z Waiting for performance profile collector pods to become ready: 3 2026-06-03T22:32:26.133106285Z Gathering data for ns/openshift-ingress-canary... 2026-06-03T22:32:26.465600582Z Gathering data for ns/openshift-insights... 2026-06-03T22:32:27.223162944Z Waiting for performance profile collector pods to become ready: 4 2026-06-03T22:32:28.228573445Z Gathering data for ns/openshift-config... 2026-06-03T22:32:28.350875708Z Waiting for performance profile collector pods to become ready: 5 2026-06-03T22:32:28.537943306Z Gathering data for ns/openshift-config-managed... 2026-06-03T22:32:28.898446153Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-06-03T22:32:29.134545925Z Gathering data for ns/openshift-kube-apiserver... 2026-06-03T22:32:29.369917807Z Gathering data for ns/openshift-kube-controller-manager... 2026-06-03T22:32:29.472052598Z Waiting for performance profile collector pods to become ready: 6 2026-06-03T22:32:29.640806331Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-06-03T22:32:29.916918718Z Gathering data for ns/openshift-kube-scheduler... 2026-06-03T22:32:29.943096903Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:30.171276351Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-06-03T22:32:30.451085199Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-06-03T22:32:30.589169582Z Waiting for performance profile collector pods to become ready: 7 2026-06-03T22:32:30.754244147Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-06-03T22:32:31.041916967Z Gathering data for ns/openshift-user-workload-monitoring... 2026-06-03T22:32:31.489703872Z Gathering data for ns/openshift-multus... 2026-06-03T22:32:31.701571881Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-06-03T22:32:31.922438132Z Collecting performance related data for node ip-10-0-143-238.ec2.internal 2026-06-03T22:32:31.929327745Z Collecting performance related data for node ip-10-0-138-219.ec2.internal 2026-06-03T22:32:31.936886364Z Collecting performance related data for node ip-10-0-136-176.ec2.internal 2026-06-03T22:32:32.971320757Z Gathering data for ns/openshift-ovn-kubernetes... 2026-06-03T22:32:34.265086646Z Gathering data for ns/openshift-host-network... 2026-06-03T22:32:34.612910577Z Gathering data for ns/openshift-network-diagnostics... 2026-06-03T22:32:34.949801587Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:35.147946156Z Gathering data for ns/openshift-network-node-identity... 2026-06-03T22:32:35.516105079Z Gathering data for ns/openshift-network-operator... 2026-06-03T22:32:35.859076447Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-06-03T22:32:36.153944542Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-06-03T22:32:36.605396110Z Gathering data for ns/openshift-apiserver-operator... 2026-06-03T22:32:36.887898307Z Gathering data for ns/openshift-apiserver... 2026-06-03T22:32:37.258832432Z Gathering data for ns/openshift-controller-manager-operator... 2026-06-03T22:32:37.501877303Z Gathering data for ns/openshift-controller-manager... 2026-06-03T22:32:37.768351126Z Gathering data for ns/openshift-cluster-samples-operator... 2026-06-03T22:32:38.399002048Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-06-03T22:32:38.663088100Z Gathering data for ns/openshift-service-ca-operator... 2026-06-03T22:32:38.969359316Z Gathering data for ns/openshift-service-ca... 2026-06-03T22:32:39.316353038Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-06-03T22:32:39.955076145Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-06-03T22:32:40.195321411Z Wrote inspect data to must-gather. 2026-06-03T22:32:40.195370478Z error: inspection completed with the errors occurred while gathering data: 2026-06-03T22:32:40.195370478Z skipping gathering secrets/support due to error: secrets "support" not found 2026-06-03T22:32:40.337677267Z Collecting kubelet logs for node ip-10-0-143-238.ec2.internal 2026-06-03T22:32:40.339348766Z Collecting kubelet logs for node ip-10-0-138-219.ec2.internal 2026-06-03T22:32:40.340959460Z Collecting kubelet logs for node ip-10-0-136-176.ec2.internal 2026-06-03T22:32:40.641698620Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-rrwbp namespace 2026-06-03T22:32:40.647027768Z INFO: Node performance data collection complete. 2026-06-03T22:32:40.776300341Z error: the server doesn't have a resource type "clusters" 2026-06-03T22:32:40.949820405Z Caches written to disk