2026-07-28T13:11:50.836175228Z [disk usage checker] Started
2026-07-28T13:11:50.839467194Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:11:51.045351767Z Gathering data for ns/openshift-cluster-version...
2026-07-28T13:11:51.188590438Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T13:11:51.285308075Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:11:51.379160829Z Gathering data for ns/default...
2026-07-28T13:11:51.708461960Z Gathering data for ns/openshift...
2026-07-28T13:11:52.137823279Z Gathering data for ns/kube-system...
2026-07-28T13:11:52.926728358Z Gathering data for ns/openshift-etcd...
2026-07-28T13:11:53.260734366Z Wrote inspect data to must-gather.
2026-07-28T13:11:53.260776926Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T13:11:53.260776926Z     namespaces "assisted-installer" not found
2026-07-28T13:11:53.562557057Z Waiting on subprocesses to finish execution.
2026-07-28T13:11:53.569006686Z INFO: Gathering HAProxy config files
2026-07-28T13:11:53.570665986Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T13:11:53.571573909Z INFO: Collecting host service logs for crio
2026-07-28T13:11:53.571948093Z INFO: Collecting host service logs for kubelet
2026-07-28T13:11:53.572296398Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T13:11:53.572624448Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T13:11:53.573056477Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T13:11:53.573406710Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T13:11:53.573746550Z INFO: Collecting host service logs for NetworkManager
2026-07-28T13:11:53.574174148Z INFO: Collecting host service logs for openvswitch
2026-07-28T13:11:53.574488524Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T13:11:53.574801232Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T13:11:53.575169237Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T13:11:53.575574694Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T13:11:53.575855972Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T13:11:53.597023291Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T13:11:53.621407197Z Executing Istio gather script
2026-07-28T13:11:53.639410902Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T13:11:53.859297495Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T13:11:54.174265952Z Wrote inspect data to must-gather.
2026-07-28T13:11:54.282381613Z No resources found in openshift-etcd namespace.
2026-07-28T13:11:54.416215222Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T13:11:54.597569375Z No resources found
2026-07-28T13:11:54.618963552Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T13:11:54.619046113Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T13:11:54.639451566Z ERROR: No running kube-apiserver pods found
2026-07-28T13:11:54.837609872Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T13:11:55.009250364Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T13:11:55.035725732Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T13:11:55.097372139Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T13:11:55.151547361Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T13:11:55.245817578Z INFO: Worker host service log collection to complete.
2026-07-28T13:11:55.360451905Z No resources found
2026-07-28T13:11:55.388939566Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-ld6p6
2026-07-28T13:11:55.389002786Z insights-runtime-extractor-b7g7l
2026-07-28T13:11:55.389026056Z insights-runtime-extractor-f58jt
2026-07-28T13:11:55.389053574Z insights-runtime-extractor-nn2gb
2026-07-28T13:11:55.511739262Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:11:55.714687632Z Wrote inspect data to must-gather.
2026-07-28T13:11:55.835711476Z Wrote inspect data to must-gather.
2026-07-28T13:11:55.856535289Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:11:55.937765315Z Wrote inspect data to must-gather.
2026-07-28T13:11:55.984906570Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.020084533Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T13:11:56.033027792Z INFO: 'previous-logs' folder not found on ip-10-0-132-15.ec2.internal, skipping...
2026-07-28T13:11:56.061879886Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T13:11:56.061879886Z See 'oc rsync -h' for help and examples
2026-07-28T13:11:56.083718679Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T13:11:56.092975640Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.127345477Z tar: Removing leading `/' from member names
2026-07-28T13:11:56.132467185Z INFO: HAProxy config collection complete.
2026-07-28T13:11:56.174679199Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.244979836Z Gathering data for ns/openshift-monitoring...
2026-07-28T13:11:56.271538960Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T13:11:56.412010144Z INFO: 'previous-logs' folder not found on ip-10-0-138-58.ec2.internal, skipping...
2026-07-28T13:11:56.478307441Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T13:11:56.483725653Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.590223074Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T13:11:56.735513274Z INFO: 'previous-logs' folder not found on ip-10-0-142-139.ec2.internal, skipping...
2026-07-28T13:11:56.735543538Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T13:11:56.735573075Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T13:11:56.768869609Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.802068586Z Wrote inspect data to must-gather.
2026-07-28T13:11:56.860950018Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T13:11:56.937090329Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T13:11:57.044273610Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T13:11:57.087668812Z Wrote inspect data to must-gather.
2026-07-28T13:11:57.159986067Z Wrote inspect data to must-gather.
2026-07-28T13:11:57.385096429Z Wrote inspect data to must-gather.
2026-07-28T13:11:57.399519119Z INFO: INTERCONNECT MODE
2026-07-28T13:11:57.399545857Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T13:11:57.492874872Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T13:11:57.583706219Z INFO: Gathering OVN_Northbound from ovnkube-node-45gqf...
2026-07-28T13:11:57.583996999Z INFO: Gathering OVN_Northbound from ovnkube-node-j9tmm...
2026-07-28T13:11:57.584243571Z INFO: Gathering OVN_Northbound from ovnkube-node-qbd9x...
2026-07-28T13:11:57.584533575Z INFO: Gathering OVN_Southbound from ovnkube-node-45gqf...
2026-07-28T13:11:57.584843670Z INFO: Gathering OVN_Southbound from ovnkube-node-j9tmm...
2026-07-28T13:11:57.585156387Z INFO: Gathering OVN_Southbound from ovnkube-node-qbd9x...
2026-07-28T13:11:57.630129009Z Wrote inspect data to must-gather.
2026-07-28T13:11:57.812961921Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T13:11:57.877117362Z tar: Removing leading `/' from member names
2026-07-28T13:11:57.934075015Z tar: Removing leading `/' from member names
2026-07-28T13:11:57.942359799Z tar: Removing leading `/' from member names
2026-07-28T13:11:57.951249486Z tar: Removing leading `/' from member names
2026-07-28T13:11:57.970708901Z tar: Removing leading `/' from member names
2026-07-28T13:11:58.001489023Z tar: Removing leading `/' from member names
2026-07-28T13:11:58.003174504Z Wrote inspect data to must-gather.
2026-07-28T13:11:58.101887315Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T13:11:58.139685700Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T13:11:58.249260366Z Wrote inspect data to must-gather.
2026-07-28T13:11:58.253067349Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T13:11:58.436851572Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T13:11:58.470320364Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:58.503741026Z INFO: Waiting for network log collection to complete ...
2026-07-28T13:11:58.503775170Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T13:11:58.503840313Z Wrote inspect data to must-gather.
2026-07-28T13:11:58.504286448Z INFO: Copying ovnk databases complete.
2026-07-28T13:11:58.505935656Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-28T13:11:58.508169322Z ovnk_database_store/
2026-07-28T13:11:58.508220679Z ovnk_database_store/ovnkube-node-45gqf_sbdb
2026-07-28T13:11:58.532996072Z ovnk_database_store/ovnkube-node-45gqf_nbdb
2026-07-28T13:11:58.540570908Z ovnk_database_store/ovnkube-node-qbd9x_sbdb
2026-07-28T13:11:58.564875877Z ovnk_database_store/ovnkube-node-qbd9x_nbdb
2026-07-28T13:11:58.574101169Z ovnk_database_store/ovnkube-node-j9tmm_sbdb
2026-07-28T13:11:58.595794558Z ovnk_database_store/ovnkube-node-j9tmm_nbdb
2026-07-28T13:11:58.605899167Z INFO: Network log collection complete.
2026-07-28T13:11:58.719766010Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:58.720450334Z W0728 13:11:58.720411    1496 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T13:11:58.722807335Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T13:11:58.725330961Z Wrote inspect data to must-gather.
2026-07-28T13:11:58.787766665Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:11:58.884282185Z INFO: Getting status from alertmanager-main-0
2026-07-28T13:11:58.956421978Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:59.021719284Z Gathering data for ns/openshift-network-console...
2026-07-28T13:11:59.027389948Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T13:11:59.030529125Z INFO: Fallback to identify the container image from release info
2026-07-28T13:11:59.179315632Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:59.181410389Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T13:11:59.375045909Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:59.480895600Z Gathering data for ns/openshift-console-operator...
2026-07-28T13:11:59.594722578Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T13:11:59.601763290Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:59.604574358Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T13:11:59.728388646Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T13:11:59.832729565Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:11:59.876710602Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T13:11:59.920027349Z Gathering data for ns/openshift-console...
2026-07-28T13:12:00.040536693Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:00.043714267Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T13:12:00.228929774Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:00.432237556Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:00.435463942Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T13:12:00.474895134Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T13:12:00.605910579Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:00.806764574Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:00.809396271Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T13:12:00.879661749Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:12:01.014361762Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:01.025186159Z Gathering data for ns/openshift-dns-operator...
2026-07-28T13:12:01.030120569Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T13:12:01.226327361Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:12:01.358543854Z Gathering data for ns/openshift-dns...
2026-07-28T13:12:01.932194294Z Done executing Istio gather script
2026-07-28T13:12:02.019186134Z Gathering data for ns/openshift-image-registry...
2026-07-28T13:12:02.139903100Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T13:12:02.504292228Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T13:12:02.872585421Z Gathering data for ns/openshift-ingress...
2026-07-28T13:12:03.231452210Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T13:12:03.269620439Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T13:12:03.641783619Z Gathering data for ns/openshift-insights...
2026-07-28T13:12:04.398716101Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T13:12:05.524226171Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T13:12:05.887507482Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:12:06.086536560Z Gathering data for ns/kserve...
2026-07-28T13:12:06.649940803Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T13:12:07.771352169Z Waiting for performance profile collector pods to become ready: 8
2026-07-28T13:12:07.846446854Z Gathering data for ns/openshift-config...
2026-07-28T13:12:08.169674724Z Gathering data for ns/openshift-config-managed...
2026-07-28T13:12:08.634994356Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T13:12:08.905194380Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T13:12:08.924182603Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T13:12:09.179257597Z Collecting performance related data for node ip-10-0-142-139.ec2.internal
2026-07-28T13:12:09.188057741Z Collecting performance related data for node ip-10-0-138-58.ec2.internal
2026-07-28T13:12:09.199096872Z Collecting performance related data for node ip-10-0-132-15.ec2.internal
2026-07-28T13:12:09.267365336Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T13:12:09.644422538Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T13:12:09.995417602Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T13:12:10.360658640Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T13:12:10.745307591Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T13:12:10.892025833Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:12:11.094395033Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T13:12:11.435490438Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T13:12:12.023540308Z Gathering data for ns/openshift-multus...
2026-07-28T13:12:13.731769631Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T13:12:15.148192327Z Gathering data for ns/openshift-host-network...
2026-07-28T13:12:15.472239035Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T13:12:15.897705313Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:12:16.010733157Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T13:12:16.470483855Z Gathering data for ns/openshift-network-operator...
2026-07-28T13:12:16.925507539Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T13:12:17.235363638Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T13:12:17.448410256Z Collecting kubelet logs for node ip-10-0-142-139.ec2.internal
2026-07-28T13:12:17.450111801Z Collecting kubelet logs for node ip-10-0-138-58.ec2.internal
2026-07-28T13:12:17.451779916Z Collecting kubelet logs for node ip-10-0-132-15.ec2.internal
2026-07-28T13:12:17.792397949Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T13:12:17.832710986Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-4r45x namespace
2026-07-28T13:12:17.839985386Z INFO: Node performance data collection complete.
2026-07-28T13:12:18.169808029Z Gathering data for ns/openshift-apiserver...
2026-07-28T13:12:18.616310878Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T13:12:18.891210515Z Gathering data for ns/openshift-controller-manager...
2026-07-28T13:12:19.187315225Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T13:12:19.821387700Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T13:12:20.115103054Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T13:12:20.454567449Z Gathering data for ns/openshift-service-ca...
2026-07-28T13:12:20.833226098Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T13:12:20.902393952Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:12:21.752861716Z Wrote inspect data to must-gather.
2026-07-28T13:12:21.752909327Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T13:12:21.752909327Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T13:12:21.879779871Z error: the server doesn't have a resource type "clusters"
2026-07-28T13:12:22.019349373Z Caches written to disk