2026-07-16T03:53:51.384704276Z [disk usage checker] Started
2026-07-16T03:53:51.388085569Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:53:51.587782187Z Gathering data for ns/openshift-cluster-version...
2026-07-16T03:53:51.727665316Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T03:53:51.817653403Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T03:53:51.898687800Z Gathering data for ns/default...
2026-07-16T03:53:52.227134332Z Gathering data for ns/openshift...
2026-07-16T03:53:52.640044047Z Gathering data for ns/kube-system...
2026-07-16T03:53:53.377421757Z Gathering data for ns/openshift-etcd...
2026-07-16T03:53:53.694081632Z Wrote inspect data to must-gather.
2026-07-16T03:53:53.694113553Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T03:53:53.694113553Z     namespaces "assisted-installer" not found
2026-07-16T03:53:53.920031863Z Waiting on subprocesses to finish execution.
2026-07-16T03:53:53.923004158Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T03:53:53.927028221Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T03:53:53.929017930Z INFO: Collecting host service logs for crio
2026-07-16T03:53:53.929383546Z INFO: Collecting host service logs for kubelet
2026-07-16T03:53:53.929739469Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T03:53:53.930073952Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T03:53:53.930501653Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T03:53:53.930842902Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T03:53:53.931188122Z INFO: Collecting host service logs for NetworkManager
2026-07-16T03:53:53.931520313Z INFO: Collecting host service logs for openvswitch
2026-07-16T03:53:53.931857547Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T03:53:53.932192426Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T03:53:53.932530751Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T03:53:53.932952557Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T03:53:53.945930154Z INFO: Gathering HAProxy config files
2026-07-16T03:53:53.957193822Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T03:53:53.959025863Z Executing Istio gather script
2026-07-16T03:53:54.095351143Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T03:53:54.304601125Z No resources found in openshift-etcd namespace.
2026-07-16T03:53:54.758869150Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-xlwgv
2026-07-16T03:53:54.758914135Z insights-runtime-extractor-brpg8
2026-07-16T03:53:54.758935142Z insights-runtime-extractor-gd9rj
2026-07-16T03:53:54.758954451Z insights-runtime-extractor-x2gfn
2026-07-16T03:53:54.890280705Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T03:53:55.011130438Z No resources found
2026-07-16T03:53:55.045843487Z Wrote inspect data to must-gather.
2026-07-16T03:53:55.281545536Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T03:53:55.287385559Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T03:53:55.329265486Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T03:53:55.343358518Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T03:53:55.444577961Z INFO: Worker host service log collection to complete.
2026-07-16T03:53:55.557232221Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T03:53:55.557232221Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T03:53:55.569322710Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T03:53:55.569322710Z See 'oc rsync -h' for help and examples
2026-07-16T03:53:55.895618206Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T03:53:55.989675386Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T03:53:55.997089611Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.025930157Z No resources found
2026-07-16T03:53:56.071178091Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T03:53:56.139180384Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T03:53:56.143619180Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.207075568Z ERROR: No running kube-apiserver pods found
2026-07-16T03:53:56.240490665Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T03:53:56.327252094Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.367070187Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.389813095Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.408897236Z INFO: 'previous-logs' folder not found on ip-10-0-130-168.ec2.internal, skipping...
2026-07-16T03:53:56.428750461Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:53:56.477633655Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.575366222Z Gathering data for ns/openshift-monitoring...
2026-07-16T03:53:56.595658309Z tar: Removing leading `/' from member names
2026-07-16T03:53:56.607715432Z INFO: HAProxy config collection complete.
2026-07-16T03:53:56.647181478Z Wrote inspect data to must-gather.
2026-07-16T03:53:56.673555220Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T03:53:56.732378426Z INFO: 'previous-logs' folder not found on ip-10-0-136-52.ec2.internal, skipping...
2026-07-16T03:53:56.777910478Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T03:53:56.860664953Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T03:53:56.953173617Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T03:53:56.955461358Z Wrote inspect data to must-gather.
2026-07-16T03:53:57.032250271Z INFO: 'previous-logs' folder not found on ip-10-0-141-197.ec2.internal, skipping...
2026-07-16T03:53:57.032409325Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T03:53:57.032469062Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T03:53:57.116695376Z Wrote inspect data to must-gather.
2026-07-16T03:53:57.187496965Z Wrote inspect data to must-gather.
2026-07-16T03:53:57.197096787Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T03:53:57.277569841Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T03:53:57.471679462Z INFO: INTERCONNECT MODE
2026-07-16T03:53:57.471679462Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T03:53:57.479025463Z Wrote inspect data to must-gather.
2026-07-16T03:53:57.587457036Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T03:53:57.690539312Z Wrote inspect data to must-gather.
2026-07-16T03:53:57.699119806Z INFO: Gathering OVN_Northbound from ovnkube-node-fm2z2...
2026-07-16T03:53:57.699378461Z INFO: Gathering OVN_Northbound from ovnkube-node-rx6vb...
2026-07-16T03:53:57.699560969Z INFO: Gathering OVN_Northbound from ovnkube-node-zt7cr...
2026-07-16T03:53:57.699822085Z INFO: Gathering OVN_Southbound from ovnkube-node-fm2z2...
2026-07-16T03:53:57.700080300Z INFO: Gathering OVN_Southbound from ovnkube-node-rx6vb...
2026-07-16T03:53:57.700383321Z INFO: Gathering OVN_Southbound from ovnkube-node-zt7cr...
2026-07-16T03:53:57.866984813Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T03:53:57.933989344Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T03:53:57.982217875Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.013825193Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.024510444Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.039079983Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.042409481Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.044141218Z tar: Removing leading `/' from member names
2026-07-16T03:53:58.081732388Z Wrote inspect data to must-gather.
2026-07-16T03:53:58.162705408Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:58.186671600Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T03:53:58.230583064Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T03:53:58.354684647Z Wrote inspect data to must-gather.
2026-07-16T03:53:58.403281848Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:58.406450946Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T03:53:58.448008167Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T03:53:58.500608574Z INFO: Waiting for network log collection to complete ...
2026-07-16T03:53:58.500650087Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T03:53:58.501165061Z INFO: Copying ovnk databases complete.
2026-07-16T03:53:58.503003592Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-16T03:53:58.507155694Z ovnk_database_store/
2026-07-16T03:53:58.507220964Z ovnk_database_store/ovnkube-node-zt7cr_nbdb
2026-07-16T03:53:58.517154285Z ovnk_database_store/ovnkube-node-fm2z2_sbdb
2026-07-16T03:53:58.535750903Z ovnk_database_store/ovnkube-node-fm2z2_nbdb
2026-07-16T03:53:58.544874514Z ovnk_database_store/ovnkube-node-zt7cr_sbdb
2026-07-16T03:53:58.565013987Z ovnk_database_store/ovnkube-node-rx6vb_sbdb
2026-07-16T03:53:58.582400465Z ovnk_database_store/ovnkube-node-rx6vb_nbdb
2026-07-16T03:53:58.593288319Z Wrote inspect data to must-gather.
2026-07-16T03:53:58.593804277Z INFO: Network log collection complete.
2026-07-16T03:53:58.625953174Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:58.796172245Z W0716 03:53:58.796128    1515 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-16T03:53:58.801677699Z Wrote inspect data to must-gather.
2026-07-16T03:53:58.838421925Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:58.841370320Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T03:53:58.856478021Z INFO: Getting status from alertmanager-main-0
2026-07-16T03:53:58.880603859Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T03:53:59.040974461Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:59.067020764Z Gathering data for ns/openshift-network-console...
2026-07-16T03:53:59.082769585Z Wrote inspect data to must-gather.
2026-07-16T03:53:59.242938863Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:59.245441712Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T03:53:59.347190915Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T03:53:59.351406347Z INFO: Fallback to identify the container image from release info
2026-07-16T03:53:59.477646554Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:59.539986800Z Gathering data for ns/openshift-console-operator...
2026-07-16T03:53:59.672094199Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:59.673940772Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T03:53:59.828204725Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:53:59.948299187Z Gathering data for ns/openshift-console...
2026-07-16T03:53:59.967173584Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-16T03:54:00.026148345Z Wrote inspect data to /must-gather/istio.
2026-07-16T03:54:00.100628039Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T03:54:00.222361105Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T03:54:00.517251967Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T03:54:00.716713454Z Done executing Istio gather script
2026-07-16T03:54:01.025058130Z Gathering data for ns/openshift-dns-operator...
2026-07-16T03:54:01.347945723Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T03:54:01.361571286Z Gathering data for ns/openshift-dns...
2026-07-16T03:54:01.444591537Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:54:01.958951430Z Gathering data for ns/openshift-image-registry...
2026-07-16T03:54:02.437423324Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T03:54:02.480636145Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T03:54:02.775766736Z Gathering data for ns/openshift-ingress...
2026-07-16T03:54:03.108377817Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T03:54:03.513527942Z Gathering data for ns/openshift-insights...
2026-07-16T03:54:03.592212949Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T03:54:04.705744637Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T03:54:05.429834202Z Gathering data for ns/openshift-jobset-operator...
2026-07-16T03:54:05.825120394Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T03:54:06.253804800Z Gathering data for ns/openshift-config...
2026-07-16T03:54:06.449074349Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:54:06.529298493Z Gathering data for ns/openshift-config-managed...
2026-07-16T03:54:06.934985848Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T03:54:07.032180082Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T03:54:07.348064661Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T03:54:07.613814103Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T03:54:07.896975897Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T03:54:08.042118448Z Waiting for performance profile collector pods to become ready: 8
2026-07-16T03:54:08.189981488Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T03:54:08.454507382Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T03:54:08.729888843Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T03:54:09.036818574Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T03:54:09.160129128Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T03:54:09.388503491Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T03:54:09.409565124Z Collecting performance related data for node ip-10-0-130-168.ec2.internal
2026-07-16T03:54:09.418134456Z Collecting performance related data for node ip-10-0-141-197.ec2.internal
2026-07-16T03:54:09.426852275Z Collecting performance related data for node ip-10-0-136-52.ec2.internal
2026-07-16T03:54:09.964358898Z Gathering data for ns/openshift-multus...
2026-07-16T03:54:11.454547617Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:54:11.594963317Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T03:54:13.003204273Z Gathering data for ns/openshift-host-network...
2026-07-16T03:54:13.368748455Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T03:54:13.921320674Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T03:54:14.344461643Z Gathering data for ns/openshift-network-operator...
2026-07-16T03:54:14.757162026Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T03:54:15.022973931Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T03:54:15.538943522Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T03:54:15.844593308Z Gathering data for ns/openshift-apiserver...
2026-07-16T03:54:16.299063737Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T03:54:16.459167255Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-16T03:54:16.602012850Z Gathering data for ns/openshift-controller-manager...
2026-07-16T03:54:16.893088336Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T03:54:17.041242587Z Collecting kubelet logs for node ip-10-0-130-168.ec2.internal
2026-07-16T03:54:17.042876168Z Collecting kubelet logs for node ip-10-0-141-197.ec2.internal
2026-07-16T03:54:17.044569041Z Collecting kubelet logs for node ip-10-0-136-52.ec2.internal
2026-07-16T03:54:17.377546893Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-58mlw namespace
2026-07-16T03:54:17.382483807Z INFO: Node performance data collection complete.
2026-07-16T03:54:17.610707854Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T03:54:17.907601752Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T03:54:18.245610714Z Gathering data for ns/openshift-service-ca...
2026-07-16T03:54:18.601507713Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T03:54:19.488521767Z Wrote inspect data to must-gather.
2026-07-16T03:54:19.488560752Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T03:54:19.488560752Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T03:54:19.606873855Z error: the server doesn't have a resource type "clusters"
2026-07-16T03:54:19.740730804Z Caches written to disk