2026-07-20T16:17:57.182332550Z [disk usage checker] Started
2026-07-20T16:17:57.186003655Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-20T16:17:57.374881776Z Gathering data for ns/openshift-cluster-version...
2026-07-20T16:17:57.492066809Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-20T16:17:57.578153759Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:17:57.656687334Z Gathering data for ns/default...
2026-07-20T16:17:57.926511078Z Gathering data for ns/openshift...
2026-07-20T16:17:58.303703981Z Gathering data for ns/kube-system...
2026-07-20T16:17:58.915710219Z Gathering data for ns/openshift-etcd...
2026-07-20T16:17:59.194361220Z Wrote inspect data to must-gather.
2026-07-20T16:17:59.194397759Z error: inspection completed with the errors occurred while gathering data:
2026-07-20T16:17:59.194397759Z     namespaces "assisted-installer" not found
2026-07-20T16:17:59.736272439Z Waiting on subprocesses to finish execution.
2026-07-20T16:17:59.744711564Z Executing Istio gather script
2026-07-20T16:17:59.745869259Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-20T16:17:59.748404450Z INFO: Gathering HAProxy config files
2026-07-20T16:17:59.752720530Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-20T16:17:59.753997548Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-20T16:17:59.769123235Z INFO: Collecting host service logs for crio
2026-07-20T16:17:59.769450171Z INFO: Collecting host service logs for kubelet
2026-07-20T16:17:59.769764507Z INFO: Collecting host service logs for rpm-ostreed
2026-07-20T16:17:59.770079372Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-20T16:17:59.770381496Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-20T16:17:59.770680115Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-20T16:17:59.770974225Z INFO: Collecting host service logs for NetworkManager
2026-07-20T16:17:59.771292026Z INFO: Collecting host service logs for openvswitch
2026-07-20T16:17:59.771589001Z INFO: Collecting host service logs for ovs-configuration
2026-07-20T16:17:59.771885158Z INFO: Collecting host service logs for ovsdb-server
2026-07-20T16:17:59.772224560Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-20T16:17:59.772584220Z INFO: Waiting for worker host service log collection to complete ...
2026-07-20T16:17:59.833400822Z INFO: Waiting for node performance related collection to complete ...
2026-07-20T16:18:00.114832506Z No resources found in openshift-etcd namespace.
2026-07-20T16:18:00.455548189Z No resources found
2026-07-20T16:18:00.652776598Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-20T16:18:00.731211983Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-20T16:18:00.764799406Z ERROR: No running kube-apiserver pods found
2026-07-20T16:18:00.903002001Z INFO: "metallb-operator" not detected. Skipping.
2026-07-20T16:18:01.103869734Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-20T16:18:01.126020727Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-2m9cf
2026-07-20T16:18:01.126262258Z insights-runtime-extractor-7nv2q
2026-07-20T16:18:01.126286577Z insights-runtime-extractor-ctcvh
2026-07-20T16:18:01.126306944Z insights-runtime-extractor-dr7hz
2026-07-20T16:18:01.191669376Z error: the server doesn't have a resource type "performanceprofile"
2026-07-20T16:18:01.223421430Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-20T16:18:01.223509938Z INFO: on-disk MachineConfig config collection complete.
2026-07-20T16:18:01.431088483Z INFO: Worker host service log collection to complete.
2026-07-20T16:18:01.526867583Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-20T16:18:01.535166373Z Wrote inspect data to must-gather.
2026-07-20T16:18:01.655746848Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-20T16:18:01.695273292Z Wrote inspect data to must-gather.
2026-07-20T16:18:01.788612269Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-20T16:18:01.932359995Z No resources found
2026-07-20T16:18:01.969859858Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.025325808Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:18:02.099444838Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-20T16:18:02.134025270Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-20T16:18:02.134025270Z See 'oc rsync -h' for help and examples
2026-07-20T16:18:02.165850431Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.167589943Z tar: Removing leading `/' from member names
2026-07-20T16:18:02.169754005Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.171926716Z INFO: HAProxy config collection complete.
2026-07-20T16:18:02.200079829Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-20T16:18:02.212555568Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.380891494Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.418244805Z INFO: 'previous-logs' folder not found on ip-10-0-134-117.ec2.internal, skipping...
2026-07-20T16:18:02.472694751Z Gathering data for ns/openshift-monitoring...
2026-07-20T16:18:02.512647820Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.585920406Z error: the server doesn't have a resource type "machineconfigs"
2026-07-20T16:18:02.588444432Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-20T16:18:02.642910511Z INFO: Getting rules from prometheus-k8s-0
2026-07-20T16:18:02.725071463Z INFO: 'previous-logs' folder not found on ip-10-0-138-194.ec2.internal, skipping...
2026-07-20T16:18:02.765966586Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-20T16:18:02.825030739Z Wrote inspect data to must-gather.
2026-07-20T16:18:02.985944575Z INFO: 'previous-logs' folder not found on ip-10-0-140-62.ec2.internal, skipping...
2026-07-20T16:18:02.985993426Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-20T16:18:02.986012709Z INFO: Machine Config Daemon termination log collection complete.
2026-07-20T16:18:03.005645503Z Wrote inspect data to must-gather.
2026-07-20T16:18:03.077688825Z INFO: Getting status/config from prometheus-k8s-0
2026-07-20T16:18:03.096945502Z Wrote inspect data to must-gather.
2026-07-20T16:18:03.163916803Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-20T16:18:03.385292638Z Wrote inspect data to must-gather.
2026-07-20T16:18:03.387719859Z INFO: INTERCONNECT MODE
2026-07-20T16:18:03.387777532Z INFO: Gathering ovn-kubernetes DBs
2026-07-20T16:18:03.456903851Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-20T16:18:03.468272492Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-20T16:18:03.551189811Z INFO: Gathering OVN_Northbound from ovnkube-node-6648b...
2026-07-20T16:18:03.551490115Z INFO: Gathering OVN_Northbound from ovnkube-node-ghc2x...
2026-07-20T16:18:03.551731257Z INFO: Gathering OVN_Northbound from ovnkube-node-xvqj9...
2026-07-20T16:18:03.552081109Z INFO: Gathering OVN_Southbound from ovnkube-node-6648b...
2026-07-20T16:18:03.552365254Z INFO: Gathering OVN_Southbound from ovnkube-node-ghc2x...
2026-07-20T16:18:03.552633207Z INFO: Gathering OVN_Southbound from ovnkube-node-xvqj9...
2026-07-20T16:18:03.617274067Z Wrote inspect data to must-gather.
2026-07-20T16:18:03.723688904Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:03.779958503Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-20T16:18:03.903131179Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.932269844Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.974125994Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.984325217Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.991601694Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.992613012Z tar: Removing leading `/' from member names
2026-07-20T16:18:03.995348852Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-20T16:18:04.034266361Z Wrote inspect data to must-gather.
2026-07-20T16:18:04.067771772Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:04.070487307Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-20T16:18:04.086070924Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-20T16:18:04.290577108Z Wrote inspect data to must-gather.
2026-07-20T16:18:04.311806088Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:04.344722999Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-20T16:18:04.445718851Z INFO: Waiting for network log collection to complete ...
2026-07-20T16:18:04.445770190Z INFO: Waiting for ovnk database copies to complete ...
2026-07-20T16:18:04.446310474Z INFO: Copying ovnk databases complete.
2026-07-20T16:18:04.448639019Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-20T16:18:04.451747841Z ovnk_database_store/
2026-07-20T16:18:04.451822405Z ovnk_database_store/ovnkube-node-xvqj9_nbdb
2026-07-20T16:18:04.468913817Z ovnk_database_store/ovnkube-node-ghc2x_nbdb
2026-07-20T16:18:04.475427215Z ovnk_database_store/ovnkube-node-6648b_nbdb
2026-07-20T16:18:04.480713523Z ovnk_database_store/ovnkube-node-6648b_sbdb
2026-07-20T16:18:04.496859727Z ovnk_database_store/ovnkube-node-ghc2x_sbdb
2026-07-20T16:18:04.512863197Z ovnk_database_store/ovnkube-node-xvqj9_sbdb
2026-07-20T16:18:04.525182297Z W0720 16:18:04.525128    1475 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-20T16:18:04.532481291Z Wrote inspect data to must-gather.
2026-07-20T16:18:04.533059419Z INFO: Network log collection complete.
2026-07-20T16:18:04.570982844Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:04.576444879Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-20T16:18:04.636239705Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:18:04.750671170Z INFO: Getting status from alertmanager-main-0
2026-07-20T16:18:04.792163320Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:04.826220901Z Gathering data for ns/openshift-network-console...
2026-07-20T16:18:04.831523158Z Wrote inspect data to must-gather.
2026-07-20T16:18:05.006631088Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:05.009017621Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-20T16:18:05.120336321Z Wrote inspect data to must-gather.
2026-07-20T16:18:05.242759416Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:05.252556096Z Gathering data for ns/openshift-console-operator...
2026-07-20T16:18:05.389385937Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-20T16:18:05.392772595Z INFO: Fallback to identify the container image from release info
2026-07-20T16:18:05.446116746Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:05.449495970Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-20T16:18:05.612567215Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:05.620891669Z Gathering data for ns/openshift-console...
2026-07-20T16:18:05.792803808Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:18:06.028616260Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-20T16:18:06.112474771Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-20T16:18:06.159929140Z daemonset.apps/perf-node-gather-daemonset created
2026-07-20T16:18:06.291787634Z Waiting for performance profile collector pods to become ready: 1
2026-07-20T16:18:06.493938979Z Done executing Istio gather script
2026-07-20T16:18:06.618383416Z Gathering data for ns/openshift-dns-operator...
2026-07-20T16:18:06.878860938Z Gathering data for ns/openshift-dns...
2026-07-20T16:18:07.207019272Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:18:07.398684691Z Gathering data for ns/openshift-image-registry...
2026-07-20T16:18:07.416111172Z Waiting for performance profile collector pods to become ready: 2
2026-07-20T16:18:07.806211138Z Gathering data for ns/openshift-ingress-operator...
2026-07-20T16:18:08.180146202Z Gathering data for ns/openshift-ingress...
2026-07-20T16:18:08.486314041Z Gathering data for ns/openshift-ingress-canary...
2026-07-20T16:18:08.532482749Z Waiting for performance profile collector pods to become ready: 3
2026-07-20T16:18:08.823704774Z Gathering data for ns/openshift-insights...
2026-07-20T16:18:09.639020389Z Waiting for performance profile collector pods to become ready: 4
2026-07-20T16:18:10.453245477Z Gathering data for ns/openshift-jobset-operator...
2026-07-20T16:18:10.767526739Z Waiting for performance profile collector pods to become ready: 5
2026-07-20T16:18:11.110574022Z Gathering data for ns/openshift-config...
2026-07-20T16:18:11.383168408Z Gathering data for ns/openshift-config-managed...
2026-07-20T16:18:11.747653377Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-20T16:18:11.875223723Z Waiting for performance profile collector pods to become ready: 6
2026-07-20T16:18:11.985657937Z Gathering data for ns/openshift-kube-apiserver...
2026-07-20T16:18:12.212443035Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:18:12.214247583Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-20T16:18:12.460347805Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-20T16:18:12.701538225Z Gathering data for ns/openshift-kube-scheduler...
2026-07-20T16:18:12.934467315Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-20T16:18:13.008293133Z Waiting for performance profile collector pods to become ready: 7
2026-07-20T16:18:13.200380002Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-20T16:18:13.505576530Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-20T16:18:13.806422010Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-20T16:18:14.126711183Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-20T16:18:14.272978577Z Gathering data for ns/openshift-multus...
2026-07-20T16:18:14.369666213Z Collecting performance related data for node ip-10-0-138-194.ec2.internal
2026-07-20T16:18:14.377315335Z Collecting performance related data for node ip-10-0-134-117.ec2.internal
2026-07-20T16:18:14.385310633Z Collecting performance related data for node ip-10-0-140-62.ec2.internal
2026-07-20T16:18:15.945074418Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-20T16:18:17.064677203Z Gathering data for ns/openshift-host-network...
2026-07-20T16:18:17.217918887Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:18:17.346443809Z Gathering data for ns/openshift-network-diagnostics...
2026-07-20T16:18:17.812749065Z Gathering data for ns/openshift-network-node-identity...
2026-07-20T16:18:18.173200996Z Gathering data for ns/openshift-network-operator...
2026-07-20T16:18:18.503025442Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-20T16:18:18.741353974Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-20T16:18:19.188351522Z Gathering data for ns/openshift-apiserver-operator...
2026-07-20T16:18:19.503693553Z Gathering data for ns/openshift-apiserver...
2026-07-20T16:18:19.856801393Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-20T16:18:20.109801254Z Gathering data for ns/openshift-controller-manager...
2026-07-20T16:18:20.394237938Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-20T16:18:21.007250595Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-20T16:18:21.266635894Z Gathering data for ns/openshift-service-ca-operator...
2026-07-20T16:18:21.537548428Z Gathering data for ns/openshift-service-ca...
2026-07-20T16:18:21.888553857Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-20T16:18:22.179096503Z Collecting kubelet logs for node ip-10-0-138-194.ec2.internal
2026-07-20T16:18:22.181004566Z Collecting kubelet logs for node ip-10-0-134-117.ec2.internal
2026-07-20T16:18:22.183380069Z Collecting kubelet logs for node ip-10-0-140-62.ec2.internal
2026-07-20T16:18:22.226525438Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:18:22.531862964Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-nps5t namespace
2026-07-20T16:18:22.536406258Z INFO: Node performance data collection complete.
2026-07-20T16:18:22.791181878Z Wrote inspect data to must-gather.
2026-07-20T16:18:22.791222122Z error: inspection completed with the errors occurred while gathering data:
2026-07-20T16:18:22.791222122Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-20T16:18:22.915024327Z error: the server doesn't have a resource type "clusters"
2026-07-20T16:18:23.046019816Z Caches written to disk