2026-07-20T16:12:54.697878960Z [disk usage checker] Started
2026-07-20T16:12:54.701085135Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:12:54.898856708Z Gathering data for ns/openshift-cluster-version...
2026-07-20T16:12:55.033331177Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-20T16:12:55.115629334Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:12:55.199552395Z Gathering data for ns/default...
2026-07-20T16:12:55.499067305Z Gathering data for ns/openshift...
2026-07-20T16:12:55.897525706Z Gathering data for ns/kube-system...
2026-07-20T16:12:56.550753554Z Gathering data for ns/openshift-etcd...
2026-07-20T16:12:56.848694269Z Wrote inspect data to must-gather.
2026-07-20T16:12:56.848729845Z error: inspection completed with the errors occurred while gathering data:
2026-07-20T16:12:56.848729845Z     namespaces "assisted-installer" not found
2026-07-20T16:12:57.334773494Z Waiting on subprocesses to finish execution.
2026-07-20T16:12:57.339753372Z INFO: Collecting host service logs for crio
2026-07-20T16:12:57.340148453Z INFO: Collecting host service logs for kubelet
2026-07-20T16:12:57.340498193Z INFO: Collecting host service logs for rpm-ostreed
2026-07-20T16:12:57.340841020Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-20T16:12:57.341202928Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-20T16:12:57.341447719Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-20T16:12:57.341587145Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-20T16:12:57.342021000Z INFO: Collecting host service logs for NetworkManager
2026-07-20T16:12:57.342367540Z INFO: Collecting host service logs for openvswitch
2026-07-20T16:12:57.342722569Z INFO: Collecting host service logs for ovs-configuration
2026-07-20T16:12:57.343083770Z INFO: Collecting host service logs for ovsdb-server
2026-07-20T16:12:57.343462560Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-20T16:12:57.343659655Z INFO: Gathering HAProxy config files
2026-07-20T16:12:57.344083047Z INFO: Waiting for worker host service log collection to complete ...
2026-07-20T16:12:57.351515441Z Executing Istio gather script
2026-07-20T16:12:57.353288570Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-20T16:12:57.387219549Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-20T16:12:57.489328684Z INFO: Waiting for node performance related collection to complete ...
2026-07-20T16:12:57.755226777Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-20T16:12:57.989147392Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-jv7sc
2026-07-20T16:12:57.989199386Z insights-runtime-extractor-hn29h
2026-07-20T16:12:57.989220975Z insights-runtime-extractor-l4qw6
2026-07-20T16:12:57.989239203Z insights-runtime-extractor-q76dr
2026-07-20T16:12:58.073093601Z No resources found in openshift-etcd namespace.
2026-07-20T16:12:58.334606994Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-20T16:12:58.334606994Z INFO: on-disk MachineConfig config collection complete.
2026-07-20T16:12:58.396551825Z INFO: "metallb-operator" not detected. Skipping.
2026-07-20T16:12:58.397953987Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-20T16:12:58.465976088Z No resources found
2026-07-20T16:12:58.528422115Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-20T16:12:58.643573289Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-20T16:12:58.709858131Z Wrote inspect data to must-gather.
2026-07-20T16:12:58.772445097Z INFO: Worker host service log collection to complete.
2026-07-20T16:12:58.819900495Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-20T16:12:58.819900495Z See 'oc rsync -h' for help and examples
2026-07-20T16:12:58.827655126Z error: the server doesn't have a resource type "performanceprofile"
2026-07-20T16:12:59.207960065Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-20T16:12:59.222065036Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-20T16:12:59.351971646Z ERROR: No running kube-apiserver pods found
2026-07-20T16:12:59.405969868Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-20T16:12:59.417626741Z No resources found
2026-07-20T16:12:59.512338976Z Wrote inspect data to must-gather.
2026-07-20T16:12:59.616088645Z Wrote inspect data to must-gather.
2026-07-20T16:12:59.632147791Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:12:59.719724018Z Gathering data for ns/openshift-monitoring...
2026-07-20T16:12:59.750604323Z Wrote inspect data to must-gather.
2026-07-20T16:12:59.778702996Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:12:59.803898184Z Wrote inspect data to must-gather.
2026-07-20T16:12:59.954454704Z Wrote inspect data to must-gather.
2026-07-20T16:12:59.969577145Z INFO: 'previous-logs' folder not found on ip-10-0-129-222.ec2.internal, skipping...
2026-07-20T16:12:59.974506034Z tar: Removing leading `/' from member names
2026-07-20T16:12:59.984188639Z INFO: HAProxy config collection complete.
2026-07-20T16:13:00.026964208Z Wrote inspect data to must-gather.
2026-07-20T16:13:00.076031140Z Wrote inspect data to must-gather.
2026-07-20T16:13:00.160616992Z error: the server doesn't have a resource type "machineconfigs"
2026-07-20T16:13:00.174729118Z INFO: Getting rules from prometheus-k8s-0
2026-07-20T16:13:00.323589401Z INFO: 'previous-logs' folder not found on ip-10-0-139-55.ec2.internal, skipping...
2026-07-20T16:13:00.368203065Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-20T16:13:00.407845842Z Wrote inspect data to must-gather.
2026-07-20T16:13:00.641202884Z INFO: 'previous-logs' folder not found on ip-10-0-142-181.ec2.internal, skipping...
2026-07-20T16:13:00.641202884Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-20T16:13:00.641269864Z INFO: Machine Config Daemon termination log collection complete.
2026-07-20T16:13:00.653593197Z Wrote inspect data to must-gather.
2026-07-20T16:13:00.664767562Z INFO: Getting status/config from prometheus-k8s-0
2026-07-20T16:13:00.688057727Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-20T16:13:00.727893301Z Wrote inspect data to must-gather.
2026-07-20T16:13:00.823851933Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-20T16:13:01.035511871Z INFO: INTERCONNECT MODE
2026-07-20T16:13:01.035719076Z INFO: Gathering ovn-kubernetes DBs
2026-07-20T16:13:01.045184702Z Wrote inspect data to must-gather.
2026-07-20T16:13:01.093387873Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-20T16:13:01.236768221Z INFO: Gathering OVN_Northbound from ovnkube-node-bvvmm...
2026-07-20T16:13:01.237071043Z INFO: Gathering OVN_Northbound from ovnkube-node-kj4n8...
2026-07-20T16:13:01.237313619Z INFO: Gathering OVN_Northbound from ovnkube-node-wbgnt...
2026-07-20T16:13:01.237646019Z INFO: Gathering OVN_Southbound from ovnkube-node-bvvmm...
2026-07-20T16:13:01.237942650Z INFO: Gathering OVN_Southbound from ovnkube-node-kj4n8...
2026-07-20T16:13:01.238229221Z INFO: Gathering OVN_Southbound from ovnkube-node-wbgnt...
2026-07-20T16:13:01.283665892Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-20T16:13:01.318086066Z Wrote inspect data to must-gather.
2026-07-20T16:13:01.410361517Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-20T16:13:01.529596204Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.576844875Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.578376455Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.591090700Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.607039792Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.622116138Z tar: Removing leading `/' from member names
2026-07-20T16:13:01.676785619Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-20T16:13:01.727504974Z Wrote inspect data to must-gather.
2026-07-20T16:13:01.745118206Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-20T16:13:01.917304733Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:01.974764105Z W0720 16:13:01.974721    1411 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-20T16:13:01.980998381Z Wrote inspect data to must-gather.
2026-07-20T16:13:02.028116919Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-20T16:13:02.061482188Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-20T16:13:02.111273442Z INFO: Waiting for network log collection to complete ...
2026-07-20T16:13:02.111396640Z INFO: Waiting for ovnk database copies to complete ...
2026-07-20T16:13:02.111967233Z INFO: Copying ovnk databases complete.
2026-07-20T16:13:02.114129376Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-20T16:13:02.118881716Z ovnk_database_store/
2026-07-20T16:13:02.118949428Z ovnk_database_store/ovnkube-node-bvvmm_nbdb
2026-07-20T16:13:02.128331118Z ovnk_database_store/ovnkube-node-wbgnt_nbdb
2026-07-20T16:13:02.135293611Z ovnk_database_store/ovnkube-node-wbgnt_sbdb
2026-07-20T16:13:02.155333600Z ovnk_database_store/ovnkube-node-kj4n8_sbdb
2026-07-20T16:13:02.171635522Z ovnk_database_store/ovnkube-node-kj4n8_nbdb
2026-07-20T16:13:02.178725076Z ovnk_database_store/ovnkube-node-bvvmm_sbdb
2026-07-20T16:13:02.194070825Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:02.197161237Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-20T16:13:02.198690169Z INFO: Network log collection complete.
2026-07-20T16:13:02.277378891Z Gathering data for ns/openshift-network-console...
2026-07-20T16:13:02.322777177Z Wrote inspect data to must-gather.
2026-07-20T16:13:02.412581558Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:02.459666189Z INFO: Getting status from alertmanager-main-0
2026-07-20T16:13:02.632941175Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:02.636955595Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-20T16:13:02.642082414Z Wrote inspect data to must-gather.
2026-07-20T16:13:02.772706200Z Gathering data for ns/openshift-console-operator...
2026-07-20T16:13:02.854404424Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:02.938873215Z Wrote inspect data to must-gather.
2026-07-20T16:13:03.052011207Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:03.054776015Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-20T16:13:03.178579373Z Gathering data for ns/openshift-console...
2026-07-20T16:13:03.211779407Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-20T16:13:03.213937115Z INFO: Fallback to identify the container image from release info
2026-07-20T16:13:03.287787917Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:03.484308770Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:03.487339396Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-20T16:13:03.654006952Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:03.722243778Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-20T16:13:03.848113492Z Wrote inspect data to /must-gather/istio.
2026-07-20T16:13:04.243637497Z Gathering data for ns/openshift-dns-operator...
2026-07-20T16:13:04.335433420Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-20T16:13:04.461213038Z daemonset.apps/perf-node-gather-daemonset created
2026-07-20T16:13:04.567301970Z Done executing Istio gather script
2026-07-20T16:13:04.577952822Z Gathering data for ns/openshift-dns...
2026-07-20T16:13:04.613265580Z Waiting for performance profile collector pods to become ready: 1
2026-07-20T16:13:04.835321687Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:13:05.212301132Z Gathering data for ns/openshift-image-registry...
2026-07-20T16:13:05.663760864Z Gathering data for ns/openshift-ingress-operator...
2026-07-20T16:13:05.738077435Z Waiting for performance profile collector pods to become ready: 2
2026-07-20T16:13:05.976815316Z Gathering data for ns/openshift-ingress...
2026-07-20T16:13:06.294966807Z Gathering data for ns/openshift-ingress-canary...
2026-07-20T16:13:06.666683746Z Gathering data for ns/openshift-insights...
2026-07-20T16:13:06.841082362Z Waiting for performance profile collector pods to become ready: 3
2026-07-20T16:13:07.947299196Z Waiting for performance profile collector pods to become ready: 4
2026-07-20T16:13:08.490274032Z Gathering data for ns/openshift-jobset-operator...
2026-07-20T16:13:09.060956137Z Waiting for performance profile collector pods to become ready: 5
2026-07-20T16:13:09.272396072Z Gathering data for ns/openshift-config...
2026-07-20T16:13:09.549668987Z Gathering data for ns/openshift-config-managed...
2026-07-20T16:13:09.841376124Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:13:09.950731965Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-20T16:13:10.187432882Z Waiting for performance profile collector pods to become ready: 6
2026-07-20T16:13:10.208382137Z Gathering data for ns/openshift-kube-apiserver...
2026-07-20T16:13:10.449027520Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-20T16:13:10.709373988Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-20T16:13:10.974014078Z Gathering data for ns/openshift-kube-scheduler...
2026-07-20T16:13:11.223930726Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-20T16:13:11.316991924Z Waiting for performance profile collector pods to become ready: 7
2026-07-20T16:13:11.499393488Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-20T16:13:11.797662577Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-20T16:13:12.092600603Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-20T16:13:12.437459329Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-20T16:13:12.584763186Z Gathering data for ns/openshift-multus...
2026-07-20T16:13:12.685111419Z Collecting performance related data for node ip-10-0-139-55.ec2.internal
2026-07-20T16:13:12.692249745Z Collecting performance related data for node ip-10-0-129-222.ec2.internal
2026-07-20T16:13:12.699590939Z Collecting performance related data for node ip-10-0-142-181.ec2.internal
2026-07-20T16:13:14.113420670Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-20T16:13:14.846440628Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:13:15.373917679Z Gathering data for ns/openshift-host-network...
2026-07-20T16:13:15.700018420Z Gathering data for ns/openshift-network-diagnostics...
2026-07-20T16:13:16.202480607Z Gathering data for ns/openshift-network-node-identity...
2026-07-20T16:13:16.641340380Z Gathering data for ns/openshift-network-operator...
2026-07-20T16:13:17.009668156Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-20T16:13:17.283494470Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-20T16:13:17.771174715Z Gathering data for ns/openshift-apiserver-operator...
2026-07-20T16:13:18.097162603Z Gathering data for ns/openshift-apiserver...
2026-07-20T16:13:18.484386448Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-20T16:13:18.732289138Z Gathering data for ns/openshift-controller-manager...
2026-07-20T16:13:18.984707613Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-20T16:13:19.630754193Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-20T16:13:19.851193033Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-20T16:13:19.977234734Z Gathering data for ns/openshift-service-ca-operator...
2026-07-20T16:13:20.278182207Z Gathering data for ns/openshift-service-ca...
2026-07-20T16:13:20.639293359Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-20T16:13:20.935953930Z Collecting kubelet logs for node ip-10-0-139-55.ec2.internal
2026-07-20T16:13:20.937738038Z Collecting kubelet logs for node ip-10-0-129-222.ec2.internal
2026-07-20T16:13:20.939557631Z Collecting kubelet logs for node ip-10-0-142-181.ec2.internal
2026-07-20T16:13:21.275611067Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-rsjp7 namespace
2026-07-20T16:13:21.280546147Z INFO: Node performance data collection complete.
2026-07-20T16:13:21.580391031Z Wrote inspect data to must-gather.
2026-07-20T16:13:21.580431452Z error: inspection completed with the errors occurred while gathering data:
2026-07-20T16:13:21.580431452Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-20T16:13:21.703791043Z error: the server doesn't have a resource type "clusters"
2026-07-20T16:13:21.853073734Z Caches written to disk