2026-07-08T17:22:51.401793106Z [disk usage checker] Started
2026-07-08T17:22:51.405376297Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-08T17:22:51.621408316Z Gathering data for ns/openshift-cluster-version...
2026-07-08T17:22:51.763050496Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T17:22:51.860495633Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T17:22:51.945330470Z Gathering data for ns/default...
2026-07-08T17:22:52.243842106Z Gathering data for ns/openshift...
2026-07-08T17:22:52.651319776Z Gathering data for ns/kube-system...
2026-07-08T17:22:53.353514666Z Gathering data for ns/openshift-etcd...
2026-07-08T17:22:53.694577231Z Wrote inspect data to must-gather.
2026-07-08T17:22:53.694601915Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T17:22:53.694601915Z     namespaces "assisted-installer" not found
2026-07-08T17:22:54.227900787Z Waiting on subprocesses to finish execution.
2026-07-08T17:22:54.232713886Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-08T17:22:54.237179170Z INFO: Gathering HAProxy config files
2026-07-08T17:22:54.237325747Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-08T17:22:54.240818205Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-08T17:22:54.240998390Z Executing Istio gather script
2026-07-08T17:22:54.243995036Z INFO: Collecting host service logs for crio
2026-07-08T17:22:54.244246014Z INFO: Collecting host service logs for kubelet
2026-07-08T17:22:54.244521103Z INFO: Collecting host service logs for rpm-ostreed
2026-07-08T17:22:54.244797626Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-08T17:22:54.245149723Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-08T17:22:54.245495196Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-08T17:22:54.246017182Z INFO: Collecting host service logs for NetworkManager
2026-07-08T17:22:54.246371586Z INFO: Collecting host service logs for openvswitch
2026-07-08T17:22:54.246734478Z INFO: Collecting host service logs for ovs-configuration
2026-07-08T17:22:54.247099374Z INFO: Collecting host service logs for ovsdb-server
2026-07-08T17:22:54.247457484Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-08T17:22:54.247874874Z INFO: Waiting for worker host service log collection to complete ...
2026-07-08T17:22:54.431778570Z INFO: Waiting for node performance related collection to complete ...
2026-07-08T17:22:54.570173942Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-xh8sq
2026-07-08T17:22:54.570228566Z insights-runtime-extractor-j58g5
2026-07-08T17:22:54.570248397Z insights-runtime-extractor-tcvsm
2026-07-08T17:22:54.570268056Z insights-runtime-extractor-xrm8g
2026-07-08T17:22:54.960807468Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-08T17:22:55.352335776Z Wrote inspect data to must-gather.
2026-07-08T17:22:55.412586977Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-08T17:22:55.455030670Z INFO: "metallb-operator" not detected. Skipping.
2026-07-08T17:22:55.506434385Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-08T17:22:55.506548073Z INFO: on-disk MachineConfig config collection complete.
2026-07-08T17:22:55.526923768Z Wrote inspect data to must-gather.
2026-07-08T17:22:55.586387973Z No resources found in openshift-etcd namespace.
2026-07-08T17:22:55.601324730Z No resources found
2026-07-08T17:22:55.637724067Z error: the server doesn't have a resource type "performanceprofile"
2026-07-08T17:22:55.710659525Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-08T17:22:55.809792619Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-08T17:22:55.809792619Z See 'oc rsync -h' for help and examples
2026-07-08T17:22:55.920422343Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-08T17:22:55.953004521Z INFO: Worker host service log collection to complete.
2026-07-08T17:22:56.017773300Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-08T17:22:56.221890280Z ERROR: No running kube-apiserver pods found
2026-07-08T17:22:56.366210977Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-08T17:22:56.466949957Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-08T17:22:56.478854206Z Gathering data for ns/openshift-monitoring...
2026-07-08T17:22:56.492883610Z tar: Removing leading `/' from member names
2026-07-08T17:22:56.503167793Z INFO: HAProxy config collection complete.
2026-07-08T17:22:56.622337326Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T17:22:56.732221756Z Wrote inspect data to must-gather.
2026-07-08T17:22:56.758576683Z Wrote inspect data to must-gather.
2026-07-08T17:22:56.778438746Z INFO: 'previous-logs' folder not found on ip-10-0-128-57.ec2.internal, skipping...
2026-07-08T17:22:56.798430510Z No resources found
2026-07-08T17:22:56.807739659Z Wrote inspect data to must-gather.
2026-07-08T17:22:56.818983709Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-08T17:22:56.846429629Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.030304085Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.038638074Z error: the server doesn't have a resource type "machineconfigs"
2026-07-08T17:22:57.161081247Z INFO: 'previous-logs' folder not found on ip-10-0-132-182.ec2.internal, skipping...
2026-07-08T17:22:57.244316509Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-08T17:22:57.340926500Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.353739892Z INFO: Getting rules from prometheus-k8s-0
2026-07-08T17:22:57.516613022Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.534306419Z INFO: 'previous-logs' folder not found on ip-10-0-138-233.ec2.internal, skipping...
2026-07-08T17:22:57.534346318Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-08T17:22:57.534346318Z INFO: Machine Config Daemon termination log collection complete.
2026-07-08T17:22:57.583723485Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-08T17:22:57.651187135Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.701325445Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-08T17:22:57.823458899Z INFO: Getting status/config from prometheus-k8s-0
2026-07-08T17:22:57.917052227Z Wrote inspect data to must-gather.
2026-07-08T17:22:57.953854595Z Wrote inspect data to must-gather.
2026-07-08T17:22:58.226287308Z Wrote inspect data to must-gather.
2026-07-08T17:22:58.255191637Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T17:22:58.277852993Z INFO: INTERCONNECT MODE
2026-07-08T17:22:58.277885720Z INFO: Gathering ovn-kubernetes DBs
2026-07-08T17:22:58.305864861Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-08T17:22:58.470766835Z INFO: Gathering OVN_Northbound from ovnkube-node-6m2cv...
2026-07-08T17:22:58.471017449Z INFO: Gathering OVN_Northbound from ovnkube-node-gx2vd...
2026-07-08T17:22:58.471257233Z INFO: Gathering OVN_Northbound from ovnkube-node-mj26x...
2026-07-08T17:22:58.471540480Z INFO: Gathering OVN_Southbound from ovnkube-node-6m2cv...
2026-07-08T17:22:58.471932279Z INFO: Gathering OVN_Southbound from ovnkube-node-gx2vd...
2026-07-08T17:22:58.472246292Z INFO: Gathering OVN_Southbound from ovnkube-node-mj26x...
2026-07-08T17:22:58.498980446Z Wrote inspect data to must-gather.
2026-07-08T17:22:58.652852467Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-08T17:22:58.837037341Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.837421057Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.864619902Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.864619902Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.882920063Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.890110434Z tar: Removing leading `/' from member names
2026-07-08T17:22:58.976084739Z W0708 17:22:58.976045    1341 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-08T17:22:58.983454557Z Wrote inspect data to must-gather.
2026-07-08T17:22:59.042445257Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-08T17:22:59.069997710Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-08T17:22:59.146777739Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T17:22:59.308168497Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:22:59.349016709Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-08T17:22:59.364202298Z Wrote inspect data to must-gather.
2026-07-08T17:22:59.373449991Z Gathering data for ns/openshift-network-console...
2026-07-08T17:22:59.428562332Z INFO: Waiting for network log collection to complete ...
2026-07-08T17:22:59.428699439Z INFO: Waiting for ovnk database copies to complete ...
2026-07-08T17:22:59.429172736Z INFO: Copying ovnk databases complete.
2026-07-08T17:22:59.431296372Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-08T17:22:59.434333258Z ovnk_database_store/
2026-07-08T17:22:59.434400944Z ovnk_database_store/ovnkube-node-mj26x_nbdb
2026-07-08T17:22:59.447112821Z ovnk_database_store/ovnkube-node-mj26x_sbdb
2026-07-08T17:22:59.474420283Z ovnk_database_store/ovnkube-node-gx2vd_nbdb
2026-07-08T17:22:59.487939101Z ovnk_database_store/ovnkube-node-gx2vd_sbdb
2026-07-08T17:22:59.519151919Z ovnk_database_store/ovnkube-node-6m2cv_nbdb
2026-07-08T17:22:59.531345158Z ovnk_database_store/ovnkube-node-6m2cv_sbdb
2026-07-08T17:22:59.559770404Z INFO: Network log collection complete.
2026-07-08T17:22:59.595558292Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:22:59.599523299Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-08T17:22:59.725981906Z Wrote inspect data to must-gather.
2026-07-08T17:22:59.822747501Z INFO: Getting status from alertmanager-main-0
2026-07-08T17:22:59.827293468Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:22:59.954266899Z Gathering data for ns/openshift-console-operator...
2026-07-08T17:23:00.068170923Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:00.072309765Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-08T17:23:00.091017909Z Wrote inspect data to must-gather.
2026-07-08T17:23:00.278345564Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:00.370590556Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-08T17:23:00.373593289Z INFO: Fallback to identify the container image from release info
2026-07-08T17:23:00.400640798Z Gathering data for ns/openshift-console...
2026-07-08T17:23:00.507649017Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:00.511284093Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-08T17:23:00.747575043Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:00.968956986Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:00.972230935Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-08T17:23:00.999982428Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-08T17:23:01.157623754Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-08T17:23:01.159717290Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:01.300211793Z daemonset.apps/perf-node-gather-daemonset created
2026-07-08T17:23:01.397031944Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:01.400721386Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-08T17:23:01.473302439Z Waiting for performance profile collector pods to become ready: 1
2026-07-08T17:23:01.477753775Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-08T17:23:01.598576415Z Gathering data for ns/openshift-dns-operator...
2026-07-08T17:23:01.599429818Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:01.856232976Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:01.860096468Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-08T17:23:01.959373366Z Gathering data for ns/openshift-dns...
2026-07-08T17:23:02.047020891Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:02.241551142Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:02.244359228Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-08T17:23:02.417924243Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:02.637532649Z Gathering data for ns/openshift-image-registry...
2026-07-08T17:23:02.640410803Z Waiting for performance profile collector pods to become ready: 2
2026-07-08T17:23:02.644450875Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:02.646973797Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-08T17:23:02.830642604Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:03.021534050Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:03.025062682Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-08T17:23:03.162348322Z Gathering data for ns/openshift-ingress-operator...
2026-07-08T17:23:03.212789444Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:03.404612320Z Wrote inspect data to /must-gather/istio.
2026-07-08T17:23:03.567781871Z Gathering data for ns/openshift-ingress...
2026-07-08T17:23:03.779118681Z Waiting for performance profile collector pods to become ready: 3
2026-07-08T17:23:03.949118332Z Gathering data for ns/openshift-ingress-canary...
2026-07-08T17:23:04.151256730Z Done executing Istio gather script
2026-07-08T17:23:04.371785788Z Gathering data for ns/openshift-insights...
2026-07-08T17:23:04.911939701Z Waiting for performance profile collector pods to become ready: 4
2026-07-08T17:23:06.030980429Z Waiting for performance profile collector pods to become ready: 5
2026-07-08T17:23:06.482241260Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-08T17:23:06.583478964Z Gathering data for ns/kserve...
2026-07-08T17:23:07.159705552Z Waiting for performance profile collector pods to become ready: 6
2026-07-08T17:23:08.141183557Z Gathering data for ns/openshift-config...
2026-07-08T17:23:08.287561334Z Waiting for performance profile collector pods to become ready: 7
2026-07-08T17:23:08.416241346Z Gathering data for ns/openshift-config-managed...
2026-07-08T17:23:08.855197775Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-08T17:23:09.157400725Z Gathering data for ns/openshift-kube-apiserver...
2026-07-08T17:23:09.424023216Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-08T17:23:09.446360364Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-08T17:23:09.715124133Z Collecting performance related data for node ip-10-0-132-182.ec2.internal
2026-07-08T17:23:09.725396036Z Collecting performance related data for node ip-10-0-128-57.ec2.internal
2026-07-08T17:23:09.733877023Z Collecting performance related data for node ip-10-0-138-233.ec2.internal
2026-07-08T17:23:09.825002916Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-08T17:23:10.202998287Z Gathering data for ns/openshift-kube-scheduler...
2026-07-08T17:23:10.544675329Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-08T17:23:10.929779489Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-08T17:23:11.347730542Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-08T17:23:11.486852940Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-08T17:23:11.696393954Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-08T17:23:12.356246908Z Gathering data for ns/openshift-multus...
2026-07-08T17:23:13.792561422Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-08T17:23:15.091272817Z Gathering data for ns/openshift-host-network...
2026-07-08T17:23:15.438534160Z Gathering data for ns/openshift-network-diagnostics...
2026-07-08T17:23:15.994920224Z Gathering data for ns/openshift-network-node-identity...
2026-07-08T17:23:16.454364162Z Gathering data for ns/openshift-network-operator...
2026-07-08T17:23:16.493385446Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-08T17:23:16.865983173Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-08T17:23:17.208958860Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-08T17:23:17.722824803Z Gathering data for ns/openshift-apiserver-operator...
2026-07-08T17:23:18.057730717Z Gathering data for ns/openshift-apiserver...
2026-07-08T17:23:18.167335379Z Collecting kubelet logs for node ip-10-0-132-182.ec2.internal
2026-07-08T17:23:18.169078112Z Collecting kubelet logs for node ip-10-0-128-57.ec2.internal
2026-07-08T17:23:18.170812271Z Collecting kubelet logs for node ip-10-0-138-233.ec2.internal
2026-07-08T17:23:18.536242112Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-08T17:23:18.579125245Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-q5dxj namespace
2026-07-08T17:23:18.585665384Z INFO: Node performance data collection complete.
2026-07-08T17:23:18.869530779Z Gathering data for ns/openshift-controller-manager...
2026-07-08T17:23:19.158646881Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-08T17:23:19.838124945Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-08T17:23:20.137502788Z Gathering data for ns/openshift-service-ca-operator...
2026-07-08T17:23:20.443314048Z Gathering data for ns/openshift-service-ca...
2026-07-08T17:23:20.815814783Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-08T17:23:21.498233534Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-08T17:23:21.725393805Z Wrote inspect data to must-gather.
2026-07-08T17:23:21.725442429Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T17:23:21.725442429Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-08T17:23:21.865583473Z error: the server doesn't have a resource type "clusters"
2026-07-08T17:23:22.151756411Z Caches written to disk