2026-07-30T16:04:49.802638649Z [disk usage checker] Started
2026-07-30T16:04:49.806106616Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:04:49.994953036Z Gathering data for ns/openshift-cluster-version...
2026-07-30T16:04:50.163113054Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T16:04:50.279948943Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T16:04:50.368291671Z Gathering data for ns/default...
2026-07-30T16:04:50.707629091Z Gathering data for ns/openshift...
2026-07-30T16:04:51.138549439Z Gathering data for ns/kube-system...
2026-07-30T16:04:52.000821203Z Gathering data for ns/openshift-etcd...
2026-07-30T16:04:52.320618491Z Wrote inspect data to must-gather.
2026-07-30T16:04:52.320675547Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T16:04:52.320675547Z     namespaces "assisted-installer" not found
2026-07-30T16:04:52.374801124Z Waiting on subprocesses to finish execution.
2026-07-30T16:04:52.382536853Z Executing Istio gather script
2026-07-30T16:04:52.389176969Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-30T16:04:52.400348639Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-30T16:04:52.414813038Z INFO: Collecting host service logs for crio
2026-07-30T16:04:52.415125216Z INFO: Collecting host service logs for kubelet
2026-07-30T16:04:52.415451469Z INFO: Collecting host service logs for rpm-ostreed
2026-07-30T16:04:52.419840002Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-30T16:04:52.420223212Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-30T16:04:52.420405581Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-30T16:04:52.420546350Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-30T16:04:52.420909564Z INFO: Collecting host service logs for NetworkManager
2026-07-30T16:04:52.421224051Z INFO: Collecting host service logs for openvswitch
2026-07-30T16:04:52.421559160Z INFO: Collecting host service logs for ovs-configuration
2026-07-30T16:04:52.421928023Z INFO: Collecting host service logs for ovsdb-server
2026-07-30T16:04:52.422275956Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-30T16:04:52.422723785Z INFO: Waiting for worker host service log collection to complete ...
2026-07-30T16:04:52.425257174Z INFO: Gathering HAProxy config files
2026-07-30T16:04:52.607329394Z INFO: Waiting for node performance related collection to complete ...
2026-07-30T16:04:52.768351466Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-30T16:04:52.768444471Z INFO: on-disk MachineConfig config collection complete.
2026-07-30T16:04:53.046210139Z No resources found
2026-07-30T16:04:53.241384171Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-30T16:04:53.347636261Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-vwt9x
2026-07-30T16:04:53.347709395Z insights-runtime-extractor-t75nq
2026-07-30T16:04:53.348715229Z insights-runtime-extractor-tjrrx
2026-07-30T16:04:53.348767483Z insights-runtime-extractor-x8gv2
2026-07-30T16:04:53.552702087Z No resources found in openshift-etcd namespace.
2026-07-30T16:04:53.577893882Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-30T16:04:53.591029172Z error: the server doesn't have a resource type "performanceprofile"
2026-07-30T16:04:53.641886376Z Wrote inspect data to must-gather.
2026-07-30T16:04:53.671590789Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-30T16:04:53.671775400Z INFO: "metallb-operator" not detected. Skipping.
2026-07-30T16:04:53.908709308Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-30T16:04:53.957554715Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-30T16:04:54.117747506Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.305214959Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T16:04:54.451154465Z INFO: Worker host service log collection to complete.
2026-07-30T16:04:54.453135069Z ERROR: No running kube-apiserver pods found
2026-07-30T16:04:54.486838708Z No resources found
2026-07-30T16:04:54.539284045Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.549262883Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-30T16:04:54.549577322Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-30T16:04:54.549577322Z See 'oc rsync -h' for help and examples
2026-07-30T16:04:54.690730437Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.715797365Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.774397475Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.856032209Z Wrote inspect data to must-gather.
2026-07-30T16:04:54.870888670Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-30T16:04:54.885282312Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:04:54.994482481Z error: the server doesn't have a resource type "machineconfigs"
2026-07-30T16:04:55.049452176Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-30T16:04:55.082910269Z Wrote inspect data to must-gather.
2026-07-30T16:04:55.110888021Z Gathering data for ns/openshift-monitoring...
2026-07-30T16:04:55.140085604Z INFO: 'previous-logs' folder not found on ip-10-0-138-172.ec2.internal, skipping...
2026-07-30T16:04:55.178836259Z tar: Removing leading `/' from member names
2026-07-30T16:04:55.185107145Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-30T16:04:55.190301254Z INFO: HAProxy config collection complete.
2026-07-30T16:04:55.279541507Z INFO: Getting rules from prometheus-k8s-0
2026-07-30T16:04:55.381432185Z Wrote inspect data to must-gather.
2026-07-30T16:04:55.417763211Z INFO: 'previous-logs' folder not found on ip-10-0-142-210.ec2.internal, skipping...
2026-07-30T16:04:55.431557287Z Wrote inspect data to must-gather.
2026-07-30T16:04:55.601808349Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-30T16:04:55.651159752Z Wrote inspect data to must-gather.
2026-07-30T16:04:55.681268002Z INFO: 'previous-logs' folder not found on ip-10-0-143-80.ec2.internal, skipping...
2026-07-30T16:04:55.681310887Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-30T16:04:55.681321644Z INFO: Machine Config Daemon termination log collection complete.
2026-07-30T16:04:55.718101044Z INFO: Getting status/config from prometheus-k8s-0
2026-07-30T16:04:55.825758430Z Wrote inspect data to must-gather.
2026-07-30T16:04:55.931148152Z INFO: INTERCONNECT MODE
2026-07-30T16:04:55.931185647Z INFO: Gathering ovn-kubernetes DBs
2026-07-30T16:04:56.037188994Z Wrote inspect data to must-gather.
2026-07-30T16:04:56.084074797Z INFO: Gathering OVN_Northbound from ovnkube-node-bj8tl...
2026-07-30T16:04:56.084280063Z INFO: Gathering OVN_Northbound from ovnkube-node-dsf5f...
2026-07-30T16:04:56.084502791Z INFO: Gathering OVN_Northbound from ovnkube-node-q7nxb...
2026-07-30T16:04:56.084793095Z INFO: Gathering OVN_Southbound from ovnkube-node-bj8tl...
2026-07-30T16:04:56.085039763Z INFO: Gathering OVN_Southbound from ovnkube-node-dsf5f...
2026-07-30T16:04:56.085340361Z INFO: Gathering OVN_Southbound from ovnkube-node-q7nxb...
2026-07-30T16:04:56.105842285Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-30T16:04:56.376195672Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-30T16:04:56.390994646Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.417667523Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.419552672Z Wrote inspect data to must-gather.
2026-07-30T16:04:56.424173636Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.444928950Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.449841738Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.461623208Z tar: Removing leading `/' from member names
2026-07-30T16:04:56.478398688Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-30T16:04:56.587509673Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:56.657745837Z Wrote inspect data to must-gather.
2026-07-30T16:04:56.748901897Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-30T16:04:56.819647437Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T16:04:56.871542282Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:56.874421801Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-30T16:04:56.911785476Z Wrote inspect data to must-gather.
2026-07-30T16:04:56.940697839Z INFO: Waiting for network log collection to complete ...
2026-07-30T16:04:56.940812863Z INFO: Waiting for ovnk database copies to complete ...
2026-07-30T16:04:56.941334785Z INFO: Copying ovnk databases complete.
2026-07-30T16:04:56.943394382Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-30T16:04:56.946311985Z ovnk_database_store/
2026-07-30T16:04:56.946365601Z ovnk_database_store/ovnkube-node-bj8tl_nbdb
2026-07-30T16:04:56.956616734Z ovnk_database_store/ovnkube-node-q7nxb_nbdb
2026-07-30T16:04:56.964400471Z ovnk_database_store/ovnkube-node-dsf5f_sbdb
2026-07-30T16:04:56.985440027Z ovnk_database_store/ovnkube-node-dsf5f_nbdb
2026-07-30T16:04:56.991380717Z ovnk_database_store/ovnkube-node-q7nxb_sbdb
2026-07-30T16:04:57.003278371Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-30T16:04:57.006791121Z ovnk_database_store/ovnkube-node-bj8tl_sbdb
2026-07-30T16:04:57.026777387Z INFO: Network log collection complete.
2026-07-30T16:04:57.094610753Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:57.158805111Z Wrote inspect data to must-gather.
2026-07-30T16:04:57.308626683Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:57.311558879Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-30T16:04:57.348960000Z W0730 16:04:57.348910    1563 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-30T16:04:57.354561614Z Wrote inspect data to must-gather.
2026-07-30T16:04:57.400728573Z INFO: Getting status from alertmanager-main-0
2026-07-30T16:04:57.474315054Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T16:04:57.520458059Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:57.621578171Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-30T16:04:57.624017842Z INFO: Fallback to identify the container image from release info
2026-07-30T16:04:57.669308640Z Gathering data for ns/openshift-network-console...
2026-07-30T16:04:57.733782408Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:57.736922961Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-30T16:04:57.945200946Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:58.103846791Z Gathering data for ns/openshift-console-operator...
2026-07-30T16:04:58.129750503Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:58.132040523Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-30T16:04:58.286759904Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:58.477160658Z Wrote inspect data to /must-gather/istio.
2026-07-30T16:04:58.509501250Z Gathering data for ns/openshift-console...
2026-07-30T16:04:58.585307278Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-30T16:04:58.728442819Z daemonset.apps/perf-node-gather-daemonset created
2026-07-30T16:04:58.864397750Z Waiting for performance profile collector pods to become ready: 1
2026-07-30T16:04:59.100646341Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-30T16:04:59.179184658Z Done executing Istio gather script
2026-07-30T16:04:59.636713113Z Gathering data for ns/openshift-dns-operator...
2026-07-30T16:04:59.890822064Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:04:59.937335845Z Gathering data for ns/openshift-dns...
2026-07-30T16:04:59.977234605Z Waiting for performance profile collector pods to become ready: 2
2026-07-30T16:05:00.554459680Z Gathering data for ns/openshift-image-registry...
2026-07-30T16:05:01.028546153Z Gathering data for ns/openshift-ingress-operator...
2026-07-30T16:05:01.114053110Z Waiting for performance profile collector pods to become ready: 3
2026-07-30T16:05:01.376439569Z Gathering data for ns/openshift-ingress...
2026-07-30T16:05:01.721556593Z Gathering data for ns/openshift-ingress-canary...
2026-07-30T16:05:02.129458058Z Gathering data for ns/openshift-insights...
2026-07-30T16:05:02.215681718Z Waiting for performance profile collector pods to become ready: 4
2026-07-30T16:05:03.320150355Z Waiting for performance profile collector pods to become ready: 5
2026-07-30T16:05:04.010409493Z Gathering data for ns/openshift-jobset-operator...
2026-07-30T16:05:04.448795144Z Waiting for performance profile collector pods to become ready: 6
2026-07-30T16:05:04.837031957Z Gathering data for ns/openshift-config...
2026-07-30T16:05:04.897858505Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:05:05.109265487Z Gathering data for ns/openshift-config-managed...
2026-07-30T16:05:05.556529333Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-30T16:05:05.569750693Z Waiting for performance profile collector pods to become ready: 7
2026-07-30T16:05:05.880119711Z Gathering data for ns/openshift-kube-apiserver...
2026-07-30T16:05:06.151797263Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-30T16:05:06.448673916Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-30T16:05:06.670054260Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-30T16:05:06.828034134Z Gathering data for ns/openshift-kube-scheduler...
2026-07-30T16:05:06.954731835Z Collecting performance related data for node ip-10-0-138-172.ec2.internal
2026-07-30T16:05:06.961315232Z Collecting performance related data for node ip-10-0-142-210.ec2.internal
2026-07-30T16:05:06.968282926Z Collecting performance related data for node ip-10-0-143-80.ec2.internal
2026-07-30T16:05:07.179169199Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-30T16:05:07.622443615Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-30T16:05:08.087087983Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-30T16:05:08.486556231Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-30T16:05:09.013016025Z Gathering data for ns/openshift-multus...
2026-07-30T16:05:09.907833204Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:05:10.546716933Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-30T16:05:11.880301912Z Gathering data for ns/openshift-host-network...
2026-07-30T16:05:12.295707396Z Gathering data for ns/openshift-network-diagnostics...
2026-07-30T16:05:12.893129813Z Gathering data for ns/openshift-network-node-identity...
2026-07-30T16:05:13.309793949Z Gathering data for ns/openshift-network-operator...
2026-07-30T16:05:13.705506777Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-30T16:05:13.988358127Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-30T16:05:14.491375583Z Gathering data for ns/openshift-apiserver-operator...
2026-07-30T16:05:14.838098024Z Gathering data for ns/openshift-apiserver...
2026-07-30T16:05:14.912982109Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-30T16:05:14.923671683Z Collecting kubelet logs for node ip-10-0-138-172.ec2.internal
2026-07-30T16:05:14.925301685Z Collecting kubelet logs for node ip-10-0-142-210.ec2.internal
2026-07-30T16:05:14.926947981Z Collecting kubelet logs for node ip-10-0-143-80.ec2.internal
2026-07-30T16:05:15.244680076Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8k5rv namespace
2026-07-30T16:05:15.250157211Z INFO: Node performance data collection complete.
2026-07-30T16:05:15.313274195Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-30T16:05:15.586363120Z Gathering data for ns/openshift-controller-manager...
2026-07-30T16:05:15.908845309Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-30T16:05:16.548885168Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-30T16:05:16.849448388Z Gathering data for ns/openshift-service-ca-operator...
2026-07-30T16:05:17.225728084Z Gathering data for ns/openshift-service-ca...
2026-07-30T16:05:17.628417406Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-30T16:05:18.544604189Z Wrote inspect data to must-gather.
2026-07-30T16:05:18.544664345Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T16:05:18.544664345Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-30T16:05:18.672728631Z error: the server doesn't have a resource type "clusters"
2026-07-30T16:05:18.776010725Z Caches written to disk