2026-07-13T20:44:24.749292250Z [disk usage checker] Started
2026-07-13T20:44:24.752639885Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:24.967055050Z Gathering data for ns/openshift-cluster-version...
2026-07-13T20:44:25.105707687Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T20:44:25.197846835Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T20:44:25.286701038Z Gathering data for ns/default...
2026-07-13T20:44:25.591939410Z Gathering data for ns/openshift...
2026-07-13T20:44:26.028989484Z Gathering data for ns/kube-system...
2026-07-13T20:44:26.752450443Z Gathering data for ns/openshift-etcd...
2026-07-13T20:44:27.094754974Z Wrote inspect data to must-gather.
2026-07-13T20:44:27.094786917Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T20:44:27.094786917Z     namespaces "assisted-installer" not found
2026-07-13T20:44:27.577782424Z Waiting on subprocesses to finish execution.
2026-07-13T20:44:27.586479756Z INFO: Gathering HAProxy config files
2026-07-13T20:44:27.587642853Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-13T20:44:27.593821287Z Executing Istio gather script
2026-07-13T20:44:27.594770242Z INFO: Collecting host service logs for crio
2026-07-13T20:44:27.595089587Z INFO: Collecting host service logs for kubelet
2026-07-13T20:44:27.595401131Z INFO: Collecting host service logs for rpm-ostreed
2026-07-13T20:44:27.595720844Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-13T20:44:27.596198369Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-13T20:44:27.596532743Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-13T20:44:27.596852608Z INFO: Collecting host service logs for NetworkManager
2026-07-13T20:44:27.597138217Z INFO: Collecting host service logs for openvswitch
2026-07-13T20:44:27.597630158Z INFO: Collecting host service logs for ovs-configuration
2026-07-13T20:44:27.597932948Z INFO: Collecting host service logs for ovsdb-server
2026-07-13T20:44:27.598209522Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-13T20:44:27.599266191Z INFO: Waiting for worker host service log collection to complete ...
2026-07-13T20:44:27.604146186Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-13T20:44:27.608672901Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-13T20:44:27.747033551Z INFO: Waiting for node performance related collection to complete ...
2026-07-13T20:44:27.842948944Z No resources found in openshift-etcd namespace.
2026-07-13T20:44:28.396678216Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-w9wm9
2026-07-13T20:44:28.396730760Z insights-runtime-extractor-b2r7q
2026-07-13T20:44:28.396750642Z insights-runtime-extractor-h8l48
2026-07-13T20:44:28.396770131Z insights-runtime-extractor-zdd9x
2026-07-13T20:44:28.429207789Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-13T20:44:28.429242646Z INFO: on-disk MachineConfig config collection complete.
2026-07-13T20:44:28.545917644Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-13T20:44:28.738963521Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-13T20:44:28.740582342Z No resources found
2026-07-13T20:44:28.749552957Z ERROR: No running kube-apiserver pods found
2026-07-13T20:44:28.981205242Z Wrote inspect data to must-gather.
2026-07-13T20:44:29.027439475Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-13T20:44:29.027439475Z See 'oc rsync -h' for help and examples
2026-07-13T20:44:29.033241993Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-13T20:44:29.078766481Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-13T20:44:29.106671614Z INFO: Worker host service log collection to complete.
2026-07-13T20:44:29.120728384Z error: the server doesn't have a resource type "performanceprofile"
2026-07-13T20:44:29.162386555Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-13T20:44:29.284163126Z INFO: "metallb-operator" not detected. Skipping.
2026-07-13T20:44:29.591000521Z INFO: 'previous-logs' folder not found on ip-10-0-128-185.ec2.internal, skipping...
2026-07-13T20:44:29.606886400Z tar: Removing leading `/' from member names
2026-07-13T20:44:29.626586659Z Wrote inspect data to must-gather.
2026-07-13T20:44:29.641337879Z INFO: HAProxy config collection complete.
2026-07-13T20:44:29.797763929Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:30.069789777Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-13T20:44:30.151969667Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-13T20:44:30.154800679Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.179396182Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T20:44:30.190844574Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.192920820Z No resources found
2026-07-13T20:44:30.214351146Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.323239122Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.341077960Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.475861436Z Gathering data for ns/openshift-monitoring...
2026-07-13T20:44:30.520188775Z error: the server doesn't have a resource type "machineconfigs"
2026-07-13T20:44:30.538634044Z INFO: 'previous-logs' folder not found on ip-10-0-131-42.ec2.internal, skipping...
2026-07-13T20:44:30.604350594Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.717817572Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-13T20:44:30.772626917Z INFO: Getting rules from prometheus-k8s-0
2026-07-13T20:44:30.848713729Z INFO: 'previous-logs' folder not found on ip-10-0-136-103.ec2.internal, skipping...
2026-07-13T20:44:30.848819014Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-13T20:44:30.848869418Z INFO: Machine Config Daemon termination log collection complete.
2026-07-13T20:44:30.927316521Z Wrote inspect data to must-gather.
2026-07-13T20:44:30.984172846Z Wrote inspect data to must-gather.
2026-07-13T20:44:31.108564361Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-13T20:44:31.165266831Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-13T20:44:31.217137666Z Wrote inspect data to must-gather.
2026-07-13T20:44:31.234990792Z INFO: Getting status/config from prometheus-k8s-0
2026-07-13T20:44:31.408313215Z Wrote inspect data to must-gather.
2026-07-13T20:44:31.527484323Z INFO: INTERCONNECT MODE
2026-07-13T20:44:31.527548093Z INFO: Gathering ovn-kubernetes DBs
2026-07-13T20:44:31.659693074Z Wrote inspect data to must-gather.
2026-07-13T20:44:31.684703348Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-13T20:44:31.716061479Z INFO: Gathering OVN_Northbound from ovnkube-node-6vz2w...
2026-07-13T20:44:31.716320153Z INFO: Gathering OVN_Northbound from ovnkube-node-7qqqv...
2026-07-13T20:44:31.716604535Z INFO: Gathering OVN_Northbound from ovnkube-node-zmz6m...
2026-07-13T20:44:31.716875647Z INFO: Gathering OVN_Southbound from ovnkube-node-6vz2w...
2026-07-13T20:44:31.717136011Z INFO: Gathering OVN_Southbound from ovnkube-node-7qqqv...
2026-07-13T20:44:31.717486548Z INFO: Gathering OVN_Southbound from ovnkube-node-zmz6m...
2026-07-13T20:44:31.985041289Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.000693904Z Wrote inspect data to must-gather.
2026-07-13T20:44:32.016656693Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-13T20:44:32.032896529Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.060425197Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.064260156Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.072836524Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.087191280Z tar: Removing leading `/' from member names
2026-07-13T20:44:32.257566096Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T20:44:32.263313776Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-13T20:44:32.295934556Z Wrote inspect data to must-gather.
2026-07-13T20:44:32.333922562Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-13T20:44:32.514761422Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:32.596422460Z Wrote inspect data to must-gather.
2026-07-13T20:44:32.633427198Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-13T20:44:32.689095320Z INFO: Waiting for network log collection to complete ...
2026-07-13T20:44:32.689131175Z INFO: Waiting for ovnk database copies to complete ...
2026-07-13T20:44:32.689651658Z INFO: Copying ovnk databases complete.
2026-07-13T20:44:32.691854224Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-13T20:44:32.694802958Z ovnk_database_store/
2026-07-13T20:44:32.694853545Z ovnk_database_store/ovnkube-node-6vz2w_nbdb
2026-07-13T20:44:32.707307966Z ovnk_database_store/ovnkube-node-zmz6m_nbdb
2026-07-13T20:44:32.716019533Z ovnk_database_store/ovnkube-node-7qqqv_nbdb
2026-07-13T20:44:32.726150424Z ovnk_database_store/ovnkube-node-6vz2w_sbdb
2026-07-13T20:44:32.750582891Z ovnk_database_store/ovnkube-node-7qqqv_sbdb
2026-07-13T20:44:32.773533935Z ovnk_database_store/ovnkube-node-zmz6m_sbdb
2026-07-13T20:44:32.782058162Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:32.784228007Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-13T20:44:32.797929872Z INFO: Network log collection complete.
2026-07-13T20:44:32.856278233Z W0713 20:44:32.856223    1504 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-13T20:44:32.861804244Z Wrote inspect data to must-gather.
2026-07-13T20:44:32.959462287Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T20:44:33.007669566Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:33.087404909Z INFO: Getting status from alertmanager-main-0
2026-07-13T20:44:33.167373527Z Gathering data for ns/openshift-network-console...
2026-07-13T20:44:33.191198640Z Wrote inspect data to must-gather.
2026-07-13T20:44:33.245267476Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:33.248752488Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-13T20:44:33.474689896Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:33.493097128Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-13T20:44:33.496276884Z INFO: Fallback to identify the container image from release info
2026-07-13T20:44:33.646576233Z Gathering data for ns/openshift-console-operator...
2026-07-13T20:44:33.702040744Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:33.705233986Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-13T20:44:33.926387502Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.034059823Z Gathering data for ns/openshift-console...
2026-07-13T20:44:34.075032898Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-13T20:44:34.147457086Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.151897159Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-13T20:44:34.228788618Z daemonset.apps/perf-node-gather-daemonset created
2026-07-13T20:44:34.344096797Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.385431615Z Waiting for performance profile collector pods to become ready: 1
2026-07-13T20:44:34.568225462Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.571048333Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-13T20:44:34.605820784Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-13T20:44:34.782343723Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.883456064Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:34.981855142Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:34.984838443Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-13T20:44:35.169867878Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:35.169998731Z Gathering data for ns/openshift-dns-operator...
2026-07-13T20:44:35.375514493Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:35.379429276Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-13T20:44:35.532312300Z Gathering data for ns/openshift-dns...
2026-07-13T20:44:35.538296250Z Waiting for performance profile collector pods to become ready: 2
2026-07-13T20:44:35.577019615Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:35.753727255Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:35.756130876Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-13T20:44:35.933346624Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:36.159038059Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:36.162683883Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-13T20:44:36.166600956Z Gathering data for ns/openshift-image-registry...
2026-07-13T20:44:36.336539592Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:36.524905971Z Wrote inspect data to /must-gather/istio.
2026-07-13T20:44:36.680208439Z Waiting for performance profile collector pods to become ready: 3
2026-07-13T20:44:36.695943275Z Gathering data for ns/openshift-ingress-operator...
2026-07-13T20:44:37.056424603Z Gathering data for ns/openshift-ingress...
2026-07-13T20:44:37.256781951Z Done executing Istio gather script
2026-07-13T20:44:37.394481528Z Gathering data for ns/openshift-ingress-canary...
2026-07-13T20:44:37.801135022Z Gathering data for ns/openshift-insights...
2026-07-13T20:44:37.821483550Z Waiting for performance profile collector pods to become ready: 4
2026-07-13T20:44:38.935300824Z Waiting for performance profile collector pods to become ready: 5
2026-07-13T20:44:39.888244491Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:39.990667031Z Gathering data for ns/kserve...
2026-07-13T20:44:40.053919482Z Waiting for performance profile collector pods to become ready: 6
2026-07-13T20:44:41.176944121Z Waiting for performance profile collector pods to become ready: 7
2026-07-13T20:44:41.533103637Z Gathering data for ns/openshift-config...
2026-07-13T20:44:41.784776673Z Gathering data for ns/openshift-config-managed...
2026-07-13T20:44:42.245988581Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-13T20:44:42.320916125Z Waiting for performance profile collector pods to become ready: 8
2026-07-13T20:44:42.553313396Z Gathering data for ns/openshift-kube-apiserver...
2026-07-13T20:44:42.818369765Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-13T20:44:43.137935220Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-13T20:44:43.423558989Z Gathering data for ns/openshift-kube-scheduler...
2026-07-13T20:44:43.465002259Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-13T20:44:43.706962452Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-13T20:44:43.742584042Z Collecting performance related data for node ip-10-0-131-42.ec2.internal
2026-07-13T20:44:43.755731373Z Collecting performance related data for node ip-10-0-128-185.ec2.internal
2026-07-13T20:44:43.764277938Z Collecting performance related data for node ip-10-0-136-103.ec2.internal
2026-07-13T20:44:44.048615418Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-13T20:44:44.449891441Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-13T20:44:44.849598922Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-13T20:44:44.895161500Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:45.484455738Z Gathering data for ns/openshift-multus...
2026-07-13T20:44:46.967980781Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-13T20:44:48.186857650Z Gathering data for ns/openshift-host-network...
2026-07-13T20:44:48.540471173Z Gathering data for ns/openshift-network-diagnostics...
2026-07-13T20:44:49.079775820Z Gathering data for ns/openshift-network-node-identity...
2026-07-13T20:44:49.502117495Z Gathering data for ns/openshift-network-operator...
2026-07-13T20:44:49.887202153Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-13T20:44:49.901821588Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-13T20:44:50.152743825Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-13T20:44:50.624549491Z Gathering data for ns/openshift-apiserver-operator...
2026-07-13T20:44:50.935989656Z Gathering data for ns/openshift-apiserver...
2026-07-13T20:44:51.312643324Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-13T20:44:51.613906638Z Gathering data for ns/openshift-controller-manager...
2026-07-13T20:44:51.901954653Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-13T20:44:52.353293207Z Collecting kubelet logs for node ip-10-0-131-42.ec2.internal
2026-07-13T20:44:52.355193726Z Collecting kubelet logs for node ip-10-0-128-185.ec2.internal
2026-07-13T20:44:52.357565508Z Collecting kubelet logs for node ip-10-0-136-103.ec2.internal
2026-07-13T20:44:52.677811606Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-13T20:44:52.733843360Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-k5kgn namespace
2026-07-13T20:44:52.740160335Z INFO: Node performance data collection complete.
2026-07-13T20:44:52.983748443Z Gathering data for ns/openshift-service-ca-operator...
2026-07-13T20:44:53.311594142Z Gathering data for ns/openshift-service-ca...
2026-07-13T20:44:53.668342119Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-13T20:44:54.554951944Z Wrote inspect data to must-gather.
2026-07-13T20:44:54.554995345Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T20:44:54.554995345Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-13T20:44:54.685766939Z error: the server doesn't have a resource type "clusters"
2026-07-13T20:44:54.832793997Z Caches written to disk