2026-09-18T15:12:09.323592942Z [disk usage checker] Started
2026-09-18T15:12:09.326896542Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:09.545681145Z Gathering data for ns/openshift-cluster-version...
2026-09-18T15:12:09.672275029Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-18T15:12:09.755106960Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T15:12:09.833306814Z Gathering data for ns/default...
2026-09-18T15:12:10.143697730Z Gathering data for ns/openshift...
2026-09-18T15:12:10.529818737Z Gathering data for ns/kube-system...
2026-09-18T15:12:11.190529764Z Gathering data for ns/openshift-etcd...
2026-09-18T15:12:11.518306978Z Wrote inspect data to must-gather.
2026-09-18T15:12:11.518360060Z error: inspection completed with the errors occurred while gathering data:
2026-09-18T15:12:11.518360060Z     namespaces "assisted-installer" not found
2026-09-18T15:12:11.869639512Z Waiting on subprocesses to finish execution.
2026-09-18T15:12:11.879762191Z INFO: Collecting host service logs for crio
2026-09-18T15:12:11.880068387Z INFO: Collecting host service logs for kubelet
2026-09-18T15:12:11.880455502Z INFO: Collecting host service logs for rpm-ostreed
2026-09-18T15:12:11.880799697Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-18T15:12:11.881153713Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-18T15:12:11.881515484Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-18T15:12:11.881878061Z INFO: Collecting host service logs for NetworkManager
2026-09-18T15:12:11.882555514Z INFO: Collecting host service logs for openvswitch
2026-09-18T15:12:11.882888672Z INFO: Collecting host service logs for ovs-configuration
2026-09-18T15:12:11.883237962Z INFO: Collecting host service logs for ovsdb-server
2026-09-18T15:12:11.883639739Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-18T15:12:11.884039046Z INFO: Waiting for worker host service log collection to complete ...
2026-09-18T15:12:11.889807593Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-18T15:12:11.890498923Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-18T15:12:11.901071480Z Executing Istio gather script
2026-09-18T15:12:11.909969342Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-18T15:12:11.910747421Z INFO: Gathering HAProxy config files
2026-09-18T15:12:11.961974745Z INFO: Waiting for node performance related collection to complete ...
2026-09-18T15:12:12.565293501Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-18T15:12:12.578477247Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-18T15:12:12.798406290Z No resources found in openshift-etcd namespace.
2026-09-18T15:12:12.871339121Z INFO: Collecting Insights Archives from insights-operator-6f9c644dd9-xjrkh
2026-09-18T15:12:12.871387317Z insights-runtime-extractor-cmll2
2026-09-18T15:12:12.871407818Z insights-runtime-extractor-n99h9
2026-09-18T15:12:12.871435087Z insights-runtime-extractor-x6m4p
2026-09-18T15:12:12.987987395Z error: the server doesn't have a resource type "performanceprofile"
2026-09-18T15:12:13.006670246Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-18T15:12:13.092038284Z No resources found
2026-09-18T15:12:13.114337049Z Wrote inspect data to must-gather.
2026-09-18T15:12:13.192210109Z INFO: "metallb-operator" not detected. Skipping.
2026-09-18T15:12:13.396489621Z INFO: Worker host service log collection to complete.
2026-09-18T15:12:13.552002736Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-18T15:12:13.552002736Z See 'oc rsync -h' for help and examples
2026-09-18T15:12:13.836648315Z ERROR: No running kube-apiserver pods found
2026-09-18T15:12:13.994244922Z Gathering data for ns/openshift-monitoring...
2026-09-18T15:12:14.008790749Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-18T15:12:14.035201622Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-18T15:12:14.035278355Z INFO: on-disk MachineConfig config collection complete.
2026-09-18T15:12:14.070634302Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T15:12:14.123047002Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.200457053Z No resources found
2026-09-18T15:12:14.203196406Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.217407923Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.272654576Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-18T15:12:14.307645760Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-18T15:12:14.379350677Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:14.470006084Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.495145074Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.502245256Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-18T15:12:14.511301655Z error: the server doesn't have a resource type "machineconfigs"
2026-09-18T15:12:14.557708415Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.583137387Z INFO: 'previous-logs' folder not found on ip-10-0-133-168.ec2.internal, skipping...
2026-09-18T15:12:14.584074990Z tar: Removing leading `/' from member names
2026-09-18T15:12:14.590297040Z INFO: HAProxy config collection complete.
2026-09-18T15:12:14.709902262Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-18T15:12:14.871183953Z Wrote inspect data to must-gather.
2026-09-18T15:12:14.901296547Z INFO: Getting rules from prometheus-k8s-0
2026-09-18T15:12:14.914873680Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-18T15:12:14.918037795Z INFO: 'previous-logs' folder not found on ip-10-0-134-28.ec2.internal, skipping...
2026-09-18T15:12:14.988220267Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.164870724Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-18T15:12:15.170025304Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.212856294Z INFO: 'previous-logs' folder not found on ip-10-0-136-56.ec2.internal, skipping...
2026-09-18T15:12:15.212987641Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-18T15:12:15.213041690Z INFO: Machine Config Daemon termination log collection complete.
2026-09-18T15:12:15.324923722Z INFO: Getting status/config from prometheus-k8s-0
2026-09-18T15:12:15.399502742Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.417604625Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.634617336Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.733673490Z INFO: INTERCONNECT MODE
2026-09-18T15:12:15.733719078Z INFO: Gathering ovn-kubernetes DBs
2026-09-18T15:12:15.750663633Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-18T15:12:15.767745317Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-18T15:12:15.865250073Z Wrote inspect data to must-gather.
2026-09-18T15:12:15.875111835Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-18T15:12:15.924062909Z INFO: Gathering OVN_Northbound from ovnkube-node-l8lcw...
2026-09-18T15:12:15.924328615Z INFO: Gathering OVN_Northbound from ovnkube-node-mhw4t...
2026-09-18T15:12:15.924538340Z INFO: Gathering OVN_Northbound from ovnkube-node-nm69k...
2026-09-18T15:12:15.924782302Z INFO: Gathering OVN_Southbound from ovnkube-node-l8lcw...
2026-09-18T15:12:15.925036841Z INFO: Gathering OVN_Southbound from ovnkube-node-mhw4t...
2026-09-18T15:12:15.925307787Z INFO: Gathering OVN_Southbound from ovnkube-node-nm69k...
2026-09-18T15:12:16.096458792Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-18T15:12:16.221493415Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.231225546Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.234242284Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:16.238448579Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.245375081Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.272246469Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.296304863Z tar: Removing leading `/' from member names
2026-09-18T15:12:16.319572068Z Wrote inspect data to must-gather.
2026-09-18T15:12:16.431819113Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-18T15:12:16.497131914Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:16.500042080Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-18T15:12:16.575448956Z W0918 15:12:16.575370    1431 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-18T15:12:16.584548062Z Wrote inspect data to must-gather.
2026-09-18T15:12:16.617639477Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T15:12:16.719277003Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-18T15:12:16.754171523Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:16.815304864Z INFO: Waiting for network log collection to complete ...
2026-09-18T15:12:16.815355351Z INFO: Waiting for ovnk database copies to complete ...
2026-09-18T15:12:16.815728072Z INFO: Copying ovnk databases complete.
2026-09-18T15:12:16.817363783Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-09-18T15:12:16.820345684Z ovnk_database_store/
2026-09-18T15:12:16.820401561Z ovnk_database_store/ovnkube-node-l8lcw_nbdb
2026-09-18T15:12:16.825522386Z Gathering data for ns/openshift-network-console...
2026-09-18T15:12:16.830486789Z ovnk_database_store/ovnkube-node-mhw4t_nbdb
2026-09-18T15:12:16.838375230Z ovnk_database_store/ovnkube-node-nm69k_sbdb
2026-09-18T15:12:16.858260055Z ovnk_database_store/ovnkube-node-l8lcw_sbdb
2026-09-18T15:12:16.882599657Z ovnk_database_store/ovnkube-node-nm69k_nbdb
2026-09-18T15:12:16.897814489Z ovnk_database_store/ovnkube-node-mhw4t_sbdb
2026-09-18T15:12:16.924693496Z INFO: Network log collection complete.
2026-09-18T15:12:16.929352864Z Wrote inspect data to must-gather.
2026-09-18T15:12:16.997238111Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:16.999751370Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-18T15:12:17.162132265Z INFO: Getting status from alertmanager-main-0
2026-09-18T15:12:17.216025154Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:17.247542670Z Wrote inspect data to must-gather.
2026-09-18T15:12:17.337557343Z Gathering data for ns/openshift-console-operator...
2026-09-18T15:12:17.467181971Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:17.470174340Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-18T15:12:17.532685789Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-18T15:12:17.534600720Z INFO: Fallback to identify the container image from release info
2026-09-18T15:12:17.680700140Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:17.748739466Z Gathering data for ns/openshift-console...
2026-09-18T15:12:17.868952846Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:17.871778526Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-18T15:12:18.016163197Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:18.140641283Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0720c8b078cc0cebb01b4c5cde21180aee7c2bb1a0d322b21890c0d3707406ea
2026-09-18T15:12:18.242895698Z Wrote inspect data to /must-gather/istio.
2026-09-18T15:12:18.275910041Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-18T15:12:18.281212759Z daemonset.apps/perf-node-gather-daemonset created
2026-09-18T15:12:18.420308198Z Waiting for performance profile collector pods to become ready: 1
2026-09-18T15:12:18.842381983Z Gathering data for ns/openshift-dns-operator...
2026-09-18T15:12:18.988466067Z Done executing Istio gather script
2026-09-18T15:12:19.123279677Z Gathering data for ns/openshift-dns...
2026-09-18T15:12:19.392869758Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:19.536336868Z Waiting for performance profile collector pods to become ready: 2
2026-09-18T15:12:19.691722544Z Gathering data for ns/openshift-image-registry...
2026-09-18T15:12:20.118990779Z Gathering data for ns/openshift-ingress-operator...
2026-09-18T15:12:20.408046325Z Gathering data for ns/openshift-ingress...
2026-09-18T15:12:20.659129702Z Waiting for performance profile collector pods to become ready: 3
2026-09-18T15:12:20.717490490Z Gathering data for ns/openshift-ingress-canary...
2026-09-18T15:12:21.108605099Z Gathering data for ns/openshift-insights...
2026-09-18T15:12:21.772694564Z Waiting for performance profile collector pods to become ready: 4
2026-09-18T15:12:22.855101328Z Gathering data for ns/openshift-jobset-operator...
2026-09-18T15:12:22.879888274Z Waiting for performance profile collector pods to become ready: 5
2026-09-18T15:12:23.647495468Z Gathering data for ns/openshift-config...
2026-09-18T15:12:23.881498714Z Gathering data for ns/openshift-config-managed...
2026-09-18T15:12:23.999599831Z Waiting for performance profile collector pods to become ready: 6
2026-09-18T15:12:24.313972917Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-18T15:12:24.397343811Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:24.546564586Z Gathering data for ns/openshift-kube-apiserver...
2026-09-18T15:12:24.789750680Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-18T15:12:25.115366168Z Waiting for performance profile collector pods to become ready: 7
2026-09-18T15:12:25.115664947Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-18T15:12:25.420151150Z Gathering data for ns/openshift-kube-scheduler...
2026-09-18T15:12:25.700073527Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-18T15:12:25.958857561Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-18T15:12:26.233259200Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-18T15:12:26.265772296Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-18T15:12:26.497749522Z Collecting performance related data for node ip-10-0-133-168.ec2.internal
2026-09-18T15:12:26.505241527Z Collecting performance related data for node ip-10-0-134-28.ec2.internal
2026-09-18T15:12:26.514257739Z Collecting performance related data for node ip-10-0-136-56.ec2.internal
2026-09-18T15:12:26.597148026Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-18T15:12:27.170287420Z Gathering data for ns/openshift-multus...
2026-09-18T15:12:28.557762550Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-18T15:12:29.403091105Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:29.789338731Z Gathering data for ns/openshift-host-network...
2026-09-18T15:12:30.061952176Z Gathering data for ns/openshift-network-diagnostics...
2026-09-18T15:12:30.566532207Z Gathering data for ns/openshift-network-node-identity...
2026-09-18T15:12:30.944875579Z Gathering data for ns/openshift-network-operator...
2026-09-18T15:12:31.355154607Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-18T15:12:31.599062963Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-18T15:12:32.092133692Z Gathering data for ns/openshift-apiserver-operator...
2026-09-18T15:12:32.369958411Z Gathering data for ns/openshift-apiserver...
2026-09-18T15:12:32.760781456Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-18T15:12:32.994979301Z Gathering data for ns/openshift-controller-manager...
2026-09-18T15:12:33.243887620Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-18T15:12:33.882940275Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-18T15:12:34.146920919Z Gathering data for ns/openshift-service-ca-operator...
2026-09-18T15:12:34.409073066Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-09-18T15:12:34.459097039Z Gathering data for ns/openshift-service-ca...
2026-09-18T15:12:34.725380476Z Collecting kubelet logs for node ip-10-0-133-168.ec2.internal
2026-09-18T15:12:34.727540149Z Collecting kubelet logs for node ip-10-0-134-28.ec2.internal
2026-09-18T15:12:34.729953130Z Collecting kubelet logs for node ip-10-0-136-56.ec2.internal
2026-09-18T15:12:34.821680592Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-18T15:12:35.063992972Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-knc85 namespace
2026-09-18T15:12:35.070378444Z INFO: Node performance data collection complete.
2026-09-18T15:12:35.714545500Z Wrote inspect data to must-gather.
2026-09-18T15:12:35.714587593Z error: inspection completed with the errors occurred while gathering data:
2026-09-18T15:12:35.714587593Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-18T15:12:35.837740565Z error: the server doesn't have a resource type "clusters"
2026-09-18T15:12:35.971650763Z Caches written to disk