2026-09-10T09:45:48.081189843Z [disk usage checker] Started
2026-09-10T09:45:48.084713185Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:45:48.280806826Z Gathering data for ns/openshift-cluster-version...
2026-09-10T09:45:48.403141282Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-10T09:45:48.484514933Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-10T09:45:48.565214963Z Gathering data for ns/default...
2026-09-10T09:45:48.833138469Z Gathering data for ns/openshift...
2026-09-10T09:45:49.246049540Z Gathering data for ns/kube-system...
2026-09-10T09:45:49.904732148Z Gathering data for ns/openshift-etcd...
2026-09-10T09:45:50.179925050Z Wrote inspect data to must-gather.
2026-09-10T09:45:50.179957786Z error: inspection completed with the errors occurred while gathering data:
2026-09-10T09:45:50.179957786Z     namespaces "assisted-installer" not found
2026-09-10T09:45:50.601242323Z Waiting on subprocesses to finish execution.
2026-09-10T09:45:50.606323354Z INFO: Collecting host service logs for crio
2026-09-10T09:45:50.606641395Z INFO: Collecting host service logs for kubelet
2026-09-10T09:45:50.606981268Z INFO: Collecting host service logs for rpm-ostreed
2026-09-10T09:45:50.607298497Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-10T09:45:50.607679053Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-10T09:45:50.608012517Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-10T09:45:50.608372525Z INFO: Collecting host service logs for NetworkManager
2026-09-10T09:45:50.608735113Z INFO: Collecting host service logs for openvswitch
2026-09-10T09:45:50.609127063Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-10T09:45:50.609127063Z INFO: Collecting host service logs for ovs-configuration
2026-09-10T09:45:50.609462400Z INFO: Collecting host service logs for ovsdb-server
2026-09-10T09:45:50.609905964Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-10T09:45:50.610347029Z INFO: Waiting for worker host service log collection to complete ...
2026-09-10T09:45:50.621680875Z Executing Istio gather script
2026-09-10T09:45:50.622310387Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-10T09:45:50.652672696Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-10T09:45:50.706202636Z INFO: Gathering HAProxy config files
2026-09-10T09:45:50.757003690Z INFO: Waiting for node performance related collection to complete ...
2026-09-10T09:45:50.913810243Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-10T09:45:51.260072928Z No resources found
2026-09-10T09:45:51.529511923Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-10T09:45:51.617491382Z No resources found in openshift-etcd namespace.
2026-09-10T09:45:51.696259547Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-10T09:45:51.755513504Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-vqh55
2026-09-10T09:45:51.755565154Z insights-runtime-extractor-89gjd
2026-09-10T09:45:51.755586009Z insights-runtime-extractor-m2b25
2026-09-10T09:45:51.755605972Z insights-runtime-extractor-zfhl5
2026-09-10T09:45:51.791952153Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-10T09:45:51.801514178Z Wrote inspect data to must-gather.
2026-09-10T09:45:51.853068100Z INFO: "metallb-operator" not detected. Skipping.
2026-09-10T09:45:51.892050775Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-10T09:45:51.892130631Z INFO: on-disk MachineConfig config collection complete.
2026-09-10T09:45:51.934882508Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-10T09:45:52.044374337Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-10T09:45:52.044374337Z See 'oc rsync -h' for help and examples
2026-09-10T09:45:52.088455057Z INFO: Worker host service log collection to complete.
2026-09-10T09:45:52.175785142Z error: the server doesn't have a resource type "performanceprofile"
2026-09-10T09:45:52.405664009Z Wrote inspect data to must-gather.
2026-09-10T09:45:52.580897058Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-10T09:45:52.624887386Z ERROR: No running kube-apiserver pods found
2026-09-10T09:45:52.665576401Z Wrote inspect data to must-gather.
2026-09-10T09:45:52.766168736Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-10T09:45:52.908573928Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-10T09:45:53.003181145Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-10T09:45:53.070557411Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.120099702Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.124664821Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:45:53.140189471Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.145393501Z No resources found
2026-09-10T09:45:53.148485083Z INFO: Getting rules from prometheus-k8s-0
2026-09-10T09:45:53.178993955Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.222448363Z INFO: 'previous-logs' folder not found on ip-10-0-138-188.ec2.internal, skipping...
2026-09-10T09:45:53.343866257Z error: the server doesn't have a resource type "machineconfigs"
2026-09-10T09:45:53.345440922Z tar: Removing leading `/' from member names
2026-09-10T09:45:53.345440922Z Gathering data for ns/openshift-monitoring...
2026-09-10T09:45:53.351260093Z INFO: HAProxy config collection complete.
2026-09-10T09:45:53.474978096Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.526169795Z INFO: 'previous-logs' folder not found on ip-10-0-139-170.ec2.internal, skipping...
2026-09-10T09:45:53.531971140Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-10T09:45:53.586088422Z INFO: Getting status/config from prometheus-k8s-0
2026-09-10T09:45:53.742644612Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.788597473Z Wrote inspect data to must-gather.
2026-09-10T09:45:53.808479929Z INFO: 'previous-logs' folder not found on ip-10-0-140-80.ec2.internal, skipping...
2026-09-10T09:45:53.808511167Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-10T09:45:53.808542508Z INFO: Machine Config Daemon termination log collection complete.
2026-09-10T09:45:53.968516419Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-10T09:45:53.996429429Z Wrote inspect data to must-gather.
2026-09-10T09:45:54.036936101Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-10T09:45:54.196904403Z Wrote inspect data to must-gather.
2026-09-10T09:45:54.256496275Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-10T09:45:54.289640112Z INFO: INTERCONNECT MODE
2026-09-10T09:45:54.289680777Z INFO: Gathering ovn-kubernetes DBs
2026-09-10T09:45:54.330883233Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-10T09:45:54.408188519Z Wrote inspect data to must-gather.
2026-09-10T09:45:54.465372621Z INFO: Gathering OVN_Northbound from ovnkube-node-dtm95...
2026-09-10T09:45:54.465688731Z INFO: Gathering OVN_Northbound from ovnkube-node-gmgxj...
2026-09-10T09:45:54.466013318Z INFO: Gathering OVN_Northbound from ovnkube-node-xsql9...
2026-09-10T09:45:54.466293897Z INFO: Gathering OVN_Southbound from ovnkube-node-dtm95...
2026-09-10T09:45:54.466594251Z INFO: Gathering OVN_Southbound from ovnkube-node-gmgxj...
2026-09-10T09:45:54.466873582Z INFO: Gathering OVN_Southbound from ovnkube-node-xsql9...
2026-09-10T09:45:54.517070942Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-10T09:45:54.591051920Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:54.785438400Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.802967969Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.810604100Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.819885562Z Wrote inspect data to must-gather.
2026-09-10T09:45:54.823936737Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-10T09:45:54.824235706Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.839072692Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.853561290Z tar: Removing leading `/' from member names
2026-09-10T09:45:54.916190034Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:54.917434903Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-10T09:45:54.918181294Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-10T09:45:55.078084932Z Wrote inspect data to must-gather.
2026-09-10T09:45:55.138887499Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:55.348816517Z Wrote inspect data to must-gather.
2026-09-10T09:45:55.353248161Z INFO: Waiting for network log collection to complete ...
2026-09-10T09:45:55.353299558Z INFO: Waiting for ovnk database copies to complete ...
2026-09-10T09:45:55.353849638Z INFO: Copying ovnk databases complete.
2026-09-10T09:45:55.355643405Z 3.6M	must-gather/network_logs/ovnk_database_store
2026-09-10T09:45:55.357323237Z INFO: Getting status from alertmanager-main-0
2026-09-10T09:45:55.357889900Z ovnk_database_store/
2026-09-10T09:45:55.357889900Z ovnk_database_store/ovnkube-node-dtm95_nbdb
2026-09-10T09:45:55.366714337Z ovnk_database_store/ovnkube-node-gmgxj_sbdb
2026-09-10T09:45:55.387442998Z ovnk_database_store/ovnkube-node-dtm95_sbdb
2026-09-10T09:45:55.399411180Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:55.401609044Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-10T09:45:55.409442282Z ovnk_database_store/ovnkube-node-xsql9_sbdb
2026-09-10T09:45:55.434497882Z ovnk_database_store/ovnkube-node-gmgxj_nbdb
2026-09-10T09:45:55.444249417Z ovnk_database_store/ovnkube-node-xsql9_nbdb
2026-09-10T09:45:55.455269838Z INFO: Network log collection complete.
2026-09-10T09:45:55.515247544Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-10T09:45:55.602216255Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:55.661004296Z Wrote inspect data to must-gather.
2026-09-10T09:45:55.705252968Z Gathering data for ns/openshift-network-console...
2026-09-10T09:45:55.793919308Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:55.797553683Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-10T09:45:55.948530456Z Wrote inspect data to must-gather.
2026-09-10T09:45:56.014528595Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:56.109646614Z Gathering data for ns/openshift-console-operator...
2026-09-10T09:45:56.225577014Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:56.227809072Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-10T09:45:56.228162104Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-10T09:45:56.230253309Z INFO: Fallback to identify the container image from release info
2026-09-10T09:45:56.393231073Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:56.476547060Z Gathering data for ns/openshift-console...
2026-09-10T09:45:56.583373483Z Wrote inspect data to /must-gather/istio.
2026-09-10T09:45:56.964519553Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-10T09:45:56.987680038Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399
2026-09-10T09:45:57.117566803Z daemonset.apps/perf-node-gather-daemonset created
2026-09-10T09:45:57.249893899Z Waiting for performance profile collector pods to become ready: 1
2026-09-10T09:45:57.255012373Z Done executing Istio gather script
2026-09-10T09:45:57.458093007Z Gathering data for ns/openshift-dns-operator...
2026-09-10T09:45:57.730255594Z Gathering data for ns/openshift-dns...
2026-09-10T09:45:58.131834931Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:45:58.284532261Z Gathering data for ns/openshift-image-registry...
2026-09-10T09:45:58.359699757Z Waiting for performance profile collector pods to become ready: 2
2026-09-10T09:45:58.708159081Z Gathering data for ns/openshift-ingress-operator...
2026-09-10T09:45:59.000060884Z Gathering data for ns/openshift-ingress...
2026-09-10T09:45:59.283677411Z Gathering data for ns/openshift-ingress-canary...
2026-09-10T09:45:59.475931056Z Waiting for performance profile collector pods to become ready: 3
2026-09-10T09:45:59.633091411Z Gathering data for ns/openshift-insights...
2026-09-10T09:46:00.586138053Z Waiting for performance profile collector pods to become ready: 4
2026-09-10T09:46:01.549978640Z Gathering data for ns/openshift-config...
2026-09-10T09:46:01.698493846Z Waiting for performance profile collector pods to become ready: 5
2026-09-10T09:46:01.792290154Z Gathering data for ns/openshift-config-managed...
2026-09-10T09:46:02.218256227Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-10T09:46:02.487468583Z Gathering data for ns/openshift-kube-apiserver...
2026-09-10T09:46:02.725325279Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-10T09:46:02.826084595Z Waiting for performance profile collector pods to become ready: 6
2026-09-10T09:46:02.988416135Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-10T09:46:03.136993968Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:46:03.246034022Z Gathering data for ns/openshift-kube-scheduler...
2026-09-10T09:46:03.519399030Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-10T09:46:03.770978374Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-10T09:46:03.943934810Z Waiting for performance profile collector pods to become ready: 7
2026-09-10T09:46:04.067011139Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-10T09:46:04.351569112Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-10T09:46:04.811210038Z Gathering data for ns/openshift-multus...
2026-09-10T09:46:05.047394487Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-10T09:46:05.265850168Z Collecting performance related data for node ip-10-0-140-80.ec2.internal
2026-09-10T09:46:05.272481504Z Collecting performance related data for node ip-10-0-139-170.ec2.internal
2026-09-10T09:46:05.279661143Z Collecting performance related data for node ip-10-0-138-188.ec2.internal
2026-09-10T09:46:06.260852365Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-10T09:46:07.410487763Z Gathering data for ns/openshift-host-network...
2026-09-10T09:46:07.759334726Z Gathering data for ns/openshift-network-diagnostics...
2026-09-10T09:46:08.143312789Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:46:08.283432461Z Gathering data for ns/openshift-network-node-identity...
2026-09-10T09:46:08.645703952Z Gathering data for ns/openshift-network-operator...
2026-09-10T09:46:08.980276319Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-10T09:46:09.210579342Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-10T09:46:09.663706832Z Gathering data for ns/openshift-apiserver-operator...
2026-09-10T09:46:09.932293854Z Gathering data for ns/openshift-apiserver...
2026-09-10T09:46:10.286640634Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-10T09:46:10.530482581Z Gathering data for ns/openshift-controller-manager...
2026-09-10T09:46:10.779573106Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-10T09:46:11.400988662Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-10T09:46:11.662224399Z Gathering data for ns/openshift-service-ca-operator...
2026-09-10T09:46:11.937284458Z Gathering data for ns/openshift-service-ca...
2026-09-10T09:46:12.290757261Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-10T09:46:13.118083138Z Wrote inspect data to must-gather.
2026-09-10T09:46:13.118342819Z error: inspection completed with the errors occurred while gathering data:
2026-09-10T09:46:13.118342819Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-10T09:46:13.149262323Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-10T09:46:13.349455637Z Collecting kubelet logs for node ip-10-0-140-80.ec2.internal
2026-09-10T09:46:13.351149643Z Collecting kubelet logs for node ip-10-0-139-170.ec2.internal
2026-09-10T09:46:13.352789278Z Collecting kubelet logs for node ip-10-0-138-188.ec2.internal
2026-09-10T09:46:13.674672021Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mnrtq namespace
2026-09-10T09:46:13.679604715Z INFO: Node performance data collection complete.
2026-09-10T09:46:13.809214590Z error: the server doesn't have a resource type "clusters"
2026-09-10T09:46:13.973308572Z Caches written to disk