2026-07-09T18:29:24.505871001Z [disk usage checker] Started
2026-07-09T18:29:24.509104468Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-09T18:29:24.698896412Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:29:24.814943175Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:29:24.905890297Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:29:24.992779961Z Gathering data for ns/default...
2026-07-09T18:29:25.355384452Z Gathering data for ns/openshift...
2026-07-09T18:29:25.796151382Z Gathering data for ns/kube-system...
2026-07-09T18:29:26.693459187Z Gathering data for ns/openshift-etcd...
2026-07-09T18:29:27.057041024Z Wrote inspect data to must-gather.
2026-07-09T18:29:27.057070060Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:29:27.057070060Z     namespaces "assisted-installer" not found
2026-07-09T18:29:27.323277831Z Waiting on subprocesses to finish execution.
2026-07-09T18:29:27.330437515Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:29:27.333899895Z Executing Istio gather script
2026-07-09T18:29:27.338101467Z INFO: Gathering HAProxy config files
2026-07-09T18:29:27.339029871Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:29:27.341268996Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:29:27.357453755Z INFO: Collecting host service logs for crio
2026-07-09T18:29:27.357783566Z INFO: Collecting host service logs for kubelet
2026-07-09T18:29:27.358137570Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:29:27.358475272Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:29:27.358802448Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:29:27.359139024Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:29:27.359478225Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:29:27.359814303Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:29:27.360136321Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:29:27.360489358Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:29:27.360944277Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:29:27.361337862Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:29:27.414564777Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:29:28.045502468Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-2xvxr
2026-07-09T18:29:28.045502468Z insights-runtime-extractor-828zp
2026-07-09T18:29:28.045502468Z insights-runtime-extractor-qxfks
2026-07-09T18:29:28.045502468Z insights-runtime-extractor-xj9gk
2026-07-09T18:29:28.286005668Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:29:28.286098045Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:29:28.364514443Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:29:28.507971243Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:29:28.507971243Z See 'oc rsync -h' for help and examples
2026-07-09T18:29:28.514575359Z Wrote inspect data to must-gather.
2026-07-09T18:29:28.535266618Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:29:28.571223431Z No resources found
2026-07-09T18:29:28.746865614Z INFO: Worker host service log collection to complete.
2026-07-09T18:29:28.787963365Z No resources found in openshift-etcd namespace.
2026-07-09T18:29:28.809648022Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:29:28.833795655Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:29:28.933922357Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:29:28.936935960Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:29:29.003494241Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:29:29.371325257Z Wrote inspect data to must-gather.
2026-07-09T18:29:29.416919613Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:29:29.487262621Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:29:29.554818593Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:29:29.566635270Z INFO: 'previous-logs' folder not found on ip-10-0-133-63.ec2.internal, skipping...
2026-07-09T18:29:29.634022077Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-09T18:29:29.739558846Z Wrote inspect data to must-gather.
2026-07-09T18:29:29.787263337Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:29:29.788641710Z Wrote inspect data to must-gather.
2026-07-09T18:29:29.806455956Z ERROR: No running kube-apiserver pods found
2026-07-09T18:29:29.915042353Z Wrote inspect data to must-gather.
2026-07-09T18:29:29.962855965Z No resources found
2026-07-09T18:29:30.029237010Z tar: Removing leading `/' from member names
2026-07-09T18:29:30.037114053Z INFO: HAProxy config collection complete.
2026-07-09T18:29:30.106823472Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.108399326Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.197803843Z INFO: 'previous-logs' folder not found on ip-10-0-136-100.ec2.internal, skipping...
2026-07-09T18:29:30.311709265Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.318414552Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:29:30.397019576Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:29:30.497220414Z INFO: 'previous-logs' folder not found on ip-10-0-141-169.ec2.internal, skipping...
2026-07-09T18:29:30.503505319Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:29:30.503580290Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:29:30.544217479Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:29:30.606593892Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.713889062Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:29:30.782607509Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.877871856Z Wrote inspect data to must-gather.
2026-07-09T18:29:30.879628457Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:29:30.954499465Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:29:31.184221538Z Wrote inspect data to must-gather.
2026-07-09T18:29:31.209142186Z INFO: INTERCONNECT MODE
2026-07-09T18:29:31.209177083Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:29:31.315017968Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:29:31.431039549Z INFO: Gathering OVN_Northbound from ovnkube-node-8xxd4...
2026-07-09T18:29:31.431256232Z INFO: Gathering OVN_Northbound from ovnkube-node-b547z...
2026-07-09T18:29:31.431526663Z INFO: Gathering OVN_Northbound from ovnkube-node-txgxp...
2026-07-09T18:29:31.431869585Z INFO: Gathering OVN_Southbound from ovnkube-node-8xxd4...
2026-07-09T18:29:31.432210976Z INFO: Gathering OVN_Southbound from ovnkube-node-b547z...
2026-07-09T18:29:31.432531410Z INFO: Gathering OVN_Southbound from ovnkube-node-txgxp...
2026-07-09T18:29:31.451492912Z Wrote inspect data to must-gather.
2026-07-09T18:29:31.500690401Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:29:31.644563157Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:29:31.738130993Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.768207240Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.788560055Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.791742240Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.856219246Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.869572471Z tar: Removing leading `/' from member names
2026-07-09T18:29:31.876814335Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:29:31.889894567Z Wrote inspect data to must-gather.
2026-07-09T18:29:31.992133110Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:29:32.107172918Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:32.147768570Z Wrote inspect data to must-gather.
2026-07-09T18:29:32.275299899Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:29:32.314110137Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:29:32.383426455Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:32.386805833Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:29:32.427399986Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:29:32.427449674Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:29:32.427916500Z INFO: Copying ovnk databases complete.
2026-07-09T18:29:32.429780300Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:29:32.432091746Z ovnk_database_store/
2026-07-09T18:29:32.432091746Z ovnk_database_store/ovnkube-node-txgxp_nbdb
2026-07-09T18:29:32.443130374Z ovnk_database_store/ovnkube-node-b547z_nbdb
2026-07-09T18:29:32.455764811Z ovnk_database_store/ovnkube-node-8xxd4_sbdb
2026-07-09T18:29:32.482496854Z ovnk_database_store/ovnkube-node-8xxd4_nbdb
2026-07-09T18:29:32.493610080Z ovnk_database_store/ovnkube-node-b547z_sbdb
2026-07-09T18:29:32.517756763Z Gathering data for ns/openshift-network-console...
2026-07-09T18:29:32.520259178Z ovnk_database_store/ovnkube-node-txgxp_sbdb
2026-07-09T18:29:32.521965984Z Wrote inspect data to must-gather.
2026-07-09T18:29:32.548384264Z INFO: Network log collection complete.
2026-07-09T18:29:32.626631592Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:32.729242242Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:29:32.851354132Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:32.855072953Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:29:32.866402793Z Wrote inspect data to must-gather.
2026-07-09T18:29:33.059324111Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:29:33.073203723Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:33.195202843Z Wrote inspect data to must-gather.
2026-07-09T18:29:33.290532636Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:33.293441053Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:29:33.463832034Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:29:33.466183195Z INFO: Fallback to identify the container image from release info
2026-07-09T18:29:33.511259098Z Gathering data for ns/openshift-console...
2026-07-09T18:29:33.555395879Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:33.763300343Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:33.765649625Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T18:29:33.948839172Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:34.007120890Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:29:34.123573074Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:29:34.167216841Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:29:34.176441534Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:34.179780827Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T18:29:34.321276862Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:29:34.373103834Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:34.577069881Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:34.579638057Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T18:29:34.712304206Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:34.734426642Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:29:34.800796001Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.000559703Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.004212441Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:29:35.080414073Z Gathering data for ns/openshift-dns...
2026-07-09T18:29:35.176194121Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.364637022Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.368966016Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:29:35.473027213Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:29:35.545535970Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.750245321Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:35.753270224Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:29:35.772228571Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:29:35.923004832Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:36.120880441Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:36.303316797Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:29:36.624715717Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:29:36.718256522Z Gathering data for ns/openshift-ingress...
2026-07-09T18:29:36.850762098Z Done executing Istio gather script
2026-07-09T18:29:37.066122185Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:29:37.470951273Z Gathering data for ns/openshift-insights...
2026-07-09T18:29:37.737229803Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:29:38.854018273Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:29:39.715490800Z Gathering data for ns/kserve...
2026-07-09T18:29:39.716951047Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:39.993383202Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:29:41.124135195Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:29:41.361601356Z Gathering data for ns/openshift-config...
2026-07-09T18:29:41.624598746Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:29:42.074698870Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:29:42.240603364Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:29:42.346087908Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:29:42.548477480Z Collecting performance related data for node ip-10-0-136-100.ec2.internal
2026-07-09T18:29:42.555511250Z Collecting performance related data for node ip-10-0-141-169.ec2.internal
2026-07-09T18:29:42.564987294Z Collecting performance related data for node ip-10-0-133-63.ec2.internal
2026-07-09T18:29:42.668839650Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:29:43.012864660Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:29:43.367012506Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:29:43.715537590Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:29:44.103668381Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:29:44.421901966Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:29:44.722198634Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:44.756632610Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:29:45.382408797Z Gathering data for ns/openshift-multus...
2026-07-09T18:29:47.031628417Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:29:48.367788942Z Gathering data for ns/openshift-host-network...
2026-07-09T18:29:48.718730903Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:29:49.243503917Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:29:49.653313214Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:29:49.726949234Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:50.047762464Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:29:50.338378095Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:29:50.846033488Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:29:50.907508950Z Collecting kubelet logs for node ip-10-0-136-100.ec2.internal
2026-07-09T18:29:50.909511306Z Collecting kubelet logs for node ip-10-0-141-169.ec2.internal
2026-07-09T18:29:50.912409562Z Collecting kubelet logs for node ip-10-0-133-63.ec2.internal
2026-07-09T18:29:51.192276305Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:29:51.304072379Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-q7djf namespace
2026-07-09T18:29:51.308339257Z INFO: Node performance data collection complete.
2026-07-09T18:29:51.603557632Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:29:51.900785738Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:29:52.207245425Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:29:52.830271747Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:29:53.115757841Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:29:53.429775580Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:29:53.789957880Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:29:54.708867746Z Wrote inspect data to must-gather.
2026-07-09T18:29:54.708912349Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:29:54.708912349Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:29:54.731619693Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:54.838183132Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:29:54.973083452Z Caches written to disk