2026-07-13T19:40:47.675516258Z [disk usage checker] Started
2026-07-13T19:40:47.678986885Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:40:47.883033493Z Gathering data for ns/openshift-cluster-version...
2026-07-13T19:40:48.033824809Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T19:40:48.131147569Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T19:40:48.216372092Z Gathering data for ns/default...
2026-07-13T19:40:48.517944295Z Gathering data for ns/openshift...
2026-07-13T19:40:48.963848325Z Gathering data for ns/kube-system...
2026-07-13T19:40:49.662252920Z Gathering data for ns/openshift-etcd...
2026-07-13T19:40:49.977018242Z Wrote inspect data to must-gather.
2026-07-13T19:40:49.977050803Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T19:40:49.977050803Z     namespaces "assisted-installer" not found
2026-07-13T19:40:50.507125353Z Waiting on subprocesses to finish execution.
2026-07-13T19:40:50.512681873Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-13T19:40:50.513979351Z INFO: Gathering HAProxy config files
2026-07-13T19:40:50.514522577Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-13T19:40:50.517011792Z INFO: Collecting host service logs for crio
2026-07-13T19:40:50.517375172Z INFO: Collecting host service logs for kubelet
2026-07-13T19:40:50.517710359Z INFO: Collecting host service logs for rpm-ostreed
2026-07-13T19:40:50.518032612Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-13T19:40:50.518391370Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-13T19:40:50.518715879Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-13T19:40:50.519050309Z INFO: Collecting host service logs for NetworkManager
2026-07-13T19:40:50.519397999Z INFO: Collecting host service logs for openvswitch
2026-07-13T19:40:50.519731096Z INFO: Collecting host service logs for ovs-configuration
2026-07-13T19:40:50.520057738Z INFO: Collecting host service logs for ovsdb-server
2026-07-13T19:40:50.520408029Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-13T19:40:50.520805295Z INFO: Waiting for worker host service log collection to complete ...
2026-07-13T19:40:50.524730455Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-13T19:40:50.538433487Z Executing Istio gather script
2026-07-13T19:40:50.569118326Z INFO: Waiting for node performance related collection to complete ...
2026-07-13T19:40:51.233236273Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-13T19:40:51.403437695Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-13T19:40:51.509541836Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-13T19:40:51.608615363Z INFO: "metallb-operator" not detected. Skipping.
2026-07-13T19:40:51.630356676Z No resources found
2026-07-13T19:40:51.779101308Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-13T19:40:51.825544365Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-8w9l7
2026-07-13T19:40:51.825544365Z insights-runtime-extractor-6wfw4
2026-07-13T19:40:51.825544365Z insights-runtime-extractor-dp9pw
2026-07-13T19:40:51.825544365Z insights-runtime-extractor-x8zfs
2026-07-13T19:40:51.830442203Z error: the server doesn't have a resource type "performanceprofile"
2026-07-13T19:40:51.873450084Z Wrote inspect data to must-gather.
2026-07-13T19:40:52.130035443Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-13T19:40:52.130114892Z INFO: on-disk MachineConfig config collection complete.
2026-07-13T19:40:52.424476503Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-13T19:40:52.512147238Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-13T19:40:52.512147238Z See 'oc rsync -h' for help and examples
2026-07-13T19:40:52.577158651Z Wrote inspect data to must-gather.
2026-07-13T19:40:52.577737026Z INFO: Worker host service log collection to complete.
2026-07-13T19:40:52.618915080Z Wrote inspect data to must-gather.
2026-07-13T19:40:52.623682230Z No resources found
2026-07-13T19:40:52.774151156Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:40:52.809358081Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T19:40:52.846892860Z No resources found in openshift-etcd namespace.
2026-07-13T19:40:52.979152263Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.049918955Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.090464616Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.138555839Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-13T19:40:53.193725185Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-13T19:40:53.274072795Z error: the server doesn't have a resource type "machineconfigs"
2026-07-13T19:40:53.311872999Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.361927319Z ERROR: No running kube-apiserver pods found
2026-07-13T19:40:53.455167988Z INFO: 'previous-logs' folder not found on ip-10-0-128-27.ec2.internal, skipping...
2026-07-13T19:40:53.499365041Z tar: Removing leading `/' from member names
2026-07-13T19:40:53.506748560Z INFO: HAProxy config collection complete.
2026-07-13T19:40:53.556937113Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-13T19:40:53.582588654Z Gathering data for ns/openshift-monitoring...
2026-07-13T19:40:53.606161397Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.759716041Z INFO: Getting rules from prometheus-k8s-0
2026-07-13T19:40:53.760950881Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-13T19:40:53.791563840Z INFO: 'previous-logs' folder not found on ip-10-0-137-93.ec2.internal, skipping...
2026-07-13T19:40:53.850121992Z Wrote inspect data to must-gather.
2026-07-13T19:40:53.935112206Z Wrote inspect data to must-gather.
2026-07-13T19:40:54.100070524Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-13T19:40:54.134841934Z INFO: 'previous-logs' folder not found on ip-10-0-139-100.ec2.internal, skipping...
2026-07-13T19:40:54.134972276Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-13T19:40:54.134995197Z INFO: Machine Config Daemon termination log collection complete.
2026-07-13T19:40:54.249339517Z Wrote inspect data to must-gather.
2026-07-13T19:40:54.262110562Z INFO: Getting status/config from prometheus-k8s-0
2026-07-13T19:40:54.358892915Z Wrote inspect data to must-gather.
2026-07-13T19:40:54.583750774Z INFO: INTERCONNECT MODE
2026-07-13T19:40:54.583784618Z INFO: Gathering ovn-kubernetes DBs
2026-07-13T19:40:54.588584417Z Wrote inspect data to must-gather.
2026-07-13T19:40:54.729072421Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-13T19:40:54.809758368Z Wrote inspect data to must-gather.
2026-07-13T19:40:54.833032101Z INFO: Gathering OVN_Northbound from ovnkube-node-7fsmf...
2026-07-13T19:40:54.833254766Z INFO: Gathering OVN_Northbound from ovnkube-node-k9k97...
2026-07-13T19:40:54.833447510Z INFO: Gathering OVN_Northbound from ovnkube-node-wv7rg...
2026-07-13T19:40:54.833681761Z INFO: Gathering OVN_Southbound from ovnkube-node-7fsmf...
2026-07-13T19:40:54.833930014Z INFO: Gathering OVN_Southbound from ovnkube-node-k9k97...
2026-07-13T19:40:54.834209697Z INFO: Gathering OVN_Southbound from ovnkube-node-wv7rg...
2026-07-13T19:40:55.059089946Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-13T19:40:55.158305365Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.161513267Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.183632420Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.185651557Z Wrote inspect data to must-gather.
2026-07-13T19:40:55.195909069Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.201028238Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.215546345Z tar: Removing leading `/' from member names
2026-07-13T19:40:55.377546333Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-13T19:40:55.385817449Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T19:40:55.389477903Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-13T19:40:55.467817130Z Wrote inspect data to must-gather.
2026-07-13T19:40:55.610195508Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:55.680736518Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-13T19:40:55.739742447Z Wrote inspect data to must-gather.
2026-07-13T19:40:55.757431951Z INFO: Waiting for network log collection to complete ...
2026-07-13T19:40:55.757483015Z INFO: Waiting for ovnk database copies to complete ...
2026-07-13T19:40:55.757948243Z INFO: Copying ovnk databases complete.
2026-07-13T19:40:55.760005530Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-13T19:40:55.763235311Z ovnk_database_store/
2026-07-13T19:40:55.763306278Z ovnk_database_store/ovnkube-node-7fsmf_nbdb
2026-07-13T19:40:55.778620745Z ovnk_database_store/ovnkube-node-wv7rg_sbdb
2026-07-13T19:40:55.806660067Z ovnk_database_store/ovnkube-node-k9k97_sbdb
2026-07-13T19:40:55.826617841Z ovnk_database_store/ovnkube-node-wv7rg_nbdb
2026-07-13T19:40:55.836054042Z ovnk_database_store/ovnkube-node-7fsmf_sbdb
2026-07-13T19:40:55.864383954Z ovnk_database_store/ovnkube-node-k9k97_nbdb
2026-07-13T19:40:55.865296886Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:55.867656839Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-13T19:40:55.878684802Z INFO: Network log collection complete.
2026-07-13T19:40:55.978153357Z W0713 19:40:55.978108    1491 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-13T19:40:55.984687419Z Wrote inspect data to must-gather.
2026-07-13T19:40:56.078282135Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T19:40:56.094886025Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:56.148357088Z INFO: Getting status from alertmanager-main-0
2026-07-13T19:40:56.295489838Z Gathering data for ns/openshift-network-console...
2026-07-13T19:40:56.310638560Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-13T19:40:56.313133752Z INFO: Fallback to identify the container image from release info
2026-07-13T19:40:56.359179117Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:56.365001105Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-13T19:40:56.570904838Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:56.746966323Z Gathering data for ns/openshift-console-operator...
2026-07-13T19:40:56.780384587Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:56.782555958Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-13T19:40:56.938645618Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-13T19:40:57.024585882Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:57.087393163Z daemonset.apps/perf-node-gather-daemonset created
2026-07-13T19:40:57.176777368Z Gathering data for ns/openshift-console...
2026-07-13T19:40:57.241537229Z Waiting for performance profile collector pods to become ready: 1
2026-07-13T19:40:57.249904399Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:57.252318097Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-13T19:40:57.425735273Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:57.659921921Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:57.662348971Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-13T19:40:57.734087224Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-13T19:40:57.820666869Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:40:57.848720103Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.044042825Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.047201804Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-13T19:40:58.238557344Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.287066329Z Gathering data for ns/openshift-dns-operator...
2026-07-13T19:40:58.401852373Z Waiting for performance profile collector pods to become ready: 2
2026-07-13T19:40:58.458892220Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.462720344Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-13T19:40:58.617133894Z Gathering data for ns/openshift-dns...
2026-07-13T19:40:58.643804540Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.818844289Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:58.821273479Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-13T19:40:58.993611624Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:59.198280409Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:59.201545998Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-13T19:40:59.250436315Z Gathering data for ns/openshift-image-registry...
2026-07-13T19:40:59.368313531Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:59.545988769Z Waiting for performance profile collector pods to become ready: 3
2026-07-13T19:40:59.586228593Z Wrote inspect data to /must-gather/istio.
2026-07-13T19:40:59.754444886Z Gathering data for ns/openshift-ingress-operator...
2026-07-13T19:41:00.137120049Z Gathering data for ns/openshift-ingress...
2026-07-13T19:41:00.312880012Z Done executing Istio gather script
2026-07-13T19:41:00.471071448Z Gathering data for ns/openshift-ingress-canary...
2026-07-13T19:41:00.676330154Z Waiting for performance profile collector pods to become ready: 4
2026-07-13T19:41:00.887231142Z Gathering data for ns/openshift-insights...
2026-07-13T19:41:01.789806408Z Waiting for performance profile collector pods to become ready: 5
2026-07-13T19:41:02.826054069Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:41:02.909522241Z Waiting for performance profile collector pods to become ready: 6
2026-07-13T19:41:03.246741453Z Gathering data for ns/kserve...
2026-07-13T19:41:04.034354563Z Waiting for performance profile collector pods to become ready: 7
2026-07-13T19:41:04.794513825Z Gathering data for ns/openshift-config...
2026-07-13T19:41:05.064994352Z Gathering data for ns/openshift-config-managed...
2026-07-13T19:41:05.174795606Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-13T19:41:05.424883235Z Collecting performance related data for node ip-10-0-128-27.ec2.internal
2026-07-13T19:41:05.437878728Z Collecting performance related data for node ip-10-0-139-100.ec2.internal
2026-07-13T19:41:05.445715446Z Collecting performance related data for node ip-10-0-137-93.ec2.internal
2026-07-13T19:41:05.609665392Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-13T19:41:05.918260786Z Gathering data for ns/openshift-kube-apiserver...
2026-07-13T19:41:06.386331602Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-13T19:41:06.732568904Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-13T19:41:07.115575369Z Gathering data for ns/openshift-kube-scheduler...
2026-07-13T19:41:07.388078111Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-13T19:41:07.664768658Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-13T19:41:07.830912352Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:41:08.026111698Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-13T19:41:08.371331941Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-13T19:41:08.907289799Z Gathering data for ns/openshift-multus...
2026-07-13T19:41:10.299012739Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-13T19:41:11.567646166Z Gathering data for ns/openshift-host-network...
2026-07-13T19:41:11.872460449Z Gathering data for ns/openshift-network-diagnostics...
2026-07-13T19:41:12.429720119Z Gathering data for ns/openshift-network-node-identity...
2026-07-13T19:41:12.838030909Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:41:12.856056348Z Gathering data for ns/openshift-network-operator...
2026-07-13T19:41:13.235744909Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-13T19:41:13.502734679Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-13T19:41:13.986832606Z Gathering data for ns/openshift-apiserver-operator...
2026-07-13T19:41:13.993006389Z Collecting kubelet logs for node ip-10-0-128-27.ec2.internal
2026-07-13T19:41:13.995101285Z Collecting kubelet logs for node ip-10-0-139-100.ec2.internal
2026-07-13T19:41:13.997329972Z Collecting kubelet logs for node ip-10-0-137-93.ec2.internal
2026-07-13T19:41:14.356161873Z Gathering data for ns/openshift-apiserver...
2026-07-13T19:41:14.380357685Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qwzz7 namespace
2026-07-13T19:41:14.386521235Z INFO: Node performance data collection complete.
2026-07-13T19:41:14.797681410Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-13T19:41:15.056465135Z Gathering data for ns/openshift-controller-manager...
2026-07-13T19:41:15.395731211Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-13T19:41:16.031371321Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-13T19:41:16.366762230Z Gathering data for ns/openshift-service-ca-operator...
2026-07-13T19:41:16.675849628Z Gathering data for ns/openshift-service-ca...
2026-07-13T19:41:17.049513476Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-13T19:41:17.842979204Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-13T19:41:17.945694539Z Wrote inspect data to must-gather.
2026-07-13T19:41:17.945735526Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T19:41:17.945735526Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-13T19:41:18.084432829Z error: the server doesn't have a resource type "clusters"
2026-07-13T19:41:18.199722827Z Caches written to disk