2026-07-09T10:23:22.948018874Z [disk usage checker] Started
2026-07-09T10:23:22.951327385Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:23.152796430Z Gathering data for ns/openshift-cluster-version...
2026-07-09T10:23:23.303055259Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T10:23:23.404187876Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T10:23:23.498612216Z Gathering data for ns/default...
2026-07-09T10:23:23.836980624Z Gathering data for ns/openshift...
2026-07-09T10:23:24.279129920Z Gathering data for ns/kube-system...
2026-07-09T10:23:25.062401504Z Gathering data for ns/openshift-etcd...
2026-07-09T10:23:25.399263490Z Wrote inspect data to must-gather.
2026-07-09T10:23:25.399296966Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T10:23:25.399296966Z     namespaces "assisted-installer" not found
2026-07-09T10:23:25.723238008Z Waiting on subprocesses to finish execution.
2026-07-09T10:23:25.732343108Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T10:23:25.733434698Z Executing Istio gather script
2026-07-09T10:23:25.735046313Z INFO: Gathering HAProxy config files
2026-07-09T10:23:25.737495226Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T10:23:25.737543640Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T10:23:25.771942474Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T10:23:25.775067625Z INFO: Collecting host service logs for crio
2026-07-09T10:23:25.775427500Z INFO: Collecting host service logs for kubelet
2026-07-09T10:23:25.775714959Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T10:23:25.776001843Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T10:23:25.776411431Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T10:23:25.776750935Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T10:23:25.777034836Z INFO: Collecting host service logs for NetworkManager
2026-07-09T10:23:25.777346498Z INFO: Collecting host service logs for openvswitch
2026-07-09T10:23:25.777636600Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T10:23:25.777939838Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T10:23:25.778911954Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T10:23:25.779294605Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T10:23:26.272097537Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T10:23:26.272097537Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T10:23:26.619570203Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T10:23:26.721376446Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T10:23:26.895787294Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-jxt6q
2026-07-09T10:23:26.895787294Z insights-runtime-extractor-456st
2026-07-09T10:23:26.895787294Z insights-runtime-extractor-glzrj
2026-07-09T10:23:26.895787294Z insights-runtime-extractor-hk2kv
2026-07-09T10:23:26.936585477Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T10:23:26.986209013Z Wrote inspect data to must-gather.
2026-07-09T10:23:27.164193223Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T10:23:27.230716057Z No resources found in openshift-etcd namespace.
2026-07-09T10:23:27.243299201Z ERROR: No running kube-apiserver pods found
2026-07-09T10:23:27.440444262Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T10:23:27.440444262Z See 'oc rsync -h' for help and examples
2026-07-09T10:23:27.660408104Z Gathering data for ns/openshift-monitoring...
2026-07-09T10:23:27.668925162Z Wrote inspect data to must-gather.
2026-07-09T10:23:27.782056655Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T10:23:27.825642048Z No resources found
2026-07-09T10:23:27.850833169Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T10:23:27.995904688Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.002205335Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.063064781Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.151900056Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T10:23:28.180284558Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:28.182025808Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T10:23:28.187868631Z INFO: 'previous-logs' folder not found on ip-10-0-135-204.ec2.internal, skipping...
2026-07-09T10:23:28.260691744Z INFO: Worker host service log collection to complete.
2026-07-09T10:23:28.277828098Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T10:23:28.389485282Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.444727541Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.551188673Z No resources found
2026-07-09T10:23:28.588838712Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T10:23:28.654764574Z tar: Removing leading `/' from member names
2026-07-09T10:23:28.664759700Z INFO: HAProxy config collection complete.
2026-07-09T10:23:28.679471077Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T10:23:28.723429061Z Wrote inspect data to must-gather.
2026-07-09T10:23:28.771996904Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T10:23:28.799437747Z INFO: 'previous-logs' folder not found on ip-10-0-141-211.ec2.internal, skipping...
2026-07-09T10:23:28.868097267Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T10:23:28.952437566Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T10:23:29.023942598Z Wrote inspect data to must-gather.
2026-07-09T10:23:29.073548496Z INFO: 'previous-logs' folder not found on ip-10-0-143-234.ec2.internal, skipping...
2026-07-09T10:23:29.073576632Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T10:23:29.073594428Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T10:23:29.145238900Z Wrote inspect data to must-gather.
2026-07-09T10:23:29.270790877Z Wrote inspect data to must-gather.
2026-07-09T10:23:29.271925926Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T10:23:29.307705188Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T10:23:29.552333333Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T10:23:29.561983844Z Wrote inspect data to must-gather.
2026-07-09T10:23:29.596138578Z INFO: INTERCONNECT MODE
2026-07-09T10:23:29.596233691Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T10:23:29.758218250Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T10:23:29.787501345Z INFO: Gathering OVN_Northbound from ovnkube-node-9479k...
2026-07-09T10:23:29.787765119Z INFO: Gathering OVN_Northbound from ovnkube-node-xdmzb...
2026-07-09T10:23:29.788008138Z INFO: Gathering OVN_Northbound from ovnkube-node-xf4sw...
2026-07-09T10:23:29.788303525Z INFO: Gathering OVN_Southbound from ovnkube-node-9479k...
2026-07-09T10:23:29.788548870Z INFO: Gathering OVN_Southbound from ovnkube-node-xdmzb...
2026-07-09T10:23:29.788768558Z INFO: Gathering OVN_Southbound from ovnkube-node-xf4sw...
2026-07-09T10:23:29.837752945Z Wrote inspect data to must-gather.
2026-07-09T10:23:30.112442200Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.115202146Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.120464084Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.146571433Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.163696846Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T10:23:30.209509890Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.223363503Z tar: Removing leading `/' from member names
2026-07-09T10:23:30.269893874Z W0709 10:23:30.269829    1341 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-09T10:23:30.276877147Z Wrote inspect data to must-gather.
2026-07-09T10:23:30.399608651Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T10:23:30.438235955Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T10:23:30.646312209Z Gathering data for ns/openshift-network-console...
2026-07-09T10:23:30.652771130Z Wrote inspect data to must-gather.
2026-07-09T10:23:30.708514305Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T10:23:30.726833178Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T10:23:30.763644593Z INFO: Waiting for network log collection to complete ...
2026-07-09T10:23:30.763809291Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T10:23:30.764400900Z INFO: Copying ovnk databases complete.
2026-07-09T10:23:30.767270332Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-07-09T10:23:30.770408209Z ovnk_database_store/
2026-07-09T10:23:30.770472051Z ovnk_database_store/ovnkube-node-xf4sw_nbdb
2026-07-09T10:23:30.786881212Z ovnk_database_store/ovnkube-node-xdmzb_nbdb
2026-07-09T10:23:30.802026185Z ovnk_database_store/ovnkube-node-9479k_nbdb
2026-07-09T10:23:30.815384687Z ovnk_database_store/ovnkube-node-9479k_sbdb
2026-07-09T10:23:30.841304981Z ovnk_database_store/ovnkube-node-xdmzb_sbdb
2026-07-09T10:23:30.870807890Z ovnk_database_store/ovnkube-node-xf4sw_sbdb
2026-07-09T10:23:30.902113521Z INFO: Network log collection complete.
2026-07-09T10:23:30.950876568Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:31.005338005Z Wrote inspect data to must-gather.
2026-07-09T10:23:31.157723349Z Gathering data for ns/openshift-console-operator...
2026-07-09T10:23:31.163500709Z INFO: Getting status from alertmanager-main-0
2026-07-09T10:23:31.188429434Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:31.191805988Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T10:23:31.345654247Z Wrote inspect data to must-gather.
2026-07-09T10:23:31.424190659Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:31.601020283Z Gathering data for ns/openshift-console...
2026-07-09T10:23:31.637101372Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:31.641228896Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T10:23:31.673422817Z Wrote inspect data to must-gather.
2026-07-09T10:23:31.856771100Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:31.949811706Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T10:23:31.952152788Z INFO: Fallback to identify the container image from release info
2026-07-09T10:23:32.072560468Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:32.076925248Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T10:23:32.156050474Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T10:23:32.308022301Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:32.506710949Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:32.509601012Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T10:23:32.649896940Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-09T10:23:32.698921710Z Gathering data for ns/openshift-dns-operator...
2026-07-09T10:23:32.704465641Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:32.807624488Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T10:23:32.930559036Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:32.933106971Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T10:23:32.961221451Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T10:23:33.039652280Z Gathering data for ns/openshift-dns...
2026-07-09T10:23:33.111242426Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:33.190195339Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:33.321529684Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:33.323631281Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T10:23:33.505111239Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:33.709046074Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:33.711982356Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T10:23:33.793737636Z Gathering data for ns/openshift-image-registry...
2026-07-09T10:23:33.879051663Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:34.093995819Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:34.097126550Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T10:23:34.114760847Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T10:23:34.278997914Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:34.417145814Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T10:23:34.486588411Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:34.489878480Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T10:23:34.650973089Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:34.796918710Z Gathering data for ns/openshift-ingress...
2026-07-09T10:23:34.881271081Z Wrote inspect data to /must-gather/istio.
2026-07-09T10:23:35.189700842Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T10:23:35.249662291Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T10:23:35.608955483Z Done executing Istio gather script
2026-07-09T10:23:35.643728343Z Gathering data for ns/openshift-insights...
2026-07-09T10:23:36.375389682Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T10:23:37.499049406Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T10:23:37.970922583Z Gathering data for ns/kserve...
2026-07-09T10:23:38.194703491Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:38.615490662Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T10:23:39.659583869Z Gathering data for ns/openshift-config...
2026-07-09T10:23:39.752991404Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T10:23:39.987829548Z Gathering data for ns/openshift-config-managed...
2026-07-09T10:23:40.425488499Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T10:23:40.691515838Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T10:23:40.865019395Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T10:23:41.034073620Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T10:23:41.146734104Z Collecting performance related data for node ip-10-0-135-204.ec2.internal
2026-07-09T10:23:41.153591376Z Collecting performance related data for node ip-10-0-143-234.ec2.internal
2026-07-09T10:23:41.160817499Z Collecting performance related data for node ip-10-0-141-211.ec2.internal
2026-07-09T10:23:41.383762990Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T10:23:41.760197303Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T10:23:42.107644355Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T10:23:42.494330959Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T10:23:42.861331840Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T10:23:43.170172502Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T10:23:43.203752898Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:43.775792660Z Gathering data for ns/openshift-multus...
2026-07-09T10:23:45.247930926Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T10:23:46.577645596Z Gathering data for ns/openshift-host-network...
2026-07-09T10:23:46.967150413Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T10:23:47.598499234Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T10:23:48.024010435Z Gathering data for ns/openshift-network-operator...
2026-07-09T10:23:48.209631305Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:48.430814492Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T10:23:48.765002274Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T10:23:49.212466664Z Collecting kubelet logs for node ip-10-0-135-204.ec2.internal
2026-07-09T10:23:49.214606809Z Collecting kubelet logs for node ip-10-0-143-234.ec2.internal
2026-07-09T10:23:49.217537552Z Collecting kubelet logs for node ip-10-0-141-211.ec2.internal
2026-07-09T10:23:49.262301374Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T10:23:49.600820280Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-q8qv5 namespace
2026-07-09T10:23:49.606233255Z INFO: Node performance data collection complete.
2026-07-09T10:23:49.638185806Z Gathering data for ns/openshift-apiserver...
2026-07-09T10:23:50.083299979Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T10:23:50.336759430Z Gathering data for ns/openshift-controller-manager...
2026-07-09T10:23:50.603106747Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T10:23:51.322148410Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T10:23:51.627388917Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T10:23:51.969717915Z Gathering data for ns/openshift-service-ca...
2026-07-09T10:23:52.341500838Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T10:23:53.214036092Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-09T10:23:53.260954258Z Wrote inspect data to must-gather.
2026-07-09T10:23:53.260997503Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T10:23:53.260997503Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T10:23:53.393831973Z error: the server doesn't have a resource type "clusters"
2026-07-09T10:23:53.533895008Z Caches written to disk