2026-09-15T09:45:48.053273685Z [disk usage checker] Started
2026-09-15T09:45:48.056360280Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:45:48.243989144Z Gathering data for ns/openshift-cluster-version...
2026-09-15T09:45:48.370260929Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-15T09:45:48.445866314Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T09:45:48.522929808Z Gathering data for ns/default...
2026-09-15T09:45:48.789637986Z Gathering data for ns/openshift...
2026-09-15T09:45:49.193805334Z Gathering data for ns/kube-system...
2026-09-15T09:45:49.832741083Z Gathering data for ns/openshift-etcd...
2026-09-15T09:45:50.116458675Z Wrote inspect data to must-gather.
2026-09-15T09:45:50.116486459Z error: inspection completed with the errors occurred while gathering data:
2026-09-15T09:45:50.116486459Z     namespaces "assisted-installer" not found
2026-09-15T09:45:50.588038450Z Waiting on subprocesses to finish execution.
2026-09-15T09:45:50.594676304Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-15T09:45:50.596771215Z Executing Istio gather script
2026-09-15T09:45:50.607538180Z INFO: Gathering HAProxy config files
2026-09-15T09:45:50.609253396Z INFO: Collecting host service logs for crio
2026-09-15T09:45:50.609722853Z INFO: Collecting host service logs for kubelet
2026-09-15T09:45:50.610151145Z INFO: Collecting host service logs for rpm-ostreed
2026-09-15T09:45:50.610556181Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-15T09:45:50.610885205Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-15T09:45:50.611200331Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-15T09:45:50.611523783Z INFO: Collecting host service logs for NetworkManager
2026-09-15T09:45:50.611865443Z INFO: Collecting host service logs for openvswitch
2026-09-15T09:45:50.612206938Z INFO: Collecting host service logs for ovs-configuration
2026-09-15T09:45:50.612573911Z INFO: Collecting host service logs for ovsdb-server
2026-09-15T09:45:50.612915624Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-15T09:45:50.613295678Z INFO: Waiting for worker host service log collection to complete ...
2026-09-15T09:45:50.619204199Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-15T09:45:50.659978705Z INFO: Waiting for node performance related collection to complete ...
2026-09-15T09:45:50.689652318Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-15T09:45:51.033988204Z No resources found
2026-09-15T09:45:51.287193390Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-15T09:45:51.572074229Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-15T09:45:51.613130245Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-15T09:45:51.670439848Z INFO: "metallb-operator" not detected. Skipping.
2026-09-15T09:45:51.720097033Z Wrote inspect data to must-gather.
2026-09-15T09:45:51.964548345Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-15T09:45:51.964631112Z INFO: on-disk MachineConfig config collection complete.
2026-09-15T09:45:51.970238439Z INFO: Collecting Insights Archives from insights-operator-5844b856db-5v5xq
2026-09-15T09:45:51.970277697Z insights-runtime-extractor-g4nxd
2026-09-15T09:45:51.970298000Z insights-runtime-extractor-l87kp
2026-09-15T09:45:51.970330018Z insights-runtime-extractor-tlzjp
2026-09-15T09:45:51.993224749Z No resources found in openshift-etcd namespace.
2026-09-15T09:45:52.105177999Z error: the server doesn't have a resource type "performanceprofile"
2026-09-15T09:45:52.228516651Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-15T09:45:52.356666591Z INFO: Worker host service log collection to complete.
2026-09-15T09:45:52.380763297Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-15T09:45:52.420705332Z Wrote inspect data to must-gather.
2026-09-15T09:45:52.458949119Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-15T09:45:52.458949119Z See 'oc rsync -h' for help and examples
2026-09-15T09:45:52.730514266Z ERROR: No running kube-apiserver pods found
2026-09-15T09:45:52.798731384Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-15T09:45:52.905380817Z Wrote inspect data to must-gather.
2026-09-15T09:45:52.907125171Z No resources found
2026-09-15T09:45:52.934295857Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-15T09:45:52.978119329Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.026023611Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.060783264Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T09:45:53.082006794Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:45:53.086137674Z INFO: 'previous-logs' folder not found on ip-10-0-132-158.ec2.internal, skipping...
2026-09-15T09:45:53.141165174Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.225566916Z tar: Removing leading `/' from member names
2026-09-15T09:45:53.233294452Z INFO: HAProxy config collection complete.
2026-09-15T09:45:53.286107750Z error: the server doesn't have a resource type "machineconfigs"
2026-09-15T09:45:53.302847521Z Gathering data for ns/openshift-monitoring...
2026-09-15T09:45:53.309034786Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.386485898Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-15T09:45:53.417009145Z INFO: 'previous-logs' folder not found on ip-10-0-136-33.ec2.internal, skipping...
2026-09-15T09:45:53.473461404Z INFO: Getting rules from prometheus-k8s-0
2026-09-15T09:45:53.479586087Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-15T09:45:53.598438770Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.692221306Z INFO: 'previous-logs' folder not found on ip-10-0-142-220.ec2.internal, skipping...
2026-09-15T09:45:53.692259186Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-15T09:45:53.692259186Z INFO: Machine Config Daemon termination log collection complete.
2026-09-15T09:45:53.700624438Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.847021388Z Wrote inspect data to must-gather.
2026-09-15T09:45:53.855768682Z INFO: Getting status/config from prometheus-k8s-0
2026-09-15T09:45:53.888241970Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-15T09:45:54.066624021Z Wrote inspect data to must-gather.
2026-09-15T09:45:54.088390335Z Wrote inspect data to must-gather.
2026-09-15T09:45:54.236549942Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-15T09:45:54.242008062Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-15T09:45:54.310891122Z Wrote inspect data to must-gather.
2026-09-15T09:45:54.367492410Z INFO: INTERCONNECT MODE
2026-09-15T09:45:54.367524918Z INFO: Gathering ovn-kubernetes DBs
2026-09-15T09:45:54.429985670Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:54.467052177Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-15T09:45:54.519104714Z Wrote inspect data to must-gather.
2026-09-15T09:45:54.521495560Z INFO: Gathering OVN_Northbound from ovnkube-node-96n7w...
2026-09-15T09:45:54.521740375Z INFO: Gathering OVN_Northbound from ovnkube-node-jshph...
2026-09-15T09:45:54.522020263Z INFO: Gathering OVN_Northbound from ovnkube-node-kzqrf...
2026-09-15T09:45:54.522270585Z INFO: Gathering OVN_Southbound from ovnkube-node-96n7w...
2026-09-15T09:45:54.522554369Z INFO: Gathering OVN_Southbound from ovnkube-node-jshph...
2026-09-15T09:45:54.522842603Z INFO: Gathering OVN_Southbound from ovnkube-node-kzqrf...
2026-09-15T09:45:54.786390115Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:54.789784852Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-15T09:45:54.841901782Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-15T09:45:54.853694866Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.865648344Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.879270373Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.886100757Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.894140023Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.925010946Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-15T09:45:54.933427204Z tar: Removing leading `/' from member names
2026-09-15T09:45:54.956148380Z Wrote inspect data to must-gather.
2026-09-15T09:45:55.037149200Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:55.111438105Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-15T09:45:55.210935961Z Wrote inspect data to must-gather.
2026-09-15T09:45:55.279447766Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:55.281704636Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-15T09:45:55.388753850Z INFO: Waiting for network log collection to complete ...
2026-09-15T09:45:55.388810306Z INFO: Waiting for ovnk database copies to complete ...
2026-09-15T09:45:55.389320755Z INFO: Copying ovnk databases complete.
2026-09-15T09:45:55.391781047Z 3.6M	must-gather/network_logs/ovnk_database_store
2026-09-15T09:45:55.394175697Z ovnk_database_store/
2026-09-15T09:45:55.394228304Z ovnk_database_store/ovnkube-node-96n7w_nbdb
2026-09-15T09:45:55.406221347Z ovnk_database_store/ovnkube-node-jshph_nbdb
2026-09-15T09:45:55.416085464Z ovnk_database_store/ovnkube-node-jshph_sbdb
2026-09-15T09:45:55.440185062Z ovnk_database_store/ovnkube-node-kzqrf_sbdb
2026-09-15T09:45:55.440532842Z W0915 09:45:55.440502    1535 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-15T09:45:55.447594968Z Wrote inspect data to must-gather.
2026-09-15T09:45:55.464753955Z ovnk_database_store/ovnkube-node-kzqrf_nbdb
2026-09-15T09:45:55.473531637Z ovnk_database_store/ovnkube-node-96n7w_sbdb
2026-09-15T09:45:55.483318084Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:55.498990187Z INFO: Network log collection complete.
2026-09-15T09:45:55.532542737Z INFO: Getting status from alertmanager-main-0
2026-09-15T09:45:55.579677618Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T09:45:55.714100373Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:55.716987704Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-15T09:45:55.754699107Z Wrote inspect data to must-gather.
2026-09-15T09:45:55.771574789Z Gathering data for ns/openshift-network-console...
2026-09-15T09:45:55.941155471Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:56.013037268Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-15T09:45:56.015216984Z INFO: Fallback to identify the container image from release info
2026-09-15T09:45:56.136279543Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:56.139284589Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-15T09:45:56.187218034Z Gathering data for ns/openshift-console-operator...
2026-09-15T09:45:56.299925517Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:56.472571345Z Wrote inspect data to /must-gather/istio.
2026-09-15T09:45:56.559567297Z Gathering data for ns/openshift-console...
2026-09-15T09:45:57.054589895Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-15T09:45:57.111266497Z Done executing Istio gather script
2026-09-15T09:45:57.573212817Z Gathering data for ns/openshift-dns-operator...
2026-09-15T09:45:57.659253254Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-15T09:45:57.778945147Z daemonset.apps/perf-node-gather-daemonset created
2026-09-15T09:45:57.875730345Z Gathering data for ns/openshift-dns...
2026-09-15T09:45:57.899205861Z Waiting for performance profile collector pods to become ready: 1
2026-09-15T09:45:58.100145870Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:45:58.430970631Z Gathering data for ns/openshift-image-registry...
2026-09-15T09:45:58.864201584Z Gathering data for ns/openshift-ingress-operator...
2026-09-15T09:45:59.009823492Z Waiting for performance profile collector pods to become ready: 2
2026-09-15T09:45:59.162331581Z Gathering data for ns/openshift-ingress...
2026-09-15T09:45:59.463945074Z Gathering data for ns/openshift-ingress-canary...
2026-09-15T09:45:59.804203233Z Gathering data for ns/openshift-insights...
2026-09-15T09:46:00.125266524Z Waiting for performance profile collector pods to become ready: 3
2026-09-15T09:46:01.245733646Z Waiting for performance profile collector pods to become ready: 4
2026-09-15T09:46:01.754145579Z Gathering data for ns/openshift-config...
2026-09-15T09:46:01.979488655Z Gathering data for ns/openshift-config-managed...
2026-09-15T09:46:02.362628523Z Waiting for performance profile collector pods to become ready: 5
2026-09-15T09:46:02.378074397Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-15T09:46:02.611875486Z Gathering data for ns/openshift-kube-apiserver...
2026-09-15T09:46:02.846746519Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-15T09:46:03.099784440Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-15T09:46:03.104728202Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:46:03.351049018Z Gathering data for ns/openshift-kube-scheduler...
2026-09-15T09:46:03.479017131Z Waiting for performance profile collector pods to become ready: 6
2026-09-15T09:46:03.590089634Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-15T09:46:03.847709045Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-15T09:46:04.143609719Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-15T09:46:04.411687078Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-15T09:46:04.587602201Z Waiting for performance profile collector pods to become ready: 7
2026-09-15T09:46:04.867766359Z Gathering data for ns/openshift-multus...
2026-09-15T09:46:05.688437979Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-15T09:46:05.939062346Z Collecting performance related data for node ip-10-0-142-220.ec2.internal
2026-09-15T09:46:05.945899648Z Collecting performance related data for node ip-10-0-136-33.ec2.internal
2026-09-15T09:46:05.954996599Z Collecting performance related data for node ip-10-0-132-158.ec2.internal
2026-09-15T09:46:06.190445616Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-15T09:46:07.484203951Z Gathering data for ns/openshift-host-network...
2026-09-15T09:46:07.748453861Z Gathering data for ns/openshift-network-diagnostics...
2026-09-15T09:46:08.111058942Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:46:08.254079330Z Gathering data for ns/openshift-network-node-identity...
2026-09-15T09:46:08.663029015Z Gathering data for ns/openshift-network-operator...
2026-09-15T09:46:09.044487007Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-15T09:46:09.278181778Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-15T09:46:09.722936967Z Gathering data for ns/openshift-apiserver-operator...
2026-09-15T09:46:09.999852405Z Gathering data for ns/openshift-apiserver...
2026-09-15T09:46:10.365283765Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-15T09:46:10.601944814Z Gathering data for ns/openshift-controller-manager...
2026-09-15T09:46:10.848969235Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-15T09:46:11.476238950Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-15T09:46:11.742755233Z Gathering data for ns/openshift-service-ca-operator...
2026-09-15T09:46:12.018077437Z Gathering data for ns/openshift-service-ca...
2026-09-15T09:46:12.334432356Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-15T09:46:13.118521979Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-09-15T09:46:13.169106319Z Wrote inspect data to must-gather.
2026-09-15T09:46:13.169150352Z error: inspection completed with the errors occurred while gathering data:
2026-09-15T09:46:13.169150352Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-15T09:46:13.912845684Z Collecting kubelet logs for node ip-10-0-142-220.ec2.internal
2026-09-15T09:46:13.914557965Z Collecting kubelet logs for node ip-10-0-136-33.ec2.internal
2026-09-15T09:46:13.916137233Z Collecting kubelet logs for node ip-10-0-132-158.ec2.internal
2026-09-15T09:46:14.226616680Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mxgxj namespace
2026-09-15T09:46:14.231725461Z INFO: Node performance data collection complete.
2026-09-15T09:46:14.350854611Z error: the server doesn't have a resource type "clusters"
2026-09-15T09:46:14.481866488Z Caches written to disk