2026-06-11T02:16:30.280565148Z [disk usage checker] Started
2026-06-11T02:16:30.284359278Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:30.481509563Z Gathering data for ns/openshift-cluster-version...
2026-06-11T02:16:30.606694480Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T02:16:30.681349274Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T02:16:30.756903031Z Gathering data for ns/default...
2026-06-11T02:16:31.009808903Z Gathering data for ns/openshift...
2026-06-11T02:16:31.375078204Z Gathering data for ns/kube-system...
2026-06-11T02:16:31.992241932Z Gathering data for ns/openshift-etcd...
2026-06-11T02:16:32.253219425Z Wrote inspect data to must-gather.
2026-06-11T02:16:32.253309089Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T02:16:32.253309089Z     namespaces "assisted-installer" not found
2026-06-11T02:16:32.832962252Z Waiting on subprocesses to finish execution.
2026-06-11T02:16:32.837681839Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-11T02:16:32.842195325Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-11T02:16:32.845900269Z Executing Istio gather script
2026-06-11T02:16:32.846678134Z INFO: Gathering HAProxy config files
2026-06-11T02:16:32.871594797Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-11T02:16:32.877561680Z INFO: Collecting host service logs for crio
2026-06-11T02:16:32.877946468Z INFO: Collecting host service logs for kubelet
2026-06-11T02:16:32.878260184Z INFO: Collecting host service logs for rpm-ostreed
2026-06-11T02:16:32.878552658Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-11T02:16:32.878841694Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-11T02:16:32.879164115Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-11T02:16:32.879453818Z INFO: Collecting host service logs for NetworkManager
2026-06-11T02:16:32.879731238Z INFO: Collecting host service logs for openvswitch
2026-06-11T02:16:32.880036579Z INFO: Collecting host service logs for ovs-configuration
2026-06-11T02:16:32.880339958Z INFO: Collecting host service logs for ovsdb-server
2026-06-11T02:16:32.880628387Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-11T02:16:32.880998386Z INFO: Waiting for worker host service log collection to complete ...
2026-06-11T02:16:33.022666401Z INFO: Waiting for node performance related collection to complete ...
2026-06-11T02:16:33.538386819Z No resources found in openshift-etcd namespace.
2026-06-11T02:16:33.553228347Z INFO: "metallb-operator" not detected. Skipping.
2026-06-11T02:16:33.620602274Z No resources found
2026-06-11T02:16:33.661416689Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-11T02:16:33.686890305Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-11T02:16:33.817142562Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-11T02:16:33.842271165Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-11T02:16:33.918168816Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-11T02:16:33.918213615Z INFO: on-disk MachineConfig config collection complete.
2026-06-11T02:16:34.050482317Z INFO: Collecting Insights Archives from insights-operator-896549976-dwjvt
2026-06-11T02:16:34.050538988Z insights-runtime-extractor-5zlgw
2026-06-11T02:16:34.050556665Z insights-runtime-extractor-dhn8q
2026-06-11T02:16:34.050576395Z insights-runtime-extractor-mbbcj
2026-06-11T02:16:34.141899347Z INFO: Worker host service log collection to complete.
2026-06-11T02:16:34.296124959Z Wrote inspect data to must-gather.
2026-06-11T02:16:34.342125357Z ERROR: No running kube-apiserver pods found
2026-06-11T02:16:34.416118724Z error: the server doesn't have a resource type "performanceprofile"
2026-06-11T02:16:34.433971037Z Wrote inspect data to must-gather.
2026-06-11T02:16:34.639827247Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-11T02:16:34.741561682Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-11T02:16:34.741561682Z See 'oc rsync -h' for help and examples
2026-06-11T02:16:34.950815469Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.067776583Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-11T02:16:35.091039118Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T02:16:35.097220815Z Gathering data for ns/openshift-monitoring...
2026-06-11T02:16:35.118590720Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.119759217Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-11T02:16:35.263820399Z No resources found
2026-06-11T02:16:35.267890464Z INFO: 'previous-logs' folder not found on ip-10-0-135-128.ec2.internal, skipping...
2026-06-11T02:16:35.308700099Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:35.319686367Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.387604240Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-11T02:16:35.397466914Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.453594241Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.464322585Z tar: Removing leading `/' from member names
2026-06-11T02:16:35.470402822Z INFO: HAProxy config collection complete.
2026-06-11T02:16:35.593049912Z error: the server doesn't have a resource type "machineconfigs"
2026-06-11T02:16:35.615106217Z INFO: Getting rules from prometheus-k8s-0
2026-06-11T02:16:35.626403599Z INFO: 'previous-logs' folder not found on ip-10-0-140-39.ec2.internal, skipping...
2026-06-11T02:16:35.744663864Z Wrote inspect data to must-gather.
2026-06-11T02:16:35.770754474Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-11T02:16:35.903763365Z INFO: 'previous-logs' folder not found on ip-10-0-142-56.ec2.internal, skipping...
2026-06-11T02:16:35.903797674Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-11T02:16:35.903833665Z INFO: Machine Config Daemon termination log collection complete.
2026-06-11T02:16:36.000517031Z Wrote inspect data to must-gather.
2026-06-11T02:16:36.014045265Z Wrote inspect data to must-gather.
2026-06-11T02:16:36.053697673Z INFO: Getting status/config from prometheus-k8s-0
2026-06-11T02:16:36.169419143Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-11T02:16:36.240095618Z Wrote inspect data to must-gather.
2026-06-11T02:16:36.397335596Z Wrote inspect data to must-gather.
2026-06-11T02:16:36.458507292Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-11T02:16:36.471366357Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-11T02:16:36.561464925Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T02:16:36.562584821Z INFO: INTERCONNECT MODE
2026-06-11T02:16:36.562607291Z INFO: Gathering ovn-kubernetes DBs
2026-06-11T02:16:36.643418845Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:36.647732194Z Wrote inspect data to must-gather.
2026-06-11T02:16:36.711103288Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-11T02:16:36.735671153Z INFO: Gathering OVN_Northbound from ovnkube-node-h8tc6...
2026-06-11T02:16:36.735947648Z INFO: Gathering OVN_Northbound from ovnkube-node-jwg8v...
2026-06-11T02:16:36.736192002Z INFO: Gathering OVN_Northbound from ovnkube-node-t4xxt...
2026-06-11T02:16:36.736446217Z INFO: Gathering OVN_Southbound from ovnkube-node-h8tc6...
2026-06-11T02:16:36.736699496Z INFO: Gathering OVN_Southbound from ovnkube-node-jwg8v...
2026-06-11T02:16:36.737322825Z INFO: Gathering OVN_Southbound from ovnkube-node-t4xxt...
2026-06-11T02:16:36.952342120Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:36.955480216Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-11T02:16:36.995404851Z Wrote inspect data to must-gather.
2026-06-11T02:16:37.077338094Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.150954583Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-11T02:16:37.155788793Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.165113882Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.165534856Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.173299810Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.180351523Z tar: Removing leading `/' from member names
2026-06-11T02:16:37.256441380Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:37.289442467Z W0611 02:16:37.289388    1388 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-11T02:16:37.299149161Z Wrote inspect data to must-gather.
2026-06-11T02:16:37.368312037Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T02:16:37.409007776Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-11T02:16:37.523066101Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:37.526911458Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-11T02:16:37.600436342Z Gathering data for ns/openshift-network-console...
2026-06-11T02:16:37.634362611Z Wrote inspect data to must-gather.
2026-06-11T02:16:37.635645320Z INFO: Waiting for network log collection to complete ...
2026-06-11T02:16:37.635713443Z INFO: Waiting for ovnk database copies to complete ...
2026-06-11T02:16:37.636228714Z INFO: Copying ovnk databases complete.
2026-06-11T02:16:37.638076940Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-06-11T02:16:37.642095749Z ovnk_database_store/
2026-06-11T02:16:37.642146566Z ovnk_database_store/ovnkube-node-h8tc6_sbdb
2026-06-11T02:16:37.664408259Z ovnk_database_store/ovnkube-node-t4xxt_nbdb
2026-06-11T02:16:37.671762452Z ovnk_database_store/ovnkube-node-t4xxt_sbdb
2026-06-11T02:16:37.689150210Z ovnk_database_store/ovnkube-node-h8tc6_nbdb
2026-06-11T02:16:37.699543982Z ovnk_database_store/ovnkube-node-jwg8v_sbdb
2026-06-11T02:16:37.721893792Z ovnk_database_store/ovnkube-node-jwg8v_nbdb
2026-06-11T02:16:37.732141444Z INFO: Network log collection complete.
2026-06-11T02:16:37.760049292Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:37.845611949Z INFO: Getting status from alertmanager-main-0
2026-06-11T02:16:37.979465915Z Wrote inspect data to must-gather.
2026-06-11T02:16:37.990278619Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:37.993382634Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-11T02:16:38.061073883Z Gathering data for ns/openshift-console-operator...
2026-06-11T02:16:38.216029824Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:38.271407564Z Wrote inspect data to must-gather.
2026-06-11T02:16:38.430965934Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:38.433653528Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-11T02:16:38.438757046Z Gathering data for ns/openshift-console...
2026-06-11T02:16:38.528566088Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-11T02:16:38.532230730Z INFO: Fallback to identify the container image from release info
2026-06-11T02:16:38.597647101Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:38.782121929Z Wrote inspect data to /must-gather/istio.
2026-06-11T02:16:38.912606755Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-11T02:16:39.131916613Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-11T02:16:39.258404663Z daemonset.apps/perf-node-gather-daemonset created
2026-06-11T02:16:39.404001736Z Waiting for performance profile collector pods to become ready: 1
2026-06-11T02:16:39.431215741Z Gathering data for ns/openshift-dns-operator...
2026-06-11T02:16:39.513636157Z Done executing Istio gather script
2026-06-11T02:16:39.697671315Z Gathering data for ns/openshift-dns...
2026-06-11T02:16:40.219832090Z Gathering data for ns/openshift-image-registry...
2026-06-11T02:16:40.317212827Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:40.515539763Z Waiting for performance profile collector pods to become ready: 2
2026-06-11T02:16:40.638071562Z Gathering data for ns/openshift-ingress-operator...
2026-06-11T02:16:40.913766669Z Gathering data for ns/openshift-ingress...
2026-06-11T02:16:41.195006506Z Gathering data for ns/openshift-ingress-canary...
2026-06-11T02:16:41.522023035Z Gathering data for ns/openshift-insights...
2026-06-11T02:16:41.628874574Z Waiting for performance profile collector pods to become ready: 3
2026-06-11T02:16:42.734605994Z Waiting for performance profile collector pods to become ready: 4
2026-06-11T02:16:43.234356463Z Gathering data for ns/openshift-config...
2026-06-11T02:16:43.456546659Z Gathering data for ns/openshift-config-managed...
2026-06-11T02:16:43.821669737Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-11T02:16:43.871065577Z Waiting for performance profile collector pods to become ready: 5
2026-06-11T02:16:44.051282467Z Gathering data for ns/openshift-kube-apiserver...
2026-06-11T02:16:44.268551150Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-11T02:16:44.507644549Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-11T02:16:44.743547369Z Gathering data for ns/openshift-kube-scheduler...
2026-06-11T02:16:44.968975915Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-11T02:16:44.993898930Z Waiting for performance profile collector pods to become ready: 6
2026-06-11T02:16:45.214493849Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-11T02:16:45.322002143Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:45.483803561Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-11T02:16:45.741005490Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-11T02:16:46.109851404Z Waiting for performance profile collector pods to become ready: 7
2026-06-11T02:16:46.167055389Z Gathering data for ns/openshift-multus...
2026-06-11T02:16:47.222262009Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-11T02:16:47.488558990Z Collecting performance related data for node ip-10-0-135-128.ec2.internal
2026-06-11T02:16:47.496321724Z Collecting performance related data for node ip-10-0-140-39.ec2.internal
2026-06-11T02:16:47.506438502Z Collecting performance related data for node ip-10-0-142-56.ec2.internal
2026-06-11T02:16:47.585220873Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-11T02:16:48.861892469Z Gathering data for ns/openshift-host-network...
2026-06-11T02:16:49.153392594Z Gathering data for ns/openshift-network-diagnostics...
2026-06-11T02:16:49.624105499Z Gathering data for ns/openshift-network-node-identity...
2026-06-11T02:16:50.022267409Z Gathering data for ns/openshift-network-operator...
2026-06-11T02:16:50.330898281Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:50.397082335Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-11T02:16:50.622306863Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-11T02:16:51.061427539Z Gathering data for ns/openshift-apiserver-operator...
2026-06-11T02:16:51.321123203Z Gathering data for ns/openshift-apiserver...
2026-06-11T02:16:51.660031168Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-11T02:16:51.889527783Z Gathering data for ns/openshift-controller-manager...
2026-06-11T02:16:52.126656831Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-11T02:16:52.753387414Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-11T02:16:52.999941593Z Gathering data for ns/openshift-service-ca-operator...
2026-06-11T02:16:53.267504066Z Gathering data for ns/openshift-service-ca...
2026-06-11T02:16:53.574053463Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-11T02:16:54.406039348Z Wrote inspect data to must-gather.
2026-06-11T02:16:54.406099807Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T02:16:54.406099807Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-11T02:16:55.336570790Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T02:16:55.658119737Z Collecting kubelet logs for node ip-10-0-135-128.ec2.internal
2026-06-11T02:16:55.659927333Z Collecting kubelet logs for node ip-10-0-140-39.ec2.internal
2026-06-11T02:16:55.661794652Z Collecting kubelet logs for node ip-10-0-142-56.ec2.internal
2026-06-11T02:16:55.954553254Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ghvqs namespace
2026-06-11T02:16:55.959420670Z INFO: Node performance data collection complete.
2026-06-11T02:16:56.090766127Z error: the server doesn't have a resource type "clusters"
2026-06-11T02:16:56.250470072Z Caches written to disk