2026-04-16T19:35:20.958934036Z [disk usage checker] Started
2026-04-16T19:35:20.962564931Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:21.179028560Z Gathering data for ns/openshift-cluster-version...
2026-04-16T19:35:21.310447777Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:35:21.393931672Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:35:21.470505939Z Gathering data for ns/default...
2026-04-16T19:35:21.759390400Z Gathering data for ns/openshift...
2026-04-16T19:35:22.154578866Z Gathering data for ns/kube-system...
2026-04-16T19:35:22.885552736Z Gathering data for ns/openshift-etcd...
2026-04-16T19:35:23.173807064Z Wrote inspect data to must-gather.
2026-04-16T19:35:23.173842725Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:35:23.173842725Z     namespaces "assisted-installer" not found
2026-04-16T19:35:23.427335362Z Waiting on subprocesses to finish execution.
2026-04-16T19:35:23.433822096Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T19:35:23.441148089Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T19:35:23.446580871Z INFO: Collecting host service logs for crio
2026-04-16T19:35:23.447020637Z INFO: Collecting host service logs for kubelet
2026-04-16T19:35:23.447363794Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T19:35:23.447511039Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T19:35:23.447748867Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T19:35:23.448041380Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T19:35:23.448339916Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T19:35:23.448655268Z INFO: Collecting host service logs for NetworkManager
2026-04-16T19:35:23.448977071Z INFO: Collecting host service logs for openvswitch
2026-04-16T19:35:23.449287143Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T19:35:23.449592450Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T19:35:23.449902905Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T19:35:23.450251985Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T19:35:23.467410768Z INFO: Gathering HAProxy config files
2026-04-16T19:35:23.511031803Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T19:35:23.951523116Z No resources found in openshift-etcd namespace.
2026-04-16T19:35:24.151778365Z INFO: Collecting Insights Archives from insights-operator-5785d4fcdd-crf8r
2026-04-16T19:35:24.151834427Z insights-runtime-extractor-99n6h
2026-04-16T19:35:24.151876253Z insights-runtime-extractor-hnhgh
2026-04-16T19:35:24.151903550Z insights-runtime-extractor-j6xwp
2026-04-16T19:35:24.229801112Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T19:35:24.517160936Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T19:35:24.517236512Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T19:35:24.631786415Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T19:35:24.651379362Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T19:35:24.666023881Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T19:35:24.743571821Z No resources found
2026-04-16T19:35:24.818440764Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T19:35:24.818440764Z See 'oc rsync -h' for help and examples
2026-04-16T19:35:24.879897804Z ERROR: No running kube-apiserver pods found
2026-04-16T19:35:24.891984892Z INFO: Worker host service log collection to complete.
2026-04-16T19:35:24.950941279Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T19:35:25.124668203Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T19:35:25.198836960Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T19:35:25.214096740Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.394624140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:35:25.454081242Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.461385701Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T19:35:25.516421983Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.531526310Z INFO: 'previous-logs' folder not found on ip-10-0-132-14.ec2.internal, skipping...
2026-04-16T19:35:25.548148885Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.583356962Z No resources found
2026-04-16T19:35:25.630646763Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.661416639Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.676636147Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T19:35:25.737114972Z Wrote inspect data to must-gather.
2026-04-16T19:35:25.900737587Z INFO: 'previous-logs' folder not found on ip-10-0-139-33.ec2.internal, skipping...
2026-04-16T19:35:25.927481064Z tar: Removing leading `/' from member names
2026-04-16T19:35:25.933052065Z INFO: HAProxy config collection complete.
2026-04-16T19:35:25.935429638Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T19:35:25.965336682Z Gathering data for ns/openshift-monitoring...
2026-04-16T19:35:25.967265272Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:25.982935961Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.088812066Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T19:35:26.194684785Z INFO: 'previous-logs' folder not found on ip-10-0-139-49.ec2.internal, skipping...
2026-04-16T19:35:26.194793756Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T19:35:26.194848626Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T19:35:26.207685345Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.248446371Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.419706669Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.483281707Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.499753531Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T19:35:26.578835631Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T19:35:26.770882658Z INFO: INTERCONNECT MODE
2026-04-16T19:35:26.770909903Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T19:35:26.784844723Z Wrote inspect data to must-gather.
2026-04-16T19:35:26.900563780Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T19:35:26.950325577Z INFO: Gathering OVN_Northbound from ovnkube-node-fdzzg...
2026-04-16T19:35:26.950576263Z INFO: Gathering OVN_Northbound from ovnkube-node-qg4jj...
2026-04-16T19:35:26.950753278Z INFO: Gathering OVN_Northbound from ovnkube-node-s62vp...
2026-04-16T19:35:26.951019201Z INFO: Gathering OVN_Southbound from ovnkube-node-fdzzg...
2026-04-16T19:35:26.951280865Z INFO: Gathering OVN_Southbound from ovnkube-node-qg4jj...
2026-04-16T19:35:26.951529082Z INFO: Gathering OVN_Southbound from ovnkube-node-s62vp...
2026-04-16T19:35:27.023769680Z Wrote inspect data to must-gather.
2026-04-16T19:35:27.212071687Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.212099836Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.228673414Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T19:35:27.241748457Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.246786525Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.282433697Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.308669929Z tar: Removing leading `/' from member names
2026-04-16T19:35:27.322161800Z Wrote inspect data to must-gather.
2026-04-16T19:35:27.508169972Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:35:27.518400635Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T19:35:27.591361646Z Wrote inspect data to must-gather.
2026-04-16T19:35:27.778971402Z INFO: Waiting for network log collection to complete ...
2026-04-16T19:35:27.778971402Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T19:35:27.779459599Z INFO: Copying ovnk databases complete.
2026-04-16T19:35:27.781369841Z 4.4M	must-gather/network_logs/ovnk_database_store
2026-04-16T19:35:27.784895427Z ovnk_database_store/
2026-04-16T19:35:27.784895427Z ovnk_database_store/ovnkube-node-s62vp_nbdb
2026-04-16T19:35:27.796993999Z ovnk_database_store/ovnkube-node-s62vp_sbdb
2026-04-16T19:35:27.799021533Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T19:35:27.821593959Z ovnk_database_store/ovnkube-node-qg4jj_sbdb
2026-04-16T19:35:27.835390303Z Wrote inspect data to must-gather.
2026-04-16T19:35:27.841694589Z ovnk_database_store/ovnkube-node-fdzzg_nbdb
2026-04-16T19:35:27.853930330Z ovnk_database_store/ovnkube-node-fdzzg_sbdb
2026-04-16T19:35:27.880534208Z ovnk_database_store/ovnkube-node-qg4jj_nbdb
2026-04-16T19:35:27.892466701Z INFO: Network log collection complete.
2026-04-16T19:35:28.052681140Z W0416 19:35:28.052637    1384 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T19:35:28.060119597Z Wrote inspect data to must-gather.
2026-04-16T19:35:28.125351524Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:35:28.203186848Z INFO: Getting status from alertmanager-main-0
2026-04-16T19:35:28.303192761Z Gathering data for ns/openshift-network-console...
2026-04-16T19:35:28.365597042Z Wrote inspect data to must-gather.
2026-04-16T19:35:28.623935236Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T19:35:28.626498895Z INFO: Fallback to identify the container image from release info
2026-04-16T19:35:28.745981135Z Gathering data for ns/openshift-console-operator...
2026-04-16T19:35:29.123950105Z Gathering data for ns/openshift-console...
2026-04-16T19:35:29.310832215Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc77fc5f1546189d6b57967ccc684e794bd846ea36c3e3c1e61d83a684f0f9b9
2026-04-16T19:35:29.438915173Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T19:35:29.587619093Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T19:35:29.609060660Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T19:35:29.981736860Z Gathering data for ns/openshift-dns-operator...
2026-04-16T19:35:30.292757843Z Gathering data for ns/openshift-dns...
2026-04-16T19:35:30.720018527Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T19:35:30.971449048Z Gathering data for ns/openshift-image-registry...
2026-04-16T19:35:30.971937573Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:31.474458185Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T19:35:31.807334040Z Gathering data for ns/openshift-ingress...
2026-04-16T19:35:31.850788575Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T19:35:32.146183911Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T19:35:32.534291116Z Gathering data for ns/openshift-insights...
2026-04-16T19:35:32.968449040Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T19:35:34.094907891Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T19:35:34.623388724Z Gathering data for ns/kserve...
2026-04-16T19:35:35.209226143Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T19:35:35.976383034Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:36.341183490Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T19:35:36.457203520Z Gathering data for ns/openshift-config...
2026-04-16T19:35:36.775155282Z Gathering data for ns/openshift-config-managed...
2026-04-16T19:35:37.227064033Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T19:35:37.479107979Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T19:35:37.497408367Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T19:35:37.775637109Z Collecting performance related data for node ip-10-0-139-33.ec2.internal
2026-04-16T19:35:37.782923775Z Collecting performance related data for node ip-10-0-139-49.ec2.internal
2026-04-16T19:35:37.783268892Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T19:35:37.792040195Z Collecting performance related data for node ip-10-0-132-14.ec2.internal
2026-04-16T19:35:38.165296510Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T19:35:38.550377372Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T19:35:38.876889324Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T19:35:39.267559007Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T19:35:39.673004630Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T19:35:40.003342621Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T19:35:40.592064671Z Gathering data for ns/openshift-multus...
2026-04-16T19:35:40.980983104Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:42.657369368Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T19:35:44.277522775Z Gathering data for ns/openshift-host-network...
2026-04-16T19:35:44.617099876Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T19:35:45.243067578Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T19:35:45.660321032Z Gathering data for ns/openshift-network-operator...
2026-04-16T19:35:45.987116032Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T19:35:46.059698336Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T19:35:46.332258824Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T19:35:46.371565615Z Collecting kubelet logs for node ip-10-0-139-33.ec2.internal
2026-04-16T19:35:46.373310155Z Collecting kubelet logs for node ip-10-0-139-49.ec2.internal
2026-04-16T19:35:46.374958784Z Collecting kubelet logs for node ip-10-0-132-14.ec2.internal
2026-04-16T19:35:46.765502954Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T19:35:46.771685844Z INFO: Node performance data collection complete.
2026-04-16T19:35:46.862789857Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T19:35:47.156879383Z Gathering data for ns/openshift-apiserver...
2026-04-16T19:35:47.588081364Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T19:35:47.831244396Z Gathering data for ns/openshift-controller-manager...
2026-04-16T19:35:48.088076334Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T19:35:48.704475692Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T19:35:49.007773577Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T19:35:49.294120903Z Gathering data for ns/openshift-service-ca...
2026-04-16T19:35:49.636791053Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T19:35:50.578881853Z Wrote inspect data to must-gather.
2026-04-16T19:35:50.578926130Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:35:50.578926130Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T19:35:50.713511289Z error: the server doesn't have a resource type "clusters"
2026-04-16T19:35:50.835476424Z Caches written to disk