2026-04-16T16:55:32.321305163Z [disk usage checker] Started
2026-04-16T16:55:32.324946525Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:32.567568348Z Gathering data for ns/openshift-cluster-version...
2026-04-16T16:55:32.772231081Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T16:55:32.966223140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T16:55:33.072998181Z Gathering data for ns/default...
2026-04-16T16:55:33.446538138Z Gathering data for ns/openshift...
2026-04-16T16:55:33.935697550Z Gathering data for ns/kube-system...
2026-04-16T16:55:34.720442408Z Gathering data for ns/openshift-etcd...
2026-04-16T16:55:35.057422310Z Waiting on subprocesses to finish execution.
2026-04-16T16:55:35.060842065Z INFO: Collecting host service logs for crio
2026-04-16T16:55:35.061205989Z INFO: Collecting host service logs for kubelet
2026-04-16T16:55:35.061549308Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T16:55:35.061874281Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T16:55:35.062250507Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T16:55:35.062598381Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T16:55:35.062941198Z INFO: Collecting host service logs for NetworkManager
2026-04-16T16:55:35.063297469Z INFO: Collecting host service logs for openvswitch
2026-04-16T16:55:35.063651654Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T16:55:35.063990581Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T16:55:35.066198719Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T16:55:35.070048522Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T16:55:35.070451563Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T16:55:35.079352902Z INFO: Gathering HAProxy config files
2026-04-16T16:55:35.089311083Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T16:55:35.101534168Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T16:55:35.212433696Z Wrote inspect data to must-gather.
2026-04-16T16:55:35.212536846Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T16:55:35.212536846Z     namespaces "assisted-installer" not found
2026-04-16T16:55:35.296951181Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T16:55:35.530361809Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T16:55:35.530461792Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T16:55:35.899454491Z No resources found
2026-04-16T16:55:35.948755172Z No resources found in openshift-etcd namespace.
2026-04-16T16:55:36.011140911Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T16:55:36.078810634Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T16:55:36.103378050Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T16:55:36.185624067Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T16:55:36.226990904Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T16:55:36.355889875Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T16:55:36.365224202Z INFO: Collecting Insights Archives from insights-operator-5785d4fcdd-hxp97
2026-04-16T16:55:36.365282917Z insights-runtime-extractor-2j4nh
2026-04-16T16:55:36.365300744Z insights-runtime-extractor-vcnhj
2026-04-16T16:55:36.365319535Z insights-runtime-extractor-wfnkm
2026-04-16T16:55:36.617402783Z INFO: Worker host service log collection to complete.
2026-04-16T16:55:36.714590788Z ERROR: No running kube-apiserver pods found
2026-04-16T16:55:36.762385409Z Wrote inspect data to must-gather.
2026-04-16T16:55:36.814723559Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T16:55:36.814723559Z See 'oc rsync -h' for help and examples
2026-04-16T16:55:36.820312787Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T16:55:37.104852804Z No resources found
2026-04-16T16:55:37.217866796Z Wrote inspect data to must-gather.
2026-04-16T16:55:37.514032481Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:37.667970338Z Wrote inspect data to must-gather.
2026-04-16T16:55:37.750991129Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T16:55:37.795435519Z Gathering data for ns/kuadrant-system...
2026-04-16T16:55:37.829729006Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T16:55:37.836460600Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T16:55:38.020660020Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.044461637Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T16:55:38.055873820Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.082137659Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.084837458Z INFO: 'previous-logs' folder not found on ip-10-0-130-165.ec2.internal, skipping...
2026-04-16T16:55:38.181991013Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T16:55:38.201643776Z tar: Removing leading `/' from member names
2026-04-16T16:55:38.209948404Z INFO: HAProxy config collection complete.
2026-04-16T16:55:38.535273960Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.566234573Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.581826752Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T16:55:38.605835753Z INFO: 'previous-logs' folder not found on ip-10-0-132-246.ec2.internal, skipping...
2026-04-16T16:55:38.919819982Z W0416 16:55:38.919702    1050 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T16:55:38.935198616Z W0416 16:55:38.935150    1042 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T16:55:38.946960472Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.957879364Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T16:55:38.964616021Z Wrote inspect data to must-gather.
2026-04-16T16:55:38.983756838Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T16:55:38.989731554Z INFO: 'previous-logs' folder not found on ip-10-0-138-125.ec2.internal, skipping...
2026-04-16T16:55:38.989769435Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T16:55:38.989800816Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T16:55:39.191891206Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T16:55:39.214533017Z Gathering data for ns/openshift-monitoring...
2026-04-16T16:55:39.385017356Z Wrote inspect data to must-gather.
2026-04-16T16:55:39.524547122Z Wrote inspect data to must-gather.
2026-04-16T16:55:39.528376522Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T16:55:39.782777103Z INFO: INTERCONNECT MODE
2026-04-16T16:55:39.782777103Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T16:55:39.807639030Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T16:55:39.844293231Z Wrote inspect data to must-gather.
2026-04-16T16:55:39.996724920Z INFO: Gathering OVN_Northbound from ovnkube-node-8zb4q...
2026-04-16T16:55:39.997009510Z INFO: Gathering OVN_Northbound from ovnkube-node-hschh...
2026-04-16T16:55:39.997295221Z INFO: Gathering OVN_Northbound from ovnkube-node-vm8pb...
2026-04-16T16:55:39.997594510Z INFO: Gathering OVN_Southbound from ovnkube-node-8zb4q...
2026-04-16T16:55:39.997867290Z INFO: Gathering OVN_Southbound from ovnkube-node-hschh...
2026-04-16T16:55:39.998191974Z INFO: Gathering OVN_Southbound from ovnkube-node-vm8pb...
2026-04-16T16:55:40.119464093Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T16:55:40.299549904Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.299549904Z Wrote inspect data to must-gather.
2026-04-16T16:55:40.331272274Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.367376218Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.390353363Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.392982469Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.424535274Z tar: Removing leading `/' from member names
2026-04-16T16:55:40.549454453Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T16:55:40.630313489Z Wrote inspect data to must-gather.
2026-04-16T16:55:40.971573441Z Wrote inspect data to must-gather.
2026-04-16T16:55:41.028835440Z INFO: Getting status from alertmanager-main-0
2026-04-16T16:55:41.039327839Z INFO: Waiting for network log collection to complete ...
2026-04-16T16:55:41.043288327Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T16:55:41.043888465Z INFO: Copying ovnk databases complete.
2026-04-16T16:55:41.046056788Z 5.0M	must-gather/network_logs/ovnk_database_store
2026-04-16T16:55:41.048422319Z ovnk_database_store/
2026-04-16T16:55:41.048422319Z ovnk_database_store/ovnkube-node-8zb4q_nbdb
2026-04-16T16:55:41.060126420Z ovnk_database_store/ovnkube-node-vm8pb_nbdb
2026-04-16T16:55:41.070633639Z ovnk_database_store/ovnkube-node-hschh_nbdb
2026-04-16T16:55:41.081582858Z ovnk_database_store/ovnkube-node-8zb4q_sbdb
2026-04-16T16:55:41.114783334Z ovnk_database_store/ovnkube-node-hschh_sbdb
2026-04-16T16:55:41.148331237Z ovnk_database_store/ovnkube-node-vm8pb_sbdb
2026-04-16T16:55:41.185509009Z INFO: Network log collection complete.
2026-04-16T16:55:41.289436796Z Wrote inspect data to must-gather.
2026-04-16T16:55:41.564826818Z W0416 16:55:41.564779    1450 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T16:55:41.571548550Z Wrote inspect data to must-gather.
2026-04-16T16:55:41.890456222Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T16:55:41.893819945Z INFO: Fallback to identify the container image from release info
2026-04-16T16:55:41.894037631Z Gathering data for ns/openshift-network-console...
2026-04-16T16:55:42.423727390Z Gathering data for ns/openshift-console-operator...
2026-04-16T16:55:42.440020998Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc77fc5f1546189d6b57967ccc684e794bd846ea36c3e3c1e61d83a684f0f9b9
2026-04-16T16:55:42.537098651Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:42.616193841Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T16:55:42.778632440Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T16:55:42.948814422Z Gathering data for ns/openshift-console...
2026-04-16T16:55:43.019204994Z Wrote inspect data to must-gather.
2026-04-16T16:55:43.491826312Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T16:55:43.931584412Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T16:55:43.965165504Z Gathering data for ns/openshift-dns-operator...
2026-04-16T16:55:44.324630662Z Gathering data for ns/openshift-dns...
2026-04-16T16:55:45.066337885Z Gathering data for ns/openshift-image-registry...
2026-04-16T16:55:45.078337256Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T16:55:45.606019526Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T16:55:45.983839371Z Gathering data for ns/openshift-ingress...
2026-04-16T16:55:46.202810848Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T16:55:46.657406152Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T16:55:47.209914737Z Gathering data for ns/openshift-insights...
2026-04-16T16:55:47.331986890Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T16:55:47.541827897Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:48.458893539Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T16:55:49.614421002Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T16:55:50.419832755Z Gathering data for ns/openshift-lws-operator...
2026-04-16T16:55:50.740373060Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T16:55:51.033906797Z Collecting performance related data for node ip-10-0-130-165.ec2.internal
2026-04-16T16:55:51.040851091Z Collecting performance related data for node ip-10-0-138-125.ec2.internal
2026-04-16T16:55:51.048886313Z Collecting performance related data for node ip-10-0-132-246.ec2.internal
2026-04-16T16:55:51.099569794Z Gathering data for ns/kserve...
2026-04-16T16:55:52.547567207Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:53.317145892Z Gathering data for ns/openshift-config...
2026-04-16T16:55:53.724622534Z Gathering data for ns/openshift-config-managed...
2026-04-16T16:55:54.289252397Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T16:55:54.696456843Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T16:55:55.139139684Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T16:55:55.574762620Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T16:55:56.004292420Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T16:55:56.363371223Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T16:55:56.755879908Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T16:55:57.199044410Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T16:55:57.552465675Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:55:57.640986326Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T16:55:58.346891529Z Gathering data for ns/openshift-multus...
2026-04-16T16:56:00.016414771Z Collecting kubelet logs for node ip-10-0-130-165.ec2.internal
2026-04-16T16:56:00.018211091Z Collecting kubelet logs for node ip-10-0-138-125.ec2.internal
2026-04-16T16:56:00.019816100Z Collecting kubelet logs for node ip-10-0-132-246.ec2.internal
2026-04-16T16:56:00.383350169Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T16:56:00.468807048Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T16:56:00.475990014Z INFO: Node performance data collection complete.
2026-04-16T16:56:02.557266590Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:56:02.558531438Z Gathering data for ns/openshift-host-network...
2026-04-16T16:56:02.972759013Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T16:56:03.642453189Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T16:56:04.127641793Z Gathering data for ns/openshift-network-operator...
2026-04-16T16:56:04.532416581Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T16:56:04.847853339Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T16:56:05.366292515Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T16:56:05.799843648Z Gathering data for ns/openshift-apiserver...
2026-04-16T16:56:06.256921718Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T16:56:06.619083883Z Gathering data for ns/openshift-controller-manager...
2026-04-16T16:56:06.981583836Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T16:56:07.561840593Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70
2026-04-16T16:56:07.667311604Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T16:56:08.011896809Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T16:56:08.380447531Z Gathering data for ns/openshift-service-ca...
2026-04-16T16:56:08.804867114Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T16:56:09.846847642Z Wrote inspect data to must-gather.
2026-04-16T16:56:09.847005868Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T16:56:09.847005868Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T16:56:09.993474626Z error: the server doesn't have a resource type "clusters"
2026-04-16T16:56:10.164811449Z Caches written to disk