2026-04-23T18:57:57.491280490Z [disk usage checker] Started
2026-04-23T18:57:57.494649499Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:57:57.838288317Z Gathering data for ns/openshift-cluster-version...
2026-04-23T18:57:57.997153494Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T18:57:58.097907584Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T18:57:58.195373404Z Gathering data for ns/default...
2026-04-23T18:57:58.528674722Z Gathering data for ns/openshift...
2026-04-23T18:57:58.972477167Z Gathering data for ns/kube-system...
2026-04-23T18:57:59.853832199Z Gathering data for ns/openshift-etcd...
2026-04-23T18:58:00.226448865Z Wrote inspect data to must-gather.
2026-04-23T18:58:00.226558608Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T18:58:00.226558608Z     namespaces "assisted-installer" not found
2026-04-23T18:58:00.228083326Z Waiting on subprocesses to finish execution.
2026-04-23T18:58:00.235350882Z INFO: Gathering HAProxy config files
2026-04-23T18:58:00.236988024Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-23T18:58:00.245392090Z INFO: Collecting host service logs for crio
2026-04-23T18:58:00.245736414Z INFO: Collecting host service logs for kubelet
2026-04-23T18:58:00.246188826Z INFO: Collecting host service logs for rpm-ostreed
2026-04-23T18:58:00.246526191Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-23T18:58:00.246863234Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-23T18:58:00.246882865Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-23T18:58:00.247226554Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-23T18:58:00.247554570Z INFO: Collecting host service logs for NetworkManager
2026-04-23T18:58:00.247877072Z INFO: Collecting host service logs for openvswitch
2026-04-23T18:58:00.248316680Z INFO: Collecting host service logs for ovs-configuration
2026-04-23T18:58:00.248756489Z INFO: Collecting host service logs for ovsdb-server
2026-04-23T18:58:00.249113538Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-23T18:58:00.249500663Z INFO: Waiting for worker host service log collection to complete ...
2026-04-23T18:58:00.266393887Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-23T18:58:00.379857873Z INFO: Waiting for node performance related collection to complete ...
2026-04-23T18:58:00.861559244Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-23T18:58:00.991672220Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-23T18:58:01.065278903Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-23T18:58:01.065307656Z INFO: on-disk MachineConfig config collection complete.
2026-04-23T18:58:01.103072808Z Wrote inspect data to must-gather.
2026-04-23T18:58:01.291485884Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-7h7vz
2026-04-23T18:58:01.291546276Z insights-runtime-extractor-mvdnw
2026-04-23T18:58:01.291567516Z insights-runtime-extractor-th8sw
2026-04-23T18:58:01.291588391Z insights-runtime-extractor-xjc8h
2026-04-23T18:58:01.316921752Z No resources found in openshift-etcd namespace.
2026-04-23T18:58:01.446795764Z error: the server doesn't have a resource type "performanceprofile"
2026-04-23T18:58:01.480145653Z INFO: "metallb-operator" not detected. Skipping.
2026-04-23T18:58:01.486106685Z No resources found
2026-04-23T18:58:01.545428562Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-23T18:58:01.545608958Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-23T18:58:01.625907995Z INFO: Worker host service log collection to complete.
2026-04-23T18:58:01.740403895Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-23T18:58:01.740403895Z See 'oc rsync -h' for help and examples
2026-04-23T18:58:01.945063862Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-23T18:58:02.142072718Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.158491856Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T18:58:02.354146983Z ERROR: No running kube-apiserver pods found
2026-04-23T18:58:02.434552487Z No resources found
2026-04-23T18:58:02.439272929Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.459826451Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-23T18:58:02.464810297Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.484359403Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-23T18:58:02.496914715Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.504575527Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:02.579235523Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.601293487Z Gathering data for ns/openshift-monitoring...
2026-04-23T18:58:02.656623945Z INFO: 'previous-logs' folder not found on ip-10-0-131-144.ec2.internal, skipping...
2026-04-23T18:58:02.690147913Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.709473980Z error: the server doesn't have a resource type "machineconfigs"
2026-04-23T18:58:02.786742590Z tar: Removing leading `/' from member names
2026-04-23T18:58:02.793632324Z INFO: HAProxy config collection complete.
2026-04-23T18:58:02.856017513Z INFO: Getting rules from prometheus-k8s-0
2026-04-23T18:58:02.872372338Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.941211739Z Wrote inspect data to must-gather.
2026-04-23T18:58:02.949031959Z INFO: 'previous-logs' folder not found on ip-10-0-135-87.ec2.internal, skipping...
2026-04-23T18:58:03.141497630Z Wrote inspect data to must-gather.
2026-04-23T18:58:03.194243766Z Wrote inspect data to must-gather.
2026-04-23T18:58:03.226481405Z INFO: 'previous-logs' folder not found on ip-10-0-142-106.ec2.internal, skipping...
2026-04-23T18:58:03.226600249Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-23T18:58:03.226653743Z INFO: Machine Config Daemon termination log collection complete.
2026-04-23T18:58:03.298511445Z INFO: Getting status/config from prometheus-k8s-0
2026-04-23T18:58:03.366967966Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-23T18:58:03.402762885Z Wrote inspect data to must-gather.
2026-04-23T18:58:03.579753109Z Wrote inspect data to must-gather.
2026-04-23T18:58:03.708127830Z INFO: INTERCONNECT MODE
2026-04-23T18:58:03.708158463Z INFO: Gathering ovn-kubernetes DBs
2026-04-23T18:58:03.722506218Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-23T18:58:03.803257649Z Wrote inspect data to must-gather.
2026-04-23T18:58:03.877809826Z INFO: Gathering OVN_Northbound from ovnkube-node-2kfrf...
2026-04-23T18:58:03.878045775Z INFO: Gathering OVN_Northbound from ovnkube-node-hhdnl...
2026-04-23T18:58:03.878229966Z INFO: Gathering OVN_Northbound from ovnkube-node-lwrk7...
2026-04-23T18:58:03.878466056Z INFO: Gathering OVN_Southbound from ovnkube-node-2kfrf...
2026-04-23T18:58:03.878720967Z INFO: Gathering OVN_Southbound from ovnkube-node-hhdnl...
2026-04-23T18:58:03.878986526Z INFO: Gathering OVN_Southbound from ovnkube-node-lwrk7...
2026-04-23T18:58:03.989988964Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-23T18:58:04.132148397Z Wrote inspect data to must-gather.
2026-04-23T18:58:04.138209720Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.154795872Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.177158239Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.186805943Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.205469947Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.209857396Z tar: Removing leading `/' from member names
2026-04-23T18:58:04.314521902Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T18:58:04.343581009Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-23T18:58:04.378251364Z Wrote inspect data to must-gather.
2026-04-23T18:58:04.628892902Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-23T18:58:04.634207204Z Wrote inspect data to must-gather.
2026-04-23T18:58:04.685904334Z INFO: Waiting for network log collection to complete ...
2026-04-23T18:58:04.685965896Z INFO: Waiting for ovnk database copies to complete ...
2026-04-23T18:58:04.686353302Z INFO: Copying ovnk databases complete.
2026-04-23T18:58:04.687911735Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-04-23T18:58:04.690071004Z ovnk_database_store/
2026-04-23T18:58:04.690071004Z ovnk_database_store/ovnkube-node-hhdnl_nbdb
2026-04-23T18:58:04.700459821Z ovnk_database_store/ovnkube-node-hhdnl_sbdb
2026-04-23T18:58:04.722028032Z ovnk_database_store/ovnkube-node-lwrk7_nbdb
2026-04-23T18:58:04.733089318Z ovnk_database_store/ovnkube-node-lwrk7_sbdb
2026-04-23T18:58:04.761701014Z ovnk_database_store/ovnkube-node-2kfrf_sbdb
2026-04-23T18:58:04.787565827Z ovnk_database_store/ovnkube-node-2kfrf_nbdb
2026-04-23T18:58:04.800605074Z INFO: Network log collection complete.
2026-04-23T18:58:04.846515602Z W0423 18:58:04.846466    1377 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-23T18:58:04.850778842Z Wrote inspect data to must-gather.
2026-04-23T18:58:04.932827915Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T18:58:05.073295823Z INFO: Getting status from alertmanager-main-0
2026-04-23T18:58:05.135492307Z Gathering data for ns/openshift-network-console...
2026-04-23T18:58:05.144243522Z Wrote inspect data to must-gather.
2026-04-23T18:58:05.413237468Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-23T18:58:05.416031566Z INFO: Fallback to identify the container image from release info
2026-04-23T18:58:05.611764557Z Gathering data for ns/openshift-console-operator...
2026-04-23T18:58:05.958502052Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-23T18:58:06.040549642Z Gathering data for ns/openshift-console...
2026-04-23T18:58:06.096307699Z daemonset.apps/perf-node-gather-daemonset created
2026-04-23T18:58:06.241672923Z Waiting for performance profile collector pods to become ready: 1
2026-04-23T18:58:06.569477714Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-23T18:58:07.024547647Z Gathering data for ns/openshift-dns-operator...
2026-04-23T18:58:07.355694911Z Gathering data for ns/openshift-dns...
2026-04-23T18:58:07.381518952Z Waiting for performance profile collector pods to become ready: 2
2026-04-23T18:58:07.524498467Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:08.030957599Z Gathering data for ns/openshift-image-registry...
2026-04-23T18:58:08.514737634Z Waiting for performance profile collector pods to become ready: 3
2026-04-23T18:58:08.567720921Z Gathering data for ns/openshift-ingress-operator...
2026-04-23T18:58:08.939146323Z Gathering data for ns/openshift-ingress...
2026-04-23T18:58:09.331661383Z Gathering data for ns/openshift-ingress-canary...
2026-04-23T18:58:09.638375232Z Waiting for performance profile collector pods to become ready: 4
2026-04-23T18:58:09.792125280Z Gathering data for ns/openshift-insights...
2026-04-23T18:58:10.757826846Z Waiting for performance profile collector pods to become ready: 5
2026-04-23T18:58:11.878608673Z Waiting for performance profile collector pods to become ready: 6
2026-04-23T18:58:12.083691536Z Gathering data for ns/kserve...
2026-04-23T18:58:12.529276159Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:13.014497788Z Waiting for performance profile collector pods to become ready: 7
2026-04-23T18:58:14.139353950Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-23T18:58:14.269680346Z Gathering data for ns/openshift-config...
2026-04-23T18:58:14.401726902Z Collecting performance related data for node ip-10-0-142-106.ec2.internal
2026-04-23T18:58:14.413082264Z Collecting performance related data for node ip-10-0-131-144.ec2.internal
2026-04-23T18:58:14.424411588Z Collecting performance related data for node ip-10-0-135-87.ec2.internal
2026-04-23T18:58:14.709734678Z Gathering data for ns/openshift-config-managed...
2026-04-23T18:58:15.309092436Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-23T18:58:15.703284730Z Gathering data for ns/openshift-kube-apiserver...
2026-04-23T18:58:16.102157944Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-23T18:58:16.426447450Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-23T18:58:16.763172435Z Gathering data for ns/openshift-kube-scheduler...
2026-04-23T18:58:17.170000568Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-23T18:58:17.537506437Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:17.625627618Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-23T18:58:18.019433697Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-23T18:58:18.393644794Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-23T18:58:19.012694036Z Gathering data for ns/openshift-multus...
2026-04-23T18:58:20.615814411Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-23T18:58:22.494131086Z Gathering data for ns/openshift-host-network...
2026-04-23T18:58:22.545656050Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:22.867286855Z Gathering data for ns/openshift-network-diagnostics...
2026-04-23T18:58:23.076694797Z Collecting kubelet logs for node ip-10-0-142-106.ec2.internal
2026-04-23T18:58:23.078510216Z Collecting kubelet logs for node ip-10-0-131-144.ec2.internal
2026-04-23T18:58:23.080378096Z Collecting kubelet logs for node ip-10-0-135-87.ec2.internal
2026-04-23T18:58:23.577748046Z Gathering data for ns/openshift-network-node-identity...
2026-04-23T18:58:23.617575061Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-23T18:58:23.625389465Z INFO: Node performance data collection complete.
2026-04-23T18:58:24.083220037Z Gathering data for ns/openshift-network-operator...
2026-04-23T18:58:24.563486767Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-23T18:58:24.896138854Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-23T18:58:25.430847613Z Gathering data for ns/openshift-apiserver-operator...
2026-04-23T18:58:25.855057128Z Gathering data for ns/openshift-apiserver...
2026-04-23T18:58:26.346658887Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-23T18:58:26.706498795Z Gathering data for ns/openshift-controller-manager...
2026-04-23T18:58:27.051195464Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-23T18:58:27.551090591Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-23T18:58:27.800025664Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-23T18:58:28.147107637Z Gathering data for ns/openshift-service-ca-operator...
2026-04-23T18:58:28.499335847Z Gathering data for ns/openshift-service-ca...
2026-04-23T18:58:28.897635896Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-23T18:58:29.872680904Z Wrote inspect data to must-gather.
2026-04-23T18:58:29.872723363Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T18:58:29.872723363Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-23T18:58:30.003472111Z error: the server doesn't have a resource type "clusters"
2026-04-23T18:58:30.152905326Z Caches written to disk