2026-04-21T04:52:42.901625640Z [disk usage checker] Started
2026-04-21T04:52:42.931510613Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:52:43.131311883Z Gathering data for ns/openshift-cluster-version...
2026-04-21T04:52:43.294752645Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T04:52:43.397070704Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:52:43.495381476Z Gathering data for ns/default...
2026-04-21T04:52:43.876547906Z Gathering data for ns/openshift...
2026-04-21T04:52:44.363765638Z Gathering data for ns/kube-system...
2026-04-21T04:52:45.115344012Z Gathering data for ns/openshift-etcd...
2026-04-21T04:52:45.311042761Z Waiting on subprocesses to finish execution.
2026-04-21T04:52:45.315181916Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T04:52:45.322629076Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T04:52:45.336679987Z INFO: Collecting host service logs for crio
2026-04-21T04:52:45.336990164Z INFO: Collecting host service logs for kubelet
2026-04-21T04:52:45.337318077Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T04:52:45.337632028Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T04:52:45.337951581Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T04:52:45.338261435Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T04:52:45.338631484Z INFO: Collecting host service logs for NetworkManager
2026-04-21T04:52:45.338931995Z INFO: Collecting host service logs for openvswitch
2026-04-21T04:52:45.339231924Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T04:52:45.339596271Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T04:52:45.339995364Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T04:52:45.340342990Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T04:52:45.350604055Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T04:52:45.356687873Z INFO: Gathering HAProxy config files
2026-04-21T04:52:45.420630957Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T04:52:46.073282620Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T04:52:46.073361444Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T04:52:46.232932660Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T04:52:46.245006701Z Wrote inspect data to must-gather.
2026-04-21T04:52:46.245048439Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T04:52:46.245048439Z     namespaces "assisted-installer" not found
2026-04-21T04:52:46.566205196Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-srd87
2026-04-21T04:52:46.566262894Z insights-runtime-extractor-cqh6m
2026-04-21T04:52:46.566282332Z insights-runtime-extractor-l4ghz
2026-04-21T04:52:46.566302485Z insights-runtime-extractor-zwvsk
2026-04-21T04:52:46.605207814Z No resources found in openshift-etcd namespace.
2026-04-21T04:52:46.665351534Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T04:52:46.758007499Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T04:52:46.820285788Z Wrote inspect data to must-gather.
2026-04-21T04:52:46.876506970Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T04:52:46.974641392Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T04:52:47.117640633Z No resources found
2026-04-21T04:52:47.234082148Z ERROR: No running kube-apiserver pods found
2026-04-21T04:52:47.262221614Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:52:47.276236369Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T04:52:47.322331830Z INFO: Worker host service log collection to complete.
2026-04-21T04:52:47.442058972Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T04:52:47.446043063Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T04:52:47.446043063Z See 'oc rsync -h' for help and examples
2026-04-21T04:52:47.481482137Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T04:52:47.704279252Z No resources found
2026-04-21T04:52:47.803281945Z Wrote inspect data to must-gather.
2026-04-21T04:52:47.809066360Z INFO: 'previous-logs' folder not found on ip-10-0-135-122.ec2.internal, skipping...
2026-04-21T04:52:47.921273941Z Wrote inspect data to must-gather.
2026-04-21T04:52:47.932359527Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T04:52:47.979083299Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:52:48.002874808Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.008868959Z tar: Removing leading `/' from member names
2026-04-21T04:52:48.016890567Z INFO: HAProxy config collection complete.
2026-04-21T04:52:48.080474970Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.149671516Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.230227155Z INFO: 'previous-logs' folder not found on ip-10-0-140-11.ec2.internal, skipping...
2026-04-21T04:52:48.405934477Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T04:52:48.491731974Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.503467829Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T04:52:48.569186159Z Gathering data for ns/kuadrant-system...
2026-04-21T04:52:48.598259324Z INFO: 'previous-logs' folder not found on ip-10-0-141-241.ec2.internal, skipping...
2026-04-21T04:52:48.598371784Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T04:52:48.598423103Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T04:52:48.721975253Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.816392377Z Wrote inspect data to must-gather.
2026-04-21T04:52:48.949969957Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T04:52:48.989144915Z Wrote inspect data to must-gather.
2026-04-21T04:52:49.133535442Z Wrote inspect data to must-gather.
2026-04-21T04:52:49.177935738Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T04:52:49.499090898Z Wrote inspect data to must-gather.
2026-04-21T04:52:49.506675372Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T04:52:49.525694387Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T04:52:49.562655833Z INFO: INTERCONNECT MODE
2026-04-21T04:52:49.562696306Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T04:52:49.748115788Z INFO: Gathering OVN_Northbound from ovnkube-node-dv4qj...
2026-04-21T04:52:49.748467958Z INFO: Gathering OVN_Northbound from ovnkube-node-jhdlv...
2026-04-21T04:52:49.748746682Z INFO: Gathering OVN_Northbound from ovnkube-node-tvl5z...
2026-04-21T04:52:49.749011714Z INFO: Gathering OVN_Southbound from ovnkube-node-dv4qj...
2026-04-21T04:52:49.749288271Z INFO: Gathering OVN_Southbound from ovnkube-node-jhdlv...
2026-04-21T04:52:49.749662205Z INFO: Gathering OVN_Southbound from ovnkube-node-tvl5z...
2026-04-21T04:52:49.838019913Z Wrote inspect data to must-gather.
2026-04-21T04:52:49.840050934Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T04:52:50.059441713Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.083847752Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.100433206Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.134711933Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.287235379Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.305554547Z tar: Removing leading `/' from member names
2026-04-21T04:52:50.305895506Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T04:52:50.390719688Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T04:52:50.426308825Z Wrote inspect data to must-gather.
2026-04-21T04:52:50.567744708Z Gathering data for ns/openshift-monitoring...
2026-04-21T04:52:50.733718653Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T04:52:50.848804273Z Wrote inspect data to must-gather.
2026-04-21T04:52:50.859693653Z INFO: Waiting for network log collection to complete ...
2026-04-21T04:52:50.859731862Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T04:52:50.860117064Z INFO: Copying ovnk databases complete.
2026-04-21T04:52:50.861875452Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-21T04:52:50.864168905Z ovnk_database_store/
2026-04-21T04:52:50.864168905Z ovnk_database_store/ovnkube-node-jhdlv_nbdb
2026-04-21T04:52:50.876282833Z ovnk_database_store/ovnkube-node-dv4qj_nbdb
2026-04-21T04:52:50.887450960Z ovnk_database_store/ovnkube-node-jhdlv_sbdb
2026-04-21T04:52:50.913075913Z ovnk_database_store/ovnkube-node-tvl5z_sbdb
2026-04-21T04:52:50.933744082Z ovnk_database_store/ovnkube-node-dv4qj_sbdb
2026-04-21T04:52:50.958033211Z ovnk_database_store/ovnkube-node-tvl5z_nbdb
2026-04-21T04:52:50.970395136Z INFO: Network log collection complete.
2026-04-21T04:52:51.156813078Z Wrote inspect data to must-gather.
2026-04-21T04:52:51.162893579Z INFO: Getting status from alertmanager-main-0
2026-04-21T04:52:51.476662138Z Wrote inspect data to must-gather.
2026-04-21T04:52:51.761142120Z Wrote inspect data to must-gather.
2026-04-21T04:52:52.066324971Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T04:52:52.068710210Z INFO: Fallback to identify the container image from release info
2026-04-21T04:52:52.696431114Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T04:52:52.848118834Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T04:52:52.966313969Z Gathering data for ns/openshift-network-console...
2026-04-21T04:52:52.989648768Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:52:52.994020341Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T04:52:53.543163739Z Gathering data for ns/openshift-console-operator...
2026-04-21T04:52:53.985440550Z Gathering data for ns/openshift-console...
2026-04-21T04:52:54.146060876Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T04:52:54.532687161Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T04:52:54.998002835Z Gathering data for ns/openshift-dns-operator...
2026-04-21T04:52:55.087677854Z Wrote inspect data to must-gather.
2026-04-21T04:52:55.276046030Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T04:52:55.353791660Z Gathering data for ns/openshift-dns...
2026-04-21T04:52:56.004439976Z Gathering data for ns/openshift-image-registry...
2026-04-21T04:52:56.399149333Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T04:52:56.509459928Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T04:52:56.901048484Z Gathering data for ns/openshift-ingress...
2026-04-21T04:52:57.523365329Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T04:52:57.685988129Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T04:52:57.995177589Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:52:58.241765488Z Gathering data for ns/openshift-insights...
2026-04-21T04:52:58.654331663Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T04:52:59.766773457Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T04:53:00.424313352Z Gathering data for ns/opendatahub...
2026-04-21T04:53:00.876100743Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T04:53:01.138379084Z Collecting performance related data for node ip-10-0-135-122.ec2.internal
2026-04-21T04:53:01.145177081Z Collecting performance related data for node ip-10-0-141-241.ec2.internal
2026-04-21T04:53:01.155399032Z Collecting performance related data for node ip-10-0-140-11.ec2.internal
2026-04-21T04:53:02.080672237Z Gathering data for ns/openshift-lws-operator...
2026-04-21T04:53:02.999706627Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:53:03.473296082Z Gathering data for ns/openshift-config...
2026-04-21T04:53:03.823275188Z Gathering data for ns/openshift-config-managed...
2026-04-21T04:53:04.390012195Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T04:53:04.730924160Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T04:53:05.043069970Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T04:53:05.419534867Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T04:53:05.799199436Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T04:53:06.132875229Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T04:53:06.512745783Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T04:53:06.909002768Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T04:53:07.292645563Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T04:53:07.895163254Z Gathering data for ns/openshift-multus...
2026-04-21T04:53:08.004084680Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:53:09.396059813Z Collecting kubelet logs for node ip-10-0-135-122.ec2.internal
2026-04-21T04:53:09.397794262Z Collecting kubelet logs for node ip-10-0-141-241.ec2.internal
2026-04-21T04:53:09.399719522Z Collecting kubelet logs for node ip-10-0-140-11.ec2.internal
2026-04-21T04:53:09.669949040Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T04:53:09.751574439Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T04:53:09.756557373Z INFO: Node performance data collection complete.
2026-04-21T04:53:11.298832601Z Gathering data for ns/openshift-host-network...
2026-04-21T04:53:11.701134681Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T04:53:12.354285705Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T04:53:12.887503002Z Gathering data for ns/openshift-network-operator...
2026-04-21T04:53:13.011173804Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:53:13.330331142Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T04:53:13.704399166Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T04:53:14.232917974Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T04:53:14.632124427Z Gathering data for ns/openshift-apiserver...
2026-04-21T04:53:15.149403881Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T04:53:15.455034915Z Gathering data for ns/openshift-controller-manager...
2026-04-21T04:53:15.793068231Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T04:53:16.608813207Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T04:53:16.979814717Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T04:53:17.377753710Z Gathering data for ns/openshift-service-ca...
2026-04-21T04:53:17.791863239Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T04:53:18.016189811Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T04:53:18.846264853Z Wrote inspect data to must-gather.
2026-04-21T04:53:18.846309851Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T04:53:18.846309851Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T04:53:18.976568137Z error: the server doesn't have a resource type "clusters"
2026-04-21T04:53:19.145570467Z Caches written to disk