2026-04-21T16:37:52.200555181Z [disk usage checker] Started
2026-04-21T16:37:52.236442465Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:37:52.432761207Z Gathering data for ns/openshift-cluster-version...
2026-04-21T16:37:52.616473506Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T16:37:52.752298954Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T16:37:52.855320023Z Gathering data for ns/default...
2026-04-21T16:37:53.241959881Z Gathering data for ns/openshift...
2026-04-21T16:37:53.840083571Z Gathering data for ns/kube-system...
2026-04-21T16:37:54.644447595Z Waiting on subprocesses to finish execution.
2026-04-21T16:37:54.646894925Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T16:37:54.648631555Z INFO: Gathering HAProxy config files
2026-04-21T16:37:54.661144571Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T16:37:54.680031193Z INFO: Collecting host service logs for crio
2026-04-21T16:37:54.680384115Z INFO: Collecting host service logs for kubelet
2026-04-21T16:37:54.680711652Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T16:37:54.681031903Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T16:37:54.681120998Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T16:37:54.681321388Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T16:37:54.681644911Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T16:37:54.681967611Z INFO: Collecting host service logs for NetworkManager
2026-04-21T16:37:54.682284538Z INFO: Collecting host service logs for openvswitch
2026-04-21T16:37:54.682604554Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T16:37:54.682917092Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T16:37:54.683212759Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T16:37:54.683568633Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T16:37:54.848555851Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T16:37:55.275798110Z Gathering data for ns/openshift-etcd...
2026-04-21T16:37:55.442949862Z No resources found
2026-04-21T16:37:55.902824033Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T16:37:55.918275867Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T16:37:56.141923561Z No resources found
2026-04-21T16:37:56.174409754Z No resources found in openshift-etcd namespace.
2026-04-21T16:37:56.325513579Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T16:37:56.368927357Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T16:37:56.412014175Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T16:37:56.412044277Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T16:37:56.543923830Z Wrote inspect data to must-gather.
2026-04-21T16:37:56.546945350Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T16:37:56.558497285Z INFO: Worker host service log collection to complete.
2026-04-21T16:37:56.670756836Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T16:37:56.675675905Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-r7jtg
2026-04-21T16:37:56.675721246Z insights-runtime-extractor-h4qvj
2026-04-21T16:37:56.675740799Z insights-runtime-extractor-rtv56
2026-04-21T16:37:56.675759753Z insights-runtime-extractor-vfgtx
2026-04-21T16:37:56.704490563Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T16:37:57.080745034Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.088150260Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T16:37:57.100762517Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T16:37:57.101778142Z ERROR: No running kube-apiserver pods found
2026-04-21T16:37:57.170485522Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T16:37:57.170696987Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.170741421Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T16:37:57.170741421Z     namespaces "assisted-installer" not found
2026-04-21T16:37:57.259093389Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T16:37:57.259093389Z See 'oc rsync -h' for help and examples
2026-04-21T16:37:57.345301643Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:37:57.430469155Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.538457189Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.593371124Z tar: Removing leading `/' from member names
2026-04-21T16:37:57.598575524Z INFO: HAProxy config collection complete.
2026-04-21T16:37:57.629289811Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.690486215Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.795682650Z Wrote inspect data to must-gather.
2026-04-21T16:37:57.803002725Z INFO: 'previous-logs' folder not found on ip-10-0-129-96.ec2.internal, skipping...
2026-04-21T16:37:57.872717523Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T16:37:57.877907618Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T16:37:58.018638342Z Gathering data for ns/kuadrant-system...
2026-04-21T16:37:58.174207950Z INFO: 'previous-logs' folder not found on ip-10-0-138-191.ec2.internal, skipping...
2026-04-21T16:37:58.190959189Z Wrote inspect data to must-gather.
2026-04-21T16:37:58.206242643Z Wrote inspect data to must-gather.
2026-04-21T16:37:58.360829579Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T16:37:58.500609639Z INFO: 'previous-logs' folder not found on ip-10-0-142-158.ec2.internal, skipping...
2026-04-21T16:37:58.500715172Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T16:37:58.500771635Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T16:37:58.528000621Z Wrote inspect data to must-gather.
2026-04-21T16:37:58.543024400Z Wrote inspect data to must-gather.
2026-04-21T16:37:58.733533912Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T16:37:58.859313855Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T16:37:58.864676739Z INFO: INTERCONNECT MODE
2026-04-21T16:37:58.864708517Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T16:37:59.001123221Z Wrote inspect data to must-gather.
2026-04-21T16:37:59.068355564Z INFO: Gathering OVN_Northbound from ovnkube-node-6rkhl...
2026-04-21T16:37:59.068607056Z INFO: Gathering OVN_Northbound from ovnkube-node-7dm6j...
2026-04-21T16:37:59.068908614Z INFO: Gathering OVN_Northbound from ovnkube-node-np85v...
2026-04-21T16:37:59.069224184Z INFO: Gathering OVN_Southbound from ovnkube-node-6rkhl...
2026-04-21T16:37:59.069489166Z INFO: Gathering OVN_Southbound from ovnkube-node-7dm6j...
2026-04-21T16:37:59.069744199Z INFO: Gathering OVN_Southbound from ovnkube-node-np85v...
2026-04-21T16:37:59.163178755Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T16:37:59.398980292Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T16:37:59.473990015Z Wrote inspect data to must-gather.
2026-04-21T16:37:59.632352291Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.666059922Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.680404111Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.689975283Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.745306914Z Wrote inspect data to must-gather.
2026-04-21T16:37:59.800977916Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.807203213Z tar: Removing leading `/' from member names
2026-04-21T16:37:59.856224081Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T16:38:00.114929934Z INFO: Waiting for network log collection to complete ...
2026-04-21T16:38:00.115017276Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T16:38:00.115433973Z INFO: Copying ovnk databases complete.
2026-04-21T16:38:00.117053921Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-21T16:38:00.119425707Z ovnk_database_store/
2026-04-21T16:38:00.119425707Z ovnk_database_store/ovnkube-node-7dm6j_sbdb
2026-04-21T16:38:00.120970041Z Wrote inspect data to must-gather.
2026-04-21T16:38:00.149415124Z ovnk_database_store/ovnkube-node-7dm6j_nbdb
2026-04-21T16:38:00.163045445Z ovnk_database_store/ovnkube-node-6rkhl_nbdb
2026-04-21T16:38:00.170446922Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T16:38:00.172897304Z ovnk_database_store/ovnkube-node-6rkhl_sbdb
2026-04-21T16:38:00.196428631Z ovnk_database_store/ovnkube-node-np85v_sbdb
2026-04-21T16:38:00.223192976Z ovnk_database_store/ovnkube-node-np85v_nbdb
2026-04-21T16:38:00.236107118Z INFO: Network log collection complete.
2026-04-21T16:38:00.244137305Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T16:38:00.474364935Z Gathering data for ns/openshift-monitoring...
2026-04-21T16:38:00.482649010Z Wrote inspect data to must-gather.
2026-04-21T16:38:00.671274767Z INFO: Getting status from alertmanager-main-0
2026-04-21T16:38:00.807724794Z Wrote inspect data to must-gather.
2026-04-21T16:38:01.086322469Z Wrote inspect data to must-gather.
2026-04-21T16:38:01.336198541Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T16:38:01.338396654Z INFO: Fallback to identify the container image from release info
2026-04-21T16:38:01.987641917Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T16:38:02.105666203Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T16:38:02.242005576Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T16:38:02.355461111Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:02.864227774Z Gathering data for ns/openshift-network-console...
2026-04-21T16:38:03.389937710Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T16:38:03.496279932Z Gathering data for ns/openshift-console-operator...
2026-04-21T16:38:03.801189773Z Wrote inspect data to must-gather.
2026-04-21T16:38:03.970946088Z Gathering data for ns/openshift-console...
2026-04-21T16:38:04.521862348Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T16:38:04.541564797Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T16:38:04.945400818Z Gathering data for ns/openshift-dns-operator...
2026-04-21T16:38:05.289464824Z Gathering data for ns/openshift-dns...
2026-04-21T16:38:05.645499588Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T16:38:05.917436108Z Gathering data for ns/openshift-image-registry...
2026-04-21T16:38:06.420995735Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T16:38:06.769232616Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T16:38:06.816581153Z Gathering data for ns/openshift-ingress...
2026-04-21T16:38:07.360333965Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:07.627378909Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T16:38:07.886003725Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T16:38:08.093043355Z Gathering data for ns/openshift-insights...
2026-04-21T16:38:09.002987286Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T16:38:10.117340579Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T16:38:10.133945099Z Gathering data for ns/opendatahub...
2026-04-21T16:38:10.367933510Z Collecting performance related data for node ip-10-0-138-191.ec2.internal
2026-04-21T16:38:10.374688098Z Collecting performance related data for node ip-10-0-142-158.ec2.internal
2026-04-21T16:38:10.382758604Z Collecting performance related data for node ip-10-0-129-96.ec2.internal
2026-04-21T16:38:11.986623835Z Gathering data for ns/openshift-lws-operator...
2026-04-21T16:38:12.366475435Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:13.313548994Z Gathering data for ns/openshift-config...
2026-04-21T16:38:13.741433912Z Gathering data for ns/openshift-config-managed...
2026-04-21T16:38:14.259745827Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T16:38:14.587608173Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T16:38:14.933487032Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T16:38:15.315146030Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T16:38:15.704268521Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T16:38:16.068883861Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T16:38:16.478893639Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T16:38:16.865251555Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T16:38:17.232720661Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T16:38:17.371277834Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:17.838161790Z Gathering data for ns/openshift-multus...
2026-04-21T16:38:18.363719882Z Collecting kubelet logs for node ip-10-0-138-191.ec2.internal
2026-04-21T16:38:18.365411326Z Collecting kubelet logs for node ip-10-0-142-158.ec2.internal
2026-04-21T16:38:18.367122539Z Collecting kubelet logs for node ip-10-0-129-96.ec2.internal
2026-04-21T16:38:18.701622786Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T16:38:18.710359748Z INFO: Node performance data collection complete.
2026-04-21T16:38:19.631111101Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T16:38:21.835285612Z Gathering data for ns/openshift-host-network...
2026-04-21T16:38:22.376256147Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:22.393504572Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T16:38:23.285502891Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T16:38:23.876410337Z Gathering data for ns/openshift-network-operator...
2026-04-21T16:38:24.430262973Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T16:38:24.898591407Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T16:38:25.625923576Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T16:38:26.048383463Z Gathering data for ns/openshift-apiserver...
2026-04-21T16:38:26.558116131Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T16:38:26.885723641Z Gathering data for ns/openshift-controller-manager...
2026-04-21T16:38:27.193869296Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T16:38:27.380762052Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-21T16:38:27.946609899Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T16:38:28.363550587Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T16:38:28.714852179Z Gathering data for ns/openshift-service-ca...
2026-04-21T16:38:29.118697321Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T16:38:30.273113121Z Wrote inspect data to must-gather.
2026-04-21T16:38:30.273156495Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T16:38:30.273156495Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T16:38:30.403512234Z error: the server doesn't have a resource type "clusters"
2026-04-21T16:38:30.580599171Z Caches written to disk