2026-04-24T23:18:24.179895670Z [disk usage checker] Started
2026-04-24T23:18:24.183102807Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-24T23:18:24.386131736Z Gathering data for ns/openshift-cluster-version...
2026-04-24T23:18:24.527988955Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T23:18:24.617036722Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T23:18:24.700526258Z Gathering data for ns/default...
2026-04-24T23:18:24.991761913Z Gathering data for ns/openshift...
2026-04-24T23:18:25.399264109Z Gathering data for ns/kube-system...
2026-04-24T23:18:26.066490433Z Gathering data for ns/openshift-etcd...
2026-04-24T23:18:26.374256853Z Wrote inspect data to must-gather.
2026-04-24T23:18:26.374286866Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T23:18:26.374286866Z     namespaces "assisted-installer" not found
2026-04-24T23:18:26.587276998Z Waiting on subprocesses to finish execution.
2026-04-24T23:18:26.592777935Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-24T23:18:26.603558715Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-24T23:18:26.608503456Z INFO: Gathering HAProxy config files
2026-04-24T23:18:26.610475659Z INFO: Collecting host service logs for crio
2026-04-24T23:18:26.610782603Z INFO: Collecting host service logs for kubelet
2026-04-24T23:18:26.611117165Z INFO: Collecting host service logs for rpm-ostreed
2026-04-24T23:18:26.611451495Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-24T23:18:26.611806992Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-24T23:18:26.611806992Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-24T23:18:26.612170832Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-24T23:18:26.612513535Z INFO: Collecting host service logs for NetworkManager
2026-04-24T23:18:26.612840067Z INFO: Collecting host service logs for openvswitch
2026-04-24T23:18:26.613195671Z INFO: Collecting host service logs for ovs-configuration
2026-04-24T23:18:26.613545005Z INFO: Collecting host service logs for ovsdb-server
2026-04-24T23:18:26.613884303Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-24T23:18:26.614435354Z INFO: Waiting for worker host service log collection to complete ...
2026-04-24T23:18:26.714920758Z INFO: Waiting for node performance related collection to complete ...
2026-04-24T23:18:27.242998211Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-24T23:18:27.329620293Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-m7xn9
2026-04-24T23:18:27.329667534Z insights-runtime-extractor-l227c
2026-04-24T23:18:27.329686854Z insights-runtime-extractor-mmbbr
2026-04-24T23:18:27.329704602Z insights-runtime-extractor-qmpbb
2026-04-24T23:18:27.465614617Z No resources found in openshift-etcd namespace.
2026-04-24T23:18:27.485846746Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-24T23:18:27.606728153Z No resources found
2026-04-24T23:18:27.706448354Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-24T23:18:27.775580238Z Wrote inspect data to must-gather.
2026-04-24T23:18:28.160723518Z INFO: "metallb-operator" not detected. Skipping.
2026-04-24T23:18:28.191326465Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-24T23:18:28.191429789Z INFO: on-disk MachineConfig config collection complete.
2026-04-24T23:18:28.256878559Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-24T23:18:28.356903695Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-24T23:18:28.477631704Z INFO: Worker host service log collection to complete.
2026-04-24T23:18:28.486266385Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-24T23:18:28.486266385Z See 'oc rsync -h' for help and examples
2026-04-24T23:18:28.512560387Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-24T23:18:28.599490920Z No resources found
2026-04-24T23:18:28.763934975Z ERROR: No running kube-apiserver pods found
2026-04-24T23:18:28.771687753Z error: the server doesn't have a resource type "performanceprofile"
2026-04-24T23:18:28.794441787Z Wrote inspect data to must-gather.
2026-04-24T23:18:28.811227557Z Wrote inspect data to must-gather.
2026-04-24T23:18:28.857390001Z INFO: 'previous-logs' folder not found on ip-10-0-132-138.ec2.internal, skipping...
2026-04-24T23:18:28.881998912Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-24T23:18:28.971892213Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T23:18:28.978279075Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.013875257Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.039268832Z Gathering data for ns/openshift-monitoring...
2026-04-24T23:18:29.193283874Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-24T23:18:29.216076237Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.221112193Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.250650224Z INFO: Getting rules from prometheus-k8s-0
2026-04-24T23:18:29.267529587Z tar: Removing leading `/' from member names
2026-04-24T23:18:29.273259009Z INFO: HAProxy config collection complete.
2026-04-24T23:18:29.274108749Z INFO: 'previous-logs' folder not found on ip-10-0-133-161.ec2.internal, skipping...
2026-04-24T23:18:29.290662173Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.399817107Z error: the server doesn't have a resource type "machineconfigs"
2026-04-24T23:18:29.537450635Z INFO: 'previous-logs' folder not found on ip-10-0-142-202.ec2.internal, skipping...
2026-04-24T23:18:29.537479237Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-24T23:18:29.537488346Z INFO: Machine Config Daemon termination log collection complete.
2026-04-24T23:18:29.553764639Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.636752381Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.670180521Z INFO: Getting status/config from prometheus-k8s-0
2026-04-24T23:18:29.797254692Z Wrote inspect data to must-gather.
2026-04-24T23:18:29.866857639Z Wrote inspect data to must-gather.
2026-04-24T23:18:30.033611378Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-24T23:18:30.078972055Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-24T23:18:30.105471617Z INFO: INTERCONNECT MODE
2026-04-24T23:18:30.105507510Z INFO: Gathering ovn-kubernetes DBs
2026-04-24T23:18:30.246817661Z Wrote inspect data to must-gather.
2026-04-24T23:18:30.259690103Z INFO: Gathering OVN_Northbound from ovnkube-node-9blws...
2026-04-24T23:18:30.259928583Z INFO: Gathering OVN_Northbound from ovnkube-node-b4zlm...
2026-04-24T23:18:30.260214437Z INFO: Gathering OVN_Northbound from ovnkube-node-sm54g...
2026-04-24T23:18:30.260530521Z INFO: Gathering OVN_Southbound from ovnkube-node-9blws...
2026-04-24T23:18:30.260810103Z INFO: Gathering OVN_Southbound from ovnkube-node-b4zlm...
2026-04-24T23:18:30.261123676Z INFO: Gathering OVN_Southbound from ovnkube-node-sm54g...
2026-04-24T23:18:30.363195660Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-24T23:18:30.547564681Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.571941777Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.574472026Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.584658237Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.625199098Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.634571359Z tar: Removing leading `/' from member names
2026-04-24T23:18:30.638323131Z Wrote inspect data to must-gather.
2026-04-24T23:18:30.651293956Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T23:18:30.695517581Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-24T23:18:30.908772477Z Wrote inspect data to must-gather.
2026-04-24T23:18:30.961818139Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-24T23:18:31.143760858Z INFO: Waiting for network log collection to complete ...
2026-04-24T23:18:31.143788588Z INFO: Waiting for ovnk database copies to complete ...
2026-04-24T23:18:31.144228541Z INFO: Copying ovnk databases complete.
2026-04-24T23:18:31.145742135Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-04-24T23:18:31.148025458Z ovnk_database_store/
2026-04-24T23:18:31.148078953Z ovnk_database_store/ovnkube-node-b4zlm_nbdb
2026-04-24T23:18:31.152632552Z W0424 23:18:31.152588    1323 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-24T23:18:31.158143485Z ovnk_database_store/ovnkube-node-sm54g_sbdb
2026-04-24T23:18:31.160110288Z Wrote inspect data to must-gather.
2026-04-24T23:18:31.178049621Z ovnk_database_store/ovnkube-node-sm54g_nbdb
2026-04-24T23:18:31.186682552Z ovnk_database_store/ovnkube-node-9blws_sbdb
2026-04-24T23:18:31.213034476Z ovnk_database_store/ovnkube-node-b4zlm_sbdb
2026-04-24T23:18:31.233383963Z ovnk_database_store/ovnkube-node-9blws_nbdb
2026-04-24T23:18:31.246836572Z INFO: Network log collection complete.
2026-04-24T23:18:31.273177296Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T23:18:31.394688503Z INFO: Getting status from alertmanager-main-0
2026-04-24T23:18:31.455259166Z Gathering data for ns/openshift-network-console...
2026-04-24T23:18:31.462924227Z Wrote inspect data to must-gather.
2026-04-24T23:18:31.733984184Z Wrote inspect data to must-gather.
2026-04-24T23:18:31.930548154Z Gathering data for ns/openshift-console-operator...
2026-04-24T23:18:32.021660149Z Wrote inspect data to must-gather.
2026-04-24T23:18:32.290223501Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-24T23:18:32.292619933Z INFO: Fallback to identify the container image from release info
2026-04-24T23:18:32.311536514Z Gathering data for ns/openshift-console...
2026-04-24T23:18:32.736357860Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-24T23:18:32.848146051Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-24T23:18:32.969256670Z daemonset.apps/perf-node-gather-daemonset created
2026-04-24T23:18:33.101084504Z Gathering data for ns/openshift-dns-operator...
2026-04-24T23:18:33.114576867Z Waiting for performance profile collector pods to become ready: 1
2026-04-24T23:18:33.448439771Z Gathering data for ns/openshift-dns...
2026-04-24T23:18:33.995524602Z Gathering data for ns/openshift-image-registry...
2026-04-24T23:18:34.201088459Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-24T23:18:34.244658809Z Waiting for performance profile collector pods to become ready: 2
2026-04-24T23:18:34.447076855Z Gathering data for ns/openshift-ingress-operator...
2026-04-24T23:18:34.794135880Z Gathering data for ns/openshift-ingress...
2026-04-24T23:18:35.105476538Z Gathering data for ns/openshift-ingress-canary...
2026-04-24T23:18:35.361055561Z Waiting for performance profile collector pods to become ready: 3
2026-04-24T23:18:35.486003443Z Gathering data for ns/openshift-insights...
2026-04-24T23:18:36.478196040Z Waiting for performance profile collector pods to become ready: 4
2026-04-24T23:18:37.596369536Z Waiting for performance profile collector pods to become ready: 5
2026-04-24T23:18:37.638934760Z Gathering data for ns/kserve...
2026-04-24T23:18:38.715579714Z Waiting for performance profile collector pods to become ready: 6
2026-04-24T23:18:39.204346606Z Gathering data for ns/openshift-config...
2026-04-24T23:18:39.205600806Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-24T23:18:39.465409609Z Gathering data for ns/openshift-config-managed...
2026-04-24T23:18:39.843523134Z Waiting for performance profile collector pods to become ready: 7
2026-04-24T23:18:39.891227467Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-24T23:18:40.183934386Z Gathering data for ns/openshift-kube-apiserver...
2026-04-24T23:18:40.433243438Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-24T23:18:40.704734433Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-24T23:18:40.977417228Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-24T23:18:41.007411499Z Gathering data for ns/openshift-kube-scheduler...
2026-04-24T23:18:41.255926157Z Collecting performance related data for node ip-10-0-142-202.ec2.internal
2026-04-24T23:18:41.264726501Z Collecting performance related data for node ip-10-0-133-161.ec2.internal
2026-04-24T23:18:41.270576856Z Collecting performance related data for node ip-10-0-132-138.ec2.internal
2026-04-24T23:18:41.336367191Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-24T23:18:41.708619299Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-24T23:18:42.093635845Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-24T23:18:42.467545155Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-24T23:18:43.051402708Z Gathering data for ns/openshift-multus...
2026-04-24T23:18:44.209892572Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-24T23:18:44.601659513Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-24T23:18:46.104689458Z Gathering data for ns/openshift-host-network...
2026-04-24T23:18:46.411311715Z Gathering data for ns/openshift-network-diagnostics...
2026-04-24T23:18:46.933534906Z Gathering data for ns/openshift-network-node-identity...
2026-04-24T23:18:47.372031660Z Gathering data for ns/openshift-network-operator...
2026-04-24T23:18:47.736621942Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-24T23:18:48.031011966Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-24T23:18:48.496748697Z Gathering data for ns/openshift-apiserver-operator...
2026-04-24T23:18:48.825298257Z Gathering data for ns/openshift-apiserver...
2026-04-24T23:18:49.215774692Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-04-24T23:18:49.221312604Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-24T23:18:49.500174973Z Gathering data for ns/openshift-controller-manager...
2026-04-24T23:18:49.646580626Z Collecting kubelet logs for node ip-10-0-142-202.ec2.internal
2026-04-24T23:18:49.648729922Z Collecting kubelet logs for node ip-10-0-133-161.ec2.internal
2026-04-24T23:18:49.651778035Z Collecting kubelet logs for node ip-10-0-132-138.ec2.internal
2026-04-24T23:18:49.827392391Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-24T23:18:50.036096436Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-24T23:18:50.042153233Z INFO: Node performance data collection complete.
2026-04-24T23:18:50.497672968Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-24T23:18:50.812488615Z Gathering data for ns/openshift-service-ca-operator...
2026-04-24T23:18:51.110588814Z Gathering data for ns/openshift-service-ca...
2026-04-24T23:18:51.475360785Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-24T23:18:52.324292571Z Wrote inspect data to must-gather.
2026-04-24T23:18:52.324576012Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T23:18:52.324576012Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-24T23:18:52.454915840Z error: the server doesn't have a resource type "clusters"
2026-04-24T23:18:52.604765329Z Caches written to disk