2026-04-20T20:53:06.605128657Z [disk usage checker] Started
2026-04-20T20:53:06.608425124Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:06.819402017Z Gathering data for ns/openshift-cluster-version...
2026-04-20T20:53:06.956760096Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T20:53:07.035128199Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T20:53:07.115389541Z Gathering data for ns/default...
2026-04-20T20:53:07.428713734Z Gathering data for ns/openshift...
2026-04-20T20:53:07.823034072Z Gathering data for ns/kube-system...
2026-04-20T20:53:08.496473896Z Gathering data for ns/openshift-etcd...
2026-04-20T20:53:08.788465432Z Wrote inspect data to must-gather.
2026-04-20T20:53:08.788498913Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T20:53:08.788498913Z     namespaces "assisted-installer" not found
2026-04-20T20:53:09.000011436Z Waiting on subprocesses to finish execution.
2026-04-20T20:53:09.004293210Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-20T20:53:09.006332447Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-20T20:53:09.008347128Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-20T20:53:09.009778079Z INFO: Gathering HAProxy config files
2026-04-20T20:53:09.044807683Z INFO: Collecting host service logs for crio
2026-04-20T20:53:09.045146696Z INFO: Collecting host service logs for kubelet
2026-04-20T20:53:09.045746898Z INFO: Collecting host service logs for rpm-ostreed
2026-04-20T20:53:09.046090959Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-20T20:53:09.046438642Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-20T20:53:09.046805585Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-20T20:53:09.047149535Z INFO: Collecting host service logs for NetworkManager
2026-04-20T20:53:09.047499638Z INFO: Collecting host service logs for openvswitch
2026-04-20T20:53:09.047841507Z INFO: Collecting host service logs for ovs-configuration
2026-04-20T20:53:09.048187118Z INFO: Collecting host service logs for ovsdb-server
2026-04-20T20:53:09.048564106Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-20T20:53:09.049014829Z INFO: Waiting for worker host service log collection to complete ...
2026-04-20T20:53:09.111429290Z INFO: Waiting for node performance related collection to complete ...
2026-04-20T20:53:09.384657875Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-20T20:53:09.599881266Z No resources found in openshift-etcd namespace.
2026-04-20T20:53:09.649605671Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-20T20:53:09.649644780Z INFO: on-disk MachineConfig config collection complete.
2026-04-20T20:53:09.824839346Z No resources found
2026-04-20T20:53:09.835625064Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-20T20:53:09.848809687Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-20T20:53:10.018111981Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-74t9l
2026-04-20T20:53:10.018157805Z insights-runtime-extractor-jpmmm
2026-04-20T20:53:10.018178387Z insights-runtime-extractor-rsxtz
2026-04-20T20:53:10.018200185Z insights-runtime-extractor-zzlvj
2026-04-20T20:53:10.130514513Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-20T20:53:10.203149835Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-20T20:53:10.352685104Z error: the server doesn't have a resource type "performanceprofile"
2026-04-20T20:53:10.380899160Z INFO: "metallb-operator" not detected. Skipping.
2026-04-20T20:53:10.655318661Z INFO: Worker host service log collection to complete.
2026-04-20T20:53:10.669609792Z Wrote inspect data to must-gather.
2026-04-20T20:53:10.710079262Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-20T20:53:10.710079262Z See 'oc rsync -h' for help and examples
2026-04-20T20:53:10.750970252Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T20:53:10.878397092Z Wrote inspect data to must-gather.
2026-04-20T20:53:10.925011513Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.027477018Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-20T20:53:11.046057803Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-20T20:53:11.076533076Z No resources found
2026-04-20T20:53:11.104600690Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.145925680Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.275861358Z tar: Removing leading `/' from member names
2026-04-20T20:53:11.284228955Z INFO: HAProxy config collection complete.
2026-04-20T20:53:11.297499693Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.329717540Z ERROR: No running kube-apiserver pods found
2026-04-20T20:53:11.360438414Z INFO: 'previous-logs' folder not found on ip-10-0-130-227.ec2.internal, skipping...
2026-04-20T20:53:11.381833078Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.464107198Z Gathering data for ns/openshift-monitoring...
2026-04-20T20:53:11.559687870Z error: the server doesn't have a resource type "machineconfigs"
2026-04-20T20:53:11.579184289Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.618415128Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:11.623440962Z INFO: Getting rules from prometheus-k8s-0
2026-04-20T20:53:11.636750249Z INFO: 'previous-logs' folder not found on ip-10-0-131-234.ec2.internal, skipping...
2026-04-20T20:53:11.793479034Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.840448780Z Wrote inspect data to must-gather.
2026-04-20T20:53:11.897985552Z INFO: 'previous-logs' folder not found on ip-10-0-140-19.ec2.internal, skipping...
2026-04-20T20:53:11.898017895Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-20T20:53:11.898017895Z INFO: Machine Config Daemon termination log collection complete.
2026-04-20T20:53:12.027009435Z Wrote inspect data to must-gather.
2026-04-20T20:53:12.033136049Z INFO: Getting status/config from prometheus-k8s-0
2026-04-20T20:53:12.082353855Z Wrote inspect data to must-gather.
2026-04-20T20:53:12.185073631Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-20T20:53:12.358888602Z INFO: INTERCONNECT MODE
2026-04-20T20:53:12.358917798Z INFO: Gathering ovn-kubernetes DBs
2026-04-20T20:53:12.397319236Z Wrote inspect data to must-gather.
2026-04-20T20:53:12.439238052Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-20T20:53:12.552870388Z INFO: Gathering OVN_Northbound from ovnkube-node-j6wvn...
2026-04-20T20:53:12.553136451Z INFO: Gathering OVN_Northbound from ovnkube-node-ks7s9...
2026-04-20T20:53:12.553364863Z INFO: Gathering OVN_Northbound from ovnkube-node-lwd67...
2026-04-20T20:53:12.553651874Z INFO: Gathering OVN_Southbound from ovnkube-node-j6wvn...
2026-04-20T20:53:12.553927178Z INFO: Gathering OVN_Southbound from ovnkube-node-ks7s9...
2026-04-20T20:53:12.554160340Z INFO: Gathering OVN_Southbound from ovnkube-node-lwd67...
2026-04-20T20:53:12.650777805Z Wrote inspect data to must-gather.
2026-04-20T20:53:12.727393832Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-20T20:53:12.843982360Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.850275642Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.856145207Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.857313299Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.879244518Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.914761476Z tar: Removing leading `/' from member names
2026-04-20T20:53:12.938679940Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T20:53:12.983104199Z Wrote inspect data to must-gather.
2026-04-20T20:53:13.016423706Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-20T20:53:13.221985592Z Wrote inspect data to must-gather.
2026-04-20T20:53:13.280511323Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-20T20:53:13.346034518Z INFO: Waiting for network log collection to complete ...
2026-04-20T20:53:13.346064042Z INFO: Waiting for ovnk database copies to complete ...
2026-04-20T20:53:13.346477067Z INFO: Copying ovnk databases complete.
2026-04-20T20:53:13.348402877Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-04-20T20:53:13.353992403Z ovnk_database_store/
2026-04-20T20:53:13.354043246Z ovnk_database_store/ovnkube-node-j6wvn_nbdb
2026-04-20T20:53:13.366685010Z ovnk_database_store/ovnkube-node-ks7s9_nbdb
2026-04-20T20:53:13.377424790Z ovnk_database_store/ovnkube-node-lwd67_nbdb
2026-04-20T20:53:13.385483663Z ovnk_database_store/ovnkube-node-j6wvn_sbdb
2026-04-20T20:53:13.408125704Z ovnk_database_store/ovnkube-node-ks7s9_sbdb
2026-04-20T20:53:13.431544687Z ovnk_database_store/ovnkube-node-lwd67_sbdb
2026-04-20T20:53:13.445809673Z W0420 20:53:13.445768    1365 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-20T20:53:13.450750933Z Wrote inspect data to must-gather.
2026-04-20T20:53:13.452658159Z INFO: Network log collection complete.
2026-04-20T20:53:13.534867968Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T20:53:13.683717556Z INFO: Getting status from alertmanager-main-0
2026-04-20T20:53:13.703186875Z Gathering data for ns/openshift-network-console...
2026-04-20T20:53:13.731950033Z Wrote inspect data to must-gather.
2026-04-20T20:53:14.031441319Z Wrote inspect data to must-gather.
2026-04-20T20:53:14.109514784Z Gathering data for ns/openshift-console-operator...
2026-04-20T20:53:14.293441698Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-20T20:53:14.295515784Z INFO: Fallback to identify the container image from release info
2026-04-20T20:53:14.464164428Z Gathering data for ns/openshift-console...
2026-04-20T20:53:14.824796945Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-20T20:53:14.858100680Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-20T20:53:14.960876710Z daemonset.apps/perf-node-gather-daemonset created
2026-04-20T20:53:15.081790645Z Waiting for performance profile collector pods to become ready: 1
2026-04-20T20:53:15.214630167Z Gathering data for ns/openshift-dns-operator...
2026-04-20T20:53:15.524771315Z Gathering data for ns/openshift-dns...
2026-04-20T20:53:16.055683376Z Gathering data for ns/openshift-image-registry...
2026-04-20T20:53:16.203872227Z Waiting for performance profile collector pods to become ready: 2
2026-04-20T20:53:16.488321675Z Gathering data for ns/openshift-ingress-operator...
2026-04-20T20:53:16.623270554Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:16.841737392Z Gathering data for ns/openshift-ingress...
2026-04-20T20:53:17.153855933Z Gathering data for ns/openshift-ingress-canary...
2026-04-20T20:53:17.332599553Z Waiting for performance profile collector pods to become ready: 3
2026-04-20T20:53:17.522731876Z Gathering data for ns/openshift-insights...
2026-04-20T20:53:18.447618491Z Waiting for performance profile collector pods to become ready: 4
2026-04-20T20:53:19.502942094Z Gathering data for ns/kserve...
2026-04-20T20:53:19.560541247Z Waiting for performance profile collector pods to become ready: 5
2026-04-20T20:53:20.673135700Z Waiting for performance profile collector pods to become ready: 6
2026-04-20T20:53:21.154042758Z Gathering data for ns/openshift-config...
2026-04-20T20:53:21.440700120Z Gathering data for ns/openshift-config-managed...
2026-04-20T20:53:21.627681061Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:21.804885247Z Waiting for performance profile collector pods to become ready: 7
2026-04-20T20:53:21.840957859Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-20T20:53:22.086411427Z Gathering data for ns/openshift-kube-apiserver...
2026-04-20T20:53:22.362066025Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-20T20:53:22.608576968Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-20T20:53:22.869313214Z Gathering data for ns/openshift-kube-scheduler...
2026-04-20T20:53:22.943582845Z Waiting for performance profile collector pods to become ready: 8
2026-04-20T20:53:23.119236241Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-20T20:53:23.397656354Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-20T20:53:23.683017355Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-20T20:53:23.956039084Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-20T20:53:24.072628805Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-20T20:53:24.347579700Z Collecting performance related data for node ip-10-0-130-227.ec2.internal
2026-04-20T20:53:24.359029554Z Collecting performance related data for node ip-10-0-140-19.ec2.internal
2026-04-20T20:53:24.366119968Z Collecting performance related data for node ip-10-0-131-234.ec2.internal
2026-04-20T20:53:24.510547745Z Gathering data for ns/openshift-multus...
2026-04-20T20:53:26.031378241Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-20T20:53:26.634505469Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:27.572733633Z Gathering data for ns/openshift-host-network...
2026-04-20T20:53:27.846328993Z Gathering data for ns/openshift-network-diagnostics...
2026-04-20T20:53:28.321844631Z Gathering data for ns/openshift-network-node-identity...
2026-04-20T20:53:28.697729403Z Gathering data for ns/openshift-network-operator...
2026-04-20T20:53:29.064336971Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-20T20:53:29.308681897Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-20T20:53:29.753323849Z Gathering data for ns/openshift-apiserver-operator...
2026-04-20T20:53:30.034883313Z Gathering data for ns/openshift-apiserver...
2026-04-20T20:53:30.435526962Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-20T20:53:30.679361005Z Gathering data for ns/openshift-controller-manager...
2026-04-20T20:53:30.921419743Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-20T20:53:31.559365150Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-20T20:53:31.639376783Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T20:53:31.817193404Z Gathering data for ns/openshift-service-ca-operator...
2026-04-20T20:53:32.101170479Z Gathering data for ns/openshift-service-ca...
2026-04-20T20:53:32.441761409Z Collecting kubelet logs for node ip-10-0-130-227.ec2.internal
2026-04-20T20:53:32.443453453Z Collecting kubelet logs for node ip-10-0-140-19.ec2.internal
2026-04-20T20:53:32.443607415Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-20T20:53:32.445160833Z Collecting kubelet logs for node ip-10-0-131-234.ec2.internal
2026-04-20T20:53:32.787386689Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-20T20:53:32.792072630Z INFO: Node performance data collection complete.
2026-04-20T20:53:33.340985896Z Wrote inspect data to must-gather.
2026-04-20T20:53:33.341025018Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T20:53:33.341025018Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-20T20:53:33.464939716Z error: the server doesn't have a resource type "clusters"
2026-04-20T20:53:33.659965559Z Caches written to disk