2026-04-16T18:41:07.913360412Z [disk usage checker] Started
2026-04-16T18:41:07.943146440Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:08.128756481Z Gathering data for ns/openshift-cluster-version...
2026-04-16T18:41:08.267999167Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T18:41:08.360967839Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T18:41:08.447127995Z Gathering data for ns/default...
2026-04-16T18:41:08.755047201Z Gathering data for ns/openshift...
2026-04-16T18:41:09.208565575Z Gathering data for ns/kube-system...
2026-04-16T18:41:09.938307360Z Gathering data for ns/openshift-etcd...
2026-04-16T18:41:10.254215462Z Wrote inspect data to must-gather.
2026-04-16T18:41:10.254289846Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T18:41:10.254289846Z     namespaces "assisted-installer" not found
2026-04-16T18:41:10.407418950Z Waiting on subprocesses to finish execution.
2026-04-16T18:41:10.411263770Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T18:41:10.417814647Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T18:41:10.429780191Z INFO: Gathering HAProxy config files
2026-04-16T18:41:10.435011390Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T18:41:10.443181676Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T18:41:10.477846573Z INFO: Collecting host service logs for crio
2026-04-16T18:41:10.478136075Z INFO: Collecting host service logs for kubelet
2026-04-16T18:41:10.478456272Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T18:41:10.478739296Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T18:41:10.479025759Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T18:41:10.479360030Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T18:41:10.479662165Z INFO: Collecting host service logs for NetworkManager
2026-04-16T18:41:10.479943814Z INFO: Collecting host service logs for openvswitch
2026-04-16T18:41:10.480270677Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T18:41:10.480570881Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T18:41:10.480863399Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T18:41:10.481223203Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T18:41:10.827401236Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T18:41:10.966047906Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T18:41:11.028779099Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T18:41:11.080078528Z INFO: Collecting Insights Archives from insights-operator-5785d4fcdd-5hlbs
2026-04-16T18:41:11.080127233Z insights-runtime-extractor-9z848
2026-04-16T18:41:11.080149137Z insights-runtime-extractor-dn8ld
2026-04-16T18:41:11.080169370Z insights-runtime-extractor-l8qct
2026-04-16T18:41:11.328757105Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T18:41:11.579029435Z No resources found in openshift-etcd namespace.
2026-04-16T18:41:11.724547489Z Wrote inspect data to must-gather.
2026-04-16T18:41:11.789846592Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T18:41:11.789846592Z See 'oc rsync -h' for help and examples
2026-04-16T18:41:11.794825784Z No resources found
2026-04-16T18:41:12.243250524Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T18:41:12.257887588Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T18:41:12.259077807Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T18:41:12.283413050Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T18:41:12.309451575Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T18:41:12.392102770Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.407059037Z No resources found
2026-04-16T18:41:12.423270785Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.469569764Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.484990889Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.487550359Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T18:41:12.562600529Z INFO: Worker host service log collection to complete.
2026-04-16T18:41:12.600702317Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T18:41:12.600792803Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T18:41:12.689009564Z ERROR: No running kube-apiserver pods found
2026-04-16T18:41:12.801887364Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.805985181Z tar: Removing leading `/' from member names
2026-04-16T18:41:12.813318171Z INFO: HAProxy config collection complete.
2026-04-16T18:41:12.859141224Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.884433897Z Wrote inspect data to must-gather.
2026-04-16T18:41:12.888315553Z INFO: 'previous-logs' folder not found on ip-10-0-128-68.ec2.internal, skipping...
2026-04-16T18:41:12.925293154Z Gathering data for ns/openshift-monitoring...
2026-04-16T18:41:12.951137732Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:13.005546238Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T18:41:13.046876745Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T18:41:13.157103004Z INFO: 'previous-logs' folder not found on ip-10-0-131-203.ec2.internal, skipping...
2026-04-16T18:41:13.168132784Z Wrote inspect data to must-gather.
2026-04-16T18:41:13.263710213Z Wrote inspect data to must-gather.
2026-04-16T18:41:13.416648208Z INFO: 'previous-logs' folder not found on ip-10-0-141-219.ec2.internal, skipping...
2026-04-16T18:41:13.416677931Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T18:41:13.416694945Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T18:41:13.429090655Z Wrote inspect data to must-gather.
2026-04-16T18:41:13.436873856Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T18:41:13.503345330Z Wrote inspect data to must-gather.
2026-04-16T18:41:13.655022721Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T18:41:13.712925609Z INFO: INTERCONNECT MODE
2026-04-16T18:41:13.712958406Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T18:41:13.844386250Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T18:41:13.863937982Z Wrote inspect data to must-gather.
2026-04-16T18:41:13.871209015Z INFO: Gathering OVN_Northbound from ovnkube-node-hls95...
2026-04-16T18:41:13.871439981Z INFO: Gathering OVN_Northbound from ovnkube-node-tj7xv...
2026-04-16T18:41:13.871640400Z INFO: Gathering OVN_Northbound from ovnkube-node-tttq4...
2026-04-16T18:41:13.871927567Z INFO: Gathering OVN_Southbound from ovnkube-node-hls95...
2026-04-16T18:41:13.872161851Z INFO: Gathering OVN_Southbound from ovnkube-node-tj7xv...
2026-04-16T18:41:13.872450562Z INFO: Gathering OVN_Southbound from ovnkube-node-tttq4...
2026-04-16T18:41:14.143048528Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.152543909Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.159792021Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.171214726Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T18:41:14.175860261Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.191476951Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.194450997Z tar: Removing leading `/' from member names
2026-04-16T18:41:14.194945248Z Wrote inspect data to must-gather.
2026-04-16T18:41:14.350104979Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T18:41:14.423703107Z Wrote inspect data to must-gather.
2026-04-16T18:41:14.427674912Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T18:41:14.675276559Z Wrote inspect data to must-gather.
2026-04-16T18:41:14.690875662Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T18:41:14.719914288Z INFO: Waiting for network log collection to complete ...
2026-04-16T18:41:14.719956954Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T18:41:14.720337792Z INFO: Copying ovnk databases complete.
2026-04-16T18:41:14.721957794Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-04-16T18:41:14.724786943Z ovnk_database_store/
2026-04-16T18:41:14.724786943Z ovnk_database_store/ovnkube-node-tj7xv_nbdb
2026-04-16T18:41:14.738068281Z ovnk_database_store/ovnkube-node-hls95_sbdb
2026-04-16T18:41:14.761570128Z ovnk_database_store/ovnkube-node-tttq4_sbdb
2026-04-16T18:41:14.784947705Z ovnk_database_store/ovnkube-node-tj7xv_sbdb
2026-04-16T18:41:14.806608327Z ovnk_database_store/ovnkube-node-tttq4_nbdb
2026-04-16T18:41:14.815930710Z ovnk_database_store/ovnkube-node-hls95_nbdb
2026-04-16T18:41:14.827767825Z INFO: Network log collection complete.
2026-04-16T18:41:14.898818610Z W0416 18:41:14.898772    1359 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T18:41:14.903821554Z Wrote inspect data to must-gather.
2026-04-16T18:41:14.983736810Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T18:41:15.089009728Z INFO: Getting status from alertmanager-main-0
2026-04-16T18:41:15.162666259Z Gathering data for ns/openshift-network-console...
2026-04-16T18:41:15.196620407Z Wrote inspect data to must-gather.
2026-04-16T18:41:15.480841922Z Wrote inspect data to must-gather.
2026-04-16T18:41:15.564062860Z Gathering data for ns/openshift-console-operator...
2026-04-16T18:41:15.745472964Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T18:41:15.747839881Z INFO: Fallback to identify the container image from release info
2026-04-16T18:41:15.970152683Z Gathering data for ns/openshift-console...
2026-04-16T18:41:16.269611299Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc77fc5f1546189d6b57967ccc684e794bd846ea36c3e3c1e61d83a684f0f9b9
2026-04-16T18:41:16.405651365Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T18:41:16.405968278Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T18:41:16.541397384Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T18:41:16.771010408Z Gathering data for ns/openshift-dns-operator...
2026-04-16T18:41:17.101443915Z Gathering data for ns/openshift-dns...
2026-04-16T18:41:17.678343645Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T18:41:17.751047001Z Gathering data for ns/openshift-image-registry...
2026-04-16T18:41:17.955966628Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:18.190079923Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T18:41:18.509786816Z Gathering data for ns/openshift-ingress...
2026-04-16T18:41:18.812918845Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T18:41:18.863827702Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T18:41:19.258206986Z Gathering data for ns/openshift-insights...
2026-04-16T18:41:19.943327475Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T18:41:21.060864505Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T18:41:21.366127937Z Gathering data for ns/kserve...
2026-04-16T18:41:22.192530019Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T18:41:22.779159010Z Gathering data for ns/openshift-config...
2026-04-16T18:41:22.961062517Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:23.087617993Z Gathering data for ns/openshift-config-managed...
2026-04-16T18:41:23.307270053Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T18:41:23.500076194Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T18:41:23.755712162Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T18:41:24.011373915Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T18:41:24.336777225Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T18:41:24.440113372Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T18:41:24.643596139Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T18:41:24.721467915Z Collecting performance related data for node ip-10-0-141-219.ec2.internal
2026-04-16T18:41:24.728251556Z Collecting performance related data for node ip-10-0-131-203.ec2.internal
2026-04-16T18:41:24.735502609Z Collecting performance related data for node ip-10-0-128-68.ec2.internal
2026-04-16T18:41:24.949021829Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T18:41:25.310204110Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T18:41:25.687993251Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T18:41:26.113469382Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T18:41:26.685924109Z Gathering data for ns/openshift-multus...
2026-04-16T18:41:27.968951457Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:28.124856987Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T18:41:29.718716515Z Gathering data for ns/openshift-host-network...
2026-04-16T18:41:30.066025008Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T18:41:30.633854937Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T18:41:31.101914056Z Gathering data for ns/openshift-network-operator...
2026-04-16T18:41:31.480337714Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T18:41:31.756301562Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T18:41:32.290622916Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T18:41:32.600059540Z Gathering data for ns/openshift-apiserver...
2026-04-16T18:41:32.869434451Z Collecting kubelet logs for node ip-10-0-141-219.ec2.internal
2026-04-16T18:41:32.871159190Z Collecting kubelet logs for node ip-10-0-131-203.ec2.internal
2026-04-16T18:41:32.872862702Z Collecting kubelet logs for node ip-10-0-128-68.ec2.internal
2026-04-16T18:41:32.977269462Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-16T18:41:33.081399331Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T18:41:33.254892814Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T18:41:33.260752776Z INFO: Node performance data collection complete.
2026-04-16T18:41:33.393939988Z Gathering data for ns/openshift-controller-manager...
2026-04-16T18:41:33.671383618Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T18:41:34.303833998Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T18:41:34.580088166Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T18:41:34.870325717Z Gathering data for ns/openshift-service-ca...
2026-04-16T18:41:35.235871303Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T18:41:36.101192150Z Wrote inspect data to must-gather.
2026-04-16T18:41:36.101254289Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T18:41:36.101254289Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T18:41:36.229351152Z error: the server doesn't have a resource type "clusters"
2026-04-16T18:41:36.367290810Z Caches written to disk