2026-04-16T22:40:08.327925984Z [disk usage checker] Started
2026-04-16T22:40:08.361766683Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:08.562122657Z Gathering data for ns/openshift-cluster-version...
2026-04-16T22:40:08.720120806Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:40:08.807362917Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:40:08.908963840Z Gathering data for ns/default...
2026-04-16T22:40:09.261353975Z Gathering data for ns/openshift...
2026-04-16T22:40:09.713917763Z Gathering data for ns/kube-system...
2026-04-16T22:40:10.443614551Z Gathering data for ns/openshift-etcd...
2026-04-16T22:40:10.802454587Z Wrote inspect data to must-gather.
2026-04-16T22:40:10.802483203Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:40:10.802483203Z     namespaces "assisted-installer" not found
2026-04-16T22:40:10.812845810Z Waiting on subprocesses to finish execution.
2026-04-16T22:40:10.825482281Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T22:40:10.827247674Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T22:40:10.829521120Z INFO: Gathering HAProxy config files
2026-04-16T22:40:10.845003861Z INFO: Collecting host service logs for crio
2026-04-16T22:40:10.845501132Z INFO: Collecting host service logs for kubelet
2026-04-16T22:40:10.845901743Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T22:40:10.846294609Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T22:40:10.846712964Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T22:40:10.847087122Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T22:40:10.847459125Z INFO: Collecting host service logs for NetworkManager
2026-04-16T22:40:10.847864336Z INFO: Collecting host service logs for openvswitch
2026-04-16T22:40:10.848257552Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T22:40:10.848672287Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T22:40:10.849064914Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T22:40:10.849585329Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T22:40:10.862206462Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T22:40:11.049737948Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T22:40:11.334572839Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T22:40:11.358075556Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T22:40:11.358164742Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T22:40:11.459779729Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-r49p7
2026-04-16T22:40:11.459779729Z insights-runtime-extractor-4x6xj
2026-04-16T22:40:11.459779729Z insights-runtime-extractor-87bvh
2026-04-16T22:40:11.459779729Z insights-runtime-extractor-tjq9f
2026-04-16T22:40:11.492944597Z No resources found
2026-04-16T22:40:11.704219156Z No resources found in openshift-etcd namespace.
2026-04-16T22:40:11.715764949Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T22:40:11.719851907Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T22:40:11.782908842Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T22:40:12.072410737Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T22:40:12.199160690Z INFO: Worker host service log collection to complete.
2026-04-16T22:40:12.222654418Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T22:40:12.222654418Z See 'oc rsync -h' for help and examples
2026-04-16T22:40:12.335758445Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T22:40:12.453993669Z Wrote inspect data to must-gather.
2026-04-16T22:40:12.461400727Z ERROR: No running kube-apiserver pods found
2026-04-16T22:40:12.746583106Z Wrote inspect data to must-gather.
2026-04-16T22:40:12.796691292Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T22:40:13.017893112Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T22:40:13.382842008Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T22:40:13.410431648Z Wrote inspect data to must-gather.
2026-04-16T22:40:13.457161712Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:13.477118738Z Wrote inspect data to must-gather.
2026-04-16T22:40:13.528900967Z No resources found
2026-04-16T22:40:13.557832206Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:40:13.603123033Z tar: Removing leading `/' from member names
2026-04-16T22:40:13.608838851Z INFO: HAProxy config collection complete.
2026-04-16T22:40:13.645561308Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T22:40:13.754702180Z Wrote inspect data to must-gather.
2026-04-16T22:40:13.780740001Z Gathering data for ns/kuadrant-system...
2026-04-16T22:40:13.851464303Z INFO: 'previous-logs' folder not found on ip-10-0-129-68.ec2.internal, skipping...
2026-04-16T22:40:13.884054295Z Wrote inspect data to must-gather.
2026-04-16T22:40:13.950405519Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T22:40:13.958758744Z Wrote inspect data to must-gather.
2026-04-16T22:40:14.224181512Z Wrote inspect data to must-gather.
2026-04-16T22:40:14.241456744Z INFO: 'previous-logs' folder not found on ip-10-0-130-26.ec2.internal, skipping...
2026-04-16T22:40:14.341936266Z Wrote inspect data to must-gather.
2026-04-16T22:40:14.478505950Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T22:40:14.592989378Z INFO: 'previous-logs' folder not found on ip-10-0-138-154.ec2.internal, skipping...
2026-04-16T22:40:14.593111612Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T22:40:14.593134969Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T22:40:14.600694621Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T22:40:14.676706173Z Wrote inspect data to must-gather.
2026-04-16T22:40:14.944114146Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:40:14.957757888Z Wrote inspect data to must-gather.
2026-04-16T22:40:15.030067314Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T22:40:15.033264539Z Wrote inspect data to must-gather.
2026-04-16T22:40:15.291097827Z Wrote inspect data to must-gather.
2026-04-16T22:40:15.314437119Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T22:40:15.476590967Z INFO: INTERCONNECT MODE
2026-04-16T22:40:15.476677206Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T22:40:15.504760011Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:40:15.616550488Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T22:40:15.699126743Z Wrote inspect data to must-gather.
2026-04-16T22:40:15.700297736Z INFO: Gathering OVN_Northbound from ovnkube-node-25llf...
2026-04-16T22:40:15.700657909Z INFO: Gathering OVN_Northbound from ovnkube-node-4tdqb...
2026-04-16T22:40:15.700938023Z INFO: Gathering OVN_Northbound from ovnkube-node-fw6md...
2026-04-16T22:40:15.701412786Z INFO: Gathering OVN_Southbound from ovnkube-node-25llf...
2026-04-16T22:40:15.701702128Z INFO: Gathering OVN_Southbound from ovnkube-node-4tdqb...
2026-04-16T22:40:15.702027980Z INFO: Gathering OVN_Southbound from ovnkube-node-fw6md...
2026-04-16T22:40:15.771788789Z Gathering data for ns/openshift-monitoring...
2026-04-16T22:40:16.050277068Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.080904020Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.086556638Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T22:40:16.120685344Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.167477429Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.188472165Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.205104821Z tar: Removing leading `/' from member names
2026-04-16T22:40:16.367471208Z Wrote inspect data to must-gather.
2026-04-16T22:40:16.632683016Z INFO: Getting status from alertmanager-main-0
2026-04-16T22:40:16.759023815Z INFO: Waiting for network log collection to complete ...
2026-04-16T22:40:16.759144844Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T22:40:16.759676009Z INFO: Copying ovnk databases complete.
2026-04-16T22:40:16.767662716Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-16T22:40:16.770401199Z ovnk_database_store/
2026-04-16T22:40:16.770401199Z ovnk_database_store/ovnkube-node-4tdqb_nbdb
2026-04-16T22:40:16.785084903Z Wrote inspect data to must-gather.
2026-04-16T22:40:16.786895007Z ovnk_database_store/ovnkube-node-25llf_sbdb
2026-04-16T22:40:16.817113061Z ovnk_database_store/ovnkube-node-25llf_nbdb
2026-04-16T22:40:16.827958483Z ovnk_database_store/ovnkube-node-fw6md_sbdb
2026-04-16T22:40:16.855315747Z ovnk_database_store/ovnkube-node-4tdqb_sbdb
2026-04-16T22:40:16.879897234Z ovnk_database_store/ovnkube-node-fw6md_nbdb
2026-04-16T22:40:16.893239292Z INFO: Network log collection complete.
2026-04-16T22:40:17.060223496Z Wrote inspect data to must-gather.
2026-04-16T22:40:17.373193090Z Wrote inspect data to must-gather.
2026-04-16T22:40:17.648796359Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T22:40:17.655809236Z INFO: Fallback to identify the container image from release info
2026-04-16T22:40:18.097236849Z Gathering data for ns/openshift-network-console...
2026-04-16T22:40:18.239341613Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T22:40:18.387070121Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T22:40:18.466128818Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:18.545883003Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T22:40:18.629760901Z Gathering data for ns/openshift-console-operator...
2026-04-16T22:40:19.112837809Z Gathering data for ns/openshift-console...
2026-04-16T22:40:19.650155962Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T22:40:19.706617545Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T22:40:20.082725778Z Gathering data for ns/openshift-dns-operator...
2026-04-16T22:40:20.291204533Z Wrote inspect data to must-gather.
2026-04-16T22:40:20.410546831Z Gathering data for ns/openshift-dns...
2026-04-16T22:40:20.826784710Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T22:40:21.022887000Z Gathering data for ns/openshift-image-registry...
2026-04-16T22:40:21.512205098Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T22:40:21.894209838Z Gathering data for ns/openshift-ingress...
2026-04-16T22:40:21.956963684Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T22:40:22.703285906Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T22:40:23.088468964Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T22:40:23.174553956Z Gathering data for ns/openshift-insights...
2026-04-16T22:40:23.473874882Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:24.197190211Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T22:40:25.111136909Z Gathering data for ns/opendatahub...
2026-04-16T22:40:25.321993138Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T22:40:26.436529952Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T22:40:26.592820959Z Gathering data for ns/openshift-lws-operator...
2026-04-16T22:40:26.674534684Z Collecting performance related data for node ip-10-0-129-68.ec2.internal
2026-04-16T22:40:26.682144746Z Collecting performance related data for node ip-10-0-138-154.ec2.internal
2026-04-16T22:40:26.709421110Z Collecting performance related data for node ip-10-0-130-26.ec2.internal
2026-04-16T22:40:28.133912527Z Gathering data for ns/openshift-config...
2026-04-16T22:40:28.481109637Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:28.567790044Z Gathering data for ns/openshift-config-managed...
2026-04-16T22:40:29.069945807Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T22:40:29.435805036Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T22:40:29.791061760Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T22:40:30.139864800Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T22:40:30.491977734Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T22:40:30.800090032Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T22:40:31.169178091Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T22:40:31.520948621Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T22:40:31.861428581Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T22:40:32.468732696Z Gathering data for ns/openshift-multus...
2026-04-16T22:40:33.485966381Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:33.975247053Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T22:40:35.535226452Z Collecting kubelet logs for node ip-10-0-129-68.ec2.internal
2026-04-16T22:40:35.537471966Z Collecting kubelet logs for node ip-10-0-138-154.ec2.internal
2026-04-16T22:40:35.539613509Z Collecting kubelet logs for node ip-10-0-130-26.ec2.internal
2026-04-16T22:40:35.608921404Z Gathering data for ns/openshift-host-network...
2026-04-16T22:40:35.939489612Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T22:40:35.944429659Z INFO: Node performance data collection complete.
2026-04-16T22:40:36.021512810Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T22:40:36.605340122Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T22:40:37.060678614Z Gathering data for ns/openshift-network-operator...
2026-04-16T22:40:37.466784022Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T22:40:37.767424321Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T22:40:38.246764766Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T22:40:38.492646762Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-04-16T22:40:38.602633813Z Gathering data for ns/openshift-apiserver...
2026-04-16T22:40:39.061692948Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T22:40:39.353338675Z Gathering data for ns/openshift-controller-manager...
2026-04-16T22:40:39.680819616Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T22:40:40.327015194Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T22:40:40.666414777Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T22:40:41.006483853Z Gathering data for ns/openshift-service-ca...
2026-04-16T22:40:41.404887422Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T22:40:42.380365386Z Wrote inspect data to must-gather.
2026-04-16T22:40:42.380408678Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:40:42.380408678Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T22:40:42.517813724Z error: the server doesn't have a resource type "clusters"
2026-04-16T22:40:42.693468571Z Caches written to disk