2026-05-25T11:25:30.407928679Z [disk usage checker] Started
2026-05-25T11:25:30.411288239Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:25:30.634745232Z Gathering data for ns/openshift-cluster-version...
2026-05-25T11:25:30.789475803Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T11:25:30.935464527Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T11:25:31.036259599Z Gathering data for ns/default...
2026-05-25T11:25:31.395495807Z Gathering data for ns/openshift...
2026-05-25T11:25:31.858605246Z Gathering data for ns/kube-system...
2026-05-25T11:25:32.664475300Z Gathering data for ns/openshift-etcd...
2026-05-25T11:25:32.793420745Z Waiting on subprocesses to finish execution.
2026-05-25T11:25:32.804270318Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-25T11:25:32.804563114Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-25T11:25:32.810212538Z INFO: Collecting host service logs for crio
2026-05-25T11:25:32.810481809Z INFO: Collecting host service logs for kubelet
2026-05-25T11:25:32.810756004Z INFO: Collecting host service logs for rpm-ostreed
2026-05-25T11:25:32.811042160Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-25T11:25:32.811370693Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-25T11:25:32.811688939Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-25T11:25:32.811988953Z INFO: Collecting host service logs for NetworkManager
2026-05-25T11:25:32.812278247Z INFO: Collecting host service logs for openvswitch
2026-05-25T11:25:32.812624151Z INFO: Collecting host service logs for ovs-configuration
2026-05-25T11:25:32.812955797Z INFO: Collecting host service logs for ovsdb-server
2026-05-25T11:25:32.813295394Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-25T11:25:32.813704498Z INFO: Waiting for worker host service log collection to complete ...
2026-05-25T11:25:32.826687464Z INFO: Waiting for node performance related collection to complete ...
2026-05-25T11:25:32.844937293Z INFO: Gathering HAProxy config files
2026-05-25T11:25:32.847750982Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-25T11:25:33.269720503Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-25T11:25:33.445238409Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-25T11:25:33.581629810Z No resources found in openshift-etcd namespace.
2026-05-25T11:25:33.774504397Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-25T11:25:33.918795965Z INFO: "metallb-operator" not detected. Skipping.
2026-05-25T11:25:33.997529112Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-25T11:25:34.110425323Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-mx9jv
2026-05-25T11:25:34.110487654Z insights-runtime-extractor-6dxvr
2026-05-25T11:25:34.110516003Z insights-runtime-extractor-b5snb
2026-05-25T11:25:34.110545014Z insights-runtime-extractor-shxpw
2026-05-25T11:25:34.125381589Z error: the server doesn't have a resource type "performanceprofile"
2026-05-25T11:25:34.288935883Z No resources found
2026-05-25T11:25:34.363840959Z Wrote inspect data to must-gather.
2026-05-25T11:25:34.404929829Z INFO: Worker host service log collection to complete.
2026-05-25T11:25:34.408541539Z Wrote inspect data to must-gather.
2026-05-25T11:25:34.408597946Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T11:25:34.408597946Z     namespaces "assisted-installer" not found
2026-05-25T11:25:34.444793481Z ERROR: No running kube-apiserver pods found
2026-05-25T11:25:34.480771199Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-25T11:25:34.480771199Z See 'oc rsync -h' for help and examples
2026-05-25T11:25:34.481805367Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-25T11:25:34.484913413Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-25T11:25:34.484995700Z INFO: on-disk MachineConfig config collection complete.
2026-05-25T11:25:34.591086520Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-25T11:25:34.943105058Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.103547325Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-25T11:25:35.175713940Z No resources found
2026-05-25T11:25:35.434247982Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.445521206Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.451848997Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:25:35.462747886Z error: the server doesn't have a resource type "machineconfigs"
2026-05-25T11:25:35.587676034Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.593445232Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.595859354Z tar: Removing leading `/' from member names
2026-05-25T11:25:35.607799652Z INFO: HAProxy config collection complete.
2026-05-25T11:25:35.708181781Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T11:25:35.867421764Z Gathering data for ns/kuadrant-system...
2026-05-25T11:25:35.906641608Z INFO: Getting rules from prometheus-k8s-0
2026-05-25T11:25:35.908622730Z Wrote inspect data to must-gather.
2026-05-25T11:25:35.912732630Z INFO: 'previous-logs' folder not found on ip-10-0-131-206.ec2.internal, skipping...
2026-05-25T11:25:36.030292924Z Wrote inspect data to must-gather.
2026-05-25T11:25:36.188932104Z Wrote inspect data to must-gather.
2026-05-25T11:25:36.243167918Z INFO: 'previous-logs' folder not found on ip-10-0-133-202.ec2.internal, skipping...
2026-05-25T11:25:36.438615631Z Wrote inspect data to must-gather.
2026-05-25T11:25:36.495503975Z INFO: Getting status/config from prometheus-k8s-0
2026-05-25T11:25:36.533017925Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-25T11:25:36.712949183Z INFO: 'previous-logs' folder not found on ip-10-0-136-188.ec2.internal, skipping...
2026-05-25T11:25:36.712982356Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-25T11:25:36.713011208Z INFO: Machine Config Daemon termination log collection complete.
2026-05-25T11:25:36.853172014Z Wrote inspect data to must-gather.
2026-05-25T11:25:36.877945149Z Wrote inspect data to must-gather.
2026-05-25T11:25:37.071357122Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-25T11:25:37.202282899Z Wrote inspect data to must-gather.
2026-05-25T11:25:37.210409584Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T11:25:37.266074195Z INFO: INTERCONNECT MODE
2026-05-25T11:25:37.266107206Z INFO: Gathering ovn-kubernetes DBs
2026-05-25T11:25:37.329504683Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-25T11:25:37.491725050Z INFO: Gathering OVN_Northbound from ovnkube-node-9jr9j...
2026-05-25T11:25:37.491965190Z INFO: Gathering OVN_Northbound from ovnkube-node-kmnzk...
2026-05-25T11:25:37.492227615Z INFO: Gathering OVN_Northbound from ovnkube-node-lg424...
2026-05-25T11:25:37.492577248Z INFO: Gathering OVN_Southbound from ovnkube-node-9jr9j...
2026-05-25T11:25:37.492859786Z INFO: Gathering OVN_Southbound from ovnkube-node-kmnzk...
2026-05-25T11:25:37.493171371Z INFO: Gathering OVN_Southbound from ovnkube-node-lg424...
2026-05-25T11:25:37.520070725Z Wrote inspect data to must-gather.
2026-05-25T11:25:37.710572004Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-25T11:25:37.780612772Z tar: Removing leading `/' from member names
2026-05-25T11:25:37.818021702Z tar: Removing leading `/' from member names
2026-05-25T11:25:37.884343427Z tar: Removing leading `/' from member names
2026-05-25T11:25:37.911232554Z tar: Removing leading `/' from member names
2026-05-25T11:25:37.931568654Z tar: Removing leading `/' from member names
2026-05-25T11:25:38.018752249Z tar: Removing leading `/' from member names
2026-05-25T11:25:38.029811893Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T11:25:38.127356491Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-25T11:25:38.138158930Z Wrote inspect data to must-gather.
2026-05-25T11:25:38.276965030Z Gathering data for ns/openshift-monitoring...
2026-05-25T11:25:38.540892213Z Wrote inspect data to must-gather.
2026-05-25T11:25:38.646559057Z INFO: Getting status from alertmanager-main-0
2026-05-25T11:25:38.649159604Z INFO: Waiting for network log collection to complete ...
2026-05-25T11:25:38.649248085Z INFO: Waiting for ovnk database copies to complete ...
2026-05-25T11:25:38.649829557Z INFO: Copying ovnk databases complete.
2026-05-25T11:25:38.651500356Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-25T11:25:38.653792300Z ovnk_database_store/
2026-05-25T11:25:38.653792300Z ovnk_database_store/ovnkube-node-lg424_sbdb
2026-05-25T11:25:38.713450660Z ovnk_database_store/ovnkube-node-kmnzk_sbdb
2026-05-25T11:25:38.739428999Z ovnk_database_store/ovnkube-node-9jr9j_nbdb
2026-05-25T11:25:38.751607044Z ovnk_database_store/ovnkube-node-kmnzk_nbdb
2026-05-25T11:25:38.766173912Z ovnk_database_store/ovnkube-node-9jr9j_sbdb
2026-05-25T11:25:38.796780399Z ovnk_database_store/ovnkube-node-lg424_nbdb
2026-05-25T11:25:38.812493166Z INFO: Network log collection complete.
2026-05-25T11:25:38.880008288Z Wrote inspect data to must-gather.
2026-05-25T11:25:39.187714014Z Wrote inspect data to must-gather.
2026-05-25T11:25:39.470197638Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-25T11:25:39.472902005Z INFO: Fallback to identify the container image from release info
2026-05-25T11:25:40.020193733Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-25T11:25:40.156779365Z daemonset.apps/perf-node-gather-daemonset created
2026-05-25T11:25:40.294136401Z Waiting for performance profile collector pods to become ready: 1
2026-05-25T11:25:40.460009087Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:25:40.466152391Z Gathering data for ns/openshift-network-console...
2026-05-25T11:25:40.994872221Z Gathering data for ns/openshift-console-operator...
2026-05-25T11:25:41.435373956Z Waiting for performance profile collector pods to become ready: 2
2026-05-25T11:25:41.459768907Z Gathering data for ns/openshift-console...
2026-05-25T11:25:41.975911217Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-25T11:25:42.235346153Z Wrote inspect data to must-gather.
2026-05-25T11:25:42.370669010Z Gathering data for ns/openshift-dns-operator...
2026-05-25T11:25:42.541840016Z Waiting for performance profile collector pods to become ready: 3
2026-05-25T11:25:42.686289163Z Gathering data for ns/openshift-dns...
2026-05-25T11:25:43.300119082Z Gathering data for ns/openshift-image-registry...
2026-05-25T11:25:43.659847643Z Waiting for performance profile collector pods to become ready: 4
2026-05-25T11:25:43.840833935Z Gathering data for ns/openshift-ingress-operator...
2026-05-25T11:25:44.195778309Z Gathering data for ns/openshift-ingress...
2026-05-25T11:25:44.772988332Z Waiting for performance profile collector pods to become ready: 5
2026-05-25T11:25:44.989177218Z Gathering data for ns/openshift-ingress-canary...
2026-05-25T11:25:45.464924423Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:25:45.471586375Z Gathering data for ns/openshift-insights...
2026-05-25T11:25:45.888667232Z Waiting for performance profile collector pods to become ready: 6
2026-05-25T11:25:47.004741500Z Waiting for performance profile collector pods to become ready: 7
2026-05-25T11:25:47.506588276Z Gathering data for ns/opendatahub...
2026-05-25T11:25:48.110902546Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-25T11:25:48.379096296Z Collecting performance related data for node ip-10-0-136-188.ec2.internal
2026-05-25T11:25:48.389498605Z Collecting performance related data for node ip-10-0-131-206.ec2.internal
2026-05-25T11:25:48.395400267Z Collecting performance related data for node ip-10-0-133-202.ec2.internal
2026-05-25T11:25:49.204962792Z Gathering data for ns/openshift-lws-operator...
2026-05-25T11:25:50.469585947Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:25:50.563778326Z Gathering data for ns/openshift-config...
2026-05-25T11:25:50.910048057Z Gathering data for ns/openshift-config-managed...
2026-05-25T11:25:51.400179444Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-25T11:25:51.752685386Z Gathering data for ns/openshift-kube-apiserver...
2026-05-25T11:25:52.060903508Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-25T11:25:52.422596124Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-25T11:25:52.780885427Z Gathering data for ns/openshift-kube-scheduler...
2026-05-25T11:25:53.120104208Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-25T11:25:53.521390886Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-25T11:25:53.886364917Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-25T11:25:54.247684925Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-25T11:25:54.822965162Z Gathering data for ns/openshift-multus...
2026-05-25T11:25:55.474148993Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-25T11:25:56.732665650Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-25T11:25:56.831692135Z Collecting kubelet logs for node ip-10-0-136-188.ec2.internal
2026-05-25T11:25:56.833532545Z Collecting kubelet logs for node ip-10-0-131-206.ec2.internal
2026-05-25T11:25:56.835202806Z Collecting kubelet logs for node ip-10-0-133-202.ec2.internal
2026-05-25T11:25:57.167215122Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-25T11:25:57.173975436Z INFO: Node performance data collection complete.
2026-05-25T11:25:58.394923746Z Gathering data for ns/openshift-host-network...
2026-05-25T11:25:58.735293556Z Gathering data for ns/openshift-network-diagnostics...
2026-05-25T11:25:59.264193004Z Gathering data for ns/openshift-network-node-identity...
2026-05-25T11:25:59.723936438Z Gathering data for ns/openshift-network-operator...
2026-05-25T11:26:00.230505550Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-25T11:26:00.478670464Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T11:26:00.593249821Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-25T11:26:01.121056359Z Gathering data for ns/openshift-apiserver-operator...
2026-05-25T11:26:01.537215103Z Gathering data for ns/openshift-apiserver...
2026-05-25T11:26:02.031884036Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-25T11:26:02.338591226Z Gathering data for ns/openshift-controller-manager...
2026-05-25T11:26:02.652519271Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-25T11:26:03.393243430Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-25T11:26:03.731968347Z Gathering data for ns/openshift-service-ca-operator...
2026-05-25T11:26:04.047135433Z Gathering data for ns/openshift-service-ca...
2026-05-25T11:26:04.420462036Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-25T11:26:05.373713329Z Wrote inspect data to must-gather.
2026-05-25T11:26:05.373750400Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T11:26:05.373750400Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-25T11:26:05.483175332Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-25T11:26:05.507520078Z error: the server doesn't have a resource type "clusters"
2026-05-25T11:26:05.637563757Z Caches written to disk