2026-04-20T22:23:24.784543933Z [disk usage checker] Started 2026-04-20T22:23:24.877619251Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:25.005529687Z Gathering data for ns/openshift-cluster-version... 2026-04-20T22:23:25.151192897Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-20T22:23:25.243782035Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T22:23:25.344543711Z Gathering data for ns/default... 2026-04-20T22:23:25.677721318Z Gathering data for ns/openshift... 2026-04-20T22:23:26.175418406Z Gathering data for ns/kube-system... 2026-04-20T22:23:26.857697532Z Gathering data for ns/openshift-etcd... 2026-04-20T22:23:27.212672594Z Wrote inspect data to must-gather. 2026-04-20T22:23:27.212794088Z error: inspection completed with the errors occurred while gathering data: 2026-04-20T22:23:27.212794088Z namespaces "assisted-installer" not found 2026-04-20T22:23:27.321028292Z Waiting on subprocesses to finish execution. 2026-04-20T22:23:27.329590000Z INFO: Collecting host service logs for crio 2026-04-20T22:23:27.329879843Z INFO: Collecting host service logs for kubelet 2026-04-20T22:23:27.330175172Z INFO: Collecting host service logs for rpm-ostreed 2026-04-20T22:23:27.330474462Z INFO: Collecting host service logs for ostree-finalize-staged 2026-04-20T22:23:27.330923844Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-04-20T22:23:27.331221687Z INFO: Collecting host service logs for machine-config-daemon-host 2026-04-20T22:23:27.331519638Z INFO: Collecting host service logs for NetworkManager 2026-04-20T22:23:27.331803958Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-04-20T22:23:27.331825180Z INFO: Collecting host service logs for openvswitch 2026-04-20T22:23:27.332122908Z INFO: Collecting host service logs for ovs-configuration 2026-04-20T22:23:27.332426287Z INFO: Collecting host service logs for ovsdb-server 2026-04-20T22:23:27.332744667Z INFO: Collecting host service logs for ovs-vswitchd 2026-04-20T22:23:27.333092460Z INFO: Waiting for worker host service log collection to complete ... 2026-04-20T22:23:27.333288713Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-04-20T22:23:27.335914191Z INFO: Gathering HAProxy config files 2026-04-20T22:23:27.339778247Z INFO: Gathering machine config daemon's old logs from all nodes 2026-04-20T22:23:27.650623030Z INFO: Waiting for node performance related collection to complete ... 2026-04-20T22:23:28.145945933Z No resources found in openshift-etcd namespace. 2026-04-20T22:23:28.155204990Z INFO: "metallb-operator" not detected. Skipping. 2026-04-20T22:23:28.224138507Z error: the server doesn't have a resource type "performanceprofile" 2026-04-20T22:23:28.583494563Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T22:23:28.684564597Z INFO: "sriov-network-operator" not detected. Skipping. 2026-04-20T22:23:28.960870883Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-bjk5j 2026-04-20T22:23:28.960870883Z insights-runtime-extractor-2m6n7 2026-04-20T22:23:28.960870883Z insights-runtime-extractor-hmbdw 2026-04-20T22:23:28.960870883Z insights-runtime-extractor-knc69 2026-04-20T22:23:29.061564212Z Wrote inspect data to must-gather. 2026-04-20T22:23:29.086152964Z Wrote inspect data to must-gather. 2026-04-20T22:23:29.240145262Z INFO: Waiting for HAProxy config collection to complete ... 2026-04-20T22:23:29.288338665Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-04-20T22:23:29.430346309Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-04-20T22:23:29.492024278Z Wrote inspect data to must-gather. 2026-04-20T22:23:29.547379612Z ERROR: No running kube-apiserver pods found 2026-04-20T22:23:29.580788373Z INFO: Found 1 replicas - prometheus-k8s-0 2026-04-20T22:23:29.608902886Z Wrote inspect data to must-gather. 2026-04-20T22:23:29.655768196Z Wrote inspect data to must-gather. 2026-04-20T22:23:29.660665225Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-04-20T22:23:29.735530200Z INFO: Worker host service log collection to complete. 2026-04-20T22:23:29.762800046Z No resources found 2026-04-20T22:23:29.840742848Z No resources found 2026-04-20T22:23:29.859244268Z INFO: 'previous-logs' folder not found on ip-10-0-136-102.ec2.internal, skipping... 2026-04-20T22:23:29.895902655Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:29.943222610Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-04-20T22:23:29.943222610Z See 'oc rsync -h' for help and examples 2026-04-20T22:23:29.950446936Z tar: Removing leading `/' from member names 2026-04-20T22:23:29.959005615Z INFO: HAProxy config collection complete. 2026-04-20T22:23:30.002092583Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-04-20T22:23:30.002092583Z INFO: on-disk MachineConfig config collection complete. 2026-04-20T22:23:30.045601897Z Wrote inspect data to must-gather. 2026-04-20T22:23:30.050798099Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-04-20T22:23:30.256385798Z Gathering data for ns/kuadrant-system... 2026-04-20T22:23:30.295925457Z Wrote inspect data to must-gather. 2026-04-20T22:23:30.316076664Z error: the server doesn't have a resource type "machineconfigs" 2026-04-20T22:23:30.339208460Z INFO: 'previous-logs' folder not found on ip-10-0-137-199.ec2.internal, skipping... 2026-04-20T22:23:30.578674656Z INFO: Getting rules from prometheus-k8s-0 2026-04-20T22:23:30.641443840Z Wrote inspect data to must-gather. 2026-04-20T22:23:30.674927371Z Wrote inspect data to must-gather. 2026-04-20T22:23:30.698220726Z INFO: 'previous-logs' folder not found on ip-10-0-140-110.ec2.internal, skipping... 2026-04-20T22:23:30.698325442Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-04-20T22:23:30.698468124Z INFO: Machine Config Daemon termination log collection complete. 2026-04-20T22:23:30.985833333Z Wrote inspect data to must-gather. 2026-04-20T22:23:31.013045582Z Wrote inspect data to must-gather. 2026-04-20T22:23:31.092293912Z INFO: Getting status/config from prometheus-k8s-0 2026-04-20T22:23:31.198267191Z error: the server doesn't have a resource type "kubeletconfigs" 2026-04-20T22:23:31.409722820Z INFO: INTERCONNECT MODE 2026-04-20T22:23:31.409722820Z INFO: Gathering ovn-kubernetes DBs 2026-04-20T22:23:31.492780454Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-20T22:23:31.511725942Z Wrote inspect data to must-gather. 2026-04-20T22:23:31.630716409Z INFO: Getting status/flags from prometheus-k8s-0 2026-04-20T22:23:31.630716409Z INFO: Gathering OVN_Northbound from ovnkube-node-jvkzb... 2026-04-20T22:23:31.631024301Z INFO: Gathering OVN_Northbound from ovnkube-node-mlfps... 2026-04-20T22:23:31.631327467Z INFO: Gathering OVN_Northbound from ovnkube-node-vf7zb... 2026-04-20T22:23:31.631669546Z INFO: Gathering OVN_Southbound from ovnkube-node-jvkzb... 2026-04-20T22:23:31.631953953Z INFO: Gathering OVN_Southbound from ovnkube-node-mlfps... 2026-04-20T22:23:31.632219836Z INFO: Gathering OVN_Southbound from ovnkube-node-vf7zb... 2026-04-20T22:23:31.957856847Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.000544034Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.004514877Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.032611882Z Wrote inspect data to must-gather. 2026-04-20T22:23:32.055222038Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-04-20T22:23:32.138470422Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.142593715Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.218672568Z tar: Removing leading `/' from member names 2026-04-20T22:23:32.276800596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T22:23:32.390223930Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-04-20T22:23:32.542060373Z Wrote inspect data to must-gather. 2026-04-20T22:23:32.567060958Z Gathering data for ns/openshift-monitoring... 2026-04-20T22:23:32.721957640Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-04-20T22:23:32.879565990Z INFO: Waiting for network log collection to complete ... 2026-04-20T22:23:32.879890817Z INFO: Waiting for ovnk database copies to complete ... 2026-04-20T22:23:32.880490526Z INFO: Copying ovnk databases complete. 2026-04-20T22:23:32.883492907Z 4.2M must-gather/network_logs/ovnk_database_store 2026-04-20T22:23:32.886492228Z ovnk_database_store/ 2026-04-20T22:23:32.886545344Z ovnk_database_store/ovnkube-node-mlfps_sbdb 2026-04-20T22:23:32.914763915Z ovnk_database_store/ovnkube-node-mlfps_nbdb 2026-04-20T22:23:32.925731058Z ovnk_database_store/ovnkube-node-jvkzb_sbdb 2026-04-20T22:23:32.950863172Z Wrote inspect data to must-gather. 2026-04-20T22:23:32.954080836Z ovnk_database_store/ovnkube-node-jvkzb_nbdb 2026-04-20T22:23:32.967764699Z ovnk_database_store/ovnkube-node-vf7zb_nbdb 2026-04-20T22:23:32.980690102Z ovnk_database_store/ovnkube-node-vf7zb_sbdb 2026-04-20T22:23:33.013812619Z INFO: Network log collection complete. 2026-04-20T22:23:33.204148599Z INFO: Getting status from alertmanager-main-0 2026-04-20T22:23:33.296429120Z Wrote inspect data to must-gather. 2026-04-20T22:23:33.606071130Z Wrote inspect data to must-gather. 2026-04-20T22:23:33.963869666Z Wrote inspect data to must-gather. 2026-04-20T22:23:34.279199031Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-04-20T22:23:34.282661179Z INFO: Fallback to identify the container image from release info 2026-04-20T22:23:34.730411438Z Gathering data for ns/openshift-network-console... 2026-04-20T22:23:34.892318999Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d 2026-04-20T22:23:34.901946159Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:35.033791137Z daemonset.apps/perf-node-gather-daemonset created 2026-04-20T22:23:35.196077054Z Waiting for performance profile collector pods to become ready: 1 2026-04-20T22:23:35.283800600Z Gathering data for ns/openshift-console-operator... 2026-04-20T22:23:35.738840682Z Gathering data for ns/openshift-console... 2026-04-20T22:23:36.248866303Z Gathering data for ns/openshift-cluster-storage-operator... 2026-04-20T22:23:36.344787448Z Waiting for performance profile collector pods to become ready: 2 2026-04-20T22:23:36.670653291Z Gathering data for ns/openshift-dns-operator... 2026-04-20T22:23:36.797082724Z Wrote inspect data to must-gather. 2026-04-20T22:23:36.976643648Z Gathering data for ns/openshift-dns... 2026-04-20T22:23:37.481091042Z Waiting for performance profile collector pods to become ready: 3 2026-04-20T22:23:37.579475285Z Gathering data for ns/openshift-image-registry... 2026-04-20T22:23:38.038444462Z Gathering data for ns/openshift-ingress-operator... 2026-04-20T22:23:38.393148972Z Gathering data for ns/openshift-ingress... 2026-04-20T22:23:38.594896961Z Waiting for performance profile collector pods to become ready: 4 2026-04-20T22:23:39.109568628Z Gathering data for ns/openshift-ingress-canary... 2026-04-20T22:23:39.600040385Z Gathering data for ns/openshift-insights... 2026-04-20T22:23:39.715671546Z Waiting for performance profile collector pods to become ready: 5 2026-04-20T22:23:39.908070109Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:40.842582234Z Waiting for performance profile collector pods to become ready: 6 2026-04-20T22:23:41.599179254Z Gathering data for ns/opendatahub... 2026-04-20T22:23:41.956373467Z Waiting for performance profile collector pods to become ready: 7 2026-04-20T22:23:43.011779020Z Gathering data for ns/openshift-lws-operator... 2026-04-20T22:23:43.074400276Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-04-20T22:23:43.366574399Z Collecting performance related data for node ip-10-0-140-110.ec2.internal 2026-04-20T22:23:43.377405975Z Collecting performance related data for node ip-10-0-137-199.ec2.internal 2026-04-20T22:23:43.400169829Z Collecting performance related data for node ip-10-0-136-102.ec2.internal 2026-04-20T22:23:44.494296930Z Gathering data for ns/openshift-config... 2026-04-20T22:23:44.883445488Z Gathering data for ns/openshift-config-managed... 2026-04-20T22:23:44.918532808Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:45.343514371Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-04-20T22:23:45.655548472Z Gathering data for ns/openshift-kube-apiserver... 2026-04-20T22:23:45.967979794Z Gathering data for ns/openshift-kube-controller-manager... 2026-04-20T22:23:46.354769891Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-04-20T22:23:46.724301851Z Gathering data for ns/openshift-kube-scheduler... 2026-04-20T22:23:47.010991967Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-04-20T22:23:47.384284467Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-04-20T22:23:47.759958375Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-04-20T22:23:48.131897635Z Gathering data for ns/openshift-user-workload-monitoring... 2026-04-20T22:23:48.796906884Z Gathering data for ns/openshift-multus... 2026-04-20T22:23:49.928440801Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:50.209259368Z Gathering data for ns/openshift-ovn-kubernetes... 2026-04-20T22:23:51.812077529Z Gathering data for ns/openshift-host-network... 2026-04-20T22:23:51.946536372Z Collecting kubelet logs for node ip-10-0-140-110.ec2.internal 2026-04-20T22:23:51.948368449Z Collecting kubelet logs for node ip-10-0-137-199.ec2.internal 2026-04-20T22:23:51.950253355Z Collecting kubelet logs for node ip-10-0-136-102.ec2.internal 2026-04-20T22:23:52.250983357Z Gathering data for ns/openshift-network-diagnostics... 2026-04-20T22:23:52.319085158Z daemonset.apps "perf-node-gather-daemonset" deleted 2026-04-20T22:23:52.323703976Z INFO: Node performance data collection complete. 2026-04-20T22:23:52.822578626Z Gathering data for ns/openshift-network-node-identity... 2026-04-20T22:23:53.258112376Z Gathering data for ns/openshift-network-operator... 2026-04-20T22:23:53.695723232Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-04-20T22:23:54.027373526Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-04-20T22:23:54.584964157Z Gathering data for ns/openshift-apiserver-operator... 2026-04-20T22:23:54.933536360Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-04-20T22:23:54.938965151Z Gathering data for ns/openshift-apiserver... 2026-04-20T22:23:55.419599119Z Gathering data for ns/openshift-controller-manager-operator... 2026-04-20T22:23:55.787994220Z Gathering data for ns/openshift-controller-manager... 2026-04-20T22:23:56.136593955Z Gathering data for ns/openshift-cluster-samples-operator... 2026-04-20T22:23:56.818077732Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-04-20T22:23:57.154193801Z Gathering data for ns/openshift-service-ca-operator... 2026-04-20T22:23:57.474798157Z Gathering data for ns/openshift-service-ca... 2026-04-20T22:23:57.849685023Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-04-20T22:23:58.830129499Z Wrote inspect data to must-gather. 2026-04-20T22:23:58.830175976Z error: inspection completed with the errors occurred while gathering data: 2026-04-20T22:23:58.830175976Z skipping gathering secrets/support due to error: secrets "support" not found 2026-04-20T22:23:58.968114379Z error: the server doesn't have a resource type "clusters" 2026-04-20T22:23:59.371697001Z Caches written to disk