2026-04-23T14:02:49.945883502Z [disk usage checker] Started 2026-04-23T14:02:49.949910819Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:02:50.200925157Z Gathering data for ns/openshift-cluster-version... 2026-04-23T14:02:50.394115543Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-23T14:02:50.528957867Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-23T14:02:50.634073786Z Gathering data for ns/default... 2026-04-23T14:02:51.024111555Z Gathering data for ns/openshift... 2026-04-23T14:02:51.501622997Z Gathering data for ns/kube-system... 2026-04-23T14:02:52.259633532Z Gathering data for ns/openshift-etcd... 2026-04-23T14:02:52.654267515Z Wrote inspect data to must-gather. 2026-04-23T14:02:52.654301201Z error: inspection completed with the errors occurred while gathering data: 2026-04-23T14:02:52.654301201Z namespaces "assisted-installer" not found 2026-04-23T14:02:52.693712908Z Waiting on subprocesses to finish execution. 2026-04-23T14:02:52.705773851Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-04-23T14:02:52.709277090Z INFO: Gathering HAProxy config files 2026-04-23T14:02:52.710011400Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-04-23T14:02:52.714967205Z INFO: Gathering machine config daemon's old logs from all nodes 2026-04-23T14:02:52.723814912Z INFO: Collecting host service logs for crio 2026-04-23T14:02:52.724175844Z INFO: Collecting host service logs for kubelet 2026-04-23T14:02:52.724530298Z INFO: Collecting host service logs for rpm-ostreed 2026-04-23T14:02:52.724862021Z INFO: Collecting host service logs for ostree-finalize-staged 2026-04-23T14:02:52.725361146Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-04-23T14:02:52.725672570Z INFO: Collecting host service logs for machine-config-daemon-host 2026-04-23T14:02:52.726038986Z INFO: Collecting host service logs for NetworkManager 2026-04-23T14:02:52.726386758Z INFO: Collecting host service logs for openvswitch 2026-04-23T14:02:52.726678663Z INFO: Collecting host service logs for ovs-configuration 2026-04-23T14:02:52.726998351Z INFO: Collecting host service logs for ovsdb-server 2026-04-23T14:02:52.727417261Z INFO: Collecting host service logs for ovs-vswitchd 2026-04-23T14:02:52.727826796Z INFO: Waiting for worker host service log collection to complete ... 2026-04-23T14:02:52.762488722Z INFO: Waiting for node performance related collection to complete ... 2026-04-23T14:02:53.014389688Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-04-23T14:02:53.014597429Z INFO: on-disk MachineConfig config collection complete. 2026-04-23T14:02:53.363544126Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-gzhr9 2026-04-23T14:02:53.363600928Z insights-runtime-extractor-4dwkv 2026-04-23T14:02:53.363621342Z insights-runtime-extractor-c5px5 2026-04-23T14:02:53.363640220Z insights-runtime-extractor-pn526 2026-04-23T14:02:53.430691534Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-04-23T14:02:53.580976329Z No resources found 2026-04-23T14:02:53.674513761Z No resources found in openshift-etcd namespace. 2026-04-23T14:02:53.830662086Z INFO: "metallb-operator" not detected. Skipping. 2026-04-23T14:02:53.847696287Z Wrote inspect data to must-gather. 2026-04-23T14:02:53.931332591Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-04-23T14:02:53.989578744Z INFO: Found 1 replicas - prometheus-k8s-0 2026-04-23T14:02:54.128665258Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-04-23T14:02:54.128665258Z See 'oc rsync -h' for help and examples 2026-04-23T14:02:54.181583922Z INFO: Worker host service log collection to complete. 2026-04-23T14:02:54.266023150Z INFO: 'previous-logs' folder not found on ip-10-0-128-108.ec2.internal, skipping... 2026-04-23T14:02:54.330110618Z INFO: "sriov-network-operator" not detected. Skipping. 2026-04-23T14:02:54.478411608Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-04-23T14:02:54.494864299Z error: the server doesn't have a resource type "performanceprofile" 2026-04-23T14:02:54.800460556Z ERROR: No running kube-apiserver pods found 2026-04-23T14:02:54.901789672Z No resources found 2026-04-23T14:02:54.940106606Z INFO: Waiting for HAProxy config collection to complete ... 2026-04-23T14:02:55.019352285Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:02:55.031050360Z Wrote inspect data to must-gather. 2026-04-23T14:02:55.221281275Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-23T14:02:55.469371853Z Gathering data for ns/kuadrant-system... 2026-04-23T14:02:55.533625651Z Wrote inspect data to must-gather. 2026-04-23T14:02:55.550532822Z INFO: 'previous-logs' folder not found on ip-10-0-133-33.ec2.internal, skipping... 2026-04-23T14:02:55.669448578Z Wrote inspect data to must-gather. 2026-04-23T14:02:55.672174907Z tar: Removing leading `/' from member names 2026-04-23T14:02:55.677433424Z Wrote inspect data to must-gather. 2026-04-23T14:02:55.682705790Z INFO: HAProxy config collection complete. 2026-04-23T14:02:55.764241969Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-04-23T14:02:55.821799564Z Wrote inspect data to must-gather. 2026-04-23T14:02:55.996479332Z INFO: Getting rules from prometheus-k8s-0 2026-04-23T14:02:56.016695097Z error: the server doesn't have a resource type "machineconfigs" 2026-04-23T14:02:56.045789305Z INFO: 'previous-logs' folder not found on ip-10-0-135-229.ec2.internal, skipping... 2026-04-23T14:02:56.045833512Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-04-23T14:02:56.045846290Z INFO: Machine Config Daemon termination log collection complete. 2026-04-23T14:02:56.155653543Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-23T14:02:56.161004030Z Wrote inspect data to must-gather. 2026-04-23T14:02:56.345605923Z Wrote inspect data to must-gather. 2026-04-23T14:02:56.477036078Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-23T14:02:56.592487069Z Wrote inspect data to must-gather. 2026-04-23T14:02:56.601237080Z INFO: Getting status/config from prometheus-k8s-0 2026-04-23T14:02:56.735534521Z Gathering data for ns/openshift-monitoring... 2026-04-23T14:02:56.741651630Z Wrote inspect data to must-gather. 2026-04-23T14:02:56.951392655Z error: the server doesn't have a resource type "kubeletconfigs" 2026-04-23T14:02:56.988217826Z Wrote inspect data to must-gather. 2026-04-23T14:02:57.153994289Z INFO: Getting status/flags from prometheus-k8s-0 2026-04-23T14:02:57.296750409Z Wrote inspect data to must-gather. 2026-04-23T14:02:57.385321129Z INFO: INTERCONNECT MODE 2026-04-23T14:02:57.385349767Z INFO: Gathering ovn-kubernetes DBs 2026-04-23T14:02:57.487052095Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-04-23T14:02:57.609440545Z INFO: Gathering OVN_Northbound from ovnkube-node-6mgfk... 2026-04-23T14:02:57.609727948Z INFO: Gathering OVN_Northbound from ovnkube-node-kgrc5... 2026-04-23T14:02:57.609968377Z INFO: Gathering OVN_Northbound from ovnkube-node-zk8wt... 2026-04-23T14:02:57.610474648Z INFO: Gathering OVN_Southbound from ovnkube-node-6mgfk... 2026-04-23T14:02:57.611556431Z INFO: Gathering OVN_Southbound from ovnkube-node-kgrc5... 2026-04-23T14:02:57.611864145Z INFO: Gathering OVN_Southbound from ovnkube-node-zk8wt... 2026-04-23T14:02:57.625395240Z Wrote inspect data to must-gather. 2026-04-23T14:02:57.860127838Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-04-23T14:02:57.956576819Z tar: Removing leading `/' from member names 2026-04-23T14:02:57.958066894Z tar: Removing leading `/' from member names 2026-04-23T14:02:57.970825509Z tar: Removing leading `/' from member names 2026-04-23T14:02:58.019368719Z tar: Removing leading `/' from member names 2026-04-23T14:02:58.035572008Z tar: Removing leading `/' from member names 2026-04-23T14:02:58.091352055Z tar: Removing leading `/' from member names 2026-04-23T14:02:58.168392032Z Wrote inspect data to must-gather. 2026-04-23T14:02:58.232413240Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-04-23T14:02:58.531317017Z Wrote inspect data to must-gather. 2026-04-23T14:02:58.728723674Z INFO: Waiting for network log collection to complete ... 2026-04-23T14:02:58.728771708Z INFO: Waiting for ovnk database copies to complete ... 2026-04-23T14:02:58.729334897Z INFO: Copying ovnk databases complete. 2026-04-23T14:02:58.731184766Z 5.0M must-gather/network_logs/ovnk_database_store 2026-04-23T14:02:58.734009721Z ovnk_database_store/ 2026-04-23T14:02:58.734009721Z ovnk_database_store/ovnkube-node-zk8wt_nbdb 2026-04-23T14:02:58.746739015Z ovnk_database_store/ovnkube-node-6mgfk_nbdb 2026-04-23T14:02:58.753962562Z INFO: Getting status from alertmanager-main-0 2026-04-23T14:02:58.760547928Z ovnk_database_store/ovnkube-node-kgrc5_sbdb 2026-04-23T14:02:58.797245835Z ovnk_database_store/ovnkube-node-6mgfk_sbdb 2026-04-23T14:02:58.836595636Z ovnk_database_store/ovnkube-node-kgrc5_nbdb 2026-04-23T14:02:58.852469995Z ovnk_database_store/ovnkube-node-zk8wt_sbdb 2026-04-23T14:02:58.855233747Z Wrote inspect data to must-gather. 2026-04-23T14:02:58.887643829Z INFO: Network log collection complete. 2026-04-23T14:02:59.159096755Z Wrote inspect data to must-gather. 2026-04-23T14:02:59.523638176Z Gathering data for ns/openshift-network-console... 2026-04-23T14:02:59.536272847Z Wrote inspect data to must-gather. 2026-04-23T14:02:59.885389607Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-04-23T14:02:59.889366455Z INFO: Fallback to identify the container image from release info 2026-04-23T14:03:00.048854255Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:00.085777039Z Gathering data for ns/openshift-console-operator... 2026-04-23T14:03:00.409204416Z Wrote inspect data to must-gather. 2026-04-23T14:03:00.537502566Z Gathering data for ns/openshift-console... 2026-04-23T14:03:00.540542458Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d 2026-04-23T14:03:00.682383864Z daemonset.apps/perf-node-gather-daemonset created 2026-04-23T14:03:00.832223071Z Waiting for performance profile collector pods to become ready: 1 2026-04-23T14:03:01.046486187Z Gathering data for ns/openshift-cluster-storage-operator... 2026-04-23T14:03:01.482877523Z Gathering data for ns/openshift-dns-operator... 2026-04-23T14:03:01.820746019Z Gathering data for ns/openshift-dns... 2026-04-23T14:03:01.961029016Z Waiting for performance profile collector pods to become ready: 2 2026-04-23T14:03:02.618463962Z Gathering data for ns/openshift-image-registry... 2026-04-23T14:03:03.107877950Z Waiting for performance profile collector pods to become ready: 3 2026-04-23T14:03:03.153572681Z Gathering data for ns/openshift-ingress-operator... 2026-04-23T14:03:03.531539423Z Gathering data for ns/openshift-ingress... 2026-04-23T14:03:04.114219968Z Gathering data for ns/openshift-ingress-canary... 2026-04-23T14:03:04.237723701Z Waiting for performance profile collector pods to become ready: 4 2026-04-23T14:03:04.612724539Z Gathering data for ns/openshift-insights... 2026-04-23T14:03:05.053821310Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:05.384120594Z Waiting for performance profile collector pods to become ready: 5 2026-04-23T14:03:06.509487171Z Waiting for performance profile collector pods to become ready: 6 2026-04-23T14:03:07.643999449Z Waiting for performance profile collector pods to become ready: 7 2026-04-23T14:03:07.687533942Z Gathering data for ns/openshift-lws-operator... 2026-04-23T14:03:08.277861812Z Gathering data for ns/kserve... 2026-04-23T14:03:08.787412218Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-04-23T14:03:09.076903596Z Collecting performance related data for node ip-10-0-128-108.ec2.internal 2026-04-23T14:03:09.085433375Z Collecting performance related data for node ip-10-0-135-229.ec2.internal 2026-04-23T14:03:09.094362985Z Collecting performance related data for node ip-10-0-133-33.ec2.internal 2026-04-23T14:03:10.059839725Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:10.372936709Z Gathering data for ns/openshift-config... 2026-04-23T14:03:10.857958475Z Gathering data for ns/openshift-config-managed... 2026-04-23T14:03:11.451606659Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-04-23T14:03:11.814947332Z Gathering data for ns/openshift-kube-apiserver... 2026-04-23T14:03:12.209111451Z Gathering data for ns/openshift-kube-controller-manager... 2026-04-23T14:03:12.621448978Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-04-23T14:03:13.036011507Z Gathering data for ns/openshift-kube-scheduler... 2026-04-23T14:03:13.439392324Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-04-23T14:03:13.894666106Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-04-23T14:03:14.479545529Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-04-23T14:03:14.906929119Z Gathering data for ns/openshift-user-workload-monitoring... 2026-04-23T14:03:15.069193797Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:15.574996830Z Gathering data for ns/openshift-multus... 2026-04-23T14:03:17.756619297Z Gathering data for ns/openshift-ovn-kubernetes... 2026-04-23T14:03:17.835094866Z Collecting kubelet logs for node ip-10-0-128-108.ec2.internal 2026-04-23T14:03:17.837068657Z Collecting kubelet logs for node ip-10-0-135-229.ec2.internal 2026-04-23T14:03:17.839121097Z Collecting kubelet logs for node ip-10-0-133-33.ec2.internal 2026-04-23T14:03:18.323859412Z daemonset.apps "perf-node-gather-daemonset" deleted 2026-04-23T14:03:18.328974978Z INFO: Node performance data collection complete. 2026-04-23T14:03:19.437829142Z Gathering data for ns/openshift-host-network... 2026-04-23T14:03:19.792731811Z Gathering data for ns/openshift-network-diagnostics... 2026-04-23T14:03:20.074337200Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:20.419370995Z Gathering data for ns/openshift-network-node-identity... 2026-04-23T14:03:20.898434958Z Gathering data for ns/openshift-network-operator... 2026-04-23T14:03:21.409895557Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-04-23T14:03:21.742798511Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-04-23T14:03:22.326844783Z Gathering data for ns/openshift-apiserver-operator... 2026-04-23T14:03:22.703608546Z Gathering data for ns/openshift-apiserver... 2026-04-23T14:03:23.229507853Z Gathering data for ns/openshift-controller-manager-operator... 2026-04-23T14:03:23.597629970Z Gathering data for ns/openshift-controller-manager... 2026-04-23T14:03:23.990201566Z Gathering data for ns/openshift-cluster-samples-operator... 2026-04-23T14:03:24.807355223Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-04-23T14:03:25.079381753Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-04-23T14:03:25.170351850Z Gathering data for ns/openshift-service-ca-operator... 2026-04-23T14:03:25.562901210Z Gathering data for ns/openshift-service-ca... 2026-04-23T14:03:26.020394940Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-04-23T14:03:27.146669395Z Wrote inspect data to must-gather. 2026-04-23T14:03:27.146717246Z error: inspection completed with the errors occurred while gathering data: 2026-04-23T14:03:27.146717246Z skipping gathering secrets/support due to error: secrets "support" not found 2026-04-23T14:03:27.292791803Z error: the server doesn't have a resource type "clusters" 2026-04-23T14:03:27.577533290Z Caches written to disk