2026-04-21T10:51:31.573473811Z [disk usage checker] Started
2026-04-21T10:51:31.607247800Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:31.785580835Z Gathering data for ns/openshift-cluster-version...
2026-04-21T10:51:31.924767361Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T10:51:32.002658079Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T10:51:32.084895010Z Gathering data for ns/default...
2026-04-21T10:51:32.408661241Z Gathering data for ns/openshift...
2026-04-21T10:51:32.819735467Z Gathering data for ns/kube-system...
2026-04-21T10:51:33.643159211Z Gathering data for ns/openshift-etcd...
2026-04-21T10:51:33.955915732Z Wrote inspect data to must-gather.
2026-04-21T10:51:33.955953725Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T10:51:33.955953725Z     namespaces "assisted-installer" not found
2026-04-21T10:51:34.039060243Z Waiting on subprocesses to finish execution.
2026-04-21T10:51:34.047022204Z INFO: Gathering HAProxy config files
2026-04-21T10:51:34.048072377Z INFO: Collecting host service logs for crio
2026-04-21T10:51:34.048546051Z INFO: Collecting host service logs for kubelet
2026-04-21T10:51:34.048895314Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T10:51:34.049223248Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T10:51:34.049639032Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T10:51:34.050399331Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T10:51:34.050796056Z INFO: Collecting host service logs for NetworkManager
2026-04-21T10:51:34.051049938Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T10:51:34.051142827Z INFO: Collecting host service logs for openvswitch
2026-04-21T10:51:34.051565062Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T10:51:34.051928945Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T10:51:34.052392877Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T10:51:34.052836868Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T10:51:34.069382200Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T10:51:34.070261373Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T10:51:34.667341273Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T10:51:34.696101721Z No resources found in openshift-etcd namespace.
2026-04-21T10:51:34.751113222Z Wrote inspect data to must-gather.
2026-04-21T10:51:34.966980594Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T10:51:35.146790252Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T10:51:35.182247381Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T10:51:35.182353650Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T10:51:35.199350052Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T10:51:35.230495965Z No resources found
2026-04-21T10:51:35.335970355Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T10:51:35.360928828Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-4hz8p
2026-04-21T10:51:35.360981128Z insights-runtime-extractor-ckmzt
2026-04-21T10:51:35.361012537Z insights-runtime-extractor-m6sp6
2026-04-21T10:51:35.361033480Z insights-runtime-extractor-x974r
2026-04-21T10:51:35.389301216Z INFO: Worker host service log collection to complete.
2026-04-21T10:51:35.423182703Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T10:51:35.489507170Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T10:51:35.653395438Z Wrote inspect data to must-gather.
2026-04-21T10:51:35.794622009Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T10:51:35.794622009Z See 'oc rsync -h' for help and examples
2026-04-21T10:51:35.800327021Z ERROR: No running kube-apiserver pods found
2026-04-21T10:51:35.983616966Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T10:51:36.230602108Z Gathering data for ns/openshift-monitoring...
2026-04-21T10:51:36.231504526Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T10:51:36.246623472Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T10:51:36.249377611Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.331393206Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.387864124Z No resources found
2026-04-21T10:51:36.391418128Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.392552554Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T10:51:36.414818871Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.465571366Z INFO: 'previous-logs' folder not found on ip-10-0-129-84.ec2.internal, skipping...
2026-04-21T10:51:36.537202345Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.570169239Z tar: Removing leading `/' from member names
2026-04-21T10:51:36.574756567Z INFO: HAProxy config collection complete.
2026-04-21T10:51:36.616376970Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:36.660237882Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.718476179Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T10:51:36.739750886Z INFO: 'previous-logs' folder not found on ip-10-0-142-243.ec2.internal, skipping...
2026-04-21T10:51:36.777073280Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T10:51:36.934142391Z Wrote inspect data to must-gather.
2026-04-21T10:51:36.950068215Z Wrote inspect data to must-gather.
2026-04-21T10:51:37.022057548Z INFO: 'previous-logs' folder not found on ip-10-0-142-73.ec2.internal, skipping...
2026-04-21T10:51:37.022086295Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T10:51:37.022123248Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T10:51:37.157115904Z Wrote inspect data to must-gather.
2026-04-21T10:51:37.167721590Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T10:51:37.169867911Z Wrote inspect data to must-gather.
2026-04-21T10:51:37.309907698Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T10:51:37.455457252Z INFO: INTERCONNECT MODE
2026-04-21T10:51:37.455493931Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T10:51:37.529452284Z Wrote inspect data to must-gather.
2026-04-21T10:51:37.581771170Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T10:51:37.642410618Z INFO: Gathering OVN_Northbound from ovnkube-node-5bsvf...
2026-04-21T10:51:37.642596236Z INFO: Gathering OVN_Northbound from ovnkube-node-l2qr9...
2026-04-21T10:51:37.642771263Z INFO: Gathering OVN_Northbound from ovnkube-node-qbn7w...
2026-04-21T10:51:37.643013623Z INFO: Gathering OVN_Southbound from ovnkube-node-5bsvf...
2026-04-21T10:51:37.643333267Z INFO: Gathering OVN_Southbound from ovnkube-node-l2qr9...
2026-04-21T10:51:37.643669121Z INFO: Gathering OVN_Southbound from ovnkube-node-qbn7w...
2026-04-21T10:51:37.648128271Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T10:51:37.833210762Z Wrote inspect data to must-gather.
2026-04-21T10:51:37.901307487Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T10:51:37.945242241Z tar: Removing leading `/' from member names
2026-04-21T10:51:37.954594008Z tar: Removing leading `/' from member names
2026-04-21T10:51:37.956220258Z tar: Removing leading `/' from member names
2026-04-21T10:51:37.962323201Z tar: Removing leading `/' from member names
2026-04-21T10:51:37.995748697Z tar: Removing leading `/' from member names
2026-04-21T10:51:38.002221756Z tar: Removing leading `/' from member names
2026-04-21T10:51:38.112209460Z Wrote inspect data to must-gather.
2026-04-21T10:51:38.171653953Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T10:51:38.358639535Z W0421 10:51:38.358591    1313 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-21T10:51:38.366310836Z Wrote inspect data to must-gather.
2026-04-21T10:51:38.385205450Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T10:51:38.421084616Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T10:51:38.426538545Z INFO: Waiting for network log collection to complete ...
2026-04-21T10:51:38.426647659Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T10:51:38.427157388Z INFO: Copying ovnk databases complete.
2026-04-21T10:51:38.429126710Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-04-21T10:51:38.432588205Z ovnk_database_store/
2026-04-21T10:51:38.432588205Z ovnk_database_store/ovnkube-node-qbn7w_sbdb
2026-04-21T10:51:38.464126108Z ovnk_database_store/ovnkube-node-5bsvf_sbdb
2026-04-21T10:51:38.490100382Z ovnk_database_store/ovnkube-node-l2qr9_nbdb
2026-04-21T10:51:38.499437317Z ovnk_database_store/ovnkube-node-l2qr9_sbdb
2026-04-21T10:51:38.522142651Z ovnk_database_store/ovnkube-node-5bsvf_nbdb
2026-04-21T10:51:38.533743644Z ovnk_database_store/ovnkube-node-qbn7w_nbdb
2026-04-21T10:51:38.547603827Z INFO: Network log collection complete.
2026-04-21T10:51:38.575344168Z Gathering data for ns/openshift-network-console...
2026-04-21T10:51:38.679715343Z Wrote inspect data to must-gather.
2026-04-21T10:51:38.814881970Z INFO: Getting status from alertmanager-main-0
2026-04-21T10:51:38.984626280Z Wrote inspect data to must-gather.
2026-04-21T10:51:38.987416984Z Gathering data for ns/openshift-console-operator...
2026-04-21T10:51:39.244475329Z Wrote inspect data to must-gather.
2026-04-21T10:51:39.348389007Z Gathering data for ns/openshift-console...
2026-04-21T10:51:39.495203063Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T10:51:39.497010390Z INFO: Fallback to identify the container image from release info
2026-04-21T10:51:39.777210963Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T10:51:39.998817804Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T10:51:40.100064534Z Gathering data for ns/openshift-dns-operator...
2026-04-21T10:51:40.130741678Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T10:51:40.270460160Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T10:51:40.372817169Z Gathering data for ns/openshift-dns...
2026-04-21T10:51:40.981151959Z Gathering data for ns/openshift-image-registry...
2026-04-21T10:51:41.395037518Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T10:51:41.451569801Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T10:51:41.622639965Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:41.760871569Z Gathering data for ns/openshift-ingress...
2026-04-21T10:51:42.079998954Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T10:51:42.439501072Z Gathering data for ns/openshift-insights...
2026-04-21T10:51:42.520023927Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T10:51:43.633570484Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T10:51:44.617644322Z Gathering data for ns/kserve...
2026-04-21T10:51:44.756480373Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T10:51:45.870436882Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T10:51:46.379580321Z Gathering data for ns/openshift-config...
2026-04-21T10:51:46.617746941Z Gathering data for ns/openshift-config-managed...
2026-04-21T10:51:46.631452960Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:47.002550731Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T10:51:47.035453695Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T10:51:47.371735502Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T10:51:47.617537857Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T10:51:47.918216604Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T10:51:48.115762973Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T10:51:48.255459567Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T10:51:48.402978471Z Collecting performance related data for node ip-10-0-142-73.ec2.internal
2026-04-21T10:51:48.411129701Z Collecting performance related data for node ip-10-0-142-243.ec2.internal
2026-04-21T10:51:48.421891589Z Collecting performance related data for node ip-10-0-129-84.ec2.internal
2026-04-21T10:51:48.549598173Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T10:51:48.879493084Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T10:51:49.244653163Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T10:51:49.612505050Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T10:51:50.169758200Z Gathering data for ns/openshift-multus...
2026-04-21T10:51:51.547875724Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T10:51:51.636760770Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:53.172506391Z Gathering data for ns/openshift-host-network...
2026-04-21T10:51:53.472419196Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T10:51:54.008604506Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T10:51:54.435599391Z Gathering data for ns/openshift-network-operator...
2026-04-21T10:51:54.768107329Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T10:51:55.011575329Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T10:51:55.455843733Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T10:51:55.747180981Z Gathering data for ns/openshift-apiserver...
2026-04-21T10:51:56.181667888Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T10:51:56.412991725Z Collecting kubelet logs for node ip-10-0-142-73.ec2.internal
2026-04-21T10:51:56.414826283Z Collecting kubelet logs for node ip-10-0-142-243.ec2.internal
2026-04-21T10:51:56.417065280Z Collecting kubelet logs for node ip-10-0-129-84.ec2.internal
2026-04-21T10:51:56.438289582Z Gathering data for ns/openshift-controller-manager...
2026-04-21T10:51:56.644577814Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T10:51:56.751835426Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T10:51:56.801986327Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T10:51:56.807951510Z INFO: Node performance data collection complete.
2026-04-21T10:51:57.422763190Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T10:51:57.718202060Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T10:51:58.002488866Z Gathering data for ns/openshift-service-ca...
2026-04-21T10:51:58.335462374Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T10:51:59.169989677Z Wrote inspect data to must-gather.
2026-04-21T10:51:59.170045970Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T10:51:59.170045970Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T10:51:59.297984916Z error: the server doesn't have a resource type "clusters"
2026-04-21T10:51:59.451476963Z Caches written to disk