2026-03-18T17:46:40.778260631Z [disk usage checker] Started
2026-03-18T17:46:40.781474561Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:46:41.003886267Z Gathering data for ns/openshift-cluster-version...
2026-03-18T17:46:41.174330874Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-03-18T17:46:41.284781349Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-03-18T17:46:41.371960592Z Gathering data for ns/default...
2026-03-18T17:46:41.714771940Z Gathering data for ns/openshift...
2026-03-18T17:46:42.157474342Z Gathering data for ns/kube-system...
2026-03-18T17:46:42.987392585Z Gathering data for ns/openshift-etcd...
2026-03-18T17:46:43.306701145Z Waiting on subprocesses to finish execution.
2026-03-18T17:46:43.308038660Z Wrote inspect data to must-gather.
2026-03-18T17:46:43.308122441Z error: inspection completed with the errors occurred while gathering data:
2026-03-18T17:46:43.308122441Z     namespaces "assisted-installer" not found
2026-03-18T17:46:43.313097864Z INFO: Gathering HAProxy config files
2026-03-18T17:46:43.316220618Z INFO: Collecting host service logs for crio
2026-03-18T17:46:43.316480320Z INFO: Collecting host service logs for kubelet
2026-03-18T17:46:43.316756332Z INFO: Collecting host service logs for rpm-ostreed
2026-03-18T17:46:43.317087635Z INFO: Collecting host service logs for ostree-finalize-staged
2026-03-18T17:46:43.317374326Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-03-18T17:46:43.317678545Z INFO: Collecting host service logs for machine-config-daemon-host
2026-03-18T17:46:43.318050589Z INFO: Collecting host service logs for NetworkManager
2026-03-18T17:46:43.318361641Z INFO: Collecting host service logs for openvswitch
2026-03-18T17:46:43.318707591Z INFO: Collecting host service logs for ovs-configuration
2026-03-18T17:46:43.319138921Z INFO: Collecting host service logs for ovsdb-server
2026-03-18T17:46:43.319462967Z INFO: Collecting host service logs for ovs-vswitchd
2026-03-18T17:46:43.319873397Z INFO: Waiting for worker host service log collection to complete ...
2026-03-18T17:46:43.324424170Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-03-18T17:46:43.325685777Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-03-18T17:46:43.360117812Z INFO: Gathering machine config daemon's old logs from all nodes
2026-03-18T17:46:43.558116653Z INFO: Waiting for node performance related collection to complete ...
2026-03-18T17:46:44.197005119Z INFO: "sriov-network-operator" not detected. Skipping.
2026-03-18T17:46:44.197400182Z INFO: Collecting Insights Archives from insights-operator-76bdd9f478-p44bj
2026-03-18T17:46:44.197430791Z insights-runtime-extractor-55qz2
2026-03-18T17:46:44.197448692Z insights-runtime-extractor-fqvd4
2026-03-18T17:46:44.197466068Z insights-runtime-extractor-vpb6s
2026-03-18T17:46:44.356904103Z No resources found in openshift-etcd namespace.
2026-03-18T17:46:44.383247611Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-03-18T17:46:44.383282710Z INFO: on-disk MachineConfig config collection complete.
2026-03-18T17:46:44.397346883Z No resources found
2026-03-18T17:46:44.417973605Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-03-18T17:46:44.481608975Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-03-18T17:46:44.487738792Z error: the server doesn't have a resource type "performanceprofile"
2026-03-18T17:46:44.604062891Z INFO: Found 1 replicas - prometheus-k8s-0
2026-03-18T17:46:44.612925453Z INFO: "metallb-operator" not detected. Skipping.
2026-03-18T17:46:44.627484565Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-03-18T17:46:44.627484565Z See 'oc rsync -h' for help and examples
2026-03-18T17:46:44.683904925Z Wrote inspect data to must-gather.
2026-03-18T17:46:44.736040711Z INFO: Worker host service log collection to complete.
2026-03-18T17:46:44.738587993Z ERROR: No running kube-apiserver pods found
2026-03-18T17:46:44.922770997Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.022072731Z INFO: Waiting for HAProxy config collection to complete ...
2026-03-18T17:46:45.167837240Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.272522627Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-03-18T17:46:45.400139005Z INFO: 'previous-logs' folder not found on ip-10-0-129-201.ec2.internal, skipping...
2026-03-18T17:46:45.411009317Z No resources found
2026-03-18T17:46:45.466371604Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-03-18T17:46:45.483002401Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-03-18T17:46:45.535018772Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.657753286Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.690620551Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.691144256Z Gathering data for ns/openshift-monitoring...
2026-03-18T17:46:45.694313427Z error: the server doesn't have a resource type "machineconfigs"
2026-03-18T17:46:45.742727181Z tar: Removing leading `/' from member names
2026-03-18T17:46:45.748154576Z INFO: HAProxy config collection complete.
2026-03-18T17:46:45.786516593Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:46:45.809075159Z Wrote inspect data to must-gather.
2026-03-18T17:46:45.869944729Z INFO: 'previous-logs' folder not found on ip-10-0-137-219.ec2.internal, skipping...
2026-03-18T17:46:45.938371153Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.043187262Z INFO: Getting rules from prometheus-k8s-0
2026-03-18T17:46:46.081087240Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.153344092Z INFO: 'previous-logs' folder not found on ip-10-0-139-49.ec2.internal, skipping...
2026-03-18T17:46:46.153471907Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-03-18T17:46:46.153492512Z INFO: Machine Config Daemon termination log collection complete.
2026-03-18T17:46:46.181273348Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.336201433Z error: the server doesn't have a resource type "kubeletconfigs"
2026-03-18T17:46:46.338606274Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.498290090Z INFO: Getting status/config from prometheus-k8s-0
2026-03-18T17:46:46.560041087Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.581771206Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.770218359Z Wrote inspect data to must-gather.
2026-03-18T17:46:46.869418260Z INFO: INTERCONNECT MODE
2026-03-18T17:46:46.869493472Z INFO: Gathering ovn-kubernetes DBs
2026-03-18T17:46:46.905630020Z INFO: Getting status/flags from prometheus-k8s-0
2026-03-18T17:46:46.988605678Z Wrote inspect data to must-gather.
2026-03-18T17:46:47.029814680Z INFO: Gathering OVN_Northbound from ovnkube-node-gk7ln...
2026-03-18T17:46:47.030093543Z INFO: Gathering OVN_Northbound from ovnkube-node-qktrm...
2026-03-18T17:46:47.030344631Z INFO: Gathering OVN_Northbound from ovnkube-node-wxnmv...
2026-03-18T17:46:47.030611469Z INFO: Gathering OVN_Southbound from ovnkube-node-gk7ln...
2026-03-18T17:46:47.030877913Z INFO: Gathering OVN_Southbound from ovnkube-node-qktrm...
2026-03-18T17:46:47.031126167Z INFO: Gathering OVN_Southbound from ovnkube-node-wxnmv...
2026-03-18T17:46:47.218081871Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-03-18T17:46:47.335235195Z Wrote inspect data to must-gather.
2026-03-18T17:46:47.364109266Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.372035457Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.372075138Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.381394147Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.384814953Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.385955019Z tar: Removing leading `/' from member names
2026-03-18T17:46:47.425080808Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-03-18T17:46:47.486015605Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-03-18T17:46:47.582690904Z Wrote inspect data to must-gather.
2026-03-18T17:46:47.768654544Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-03-18T17:46:47.814721909Z Wrote inspect data to must-gather.
2026-03-18T17:46:47.872468378Z INFO: Waiting for network log collection to complete ...
2026-03-18T17:46:47.872595112Z INFO: Waiting for ovnk database copies to complete ...
2026-03-18T17:46:47.873138656Z INFO: Copying ovnk databases complete.
2026-03-18T17:46:47.875247441Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-03-18T17:46:47.877793640Z ovnk_database_store/
2026-03-18T17:46:47.877862350Z ovnk_database_store/ovnkube-node-gk7ln_nbdb
2026-03-18T17:46:47.892193684Z ovnk_database_store/ovnkube-node-qktrm_sbdb
2026-03-18T17:46:47.919391882Z ovnk_database_store/ovnkube-node-qktrm_nbdb
2026-03-18T17:46:47.933042067Z ovnk_database_store/ovnkube-node-wxnmv_sbdb
2026-03-18T17:46:47.953513559Z ovnk_database_store/ovnkube-node-gk7ln_sbdb
2026-03-18T17:46:47.973507114Z ovnk_database_store/ovnkube-node-wxnmv_nbdb
2026-03-18T17:46:47.985762407Z INFO: Network log collection complete.
2026-03-18T17:46:48.029374135Z W0318 17:46:48.029331    1414 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-03-18T17:46:48.034336452Z Wrote inspect data to must-gather.
2026-03-18T17:46:48.057511358Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-03-18T17:46:48.168015756Z INFO: Getting status from alertmanager-main-0
2026-03-18T17:46:48.248244233Z Gathering data for ns/openshift-network-console...
2026-03-18T17:46:48.301890265Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-03-18T17:46:48.303878577Z INFO: Fallback to identify the container image from release info
2026-03-18T17:46:48.702120816Z Gathering data for ns/openshift-console-operator...
2026-03-18T17:46:49.013524649Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf8869933b4dc87cede910550d284a996d1bbf346e2c331b91a37d5ca9275599
2026-03-18T17:46:49.084052353Z Gathering data for ns/openshift-console...
2026-03-18T17:46:49.138023880Z daemonset.apps/perf-node-gather-daemonset created
2026-03-18T17:46:49.274318857Z Waiting for performance profile collector pods to become ready: 1
2026-03-18T17:46:49.557568384Z Gathering data for ns/openshift-cluster-storage-operator...
2026-03-18T17:46:49.941498461Z Gathering data for ns/openshift-dns-operator...
2026-03-18T17:46:50.249359754Z Gathering data for ns/openshift-dns...
2026-03-18T17:46:50.387762812Z Waiting for performance profile collector pods to become ready: 2
2026-03-18T17:46:50.792431993Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:46:50.887458827Z Gathering data for ns/openshift-image-registry...
2026-03-18T17:46:51.439852818Z Gathering data for ns/openshift-ingress-operator...
2026-03-18T17:46:51.530083559Z Waiting for performance profile collector pods to become ready: 3
2026-03-18T17:46:51.845073358Z Gathering data for ns/openshift-ingress...
2026-03-18T17:46:52.194215944Z Gathering data for ns/openshift-ingress-canary...
2026-03-18T17:46:52.589116578Z Gathering data for ns/openshift-insights...
2026-03-18T17:46:52.657124203Z Waiting for performance profile collector pods to become ready: 4
2026-03-18T17:46:53.783877256Z Waiting for performance profile collector pods to become ready: 5
2026-03-18T17:46:54.727531542Z Gathering data for ns/kserve...
2026-03-18T17:46:54.910235771Z Waiting for performance profile collector pods to become ready: 6
2026-03-18T17:46:55.797569487Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:46:56.034369041Z Waiting for performance profile collector pods to become ready: 7
2026-03-18T17:46:56.590460376Z Gathering data for ns/openshift-config...
2026-03-18T17:46:56.874461763Z Gathering data for ns/openshift-config-managed...
2026-03-18T17:46:57.155335895Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-03-18T17:46:57.310033810Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-03-18T17:46:57.443004670Z Collecting performance related data for node ip-10-0-129-201.ec2.internal
2026-03-18T17:46:57.449542056Z Collecting performance related data for node ip-10-0-139-49.ec2.internal
2026-03-18T17:46:57.458905524Z Collecting performance related data for node ip-10-0-137-219.ec2.internal
2026-03-18T17:46:57.650619856Z Gathering data for ns/openshift-kube-apiserver...
2026-03-18T17:46:57.989771530Z Gathering data for ns/openshift-kube-controller-manager...
2026-03-18T17:46:58.364468596Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-03-18T17:46:58.753593350Z Gathering data for ns/openshift-kube-scheduler...
2026-03-18T17:46:59.139700103Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-03-18T17:46:59.475669610Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-03-18T17:46:59.848726202Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-03-18T17:47:00.303682284Z Gathering data for ns/openshift-user-workload-monitoring...
2026-03-18T17:47:00.802723948Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:47:00.912138069Z Gathering data for ns/openshift-multus...
2026-03-18T17:47:02.792185343Z Gathering data for ns/openshift-ovn-kubernetes...
2026-03-18T17:47:04.391299970Z Gathering data for ns/openshift-host-network...
2026-03-18T17:47:04.771265940Z Gathering data for ns/openshift-network-diagnostics...
2026-03-18T17:47:05.388306946Z Gathering data for ns/openshift-network-node-identity...
2026-03-18T17:47:05.738117483Z Collecting kubelet logs for node ip-10-0-129-201.ec2.internal
2026-03-18T17:47:05.740850858Z Collecting kubelet logs for node ip-10-0-139-49.ec2.internal
2026-03-18T17:47:05.742599615Z Collecting kubelet logs for node ip-10-0-137-219.ec2.internal
2026-03-18T17:47:05.808857600Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:47:05.940303636Z Gathering data for ns/openshift-network-operator...
2026-03-18T17:47:06.160650221Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-03-18T17:47:06.165670823Z INFO: Node performance data collection complete.
2026-03-18T17:47:06.382950966Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-03-18T17:47:06.672166259Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-03-18T17:47:07.158294198Z Gathering data for ns/openshift-apiserver-operator...
2026-03-18T17:47:07.467564428Z Gathering data for ns/openshift-apiserver...
2026-03-18T17:47:07.875164537Z Gathering data for ns/openshift-controller-manager-operator...
2026-03-18T17:47:08.141219535Z Gathering data for ns/openshift-controller-manager...
2026-03-18T17:47:08.430806009Z Gathering data for ns/openshift-cluster-samples-operator...
2026-03-18T17:47:09.097935530Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-03-18T17:47:09.406699685Z Gathering data for ns/openshift-service-ca-operator...
2026-03-18T17:47:09.738766932Z Gathering data for ns/openshift-service-ca...
2026-03-18T17:47:10.122385806Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-03-18T17:47:10.814486724Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-03-18T17:47:11.301126030Z Wrote inspect data to must-gather.
2026-03-18T17:47:11.301176719Z error: inspection completed with the errors occurred while gathering data:
2026-03-18T17:47:11.301176719Z     [skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering customresourcedefinitions.apiextensions.k8s.io due to error: skipping gathering namespaces/kserve due to error: one or more errors occurred while gathering pod-specific data for namespace: kserve
2026-03-18T17:47:11.301176719Z 
2026-03-18T17:47:11.301176719Z     one or more errors occurred while gathering container data for pod model-serving-api-9699c8d45-ttb7x:
2026-03-18T17:47:11.301176719Z 
2026-03-18T17:47:11.301176719Z     [previous terminated container "server" in pod "model-serving-api-9699c8d45-ttb7x" not found, container "server" in pod "model-serving-api-9699c8d45-ttb7x" is waiting to start: trying and failing to pull image, the server could not find the requested resource]]
2026-03-18T17:47:11.441668172Z error: the server doesn't have a resource type "clusters"
2026-03-18T17:47:11.590349027Z Caches written to disk