2026-10-31T08:10:10.921528642Z [disk usage checker] Started
2026-10-31T08:10:10.925183514Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:11.306839862Z Gathering data for ns/openshift-cluster-version...
2026-10-31T08:10:11.580489107Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-31T08:10:11.778448497Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T08:10:11.966565408Z Gathering data for ns/default...
2026-10-31T08:10:12.633962098Z Gathering data for ns/openshift...
2026-10-31T08:10:13.401682669Z Gathering data for ns/kube-system...
2026-10-31T08:10:14.355562203Z Waiting on subprocesses to finish execution.
2026-10-31T08:10:14.368073363Z INFO: Gathering HAProxy config files
2026-10-31T08:10:14.368937756Z INFO: Gathering machine config daemon's old logs from all nodes
2026-10-31T08:10:14.375016903Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-10-31T08:10:14.378525519Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-10-31T08:10:14.378919199Z Executing Istio gather script
2026-10-31T08:10:14.388934886Z INFO: Collecting host service logs for crio
2026-10-31T08:10:14.389159346Z INFO: Collecting host service logs for kubelet
2026-10-31T08:10:14.389478769Z INFO: Collecting host service logs for rpm-ostreed
2026-10-31T08:10:14.389787128Z INFO: Collecting host service logs for ostree-finalize-staged
2026-10-31T08:10:14.390142312Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-10-31T08:10:14.390437592Z INFO: Collecting host service logs for machine-config-daemon-host
2026-10-31T08:10:14.390743859Z INFO: Collecting host service logs for NetworkManager
2026-10-31T08:10:14.391050348Z INFO: Collecting host service logs for openvswitch
2026-10-31T08:10:14.391358516Z INFO: Collecting host service logs for ovs-configuration
2026-10-31T08:10:14.391640124Z INFO: Collecting host service logs for ovsdb-server
2026-10-31T08:10:14.391902943Z INFO: Collecting host service logs for ovs-vswitchd
2026-10-31T08:10:14.392063766Z INFO: Waiting for node performance related collection to complete ...
2026-10-31T08:10:14.392242331Z INFO: Waiting for worker host service log collection to complete ...
2026-10-31T08:10:14.852393737Z INFO: "sriov-network-operator" not detected. Skipping.
2026-10-31T08:10:14.981989442Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-10-31T08:10:14.981989442Z INFO: on-disk MachineConfig config collection complete.
2026-10-31T08:10:15.193580985Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-10-31T08:10:15.261681770Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-kb6zm
2026-10-31T08:10:15.261732981Z insights-runtime-extractor-cbxxr
2026-10-31T08:10:15.261754374Z insights-runtime-extractor-jsvl5
2026-10-31T08:10:15.261775748Z insights-runtime-extractor-k56zj
2026-10-31T08:10:15.359715961Z Wrote inspect data to must-gather.
2026-10-31T08:10:15.405815935Z error: the server doesn't have a resource type "performanceprofile"
2026-10-31T08:10:15.407944777Z No resources found in openshift-etcd namespace.
2026-10-31T08:10:15.685563602Z INFO: "nfd" not detected. Skipping.
2026-10-31T08:10:15.811706634Z ERROR: No running kube-apiserver pods found
2026-10-31T08:10:15.841811700Z INFO: "metallb-operator" not detected. Skipping.
2026-10-31T08:10:15.850868616Z No resources found
2026-10-31T08:10:15.994487766Z INFO: Worker host service log collection to complete.
2026-10-31T08:10:16.020631719Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-10-31T08:10:16.031982329Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:16.036877830Z INFO: Found 1 replicas - prometheus-k8s-0
2026-10-31T08:10:16.116522005Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-10-31T08:10:16.116522005Z See 'oc rsync -h' for help and examples
2026-10-31T08:10:16.334600820Z INFO: Waiting for HAProxy config collection to complete ...
2026-10-31T08:10:16.338641647Z No resources found
2026-10-31T08:10:16.550013351Z W1031 08:10:16.549963     738 util.go:224] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-10-31T08:10:16.564225863Z Wrote inspect data to must-gather.
2026-10-31T08:10:16.650291176Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T08:10:16.694869183Z Wrote inspect data to must-gather.
2026-10-31T08:10:16.911227279Z Wrote inspect data to must-gather.
2026-10-31T08:10:16.991946432Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-10-31T08:10:17.001367332Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-10-31T08:10:17.004528833Z Wrote inspect data to must-gather.
2026-10-31T08:10:17.010882244Z Wrote inspect data to must-gather.
2026-10-31T08:10:17.041849996Z INFO: 'previous-logs' folder not found on ip-10-0-131-97.ec2.internal, skipping...
2026-10-31T08:10:17.056800108Z tar: Removing leading `/' from member names
2026-10-31T08:10:17.088739710Z INFO: HAProxy config collection complete.
2026-10-31T08:10:17.120599434Z Gathering data for ns/openshift-etcd...
2026-10-31T08:10:17.127063903Z Wrote inspect data to must-gather.
2026-10-31T08:10:17.343461037Z Wrote inspect data to must-gather.
2026-10-31T08:10:17.414527880Z INFO: 'previous-logs' folder not found on ip-10-0-133-56.ec2.internal, skipping...
2026-10-31T08:10:17.423947819Z error: the server doesn't have a resource type "machineconfigs"
2026-10-31T08:10:17.441470588Z Gathering data for ns/openshift-cloud-credential-operator...
2026-10-31T08:10:17.714687991Z error: the server doesn't have a resource type "machineconfigpools"
2026-10-31T08:10:17.780849462Z INFO: 'previous-logs' folder not found on ip-10-0-143-45.ec2.internal, skipping...
2026-10-31T08:10:17.780976873Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-10-31T08:10:17.780998524Z INFO: Machine Config Daemon termination log collection complete.
2026-10-31T08:10:17.810968095Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-31T08:10:17.818151473Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-10-31T08:10:17.821048774Z INFO: Getting rules from prometheus-k8s-0
2026-10-31T08:10:17.829444493Z Wrote inspect data to must-gather.
2026-10-31T08:10:18.046269623Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T08:10:18.048765613Z Wrote inspect data to must-gather.
2026-10-31T08:10:18.066121667Z Wrote inspect data to must-gather.
2026-10-31T08:10:18.066155212Z error: inspection completed with the errors occurred while gathering data:
2026-10-31T08:10:18.066155212Z     namespaces "assisted-installer" not found
2026-10-31T08:10:18.254480706Z Wrote inspect data to must-gather.
2026-10-31T08:10:18.269910524Z error: the server doesn't have a resource type "kubeletconfigs"
2026-10-31T08:10:18.510578672Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-10-31T08:10:18.520906413Z INFO: Getting status/config from prometheus-k8s-0
2026-10-31T08:10:18.613912122Z Wrote inspect data to must-gather.
2026-10-31T08:10:18.648475289Z INFO: LEGACY MODE
2026-10-31T08:10:18.648509613Z INFO: Gathering ovn-kubernetes master data
2026-10-31T08:10:18.679853200Z Gathering data for ns/openshift-monitoring...
2026-10-31T08:10:18.795473499Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:18.878492871Z Wrote inspect data to must-gather.
2026-10-31T08:10:19.123855828Z INFO: Getting status/flags from prometheus-k8s-0
2026-10-31T08:10:19.123855828Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:19.127291345Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-10-31T08:10:19.178114276Z Wrote inspect data to must-gather.
2026-10-31T08:10:19.409784619Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:19.457128635Z Wrote inspect data to must-gather.
2026-10-31T08:10:19.465700626Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-10-31T08:10:19.680675092Z INFO: Waiting for network log collection to complete ...
2026-10-31T08:10:19.680717978Z INFO: Waiting for ovnk database copies to complete ...
2026-10-31T08:10:19.681210783Z INFO: Copying ovnk databases complete.
2026-10-31T08:10:19.682868955Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 708 is not a child of this shell
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 709 is not a child of this shell
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 710 is not a child of this shell
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 711 is not a child of this shell
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 712 is not a child of this shell
2026-10-31T08:10:19.683407085Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 713 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 714 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 715 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 730 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 731 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 732 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 733 is not a child of this shell
2026-10-31T08:10:19.683433568Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 734 is not a child of this shell
2026-10-31T08:10:19.687566413Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-10-31T08:10:19.691732549Z tar: Exiting with failure status due to previous errors
2026-10-31T08:10:19.692437479Z INFO: Network log collection complete.
2026-10-31T08:10:19.713580363Z Wrote inspect data to must-gather.
2026-10-31T08:10:19.716513401Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:19.718809478Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-10-31T08:10:19.838983962Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-10-31T08:10:19.955583184Z Wrote inspect data to must-gather.
2026-10-31T08:10:19.961148526Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:20.201425008Z Wrote inspect data to must-gather.
2026-10-31T08:10:20.243238434Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-10-31T08:10:20.272692827Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:20.275179233Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-10-31T08:10:20.504447019Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:20.584147550Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-10-31T08:10:20.750610462Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-10-31T08:10:20.778275886Z INFO: Getting status from alertmanager-main-0
2026-10-31T08:10:20.815555080Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:20.819356734Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-10-31T08:10:20.939523249Z daemonset.apps/perf-node-gather-daemonset created
2026-10-31T08:10:21.038557660Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:21.090860992Z Waiting for performance profile collector pods to become ready: 1
2026-10-31T08:10:21.094512437Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:21.376724112Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:21.379527543Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-10-31T08:10:21.578836104Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:21.831420747Z Wrote inspect data to /must-gather/istio.
2026-10-31T08:10:22.227669794Z Waiting for performance profile collector pods to become ready: 2
2026-10-31T08:10:22.751705247Z Done executing Istio gather script
2026-10-31T08:10:23.352168253Z Waiting for performance profile collector pods to become ready: 3
2026-10-31T08:10:24.493359569Z Waiting for performance profile collector pods to become ready: 4
2026-10-31T08:10:24.532939910Z Gathering data for ns/openshift-network-console...
2026-10-31T08:10:25.347519360Z Gathering data for ns/openshift-console-operator...
2026-10-31T08:10:25.645945758Z Waiting for performance profile collector pods to become ready: 5
2026-10-31T08:10:26.044945317Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:26.097769147Z Gathering data for ns/openshift-console...
2026-10-31T08:10:26.787724795Z Waiting for performance profile collector pods to become ready: 6
2026-10-31T08:10:27.078514820Z Gathering data for ns/openshift-cluster-storage-operator...
2026-10-31T08:10:27.935828365Z Waiting for performance profile collector pods to become ready: 7
2026-10-31T08:10:28.089071882Z Gathering data for ns/openshift-dns-operator...
2026-10-31T08:10:28.742050001Z Gathering data for ns/openshift-dns...
2026-10-31T08:10:29.075781096Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-10-31T08:10:29.365300644Z Collecting performance related data for node ip-10-0-131-97.ec2.internal
2026-10-31T08:10:29.372036964Z Collecting performance related data for node ip-10-0-143-45.ec2.internal
2026-10-31T08:10:29.379211934Z Collecting performance related data for node ip-10-0-133-56.ec2.internal
2026-10-31T08:10:30.569765445Z Gathering data for ns/openshift-image-registry...
2026-10-31T08:10:31.049897109Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:31.786564546Z Gathering data for ns/openshift-ingress-operator...
2026-10-31T08:10:32.435804573Z Gathering data for ns/openshift-ingress...
2026-10-31T08:10:33.220820991Z Gathering data for ns/openshift-ingress-canary...
2026-10-31T08:10:34.180833738Z Gathering data for ns/openshift-insights...
2026-10-31T08:10:36.054397075Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:37.514334904Z Gathering data for ns/openshift-jobset-operator...
2026-10-31T08:10:37.675295877Z Collecting kubelet logs for node ip-10-0-131-97.ec2.internal
2026-10-31T08:10:37.677137657Z Collecting kubelet logs for node ip-10-0-143-45.ec2.internal
2026-10-31T08:10:37.679254526Z Collecting kubelet logs for node ip-10-0-133-56.ec2.internal
2026-10-31T08:10:38.136598181Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-2fbdj namespace
2026-10-31T08:10:38.157584800Z INFO: Node performance data collection complete.
2026-10-31T08:10:38.851633040Z Gathering data for ns/openshift-config...
2026-10-31T08:10:39.416281419Z Gathering data for ns/openshift-config-managed...
2026-10-31T08:10:40.110285496Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-10-31T08:10:40.672488899Z Gathering data for ns/openshift-kube-apiserver...
2026-10-31T08:10:41.061347238Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:41.232043093Z Gathering data for ns/openshift-kube-controller-manager...
2026-10-31T08:10:41.813384519Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-10-31T08:10:42.418419123Z Gathering data for ns/openshift-kube-scheduler...
2026-10-31T08:10:42.968501668Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-10-31T08:10:43.569305711Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-10-31T08:10:44.370177151Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-10-31T08:10:45.056695345Z Gathering data for ns/openshift-user-workload-monitoring...
2026-10-31T08:10:45.959917000Z Gathering data for ns/openshift-multus...
2026-10-31T08:10:46.065771358Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:50.501201869Z Gathering data for ns/openshift-ovn-kubernetes...
2026-10-31T08:10:51.070138651Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:54.226022106Z Gathering data for ns/openshift-host-network...
2026-10-31T08:10:54.881538992Z Gathering data for ns/openshift-network-diagnostics...
2026-10-31T08:10:56.074472661Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:10:56.226863868Z Gathering data for ns/openshift-network-node-identity...
2026-10-31T08:10:57.157852246Z Gathering data for ns/openshift-network-operator...
2026-10-31T08:10:58.085462333Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-10-31T08:10:58.643685296Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-10-31T08:10:59.730940649Z Gathering data for ns/openshift-apiserver-operator...
2026-10-31T08:11:00.321221189Z Gathering data for ns/openshift-apiserver...
2026-10-31T08:11:01.079725207Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:11:01.151457384Z Gathering data for ns/openshift-controller-manager-operator...
2026-10-31T08:11:01.715557079Z Gathering data for ns/openshift-controller-manager...
2026-10-31T08:11:02.302073888Z Gathering data for ns/openshift-cluster-samples-operator...
2026-10-31T08:11:03.427848833Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-10-31T08:11:04.019617463Z Gathering data for ns/openshift-service-ca-operator...
2026-10-31T08:11:04.708579807Z Gathering data for ns/openshift-service-ca...
2026-10-31T08:11:05.505612066Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-10-31T08:11:06.084471626Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-10-31T08:11:07.464890401Z Wrote inspect data to must-gather.
2026-10-31T08:11:07.464931841Z error: inspection completed with the errors occurred while gathering data:
2026-10-31T08:11:07.464931841Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-10-31T08:11:07.657139676Z error: the server doesn't have a resource type "clusters"
2026-10-31T08:11:07.755910877Z Caches written to disk