2026-11-04T08:29:55.584333969Z [disk usage checker] Started
2026-11-04T08:29:55.587522010Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:29:55.940917982Z Gathering data for ns/openshift-cluster-version...
2026-11-04T08:29:56.224288481Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T08:29:56.419871509Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:29:56.611850371Z Gathering data for ns/default...
2026-11-04T08:29:57.286151394Z Gathering data for ns/openshift...
2026-11-04T08:29:58.073031024Z Gathering data for ns/kube-system...
2026-11-04T08:29:59.060503603Z Waiting on subprocesses to finish execution.
2026-11-04T08:29:59.067774880Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-04T08:29:59.073513540Z Executing Istio gather script
2026-11-04T08:29:59.075087841Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-04T08:29:59.077115880Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-04T08:29:59.085080799Z INFO: Collecting host service logs for crio
2026-11-04T08:29:59.085401088Z INFO: Collecting host service logs for kubelet
2026-11-04T08:29:59.085730836Z INFO: Collecting host service logs for rpm-ostreed
2026-11-04T08:29:59.086053022Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-04T08:29:59.086331221Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-04T08:29:59.086630699Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-04T08:29:59.086966998Z INFO: Collecting host service logs for NetworkManager
2026-11-04T08:29:59.087304749Z INFO: Collecting host service logs for openvswitch
2026-11-04T08:29:59.087599547Z INFO: Collecting host service logs for ovs-configuration
2026-11-04T08:29:59.087910926Z INFO: Collecting host service logs for ovsdb-server
2026-11-04T08:29:59.088215825Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-04T08:29:59.088568395Z INFO: Waiting for worker host service log collection to complete ...
2026-11-04T08:29:59.132920287Z INFO: Waiting for node performance related collection to complete ...
2026-11-04T08:29:59.382626312Z INFO: Gathering HAProxy config files
2026-11-04T08:29:59.703529228Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-04T08:29:59.703627614Z INFO: on-disk MachineConfig config collection complete.
2026-11-04T08:29:59.816869017Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-4n58p
2026-11-04T08:29:59.816924866Z insights-runtime-extractor-5g6j4
2026-11-04T08:29:59.816944428Z insights-runtime-extractor-ggc6n
2026-11-04T08:29:59.816962937Z insights-runtime-extractor-wfgtd
2026-11-04T08:29:59.824791723Z No resources found in openshift-etcd namespace.
2026-11-04T08:30:00.323173958Z No resources found
2026-11-04T08:30:00.363498945Z Wrote inspect data to must-gather.
2026-11-04T08:30:00.425476496Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-04T08:30:00.458883386Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-04T08:30:00.525776201Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-04T08:30:00.600380825Z error: the server doesn't have a resource type "performanceprofile"
2026-11-04T08:30:00.625802042Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:00.640082682Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-04T08:30:00.667269315Z INFO: "metallb-operator" not detected. Skipping.
2026-11-04T08:30:00.697459627Z INFO: "nfd" not detected. Skipping.
2026-11-04T08:30:00.706747860Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-04T08:30:00.706747860Z See 'oc rsync -h' for help and examples
2026-11-04T08:30:00.736540322Z INFO: Worker host service log collection to complete.
2026-11-04T08:30:01.124574282Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-04T08:30:01.139720300Z ERROR: No running kube-apiserver pods found
2026-11-04T08:30:01.239546176Z W1104 08:30:01.239488     763 util.go:224] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-11-04T08:30:01.270775924Z No resources found
2026-11-04T08:30:01.285743467Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.391877389Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-04T08:30:01.415148816Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-04T08:30:01.466907617Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.549805888Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:30:01.614046215Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.680281690Z INFO: 'previous-logs' folder not found on ip-10-0-129-58.ec2.internal, skipping...
2026-11-04T08:30:01.683506322Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.718258234Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.891390186Z Wrote inspect data to must-gather.
2026-11-04T08:30:01.926209147Z tar: Removing leading `/' from member names
2026-11-04T08:30:01.941122519Z Gathering data for ns/openshift-etcd...
2026-11-04T08:30:01.958096197Z INFO: HAProxy config collection complete.
2026-11-04T08:30:02.098556898Z error: the server doesn't have a resource type "machineconfigs"
2026-11-04T08:30:02.156977678Z INFO: 'previous-logs' folder not found on ip-10-0-135-138.ec2.internal, skipping...
2026-11-04T08:30:02.268289337Z Wrote inspect data to must-gather.
2026-11-04T08:30:02.284913919Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-04T08:30:02.350962496Z INFO: Getting rules from prometheus-k8s-0
2026-11-04T08:30:02.360867339Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-04T08:30:02.502920385Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-04T08:30:02.521603156Z INFO: 'previous-logs' folder not found on ip-10-0-135-180.ec2.internal, skipping...
2026-11-04T08:30:02.521704762Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-04T08:30:02.521756713Z INFO: Machine Config Daemon termination log collection complete.
2026-11-04T08:30:02.696444562Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T08:30:02.700644219Z Wrote inspect data to must-gather.
2026-11-04T08:30:02.769786989Z Wrote inspect data to must-gather.
2026-11-04T08:30:02.812142542Z Wrote inspect data to must-gather.
2026-11-04T08:30:02.812229980Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T08:30:02.812229980Z     namespaces "assisted-installer" not found
2026-11-04T08:30:02.943492298Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:30:02.973811472Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-04T08:30:03.088861689Z Wrote inspect data to must-gather.
2026-11-04T08:30:03.116663339Z INFO: Getting status/config from prometheus-k8s-0
2026-11-04T08:30:03.263311260Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-04T08:30:03.272170943Z Wrote inspect data to must-gather.
2026-11-04T08:30:03.506650853Z INFO: LEGACY MODE
2026-11-04T08:30:03.506678757Z INFO: Gathering ovn-kubernetes master data
2026-11-04T08:30:03.533890156Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:03.552228931Z Wrote inspect data to must-gather.
2026-11-04T08:30:03.688453532Z Gathering data for ns/openshift-monitoring...
2026-11-04T08:30:03.773254858Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-04T08:30:03.834415581Z Wrote inspect data to must-gather.
2026-11-04T08:30:03.884410123Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:03.887694223Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-04T08:30:04.110201065Z Wrote inspect data to must-gather.
2026-11-04T08:30:04.132768759Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:04.141394917Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-04T08:30:04.436444921Z Wrote inspect data to must-gather.
2026-11-04T08:30:04.474107650Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:04.477173207Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-04T08:30:04.523640453Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-04T08:30:04.567795125Z INFO: Waiting for network log collection to complete ...
2026-11-04T08:30:04.567851842Z INFO: Waiting for ovnk database copies to complete ...
2026-11-04T08:30:04.568350383Z INFO: Copying ovnk databases complete.
2026-11-04T08:30:04.570251333Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-04T08:30:04.571693861Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 734 is not a child of this shell
2026-11-04T08:30:04.571693861Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 735 is not a child of this shell
2026-11-04T08:30:04.571693861Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 736 is not a child of this shell
2026-11-04T08:30:04.571693861Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 737 is not a child of this shell
2026-11-04T08:30:04.571693861Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 738 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 739 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 740 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 741 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 746 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 747 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 748 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 749 is not a child of this shell
2026-11-04T08:30:04.571730536Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 750 is not a child of this shell
2026-11-04T08:30:04.577325488Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-04T08:30:04.581434027Z tar: Exiting with failure status due to previous errors
2026-11-04T08:30:04.582044814Z INFO: Network log collection complete.
2026-11-04T08:30:04.715521958Z Wrote inspect data to must-gather.
2026-11-04T08:30:04.734361363Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:04.937283165Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-04T08:30:04.959822320Z Wrote inspect data to must-gather.
2026-11-04T08:30:05.043539627Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:05.046231455Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-04T08:30:05.304463432Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:05.312737479Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-04T08:30:05.496218882Z INFO: Getting status from alertmanager-main-0
2026-11-04T08:30:05.509448110Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-04T08:30:05.595329466Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:05.598074040Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-04T08:30:05.647878228Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:05.697579499Z daemonset.apps/perf-node-gather-daemonset created
2026-11-04T08:30:05.848245051Z Waiting for performance profile collector pods to become ready: 1
2026-11-04T08:30:05.873744015Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:06.158373890Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:06.161320259Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-04T08:30:06.379526794Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:06.664515179Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:30:06.986735640Z Waiting for performance profile collector pods to become ready: 2
2026-11-04T08:30:07.620858874Z Done executing Istio gather script
2026-11-04T08:30:08.120413501Z Waiting for performance profile collector pods to become ready: 3
2026-11-04T08:30:09.254661507Z Waiting for performance profile collector pods to become ready: 4
2026-11-04T08:30:10.195766071Z Gathering data for ns/openshift-network-console...
2026-11-04T08:30:10.395347286Z Waiting for performance profile collector pods to become ready: 5
2026-11-04T08:30:10.652384034Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:11.163450068Z Gathering data for ns/openshift-console-operator...
2026-11-04T08:30:11.537165757Z Waiting for performance profile collector pods to become ready: 6
2026-11-04T08:30:11.994857208Z Gathering data for ns/openshift-console...
2026-11-04T08:30:12.677727997Z Waiting for performance profile collector pods to become ready: 7
2026-11-04T08:30:13.102077994Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-04T08:30:13.816632147Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-04T08:30:14.120735537Z Collecting performance related data for node ip-10-0-129-58.ec2.internal
2026-11-04T08:30:14.130141643Z Collecting performance related data for node ip-10-0-135-180.ec2.internal
2026-11-04T08:30:14.137275985Z Collecting performance related data for node ip-10-0-135-138.ec2.internal
2026-11-04T08:30:14.471208788Z Gathering data for ns/openshift-dns-operator...
2026-11-04T08:30:15.266811789Z Gathering data for ns/openshift-dns...
2026-11-04T08:30:15.656901129Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:17.266626893Z Gathering data for ns/openshift-image-registry...
2026-11-04T08:30:18.545250489Z Gathering data for ns/openshift-ingress-operator...
2026-11-04T08:30:19.301174008Z Gathering data for ns/openshift-ingress...
2026-11-04T08:30:20.133757256Z Gathering data for ns/openshift-ingress-canary...
2026-11-04T08:30:20.661613401Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:21.215119830Z Gathering data for ns/openshift-insights...
2026-11-04T08:30:22.088072831Z Collecting kubelet logs for node ip-10-0-129-58.ec2.internal
2026-11-04T08:30:22.089808526Z Collecting kubelet logs for node ip-10-0-135-180.ec2.internal
2026-11-04T08:30:22.091599401Z Collecting kubelet logs for node ip-10-0-135-138.ec2.internal
2026-11-04T08:30:22.564517929Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tz4jt namespace
2026-11-04T08:30:22.582691955Z INFO: Node performance data collection complete.
2026-11-04T08:30:24.884941416Z Gathering data for ns/openshift-jobset-operator...
2026-11-04T08:30:25.666659525Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:26.268886391Z Gathering data for ns/openshift-config...
2026-11-04T08:30:26.916433537Z Gathering data for ns/openshift-config-managed...
2026-11-04T08:30:27.683354836Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-04T08:30:28.369931282Z Gathering data for ns/openshift-kube-apiserver...
2026-11-04T08:30:29.063165253Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-04T08:30:29.731790665Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-04T08:30:30.420552495Z Gathering data for ns/openshift-kube-scheduler...
2026-11-04T08:30:30.672187143Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:31.103224751Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-04T08:30:31.804036360Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-04T08:30:32.729803540Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-04T08:30:33.543461201Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-04T08:30:34.591034065Z Gathering data for ns/openshift-multus...
2026-11-04T08:30:35.676406019Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:39.681222081Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-04T08:30:40.680830376Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:43.993917438Z Gathering data for ns/openshift-host-network...
2026-11-04T08:30:44.759295025Z Gathering data for ns/openshift-network-diagnostics...
2026-11-04T08:30:45.685272704Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:46.298745350Z Gathering data for ns/openshift-network-node-identity...
2026-11-04T08:30:47.416368677Z Gathering data for ns/openshift-network-operator...
2026-11-04T08:30:48.487777589Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-04T08:30:49.137177280Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-04T08:30:50.419376542Z Gathering data for ns/openshift-apiserver-operator...
2026-11-04T08:30:50.689751109Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:51.101419527Z Gathering data for ns/openshift-apiserver...
2026-11-04T08:30:52.023403906Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-04T08:30:52.677038759Z Gathering data for ns/openshift-controller-manager...
2026-11-04T08:30:53.358167906Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-04T08:30:54.600715778Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-04T08:30:55.289224940Z Gathering data for ns/openshift-service-ca-operator...
2026-11-04T08:30:55.694101881Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-04T08:30:56.093411144Z Gathering data for ns/openshift-service-ca...
2026-11-04T08:30:57.006708843Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-04T08:30:59.194740055Z Wrote inspect data to must-gather.
2026-11-04T08:30:59.194781015Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T08:30:59.194781015Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-04T08:30:59.385259890Z error: the server doesn't have a resource type "clusters"
2026-11-04T08:30:59.463025247Z Caches written to disk