2026-11-03T15:47:46.510137156Z [disk usage checker] Started
2026-11-03T15:47:46.513416439Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:47:47.356061513Z Gathering data for ns/openshift-cluster-version...
2026-11-03T15:47:48.079595061Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T15:47:48.600408751Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T15:47:49.079877762Z Gathering data for ns/default...
2026-11-03T15:47:50.861425655Z Gathering data for ns/openshift...
2026-11-03T15:47:51.517753326Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:47:52.718939168Z Gathering data for ns/kube-system...
2026-11-03T15:47:53.286336183Z Waiting on subprocesses to finish execution.
2026-11-03T15:47:53.297372197Z Executing Istio gather script
2026-11-03T15:47:53.303967915Z INFO: Gathering HAProxy config files
2026-11-03T15:47:53.306852985Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-03T15:47:53.312340126Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-03T15:47:53.313536746Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-03T15:47:53.324512899Z INFO: Collecting host service logs for crio
2026-11-03T15:47:53.324775873Z INFO: Collecting host service logs for kubelet
2026-11-03T15:47:53.325129519Z INFO: Collecting host service logs for rpm-ostreed
2026-11-03T15:47:53.325457822Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-03T15:47:53.325744387Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-03T15:47:53.326033978Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-03T15:47:53.326353484Z INFO: Collecting host service logs for NetworkManager
2026-11-03T15:47:53.326642859Z INFO: Collecting host service logs for openvswitch
2026-11-03T15:47:53.326929398Z INFO: Collecting host service logs for ovs-configuration
2026-11-03T15:47:53.327225134Z INFO: Collecting host service logs for ovsdb-server
2026-11-03T15:47:53.327611858Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-03T15:47:53.327952849Z INFO: Waiting for worker host service log collection to complete ...
2026-11-03T15:47:53.416478549Z INFO: Waiting for node performance related collection to complete ...
2026-11-03T15:47:53.714598300Z No resources found in openshift-etcd namespace.
2026-11-03T15:47:54.070786695Z INFO: "metallb-operator" not detected. Skipping.
2026-11-03T15:47:54.417730249Z Wrote inspect data to must-gather.
2026-11-03T15:47:54.450706890Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-03T15:47:54.519861011Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-zrn4t
2026-11-03T15:47:54.519861011Z insights-runtime-extractor-8fksr
2026-11-03T15:47:54.519861011Z insights-runtime-extractor-ll5rf
2026-11-03T15:47:54.519861011Z insights-runtime-extractor-xbjr5
2026-11-03T15:47:54.531767470Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-03T15:47:54.610030775Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-03T15:47:54.610080658Z INFO: on-disk MachineConfig config collection complete.
2026-11-03T15:47:54.636349089Z No resources found
2026-11-03T15:47:54.658231829Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-03T15:47:54.690527364Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-03T15:47:54.801541142Z INFO: "nfd" not detected. Skipping.
2026-11-03T15:47:54.812794625Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-03T15:47:54.812794625Z See 'oc rsync -h' for help and examples
2026-11-03T15:47:54.826180799Z error: the server doesn't have a resource type "performanceprofile"
2026-11-03T15:47:54.915204390Z INFO: Worker host service log collection to complete.
2026-11-03T15:47:54.991529519Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-03T15:47:54.999865495Z ERROR: No running kube-apiserver pods found
2026-11-03T15:47:55.510435266Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-03T15:47:55.526529176Z Wrote inspect data to must-gather.
2026-11-03T15:47:55.549656203Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T15:47:55.625631643Z Wrote inspect data to must-gather.
2026-11-03T15:47:55.642513699Z No resources found
2026-11-03T15:47:55.692634185Z Wrote inspect data to must-gather.
2026-11-03T15:47:55.700951627Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-03T15:47:55.704099065Z INFO: 'previous-logs' folder not found on ip-10-0-133-102.us-west-2.compute.internal, skipping...
2026-11-03T15:47:55.876561954Z Wrote inspect data to must-gather.
2026-11-03T15:47:55.913862639Z Wrote inspect data to must-gather.
2026-11-03T15:47:55.998157679Z Wrote inspect data to must-gather.
2026-11-03T15:47:56.078583302Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-03T15:47:56.110158681Z INFO: 'previous-logs' folder not found on ip-10-0-133-228.us-west-2.compute.internal, skipping...
2026-11-03T15:47:56.274988867Z tar: Removing leading `/' from member names
2026-11-03T15:47:56.364725581Z error: the server doesn't have a resource type "machineconfigs"
2026-11-03T15:47:56.372433542Z INFO: HAProxy config collection complete.
2026-11-03T15:47:56.461243445Z INFO: 'previous-logs' folder not found on ip-10-0-136-152.us-west-2.compute.internal, skipping...
2026-11-03T15:47:56.461393338Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-03T15:47:56.461475685Z INFO: Machine Config Daemon termination log collection complete.
2026-11-03T15:47:56.521932672Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:47:56.565210559Z Wrote inspect data to must-gather.
2026-11-03T15:47:56.831095688Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-03T15:47:57.066570444Z INFO: Getting rules from prometheus-k8s-0
2026-11-03T15:47:57.201854559Z Wrote inspect data to must-gather.
2026-11-03T15:47:57.336520516Z Wrote inspect data to must-gather.
2026-11-03T15:47:57.602288258Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-03T15:47:57.661279330Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-03T15:47:57.807558988Z Wrote inspect data to must-gather.
2026-11-03T15:47:57.948838686Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-03T15:47:58.079699598Z Wrote inspect data to must-gather.
2026-11-03T15:47:58.083524040Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:47:58.494330319Z INFO: LEGACY MODE
2026-11-03T15:47:58.494368949Z INFO: Gathering ovn-kubernetes master data
2026-11-03T15:47:58.511108725Z Wrote inspect data to must-gather.
2026-11-03T15:47:58.577088453Z INFO: Getting status/config from prometheus-k8s-0
2026-11-03T15:47:58.693552369Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T15:47:58.726210336Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:47:58.728612589Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-03T15:47:58.982569937Z Wrote inspect data to must-gather.
2026-11-03T15:47:59.175024869Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:47:59.259015635Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T15:47:59.471398115Z Wrote inspect data to must-gather.
2026-11-03T15:47:59.837488985Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:47:59.840420229Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-03T15:48:00.018318208Z Wrote inspect data to must-gather.
2026-11-03T15:48:00.138333491Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-03T15:48:00.351027964Z Gathering data for ns/openshift-etcd...
2026-11-03T15:48:00.395210314Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:00.433234472Z Wrote inspect data to must-gather.
2026-11-03T15:48:00.580391178Z INFO: Waiting for network log collection to complete ...
2026-11-03T15:48:00.580429990Z INFO: Waiting for ovnk database copies to complete ...
2026-11-03T15:48:00.580846761Z INFO: Copying ovnk databases complete.
2026-11-03T15:48:00.582354899Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-03T15:48:00.582848972Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 731 is not a child of this shell
2026-11-03T15:48:00.582848972Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 732 is not a child of this shell
2026-11-03T15:48:00.582848972Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 733 is not a child of this shell
2026-11-03T15:48:00.582848972Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 735 is not a child of this shell
2026-11-03T15:48:00.582848972Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 736 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 737 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 739 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 740 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 748 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 749 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 750 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 753 is not a child of this shell
2026-11-03T15:48:00.582870512Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 754 is not a child of this shell
2026-11-03T15:48:00.585624376Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-03T15:48:00.588621012Z tar: Exiting with failure status due to previous errors
2026-11-03T15:48:00.589171529Z INFO: Network log collection complete.
2026-11-03T15:48:00.792230904Z Gathering data for ns/openshift-monitoring...
2026-11-03T15:48:01.003753959Z Wrote inspect data to must-gather.
2026-11-03T15:48:01.009696228Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-03T15:48:01.018247879Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:01.022069883Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-03T15:48:01.518727365Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:01.526345550Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:01.821258803Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-03T15:48:01.908380549Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-03T15:48:02.120105958Z Wrote inspect data to must-gather.
2026-11-03T15:48:02.120147838Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T15:48:02.120147838Z     namespaces "assisted-installer" not found
2026-11-03T15:48:02.163225359Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:02.166175247Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-03T15:48:02.206751022Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-03T15:48:02.549910487Z daemonset.apps/perf-node-gather-daemonset created
2026-11-03T15:48:02.723688494Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:02.808841386Z Waiting for performance profile collector pods to become ready: 1
2026-11-03T15:48:02.967897575Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-03T15:48:03.404252014Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:03.407395228Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-03T15:48:03.801206999Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:04.067562222Z Waiting for performance profile collector pods to become ready: 2
2026-11-03T15:48:04.200952699Z INFO: Getting status from alertmanager-main-0
2026-11-03T15:48:04.518104888Z Wrote inspect data to /must-gather/istio.
2026-11-03T15:48:05.313793204Z Waiting for performance profile collector pods to become ready: 3
2026-11-03T15:48:06.364010707Z Done executing Istio gather script
2026-11-03T15:48:06.530585691Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:06.571745867Z Waiting for performance profile collector pods to become ready: 4
2026-11-03T15:48:07.824274085Z Waiting for performance profile collector pods to become ready: 5
2026-11-03T15:48:09.065103263Z Waiting for performance profile collector pods to become ready: 6
2026-11-03T15:48:10.301423062Z Waiting for performance profile collector pods to become ready: 7
2026-11-03T15:48:11.534894112Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:11.555418257Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-03T15:48:12.125421408Z Collecting performance related data for node ip-10-0-133-102.us-west-2.compute.internal
2026-11-03T15:48:12.132094476Z Collecting performance related data for node ip-10-0-133-228.us-west-2.compute.internal
2026-11-03T15:48:12.138810969Z Collecting performance related data for node ip-10-0-136-152.us-west-2.compute.internal
2026-11-03T15:48:16.539248796Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:20.828182530Z Gathering data for ns/openshift-network-console...
2026-11-03T15:48:21.543968341Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:23.351329765Z Gathering data for ns/openshift-console-operator...
2026-11-03T15:48:25.577433974Z Gathering data for ns/openshift-console...
2026-11-03T15:48:25.683086561Z Collecting kubelet logs for node ip-10-0-133-102.us-west-2.compute.internal
2026-11-03T15:48:25.684853582Z Collecting kubelet logs for node ip-10-0-133-228.us-west-2.compute.internal
2026-11-03T15:48:25.686947204Z Collecting kubelet logs for node ip-10-0-136-152.us-west-2.compute.internal
2026-11-03T15:48:26.530090259Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-hw2fs namespace
2026-11-03T15:48:26.549513968Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:26.587759166Z INFO: Node performance data collection complete.
2026-11-03T15:48:28.585816914Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-03T15:48:31.554147192Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:31.585801975Z Gathering data for ns/openshift-dns-operator...
2026-11-03T15:48:33.572615051Z Gathering data for ns/openshift-dns...
2026-11-03T15:48:36.558985394Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:39.350594472Z Gathering data for ns/openshift-image-registry...
2026-11-03T15:48:41.563440053Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:42.831655107Z Gathering data for ns/openshift-ingress-operator...
2026-11-03T15:48:44.722525732Z Gathering data for ns/openshift-ingress...
2026-11-03T15:48:46.567825945Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:47.011939331Z Gathering data for ns/openshift-ingress-canary...
2026-11-03T15:48:50.114204118Z Gathering data for ns/openshift-insights...
2026-11-03T15:48:51.572146356Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:56.576416085Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:48:57.999812706Z Gathering data for ns/openshift-jobset-operator...
2026-11-03T15:49:01.077535957Z Gathering data for ns/openshift-config...
2026-11-03T15:49:01.580544902Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:02.812642322Z Gathering data for ns/openshift-config-managed...
2026-11-03T15:49:04.729835934Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-03T15:49:06.473623046Z Gathering data for ns/openshift-kube-apiserver...
2026-11-03T15:49:06.587056855Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:08.216132900Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-03T15:49:09.993909743Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-03T15:49:11.591809298Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:11.849746674Z Gathering data for ns/openshift-kube-scheduler...
2026-11-03T15:49:13.582731598Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-03T15:49:15.434713828Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-03T15:49:16.596318552Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:17.989952595Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-03T15:49:20.142266395Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-03T15:49:21.600760830Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:22.716621602Z Gathering data for ns/openshift-multus...
2026-11-03T15:49:26.605128940Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:31.609628667Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:36.613872120Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:38.509410643Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-03T15:49:41.618491869Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:46.622912661Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:51.491444941Z Gathering data for ns/openshift-host-network...
2026-11-03T15:49:51.628809476Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:53.502512885Z Gathering data for ns/openshift-network-diagnostics...
2026-11-03T15:49:56.633516114Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:49:57.924022375Z Gathering data for ns/openshift-network-node-identity...
2026-11-03T15:50:00.810861115Z Gathering data for ns/openshift-network-operator...
2026-11-03T15:50:01.638816681Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:03.823733556Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-03T15:50:05.559971892Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-03T15:50:06.643164222Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:08.969531027Z Gathering data for ns/openshift-apiserver-operator...
2026-11-03T15:50:10.718072548Z Gathering data for ns/openshift-apiserver...
2026-11-03T15:50:11.647719543Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:13.129123087Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-03T15:50:14.861770960Z Gathering data for ns/openshift-controller-manager...
2026-11-03T15:50:16.652014504Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:16.679368681Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-03T15:50:19.696758549Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-03T15:50:21.492838372Z Gathering data for ns/openshift-service-ca-operator...
2026-11-03T15:50:21.656329910Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:23.647621334Z Gathering data for ns/openshift-service-ca...
2026-11-03T15:50:26.142586739Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-03T15:50:26.660664394Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:31.665080471Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T15:50:31.928056267Z Wrote inspect data to must-gather.
2026-11-03T15:50:31.928097097Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T15:50:31.928097097Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-03T15:50:32.361191672Z error: the server doesn't have a resource type "clusters"
2026-11-03T15:50:32.404701013Z Caches written to disk