2026-11-03T13:48:01.006718544Z [disk usage checker] Started
2026-11-03T13:48:01.010007077Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:01.901120523Z Gathering data for ns/openshift-cluster-version...
2026-11-03T13:48:02.790983180Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T13:48:03.336703498Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T13:48:03.837977129Z Gathering data for ns/default...
2026-11-03T13:48:05.643887324Z Gathering data for ns/openshift...
2026-11-03T13:48:06.014761305Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:07.560969641Z Gathering data for ns/kube-system...
2026-11-03T13:48:07.622663846Z Waiting on subprocesses to finish execution.
2026-11-03T13:48:07.628462065Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-03T13:48:07.639959627Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-03T13:48:07.643976382Z INFO: Gathering HAProxy config files
2026-11-03T13:48:07.649227569Z Executing Istio gather script
2026-11-03T13:48:07.652984998Z INFO: Collecting host service logs for crio
2026-11-03T13:48:07.653328225Z INFO: Collecting host service logs for kubelet
2026-11-03T13:48:07.653678868Z INFO: Collecting host service logs for rpm-ostreed
2026-11-03T13:48:07.653717374Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-03T13:48:07.654152858Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-03T13:48:07.654473437Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-03T13:48:07.654830863Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-03T13:48:07.655152425Z INFO: Collecting host service logs for NetworkManager
2026-11-03T13:48:07.655462579Z INFO: Collecting host service logs for openvswitch
2026-11-03T13:48:07.655796264Z INFO: Collecting host service logs for ovs-configuration
2026-11-03T13:48:07.656196982Z INFO: Collecting host service logs for ovsdb-server
2026-11-03T13:48:07.656532695Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-03T13:48:07.656933946Z INFO: Waiting for worker host service log collection to complete ...
2026-11-03T13:48:07.950695080Z INFO: Waiting for node performance related collection to complete ...
2026-11-03T13:48:08.051870272Z No resources found
2026-11-03T13:48:08.078830015Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-03T13:48:08.257001716Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-7qrjx
2026-11-03T13:48:08.257055487Z insights-runtime-extractor-knhvp
2026-11-03T13:48:08.257078660Z insights-runtime-extractor-s8h77
2026-11-03T13:48:08.257099313Z insights-runtime-extractor-ztj9j
2026-11-03T13:48:08.292126452Z INFO: "nfd" not detected. Skipping.
2026-11-03T13:48:08.404420631Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-03T13:48:08.639105093Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-03T13:48:08.639181610Z INFO: on-disk MachineConfig config collection complete.
2026-11-03T13:48:08.654904910Z INFO: "metallb-operator" not detected. Skipping.
2026-11-03T13:48:08.745605399Z No resources found in openshift-etcd namespace.
2026-11-03T13:48:08.780391769Z Wrote inspect data to must-gather.
2026-11-03T13:48:08.798399726Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-03T13:48:08.798399726Z See 'oc rsync -h' for help and examples
2026-11-03T13:48:08.929629509Z error: the server doesn't have a resource type "performanceprofile"
2026-11-03T13:48:09.087086278Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-03T13:48:09.090574773Z INFO: Worker host service log collection to complete.
2026-11-03T13:48:09.310316760Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-03T13:48:09.558168012Z Wrote inspect data to must-gather.
2026-11-03T13:48:09.728034331Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-03T13:48:09.883436342Z INFO: 'previous-logs' folder not found on ip-10-0-130-143.us-west-1.compute.internal, skipping...
2026-11-03T13:48:09.898899460Z No resources found
2026-11-03T13:48:09.936175339Z ERROR: No running kube-apiserver pods found
2026-11-03T13:48:09.983800733Z Wrote inspect data to must-gather.
2026-11-03T13:48:09.997357210Z Wrote inspect data to must-gather.
2026-11-03T13:48:10.016001271Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-03T13:48:10.052807779Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T13:48:10.077644905Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-03T13:48:10.175216682Z Wrote inspect data to must-gather.
2026-11-03T13:48:10.237232086Z Wrote inspect data to must-gather.
2026-11-03T13:48:10.238521103Z Wrote inspect data to must-gather.
2026-11-03T13:48:10.310014638Z INFO: 'previous-logs' folder not found on ip-10-0-136-83.us-west-1.compute.internal, skipping...
2026-11-03T13:48:10.531391862Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-03T13:48:10.570162281Z error: the server doesn't have a resource type "machineconfigs"
2026-11-03T13:48:10.660045487Z INFO: 'previous-logs' folder not found on ip-10-0-142-136.us-west-1.compute.internal, skipping...
2026-11-03T13:48:10.660102422Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-03T13:48:10.660102422Z INFO: Machine Config Daemon termination log collection complete.
2026-11-03T13:48:10.728008856Z Wrote inspect data to must-gather.
2026-11-03T13:48:10.960230600Z tar: Removing leading `/' from member names
2026-11-03T13:48:11.015490069Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-03T13:48:11.019494262Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:11.065062622Z INFO: HAProxy config collection complete.
2026-11-03T13:48:11.415443334Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-03T13:48:11.415488069Z Wrote inspect data to must-gather.
2026-11-03T13:48:11.444610770Z INFO: Getting rules from prometheus-k8s-0
2026-11-03T13:48:11.541387632Z Wrote inspect data to must-gather.
2026-11-03T13:48:11.940219387Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:11.963405123Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-03T13:48:12.063151538Z Wrote inspect data to must-gather.
2026-11-03T13:48:12.378253436Z Wrote inspect data to must-gather.
2026-11-03T13:48:12.563726811Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:12.566225720Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-03T13:48:12.732492038Z INFO: LEGACY MODE
2026-11-03T13:48:12.732532299Z INFO: Gathering ovn-kubernetes master data
2026-11-03T13:48:12.786027223Z Wrote inspect data to must-gather.
2026-11-03T13:48:12.881537948Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-03T13:48:13.004056156Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:13.204459439Z Wrote inspect data to must-gather.
2026-11-03T13:48:13.228177846Z INFO: Getting status/config from prometheus-k8s-0
2026-11-03T13:48:13.701068610Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:13.703739007Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-03T13:48:13.852452467Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T13:48:13.865498327Z Wrote inspect data to must-gather.
2026-11-03T13:48:14.374924428Z Wrote inspect data to must-gather.
2026-11-03T13:48:14.388632021Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:14.602878517Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T13:48:14.797822403Z Wrote inspect data to must-gather.
2026-11-03T13:48:14.807446580Z INFO: Waiting for network log collection to complete ...
2026-11-03T13:48:14.807571899Z INFO: Waiting for ovnk database copies to complete ...
2026-11-03T13:48:14.808263085Z INFO: Copying ovnk databases complete.
2026-11-03T13:48:14.810438393Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-03T13:48:14.811087898Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 717 is not a child of this shell
2026-11-03T13:48:14.811149549Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 718 is not a child of this shell
2026-11-03T13:48:14.811175805Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 720 is not a child of this shell
2026-11-03T13:48:14.811210079Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 723 is not a child of this shell
2026-11-03T13:48:14.811245053Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 725 is not a child of this shell
2026-11-03T13:48:14.811269055Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 726 is not a child of this shell
2026-11-03T13:48:14.811292292Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 727 is not a child of this shell
2026-11-03T13:48:14.811334225Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 728 is not a child of this shell
2026-11-03T13:48:14.811334225Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 738 is not a child of this shell
2026-11-03T13:48:14.811334225Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 739 is not a child of this shell
2026-11-03T13:48:14.811363629Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 741 is not a child of this shell
2026-11-03T13:48:14.811363629Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 742 is not a child of this shell
2026-11-03T13:48:14.811363629Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 743 is not a child of this shell
2026-11-03T13:48:14.815313383Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-03T13:48:14.819006694Z tar: Exiting with failure status due to previous errors
2026-11-03T13:48:14.819562208Z INFO: Network log collection complete.
2026-11-03T13:48:15.394614956Z Wrote inspect data to must-gather.
2026-11-03T13:48:15.410425614Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:15.413845011Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-03T13:48:15.660452307Z Gathering data for ns/openshift-etcd...
2026-11-03T13:48:15.768883303Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-03T13:48:15.961596324Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:16.024204413Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:16.369245428Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-03T13:48:16.655161904Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-03T13:48:16.671365347Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:16.674163919Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-03T13:48:16.729403091Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-03T13:48:16.888936187Z Gathering data for ns/openshift-monitoring...
2026-11-03T13:48:17.099663434Z daemonset.apps/perf-node-gather-daemonset created
2026-11-03T13:48:17.227430859Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:17.363443380Z Waiting for performance profile collector pods to become ready: 1
2026-11-03T13:48:17.551033259Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-03T13:48:17.585848330Z Wrote inspect data to must-gather.
2026-11-03T13:48:17.585887906Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T13:48:17.585887906Z     namespaces "assisted-installer" not found
2026-11-03T13:48:17.877428809Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:17.880303123Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-03T13:48:18.267019293Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:18.608936123Z Waiting for performance profile collector pods to become ready: 2
2026-11-03T13:48:18.875201504Z Wrote inspect data to /must-gather/istio.
2026-11-03T13:48:19.015985337Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-03T13:48:19.861192074Z Waiting for performance profile collector pods to become ready: 3
2026-11-03T13:48:20.219614268Z INFO: Getting status from alertmanager-main-0
2026-11-03T13:48:20.678808702Z Done executing Istio gather script
2026-11-03T13:48:21.028834406Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:21.109287309Z Waiting for performance profile collector pods to become ready: 4
2026-11-03T13:48:22.357620147Z Waiting for performance profile collector pods to become ready: 5
2026-11-03T13:48:23.603199667Z Waiting for performance profile collector pods to become ready: 6
2026-11-03T13:48:24.849917496Z Waiting for performance profile collector pods to become ready: 7
2026-11-03T13:48:26.033430227Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:26.099335491Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-03T13:48:26.654416508Z Collecting performance related data for node ip-10-0-130-143.us-west-1.compute.internal
2026-11-03T13:48:26.661370645Z Collecting performance related data for node ip-10-0-142-136.us-west-1.compute.internal
2026-11-03T13:48:26.668572292Z Collecting performance related data for node ip-10-0-136-83.us-west-1.compute.internal
2026-11-03T13:48:31.038160101Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:36.043689871Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:39.199731000Z Gathering data for ns/openshift-network-console...
2026-11-03T13:48:39.902476365Z Collecting kubelet logs for node ip-10-0-130-143.us-west-1.compute.internal
2026-11-03T13:48:39.904374113Z Collecting kubelet logs for node ip-10-0-142-136.us-west-1.compute.internal
2026-11-03T13:48:39.906505122Z Collecting kubelet logs for node ip-10-0-136-83.us-west-1.compute.internal
2026-11-03T13:48:40.783478636Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-p55h4 namespace
2026-11-03T13:48:40.839145484Z INFO: Node performance data collection complete.
2026-11-03T13:48:41.048956711Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:41.875840042Z Gathering data for ns/openshift-console-operator...
2026-11-03T13:48:44.487756866Z Gathering data for ns/openshift-console...
2026-11-03T13:48:46.053728646Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:47.857280063Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-03T13:48:51.058546414Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:51.491801484Z Gathering data for ns/openshift-dns-operator...
2026-11-03T13:48:53.594228296Z Gathering data for ns/openshift-dns...
2026-11-03T13:48:56.063256012Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:48:59.815913282Z Gathering data for ns/openshift-image-registry...
2026-11-03T13:49:01.067990601Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:03.718051445Z Gathering data for ns/openshift-ingress-operator...
2026-11-03T13:49:05.844162622Z Gathering data for ns/openshift-ingress...
2026-11-03T13:49:06.072785179Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:08.275806213Z Gathering data for ns/openshift-ingress-canary...
2026-11-03T13:49:11.077427531Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:11.542452644Z Gathering data for ns/openshift-insights...
2026-11-03T13:49:16.082230416Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:21.087037356Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:21.358939837Z Gathering data for ns/openshift-jobset-operator...
2026-11-03T13:49:24.649476833Z Gathering data for ns/openshift-config...
2026-11-03T13:49:26.093033031Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:26.558272235Z Gathering data for ns/openshift-config-managed...
2026-11-03T13:49:28.662333157Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-03T13:49:30.505974345Z Gathering data for ns/openshift-kube-apiserver...
2026-11-03T13:49:31.098113268Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:32.348775816Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-03T13:49:34.264250612Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-03T13:49:36.102741782Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:36.246457836Z Gathering data for ns/openshift-kube-scheduler...
2026-11-03T13:49:38.089134863Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-03T13:49:40.068549545Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-03T13:49:41.107426818Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:42.775530714Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-03T13:49:45.168386916Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-03T13:49:46.112202297Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:47.948479296Z Gathering data for ns/openshift-multus...
2026-11-03T13:49:51.116866231Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:49:56.121509038Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:01.126253075Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:04.879562689Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-03T13:50:06.131070292Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:11.135693320Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:16.140258842Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:19.584789266Z Gathering data for ns/openshift-host-network...
2026-11-03T13:50:21.144989034Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:21.724404888Z Gathering data for ns/openshift-network-diagnostics...
2026-11-03T13:50:26.149732788Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:26.351098314Z Gathering data for ns/openshift-network-node-identity...
2026-11-03T13:50:29.430188286Z Gathering data for ns/openshift-network-operator...
2026-11-03T13:50:31.154324092Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:32.636181395Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-03T13:50:34.480384649Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-03T13:50:36.158951001Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:38.150303616Z Gathering data for ns/openshift-apiserver-operator...
2026-11-03T13:50:40.075679365Z Gathering data for ns/openshift-apiserver...
2026-11-03T13:50:41.163937074Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:42.653445746Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-03T13:50:44.548098858Z Gathering data for ns/openshift-controller-manager...
2026-11-03T13:50:46.168644581Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:46.450734572Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-03T13:50:49.732685953Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-03T13:50:51.173297448Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:51.644051788Z Gathering data for ns/openshift-service-ca-operator...
2026-11-03T13:50:53.993684822Z Gathering data for ns/openshift-service-ca...
2026-11-03T13:50:56.177938732Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:50:56.646528892Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-03T13:51:01.182589528Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T13:51:02.875990872Z Wrote inspect data to must-gather.
2026-11-03T13:51:02.876048416Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T13:51:02.876048416Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-03T13:51:03.297667884Z error: the server doesn't have a resource type "clusters"
2026-11-03T13:51:03.342881008Z Caches written to disk