2026-10-31T01:20:40.675494532Z [disk usage checker] Started 2026-10-31T01:20:40.679212724Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:20:41.082430575Z Gathering data for ns/openshift-cluster-version... 2026-10-31T01:20:41.360831741Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T01:20:41.563929966Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:20:41.762654529Z Gathering data for ns/default... 2026-10-31T01:20:42.460751099Z Gathering data for ns/openshift... 2026-10-31T01:20:43.281304833Z Gathering data for ns/kube-system... 2026-10-31T01:20:44.972198901Z Waiting on subprocesses to finish execution. 2026-10-31T01:20:44.983599098Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-10-31T01:20:44.984751574Z Executing Istio gather script 2026-10-31T01:20:44.989116566Z INFO: Gathering machine config daemon's old logs from all nodes 2026-10-31T01:20:44.994135578Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-10-31T01:20:44.998366622Z INFO: Gathering HAProxy config files 2026-10-31T01:20:45.028530376Z INFO: Collecting host service logs for crio 2026-10-31T01:20:45.028828237Z INFO: Collecting host service logs for kubelet 2026-10-31T01:20:45.029145617Z INFO: Collecting host service logs for rpm-ostreed 2026-10-31T01:20:45.029427632Z INFO: Collecting host service logs for ostree-finalize-staged 2026-10-31T01:20:45.029697845Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-10-31T01:20:45.030035202Z INFO: Collecting host service logs for machine-config-daemon-host 2026-10-31T01:20:45.030333787Z INFO: Collecting host service logs for NetworkManager 2026-10-31T01:20:45.030830104Z INFO: Collecting host service logs for openvswitch 2026-10-31T01:20:45.031191376Z INFO: Collecting host service logs for ovs-configuration 2026-10-31T01:20:45.031543838Z INFO: Collecting host service logs for ovsdb-server 2026-10-31T01:20:45.031974552Z INFO: Collecting host service logs for ovs-vswitchd 2026-10-31T01:20:45.032476161Z INFO: Waiting for worker host service log collection to complete ... 2026-10-31T01:20:45.072732450Z INFO: Waiting for node performance related collection to complete ... 2026-10-31T01:20:46.011856966Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-10-31T01:20:46.032381517Z INFO: Found 1 replicas - prometheus-k8s-0 2026-10-31T01:20:46.034904707Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:20:46.070133759Z error: the server doesn't have a resource type "performanceprofile" 2026-10-31T01:20:46.187544639Z Wrote inspect data to must-gather. 2026-10-31T01:20:46.197908676Z INFO: "metallb-operator" not detected. Skipping. 2026-10-31T01:20:46.349480179Z No resources found 2026-10-31T01:20:46.395534355Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-10-31T01:20:46.614900864Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-10-31T01:20:46.615001261Z INFO: on-disk MachineConfig config collection complete. 2026-10-31T01:20:46.800343179Z Gathering data for ns/openshift-etcd... 2026-10-31T01:20:46.870595349Z INFO: Waiting for HAProxy config collection to complete ... 2026-10-31T01:20:47.036977812Z INFO: "nfd" not detected. Skipping. 2026-10-31T01:20:47.048467216Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-10-31T01:20:47.176334158Z No resources found 2026-10-31T01:20:47.207442969Z INFO: "sriov-network-operator" not detected. Skipping. 2026-10-31T01:20:47.388245184Z INFO: Worker host service log collection to complete. 2026-10-31T01:20:47.393014258Z Wrote inspect data to must-gather. 2026-10-31T01:20:47.411474576Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-6f4gc 2026-10-31T01:20:47.411522552Z insights-runtime-extractor-h2d2j 2026-10-31T01:20:47.411541869Z insights-runtime-extractor-nqpjq 2026-10-31T01:20:47.411561072Z insights-runtime-extractor-qf2jx 2026-10-31T01:20:47.502161803Z Wrote inspect data to must-gather. 2026-10-31T01:20:47.530055989Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-10-31T01:20:47.566326597Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:20:47.628426952Z ERROR: No running kube-apiserver pods found 2026-10-31T01:20:47.629080168Z No resources found in openshift-etcd namespace. 2026-10-31T01:20:47.667147264Z Wrote inspect data to must-gather. 2026-10-31T01:20:47.806199691Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-10-31T01:20:47.806199691Z See 'oc rsync -h' for help and examples 2026-10-31T01:20:47.917110931Z Wrote inspect data to must-gather. 2026-10-31T01:20:48.064103362Z error: the server doesn't have a resource type "machineconfigs" 2026-10-31T01:20:48.074060499Z Wrote inspect data to must-gather. 2026-10-31T01:20:48.130649828Z tar: Removing leading `/' from member names 2026-10-31T01:20:48.156459653Z INFO: HAProxy config collection complete. 2026-10-31T01:20:48.201262290Z Wrote inspect data to must-gather. 2026-10-31T01:20:48.212069157Z Wrote inspect data to must-gather. 2026-10-31T01:20:48.212101057Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T01:20:48.212101057Z namespaces "assisted-installer" not found 2026-10-31T01:20:48.251361004Z INFO: 'previous-logs' folder not found on ip-10-0-131-133.ec2.internal, skipping... 2026-10-31T01:20:48.334670618Z INFO: Getting rules from prometheus-k8s-0 2026-10-31T01:20:48.351317435Z Gathering data for ns/openshift-cloud-credential-operator... 2026-10-31T01:20:48.388753953Z error: the server doesn't have a resource type "machineconfigpools" 2026-10-31T01:20:48.647571203Z INFO: 'previous-logs' folder not found on ip-10-0-131-90.ec2.internal, skipping... 2026-10-31T01:20:48.678934964Z Wrote inspect data to must-gather. 2026-10-31T01:20:48.808304046Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T01:20:48.826536394Z Wrote inspect data to must-gather. 2026-10-31T01:20:49.070590062Z INFO: 'previous-logs' folder not found on ip-10-0-139-153.ec2.internal, skipping... 2026-10-31T01:20:49.070625768Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-10-31T01:20:49.070653980Z INFO: Machine Config Daemon termination log collection complete. 2026-10-31T01:20:49.121182982Z Wrote inspect data to must-gather. 2026-10-31T01:20:49.124477761Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:20:49.153106038Z error: the server doesn't have a resource type "kubeletconfigs" 2026-10-31T01:20:49.194489387Z INFO: Getting status/config from prometheus-k8s-0 2026-10-31T01:20:49.254001143Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-10-31T01:20:49.559809901Z Wrote inspect data to must-gather. 2026-10-31T01:20:49.605096605Z Wrote inspect data to must-gather. 2026-10-31T01:20:49.907463719Z Gathering data for ns/openshift-monitoring... 2026-10-31T01:20:49.932588544Z Wrote inspect data to must-gather. 2026-10-31T01:20:49.982505699Z INFO: Getting status/flags from prometheus-k8s-0 2026-10-31T01:20:50.077437679Z INFO: LEGACY MODE 2026-10-31T01:20:50.077483364Z INFO: Gathering ovn-kubernetes master data 2026-10-31T01:20:50.101063671Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io 2026-10-31T01:20:50.264417133Z Wrote inspect data to must-gather. 2026-10-31T01:20:50.384745676Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-10-31T01:20:50.441478668Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:50.612147670Z Wrote inspect data to must-gather. 2026-10-31T01:20:50.791714546Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:50.794396583Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-10-31T01:20:50.826219134Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-10-31T01:20:50.966456923Z Wrote inspect data to must-gather. 2026-10-31T01:20:51.054297031Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:20:51.120557347Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:51.298905226Z Wrote inspect data to must-gather. 2026-10-31T01:20:51.313976842Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-10-31T01:20:51.380505621Z INFO: Waiting for network log collection to complete ... 2026-10-31T01:20:51.380539234Z INFO: Waiting for ovnk database copies to complete ... 2026-10-31T01:20:51.381100845Z INFO: Copying ovnk databases complete. 2026-10-31T01:20:51.382953431Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory 2026-10-31T01:20:51.383509272Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 698 is not a child of this shell 2026-10-31T01:20:51.383509272Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 699 is not a child of this shell 2026-10-31T01:20:51.383509272Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 700 is not a child of this shell 2026-10-31T01:20:51.383509272Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 701 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 702 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 703 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 704 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 705 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 741 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 742 is not a child of this shell 2026-10-31T01:20:51.383533191Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 743 is not a child of this shell 2026-10-31T01:20:51.383543304Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 744 is not a child of this shell 2026-10-31T01:20:51.383543304Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 745 is not a child of this shell 2026-10-31T01:20:51.388712013Z tar: ovnk_database_store: Cannot stat: No such file or directory 2026-10-31T01:20:51.394059217Z tar: Exiting with failure status due to previous errors 2026-10-31T01:20:51.394837849Z INFO: Network log collection complete. 2026-10-31T01:20:51.494324573Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:51.498141203Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-10-31T01:20:51.667395253Z Wrote inspect data to must-gather. 2026-10-31T01:20:51.797964625Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:51.948853481Z INFO: Getting status from alertmanager-main-0 2026-10-31T01:20:52.139838414Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:52.142507693Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-10-31T01:20:52.193020329Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod 2026-10-31T01:20:52.388712376Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c 2026-10-31T01:20:52.447272445Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:52.617578400Z daemonset.apps/perf-node-gather-daemonset created 2026-10-31T01:20:52.787433394Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:52.790121000Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-10-31T01:20:52.836732339Z Waiting for performance profile collector pods to become ready: 1 2026-10-31T01:20:53.113345128Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:53.444999006Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:53.447703114Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-10-31T01:20:53.706316683Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:54.024744502Z Waiting for performance profile collector pods to become ready: 2 2026-10-31T01:20:54.059732277Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:54.063326621Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-10-31T01:20:54.318549105Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:54.656763923Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:54.659838455Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-10-31T01:20:54.904359385Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:55.203702262Z Waiting for performance profile collector pods to become ready: 3 2026-10-31T01:20:55.214614082Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:20:56.059014218Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:20:56.393756737Z Waiting for performance profile collector pods to become ready: 4 2026-10-31T01:20:56.429525106Z Done executing Istio gather script 2026-10-31T01:20:57.062229252Z Gathering data for ns/openshift-network-console... 2026-10-31T01:20:57.570847109Z Waiting for performance profile collector pods to become ready: 5 2026-10-31T01:20:58.072055168Z Gathering data for ns/openshift-console-operator... 2026-10-31T01:20:58.755470186Z Waiting for performance profile collector pods to become ready: 6 2026-10-31T01:20:58.997795434Z Gathering data for ns/openshift-console... 2026-10-31T01:20:59.935438289Z Waiting for performance profile collector pods to become ready: 7 2026-10-31T01:21:00.190769749Z Gathering data for ns/openshift-cluster-storage-operator... 2026-10-31T01:21:01.065605814Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:01.114804184Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-10-31T01:21:01.425193738Z Gathering data for ns/openshift-dns-operator... 2026-10-31T01:21:01.518717706Z Collecting performance related data for node ip-10-0-131-133.ec2.internal 2026-10-31T01:21:01.526572917Z Collecting performance related data for node ip-10-0-131-90.ec2.internal 2026-10-31T01:21:01.533715951Z Collecting performance related data for node ip-10-0-139-153.ec2.internal 2026-10-31T01:21:02.287516060Z Gathering data for ns/openshift-dns... 2026-10-31T01:21:04.487320792Z Gathering data for ns/openshift-image-registry... 2026-10-31T01:21:05.887871452Z Gathering data for ns/openshift-ingress-operator... 2026-10-31T01:21:06.071340931Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:06.680423769Z Gathering data for ns/openshift-ingress... 2026-10-31T01:21:07.588063030Z Gathering data for ns/openshift-ingress-canary... 2026-10-31T01:21:08.773903386Z Gathering data for ns/openshift-insights... 2026-10-31T01:21:09.881463474Z Collecting kubelet logs for node ip-10-0-131-133.ec2.internal 2026-10-31T01:21:09.883271202Z Collecting kubelet logs for node ip-10-0-131-90.ec2.internal 2026-10-31T01:21:09.885098531Z Collecting kubelet logs for node ip-10-0-139-153.ec2.internal 2026-10-31T01:21:10.434567927Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-dxkrx namespace 2026-10-31T01:21:10.456301431Z INFO: Node performance data collection complete. 2026-10-31T01:21:11.075955643Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:13.224203837Z Gathering data for ns/kserve... 2026-10-31T01:21:16.085794244Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:16.136517113Z Gathering data for ns/openshift-config... 2026-10-31T01:21:16.841601378Z Gathering data for ns/openshift-config-managed... 2026-10-31T01:21:17.710363556Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-10-31T01:21:18.452764533Z Gathering data for ns/openshift-kube-apiserver... 2026-10-31T01:21:19.173704445Z Gathering data for ns/openshift-kube-controller-manager... 2026-10-31T01:21:19.897836237Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-10-31T01:21:20.643903737Z Gathering data for ns/openshift-kube-scheduler... 2026-10-31T01:21:21.091637891Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:21.353826515Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-10-31T01:21:22.113528789Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-10-31T01:21:23.095170461Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-10-31T01:21:23.940390435Z Gathering data for ns/openshift-user-workload-monitoring... 2026-10-31T01:21:25.068306572Z Gathering data for ns/openshift-multus... 2026-10-31T01:21:26.096809971Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:30.526149428Z Gathering data for ns/openshift-ovn-kubernetes... 2026-10-31T01:21:31.101446056Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:35.024063278Z Gathering data for ns/openshift-host-network... 2026-10-31T01:21:35.843113022Z Gathering data for ns/openshift-network-diagnostics... 2026-10-31T01:21:36.106074753Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:37.527039245Z Gathering data for ns/openshift-network-node-identity... 2026-10-31T01:21:38.706976257Z Gathering data for ns/openshift-network-operator... 2026-10-31T01:21:39.858323180Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-10-31T01:21:40.591257965Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-10-31T01:21:41.110842781Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:41.919351503Z Gathering data for ns/openshift-apiserver-operator... 2026-10-31T01:21:42.663923433Z Gathering data for ns/openshift-apiserver... 2026-10-31T01:21:43.684033765Z Gathering data for ns/openshift-controller-manager-operator... 2026-10-31T01:21:44.389504222Z Gathering data for ns/openshift-controller-manager... 2026-10-31T01:21:45.112981511Z Gathering data for ns/openshift-cluster-samples-operator... 2026-10-31T01:21:46.116874875Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:46.452252363Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-10-31T01:21:47.215083777Z Gathering data for ns/openshift-service-ca-operator... 2026-10-31T01:21:48.081650410Z Gathering data for ns/openshift-service-ca... 2026-10-31T01:21:49.066372712Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-10-31T01:21:51.121610702Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-10-31T01:21:51.363008142Z Wrote inspect data to must-gather. 2026-10-31T01:21:51.363055856Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T01:21:51.363055856Z skipping gathering secrets/support due to error: secrets "support" not found 2026-10-31T01:21:51.568409613Z error: the server doesn't have a resource type "clusters" 2026-10-31T01:21:51.651656888Z Caches written to disk