2026-07-30T08:40:25.246916950Z [disk usage checker] Started
2026-07-30T08:40:25.250193275Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:25.428768922Z Gathering data for ns/openshift-cluster-version...
2026-07-30T08:40:25.547874319Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T08:40:25.625555605Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T08:40:25.696387709Z Gathering data for ns/default...
2026-07-30T08:40:25.957927906Z Gathering data for ns/openshift...
2026-07-30T08:40:26.365880010Z Gathering data for ns/kube-system...
2026-07-30T08:40:26.999428722Z Gathering data for ns/openshift-etcd...
2026-07-30T08:40:27.267710041Z Wrote inspect data to must-gather.
2026-07-30T08:40:27.267743547Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T08:40:27.267743547Z     namespaces "assisted-installer" not found
2026-07-30T08:40:27.629632529Z Waiting on subprocesses to finish execution.
2026-07-30T08:40:27.635251896Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-30T08:40:27.635733131Z INFO: Gathering HAProxy config files
2026-07-30T08:40:27.642339591Z Executing Istio gather script
2026-07-30T08:40:27.646153720Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-30T08:40:27.661700696Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-30T08:40:27.662830983Z INFO: Waiting for node performance related collection to complete ...
2026-07-30T08:40:27.666682604Z INFO: Collecting host service logs for crio
2026-07-30T08:40:27.667015933Z INFO: Collecting host service logs for kubelet
2026-07-30T08:40:27.667337780Z INFO: Collecting host service logs for rpm-ostreed
2026-07-30T08:40:27.667709359Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-30T08:40:27.668058053Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-30T08:40:27.668390970Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-30T08:40:27.668729306Z INFO: Collecting host service logs for NetworkManager
2026-07-30T08:40:27.669049656Z INFO: Collecting host service logs for openvswitch
2026-07-30T08:40:27.669388097Z INFO: Collecting host service logs for ovs-configuration
2026-07-30T08:40:27.669722472Z INFO: Collecting host service logs for ovsdb-server
2026-07-30T08:40:27.670051456Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-30T08:40:27.670478644Z INFO: Waiting for worker host service log collection to complete ...
2026-07-30T08:40:28.164931076Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-30T08:40:28.165013700Z INFO: on-disk MachineConfig config collection complete.
2026-07-30T08:40:28.270730003Z No resources found in openshift-etcd namespace.
2026-07-30T08:40:28.324367114Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-30T08:40:28.351918797Z error: the server doesn't have a resource type "performanceprofile"
2026-07-30T08:40:28.480171207Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-30T08:40:28.531477669Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-30T08:40:28.746066388Z Wrote inspect data to must-gather.
2026-07-30T08:40:28.833401629Z No resources found
2026-07-30T08:40:28.963391891Z Wrote inspect data to must-gather.
2026-07-30T08:40:29.051853682Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-30T08:40:29.178406926Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-75lsm
2026-07-30T08:40:29.178460283Z insights-runtime-extractor-4twmv
2026-07-30T08:40:29.178479574Z insights-runtime-extractor-fvtcf
2026-07-30T08:40:29.178498567Z insights-runtime-extractor-wd5z5
2026-07-30T08:40:29.249454140Z INFO: "metallb-operator" not detected. Skipping.
2026-07-30T08:40:29.259098755Z INFO: Worker host service log collection to complete.
2026-07-30T08:40:29.521623658Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T08:40:29.643582617Z INFO: 'previous-logs' folder not found on ip-10-0-128-223.ec2.internal, skipping...
2026-07-30T08:40:29.753427950Z Wrote inspect data to must-gather.
2026-07-30T08:40:29.756520448Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-30T08:40:29.759225349Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-30T08:40:29.759225349Z See 'oc rsync -h' for help and examples
2026-07-30T08:40:29.771767241Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-30T08:40:29.791891854Z Wrote inspect data to must-gather.
2026-07-30T08:40:29.896519784Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-30T08:40:29.904964915Z No resources found
2026-07-30T08:40:29.938390074Z ERROR: No running kube-apiserver pods found
2026-07-30T08:40:30.143589449Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.171057643Z error: the server doesn't have a resource type "machineconfigs"
2026-07-30T08:40:30.186142425Z Gathering data for ns/openshift-monitoring...
2026-07-30T08:40:30.190471352Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.243727940Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.257280183Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-30T08:40:30.257475122Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:30.278287812Z INFO: 'previous-logs' folder not found on ip-10-0-136-119.ec2.internal, skipping...
2026-07-30T08:40:30.303822441Z tar: Removing leading `/' from member names
2026-07-30T08:40:30.308972531Z INFO: HAProxy config collection complete.
2026-07-30T08:40:30.378481015Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-30T08:40:30.481826076Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.528553448Z INFO: 'previous-logs' folder not found on ip-10-0-143-99.ec2.internal, skipping...
2026-07-30T08:40:30.528658398Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-30T08:40:30.528715506Z INFO: Machine Config Daemon termination log collection complete.
2026-07-30T08:40:30.533935650Z INFO: Getting rules from prometheus-k8s-0
2026-07-30T08:40:30.604136346Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.719773957Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.744272008Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-30T08:40:30.888989944Z INFO: Getting status/config from prometheus-k8s-0
2026-07-30T08:40:30.948351892Z Wrote inspect data to must-gather.
2026-07-30T08:40:30.955534540Z Wrote inspect data to must-gather.
2026-07-30T08:40:31.146653683Z Wrote inspect data to must-gather.
2026-07-30T08:40:31.203674378Z INFO: INTERCONNECT MODE
2026-07-30T08:40:31.203715308Z INFO: Gathering ovn-kubernetes DBs
2026-07-30T08:40:31.230664735Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-30T08:40:31.321033740Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-30T08:40:31.349105281Z Wrote inspect data to must-gather.
2026-07-30T08:40:31.360416614Z INFO: Gathering OVN_Northbound from ovnkube-node-78wc5...
2026-07-30T08:40:31.360620127Z INFO: Gathering OVN_Northbound from ovnkube-node-dqt85...
2026-07-30T08:40:31.360851875Z INFO: Gathering OVN_Northbound from ovnkube-node-nzfgq...
2026-07-30T08:40:31.361195742Z INFO: Gathering OVN_Southbound from ovnkube-node-78wc5...
2026-07-30T08:40:31.361465574Z INFO: Gathering OVN_Southbound from ovnkube-node-dqt85...
2026-07-30T08:40:31.361758436Z INFO: Gathering OVN_Southbound from ovnkube-node-nzfgq...
2026-07-30T08:40:31.456020251Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:31.635603111Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-30T08:40:31.647622186Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.648045068Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.657690278Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.695871779Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.700085320Z Wrote inspect data to must-gather.
2026-07-30T08:40:31.705801762Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.706968406Z tar: Removing leading `/' from member names
2026-07-30T08:40:31.796423040Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:31.798811316Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-30T08:40:31.860006060Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T08:40:31.919686643Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-30T08:40:31.939227167Z Wrote inspect data to must-gather.
2026-07-30T08:40:32.012838461Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:32.171322029Z INFO: Waiting for network log collection to complete ...
2026-07-30T08:40:32.171351487Z INFO: Waiting for ovnk database copies to complete ...
2026-07-30T08:40:32.171799472Z INFO: Copying ovnk databases complete.
2026-07-30T08:40:32.172565529Z Wrote inspect data to must-gather.
2026-07-30T08:40:32.174277532Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-30T08:40:32.176972250Z ovnk_database_store/
2026-07-30T08:40:32.177016869Z ovnk_database_store/ovnkube-node-78wc5_nbdb
2026-07-30T08:40:32.186258504Z ovnk_database_store/ovnkube-node-78wc5_sbdb
2026-07-30T08:40:32.188871683Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-30T08:40:32.205459614Z ovnk_database_store/ovnkube-node-nzfgq_sbdb
2026-07-30T08:40:32.226549862Z ovnk_database_store/ovnkube-node-nzfgq_nbdb
2026-07-30T08:40:32.236346726Z ovnk_database_store/ovnkube-node-dqt85_nbdb
2026-07-30T08:40:32.238796430Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:32.241035520Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-30T08:40:32.245587704Z ovnk_database_store/ovnkube-node-dqt85_sbdb
2026-07-30T08:40:32.269400884Z INFO: Network log collection complete.
2026-07-30T08:40:32.372848240Z W0730 08:40:32.372805    1532 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-30T08:40:32.378004510Z Wrote inspect data to must-gather.
2026-07-30T08:40:32.451419057Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:32.483254191Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T08:40:32.556290937Z INFO: Getting status from alertmanager-main-0
2026-07-30T08:40:32.646656114Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-30T08:40:32.649627179Z INFO: Fallback to identify the container image from release info
2026-07-30T08:40:32.674647208Z Gathering data for ns/openshift-network-console...
2026-07-30T08:40:32.675589270Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:32.678596266Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-30T08:40:32.877489740Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:33.049408562Z Gathering data for ns/openshift-console-operator...
2026-07-30T08:40:33.051232617Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:33.053672732Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-30T08:40:33.200987592Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:33.291099480Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-30T08:40:33.395937117Z Wrote inspect data to /must-gather/istio.
2026-07-30T08:40:33.424752938Z daemonset.apps/perf-node-gather-daemonset created
2026-07-30T08:40:33.428176811Z Gathering data for ns/openshift-console...
2026-07-30T08:40:33.549845010Z Waiting for performance profile collector pods to become ready: 1
2026-07-30T08:40:33.974243179Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-30T08:40:34.054122739Z Done executing Istio gather script
2026-07-30T08:40:34.425633934Z Gathering data for ns/openshift-dns-operator...
2026-07-30T08:40:34.660431923Z Waiting for performance profile collector pods to become ready: 2
2026-07-30T08:40:34.718122673Z Gathering data for ns/openshift-dns...
2026-07-30T08:40:35.260624407Z Gathering data for ns/openshift-image-registry...
2026-07-30T08:40:35.262435359Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:35.674691241Z Gathering data for ns/openshift-ingress-operator...
2026-07-30T08:40:35.776117673Z Waiting for performance profile collector pods to become ready: 3
2026-07-30T08:40:35.971671524Z Gathering data for ns/openshift-ingress...
2026-07-30T08:40:36.279425002Z Gathering data for ns/openshift-ingress-canary...
2026-07-30T08:40:36.628591325Z Gathering data for ns/openshift-insights...
2026-07-30T08:40:36.872585439Z Waiting for performance profile collector pods to become ready: 4
2026-07-30T08:40:37.970942821Z Waiting for performance profile collector pods to become ready: 5
2026-07-30T08:40:38.494900859Z Gathering data for ns/openshift-config...
2026-07-30T08:40:38.716705733Z Gathering data for ns/openshift-config-managed...
2026-07-30T08:40:39.084508764Z Waiting for performance profile collector pods to become ready: 6
2026-07-30T08:40:39.097326917Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-30T08:40:39.375442350Z Gathering data for ns/openshift-kube-apiserver...
2026-07-30T08:40:39.602547356Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-30T08:40:39.849151320Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-30T08:40:40.083222509Z Gathering data for ns/openshift-kube-scheduler...
2026-07-30T08:40:40.202329896Z Waiting for performance profile collector pods to become ready: 7
2026-07-30T08:40:40.267568795Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:40.357615047Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-30T08:40:40.604762484Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-30T08:40:40.930598168Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-30T08:40:41.206070684Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-30T08:40:41.313519380Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-30T08:40:41.546430178Z Collecting performance related data for node ip-10-0-136-119.ec2.internal
2026-07-30T08:40:41.555178042Z Collecting performance related data for node ip-10-0-128-223.ec2.internal
2026-07-30T08:40:41.562177253Z Collecting performance related data for node ip-10-0-143-99.ec2.internal
2026-07-30T08:40:41.707351230Z Gathering data for ns/openshift-multus...
2026-07-30T08:40:43.372332515Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-30T08:40:44.619935438Z Gathering data for ns/openshift-host-network...
2026-07-30T08:40:44.898172857Z Gathering data for ns/openshift-network-diagnostics...
2026-07-30T08:40:45.272305879Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:45.404896343Z Gathering data for ns/openshift-network-node-identity...
2026-07-30T08:40:45.783246441Z Gathering data for ns/openshift-network-operator...
2026-07-30T08:40:46.137097920Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-30T08:40:46.421076362Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-30T08:40:46.904573009Z Gathering data for ns/openshift-apiserver-operator...
2026-07-30T08:40:47.185570077Z Gathering data for ns/openshift-apiserver...
2026-07-30T08:40:47.587307073Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-30T08:40:47.845522230Z Gathering data for ns/openshift-controller-manager...
2026-07-30T08:40:48.087293586Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-30T08:40:48.711748387Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-30T08:40:49.017258530Z Gathering data for ns/openshift-service-ca-operator...
2026-07-30T08:40:49.134443945Z Collecting kubelet logs for node ip-10-0-136-119.ec2.internal
2026-07-30T08:40:49.136047893Z Collecting kubelet logs for node ip-10-0-128-223.ec2.internal
2026-07-30T08:40:49.137648236Z Collecting kubelet logs for node ip-10-0-143-99.ec2.internal
2026-07-30T08:40:49.343893452Z Gathering data for ns/openshift-service-ca...
2026-07-30T08:40:49.494552415Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-bh2gv namespace
2026-07-30T08:40:49.502423958Z INFO: Node performance data collection complete.
2026-07-30T08:40:49.754628503Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-30T08:40:50.277631399Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-30T08:40:50.606336974Z Wrote inspect data to must-gather.
2026-07-30T08:40:50.606395184Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T08:40:50.606395184Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-30T08:40:50.716527348Z error: the server doesn't have a resource type "clusters"
2026-07-30T08:40:50.830325831Z Caches written to disk