2026-07-02T15:52:33.804201794Z [disk usage checker] Started
2026-07-02T15:52:33.807476808Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:33.997726912Z Gathering data for ns/openshift-cluster-version...
2026-07-02T15:52:34.124124320Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T15:52:34.212020192Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T15:52:34.292240127Z Gathering data for ns/default...
2026-07-02T15:52:34.579129736Z Gathering data for ns/openshift...
2026-07-02T15:52:34.959072494Z Gathering data for ns/kube-system...
2026-07-02T15:52:35.704873921Z Gathering data for ns/openshift-etcd...
2026-07-02T15:52:35.974947744Z Wrote inspect data to must-gather.
2026-07-02T15:52:35.974981254Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T15:52:35.974981254Z     namespaces "assisted-installer" not found
2026-07-02T15:52:36.346115280Z Waiting on subprocesses to finish execution.
2026-07-02T15:52:36.356564866Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T15:52:36.358378396Z INFO: Gathering HAProxy config files
2026-07-02T15:52:36.359281682Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T15:52:36.365279575Z Executing Istio gather script
2026-07-02T15:52:36.369317827Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T15:52:36.371033777Z INFO: Collecting host service logs for crio
2026-07-02T15:52:36.371279781Z INFO: Collecting host service logs for kubelet
2026-07-02T15:52:36.371554633Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T15:52:36.371939061Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T15:52:36.372249544Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T15:52:36.372547443Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T15:52:36.372856852Z INFO: Collecting host service logs for NetworkManager
2026-07-02T15:52:36.373149436Z INFO: Collecting host service logs for openvswitch
2026-07-02T15:52:36.373457676Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T15:52:36.373775364Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T15:52:36.374064383Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T15:52:36.374440402Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T15:52:36.385822142Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T15:52:36.689641280Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-hv72j
2026-07-02T15:52:36.689698692Z insights-runtime-extractor-d5nkf
2026-07-02T15:52:36.689721842Z insights-runtime-extractor-jsqqg
2026-07-02T15:52:36.689742190Z insights-runtime-extractor-rbg7d
2026-07-02T15:52:36.833247787Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T15:52:36.923278957Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T15:52:36.923426808Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T15:52:37.140941332Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T15:52:37.149262322Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T15:52:37.220709008Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T15:52:37.220709008Z See 'oc rsync -h' for help and examples
2026-07-02T15:52:37.334455741Z No resources found
2026-07-02T15:52:37.404494627Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T15:52:37.517372454Z No resources found in openshift-etcd namespace.
2026-07-02T15:52:37.535451921Z Wrote inspect data to must-gather.
2026-07-02T15:52:37.599018599Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T15:52:37.711730029Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T15:52:37.726697153Z Wrote inspect data to must-gather.
2026-07-02T15:52:37.853117857Z ERROR: No running kube-apiserver pods found
2026-07-02T15:52:37.859820474Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T15:52:37.904069834Z INFO: Worker host service log collection to complete.
2026-07-02T15:52:37.922147590Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T15:52:38.118504304Z Wrote inspect data to must-gather.
2026-07-02T15:52:38.490096481Z Gathering data for ns/openshift-monitoring...
2026-07-02T15:52:38.610342300Z No resources found
2026-07-02T15:52:38.650346136Z Wrote inspect data to must-gather.
2026-07-02T15:52:38.753314546Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T15:52:38.853768572Z Wrote inspect data to must-gather.
2026-07-02T15:52:38.861273002Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T15:52:38.872736271Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:38.880652220Z tar: Removing leading `/' from member names
2026-07-02T15:52:38.886625259Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T15:52:38.887242964Z INFO: HAProxy config collection complete.
2026-07-02T15:52:38.912663670Z INFO: 'previous-logs' folder not found on ip-10-0-134-12.ec2.internal, skipping...
2026-07-02T15:52:38.935660215Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T15:52:38.937830242Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T15:52:38.954647833Z Wrote inspect data to must-gather.
2026-07-02T15:52:38.957930417Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.093814035Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T15:52:39.101878439Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.206740710Z INFO: 'previous-logs' folder not found on ip-10-0-136-55.ec2.internal, skipping...
2026-07-02T15:52:39.326815619Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.373177974Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.376473395Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T15:52:39.481167824Z INFO: 'previous-logs' folder not found on ip-10-0-140-71.ec2.internal, skipping...
2026-07-02T15:52:39.481199114Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T15:52:39.481208153Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T15:52:39.497536674Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T15:52:39.635134788Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.716639952Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.787887144Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T15:52:39.922107700Z INFO: INTERCONNECT MODE
2026-07-02T15:52:39.922143293Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T15:52:39.939142863Z Wrote inspect data to must-gather.
2026-07-02T15:52:39.944696596Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T15:52:40.020100635Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T15:52:40.102657034Z INFO: Gathering OVN_Northbound from ovnkube-node-kdqrl...
2026-07-02T15:52:40.103045674Z INFO: Gathering OVN_Northbound from ovnkube-node-mj5qr...
2026-07-02T15:52:40.103308926Z INFO: Gathering OVN_Northbound from ovnkube-node-z2ztz...
2026-07-02T15:52:40.103592753Z INFO: Gathering OVN_Southbound from ovnkube-node-kdqrl...
2026-07-02T15:52:40.103890646Z INFO: Gathering OVN_Southbound from ovnkube-node-mj5qr...
2026-07-02T15:52:40.104158455Z INFO: Gathering OVN_Southbound from ovnkube-node-z2ztz...
2026-07-02T15:52:40.154576051Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:40.180802253Z Wrote inspect data to must-gather.
2026-07-02T15:52:40.350180673Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T15:52:40.401188014Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T15:52:40.446727240Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.454296507Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.455275053Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.464553076Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.503338233Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:40.507799246Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T15:52:40.511052159Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.518029499Z tar: Removing leading `/' from member names
2026-07-02T15:52:40.544843883Z Wrote inspect data to must-gather.
2026-07-02T15:52:40.702773423Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T15:52:40.753361333Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:40.788719796Z Wrote inspect data to must-gather.
2026-07-02T15:52:40.951362533Z INFO: Waiting for network log collection to complete ...
2026-07-02T15:52:40.951454668Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T15:52:40.951854507Z INFO: Copying ovnk databases complete.
2026-07-02T15:52:40.953447500Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-02T15:52:40.955648421Z ovnk_database_store/
2026-07-02T15:52:40.955648421Z ovnk_database_store/ovnkube-node-kdqrl_sbdb
2026-07-02T15:52:40.972105373Z ovnk_database_store/ovnkube-node-kdqrl_nbdb
2026-07-02T15:52:40.979683694Z ovnk_database_store/ovnkube-node-z2ztz_sbdb
2026-07-02T15:52:40.992384487Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:40.995298696Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T15:52:40.997574762Z ovnk_database_store/ovnkube-node-z2ztz_nbdb
2026-07-02T15:52:41.005459679Z ovnk_database_store/ovnkube-node-mj5qr_sbdb
2026-07-02T15:52:41.018097250Z W0702 15:52:41.018055    1497 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T15:52:41.023224723Z Wrote inspect data to must-gather.
2026-07-02T15:52:41.023250917Z ovnk_database_store/ovnkube-node-mj5qr_nbdb
2026-07-02T15:52:41.035169521Z INFO: Network log collection complete.
2026-07-02T15:52:41.058548702Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T15:52:41.137092736Z INFO: Getting status from alertmanager-main-0
2026-07-02T15:52:41.200893146Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:41.238766886Z Gathering data for ns/openshift-network-console...
2026-07-02T15:52:41.322735955Z Wrote inspect data to must-gather.
2026-07-02T15:52:41.407066394Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:41.409253390Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T15:52:41.589580909Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T15:52:41.591598304Z INFO: Fallback to identify the container image from release info
2026-07-02T15:52:41.634859792Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:41.668902581Z Gathering data for ns/openshift-console-operator...
2026-07-02T15:52:41.825609349Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:41.827757314Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T15:52:41.986625707Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:42.047378814Z Gathering data for ns/openshift-console...
2026-07-02T15:52:42.160452561Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-02T15:52:42.180935421Z Wrote inspect data to /must-gather/istio.
2026-07-02T15:52:42.287896785Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T15:52:42.409799065Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T15:52:42.585929433Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T15:52:42.867565475Z Done executing Istio gather script
2026-07-02T15:52:43.084840753Z Gathering data for ns/openshift-dns-operator...
2026-07-02T15:52:43.326473127Z Gathering data for ns/openshift-dns...
2026-07-02T15:52:43.508235555Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T15:52:43.867374858Z Gathering data for ns/openshift-image-registry...
2026-07-02T15:52:43.891315460Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:44.311562547Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T15:52:44.602849854Z Gathering data for ns/openshift-ingress...
2026-07-02T15:52:44.630362964Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T15:52:44.891916512Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T15:52:45.239306362Z Gathering data for ns/openshift-insights...
2026-07-02T15:52:45.742309372Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T15:52:46.853304051Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T15:52:46.984680139Z Gathering data for ns/openshift-config...
2026-07-02T15:52:47.253550263Z Gathering data for ns/openshift-config-managed...
2026-07-02T15:52:47.628447121Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T15:52:47.863177776Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T15:52:47.976083262Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T15:52:48.098587212Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T15:52:48.345356067Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T15:52:48.640928623Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T15:52:48.859828600Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T15:52:48.899526930Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:49.100965985Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T15:52:49.130511896Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T15:52:49.454059613Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T15:52:49.742485640Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T15:52:50.183310435Z Gathering data for ns/openshift-multus...
2026-07-02T15:52:50.221842098Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T15:52:50.440065569Z Collecting performance related data for node ip-10-0-136-55.ec2.internal
2026-07-02T15:52:50.447315749Z Collecting performance related data for node ip-10-0-140-71.ec2.internal
2026-07-02T15:52:50.454800834Z Collecting performance related data for node ip-10-0-134-12.ec2.internal
2026-07-02T15:52:52.061802937Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T15:52:53.324334526Z Gathering data for ns/openshift-host-network...
2026-07-02T15:52:53.622949324Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T15:52:53.905095782Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:54.112566583Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T15:52:54.493915616Z Gathering data for ns/openshift-network-operator...
2026-07-02T15:52:54.842580363Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T15:52:55.090890830Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T15:52:55.551276522Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T15:52:55.862042468Z Gathering data for ns/openshift-apiserver...
2026-07-02T15:52:56.243073473Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T15:52:56.514142066Z Gathering data for ns/openshift-controller-manager...
2026-07-02T15:52:56.756312026Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T15:52:57.363179584Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T15:52:57.623645504Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T15:52:57.909243754Z Gathering data for ns/openshift-service-ca...
2026-07-02T15:52:58.238504811Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T15:52:58.790616128Z Collecting kubelet logs for node ip-10-0-136-55.ec2.internal
2026-07-02T15:52:58.792598547Z Collecting kubelet logs for node ip-10-0-140-71.ec2.internal
2026-07-02T15:52:58.794870429Z Collecting kubelet logs for node ip-10-0-134-12.ec2.internal
2026-07-02T15:52:58.912481386Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T15:52:59.106216285Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-h54n7 namespace
2026-07-02T15:52:59.111170530Z INFO: Node performance data collection complete.
2026-07-02T15:52:59.149825119Z Wrote inspect data to must-gather.
2026-07-02T15:52:59.149860175Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T15:52:59.149860175Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T15:52:59.272972722Z error: the server doesn't have a resource type "clusters"
2026-07-02T15:52:59.399312720Z Caches written to disk