2026-06-09T15:00:04.839503701Z [disk usage checker] Started
2026-06-09T15:00:04.843565254Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:05.020437285Z Gathering data for ns/openshift-cluster-version...
2026-06-09T15:00:05.158412773Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-09T15:00:05.250783931Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T15:00:05.404784959Z Gathering data for ns/default...
2026-06-09T15:00:05.688398765Z Gathering data for ns/openshift...
2026-06-09T15:00:06.151756026Z Gathering data for ns/kube-system...
2026-06-09T15:00:06.830638696Z Gathering data for ns/openshift-etcd...
2026-06-09T15:00:07.115935373Z Wrote inspect data to must-gather.
2026-06-09T15:00:07.115961941Z error: inspection completed with the errors occurred while gathering data:
2026-06-09T15:00:07.115961941Z     namespaces "assisted-installer" not found
2026-06-09T15:00:07.344158767Z Waiting on subprocesses to finish execution.
2026-06-09T15:00:07.348162127Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-09T15:00:07.348739889Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-09T15:00:07.350145900Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-09T15:00:07.352117212Z INFO: Gathering HAProxy config files
2026-06-09T15:00:07.375019031Z Executing Istio gather script
2026-06-09T15:00:07.394145662Z INFO: Collecting host service logs for crio
2026-06-09T15:00:07.394542694Z INFO: Collecting host service logs for kubelet
2026-06-09T15:00:07.394981843Z INFO: Collecting host service logs for rpm-ostreed
2026-06-09T15:00:07.395304925Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-09T15:00:07.395624978Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-09T15:00:07.395967850Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-09T15:00:07.396303489Z INFO: Collecting host service logs for NetworkManager
2026-06-09T15:00:07.396635945Z INFO: Collecting host service logs for openvswitch
2026-06-09T15:00:07.396985335Z INFO: Collecting host service logs for ovs-configuration
2026-06-09T15:00:07.397326173Z INFO: Collecting host service logs for ovsdb-server
2026-06-09T15:00:07.397637225Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-09T15:00:07.398051778Z INFO: Waiting for worker host service log collection to complete ...
2026-06-09T15:00:07.472993209Z INFO: Waiting for node performance related collection to complete ...
2026-06-09T15:00:07.748258708Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-09T15:00:07.805282092Z No resources found
2026-06-09T15:00:07.807884545Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-09T15:00:07.854286165Z INFO: Collecting Insights Archives from insights-operator-896549976-mxzvn
2026-06-09T15:00:07.854286165Z insights-runtime-extractor-8t2tx
2026-06-09T15:00:07.854286165Z insights-runtime-extractor-crpgl
2026-06-09T15:00:07.854286165Z insights-runtime-extractor-jxgng
2026-06-09T15:00:08.060658913Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-09T15:00:08.060658913Z INFO: on-disk MachineConfig config collection complete.
2026-06-09T15:00:08.350935581Z INFO: "metallb-operator" not detected. Skipping.
2026-06-09T15:00:08.618317846Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-09T15:00:08.687763923Z Wrote inspect data to must-gather.
2026-06-09T15:00:08.718163907Z INFO: 'previous-logs' folder not found on ip-10-0-129-81.ec2.internal, skipping...
2026-06-09T15:00:08.807995476Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T15:00:08.848060098Z No resources found
2026-06-09T15:00:08.984484888Z ERROR: No running kube-apiserver pods found
2026-06-09T15:00:09.030884546Z tar: Removing leading `/' from member names
2026-06-09T15:00:09.053150559Z INFO: HAProxy config collection complete.
2026-06-09T15:00:09.119970372Z No resources found in openshift-etcd namespace.
2026-06-09T15:00:09.147154453Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-09T15:00:09.180564140Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.213746792Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.337964944Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.355032896Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.358135990Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-09T15:00:09.454594405Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-09T15:00:09.454594405Z See 'oc rsync -h' for help and examples
2026-06-09T15:00:09.526237311Z INFO: Worker host service log collection to complete.
2026-06-09T15:00:09.585940889Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.587556996Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-09T15:00:09.603344074Z INFO: 'previous-logs' folder not found on ip-10-0-136-76.ec2.internal, skipping...
2026-06-09T15:00:09.659616937Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-09T15:00:09.664259880Z error: the server doesn't have a resource type "performanceprofile"
2026-06-09T15:00:09.700296826Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.856921226Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:09.915533987Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-09T15:00:09.928803927Z INFO: 'previous-logs' folder not found on ip-10-0-142-176.ec2.internal, skipping...
2026-06-09T15:00:09.928845387Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-09T15:00:09.928876928Z INFO: Machine Config Daemon termination log collection complete.
2026-06-09T15:00:09.956752459Z Gathering data for ns/openshift-monitoring...
2026-06-09T15:00:09.967188860Z Wrote inspect data to must-gather.
2026-06-09T15:00:09.987908855Z Wrote inspect data to must-gather.
2026-06-09T15:00:10.063410170Z INFO: Getting rules from prometheus-k8s-0
2026-06-09T15:00:10.120416237Z error: the server doesn't have a resource type "machineconfigs"
2026-06-09T15:00:10.220798326Z Wrote inspect data to must-gather.
2026-06-09T15:00:10.270853733Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-09T15:00:10.462474506Z INFO: Getting status/config from prometheus-k8s-0
2026-06-09T15:00:10.484173488Z INFO: INTERCONNECT MODE
2026-06-09T15:00:10.484203992Z INFO: Gathering ovn-kubernetes DBs
2026-06-09T15:00:10.489058381Z Wrote inspect data to must-gather.
2026-06-09T15:00:10.585898934Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-09T15:00:10.648491224Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-09T15:00:10.648789594Z INFO: Gathering OVN_Northbound from ovnkube-node-bm2tm...
2026-06-09T15:00:10.649013631Z INFO: Gathering OVN_Northbound from ovnkube-node-ntf9s...
2026-06-09T15:00:10.649273066Z INFO: Gathering OVN_Northbound from ovnkube-node-v6w98...
2026-06-09T15:00:10.649545531Z INFO: Gathering OVN_Southbound from ovnkube-node-bm2tm...
2026-06-09T15:00:10.649790135Z INFO: Gathering OVN_Southbound from ovnkube-node-ntf9s...
2026-06-09T15:00:10.650049788Z INFO: Gathering OVN_Southbound from ovnkube-node-v6w98...
2026-06-09T15:00:10.869650208Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:10.921774669Z tar: Removing leading `/' from member names
2026-06-09T15:00:10.931332407Z tar: Removing leading `/' from member names
2026-06-09T15:00:10.935347615Z tar: Removing leading `/' from member names
2026-06-09T15:00:10.936890569Z tar: Removing leading `/' from member names
2026-06-09T15:00:10.952320461Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-09T15:00:10.986181641Z tar: Removing leading `/' from member names
2026-06-09T15:00:11.001973988Z tar: Removing leading `/' from member names
2026-06-09T15:00:11.010270571Z Wrote inspect data to must-gather.
2026-06-09T15:00:11.113283528Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:11.115225551Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-09T15:00:11.215625918Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-09T15:00:11.256119772Z Wrote inspect data to must-gather.
2026-06-09T15:00:11.347854547Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:11.363791713Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-09T15:00:11.457110659Z INFO: Waiting for network log collection to complete ...
2026-06-09T15:00:11.457145377Z INFO: Waiting for ovnk database copies to complete ...
2026-06-09T15:00:11.457542896Z INFO: Copying ovnk databases complete.
2026-06-09T15:00:11.459190515Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-06-09T15:00:11.461795848Z ovnk_database_store/
2026-06-09T15:00:11.461795848Z ovnk_database_store/ovnkube-node-bm2tm_nbdb
2026-06-09T15:00:11.461856220Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-09T15:00:11.475033876Z ovnk_database_store/ovnkube-node-v6w98_nbdb
2026-06-09T15:00:11.481836335Z ovnk_database_store/ovnkube-node-bm2tm_sbdb
2026-06-09T15:00:11.490701997Z Wrote inspect data to must-gather.
2026-06-09T15:00:11.498100668Z ovnk_database_store/ovnkube-node-v6w98_sbdb
2026-06-09T15:00:11.514269229Z ovnk_database_store/ovnkube-node-ntf9s_nbdb
2026-06-09T15:00:11.521211319Z ovnk_database_store/ovnkube-node-ntf9s_sbdb
2026-06-09T15:00:11.544265874Z INFO: Network log collection complete.
2026-06-09T15:00:11.553764327Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:11.559042782Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-09T15:00:11.687130422Z Wrote inspect data to must-gather.
2026-06-09T15:00:11.697186527Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-09T15:00:11.752152777Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:11.882486517Z W0609 15:00:11.882442    1506 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-09T15:00:11.888188456Z Wrote inspect data to must-gather.
2026-06-09T15:00:11.945821912Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:11.948659486Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-09T15:00:12.001204608Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T15:00:12.067948973Z INFO: Getting status from alertmanager-main-0
2026-06-09T15:00:12.189522651Z Wrote inspect data to must-gather.
2026-06-09T15:00:12.189987375Z Gathering data for ns/openshift-network-console...
2026-06-09T15:00:12.191605812Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:12.384275663Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:12.386988825Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-09T15:00:12.471645953Z Wrote inspect data to must-gather.
2026-06-09T15:00:12.554049237Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:12.616077970Z Gathering data for ns/openshift-console-operator...
2026-06-09T15:00:12.740070752Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-09T15:00:12.742008046Z INFO: Fallback to identify the container image from release info
2026-06-09T15:00:12.760790960Z Wrote inspect data to /must-gather/istio.
2026-06-09T15:00:13.012974878Z Gathering data for ns/openshift-console...
2026-06-09T15:00:13.394240958Z Done executing Istio gather script
2026-06-09T15:00:13.406599736Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-09T15:00:13.484920273Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-09T15:00:13.528083456Z daemonset.apps/perf-node-gather-daemonset created
2026-06-09T15:00:13.641402144Z Waiting for performance profile collector pods to become ready: 1
2026-06-09T15:00:13.915513876Z Gathering data for ns/openshift-dns-operator...
2026-06-09T15:00:14.192633197Z Gathering data for ns/openshift-dns...
2026-06-09T15:00:14.713787888Z Gathering data for ns/openshift-image-registry...
2026-06-09T15:00:14.750852221Z Waiting for performance profile collector pods to become ready: 2
2026-06-09T15:00:14.872848609Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:15.119376376Z Gathering data for ns/openshift-ingress-operator...
2026-06-09T15:00:15.399442157Z Gathering data for ns/openshift-ingress...
2026-06-09T15:00:15.692280274Z Gathering data for ns/openshift-ingress-canary...
2026-06-09T15:00:15.868639743Z Waiting for performance profile collector pods to become ready: 3
2026-06-09T15:00:16.031125423Z Gathering data for ns/openshift-insights...
2026-06-09T15:00:16.970658461Z Waiting for performance profile collector pods to become ready: 4
2026-06-09T15:00:17.803571390Z Gathering data for ns/openshift-config...
2026-06-09T15:00:18.086778986Z Waiting for performance profile collector pods to become ready: 5
2026-06-09T15:00:18.130888106Z Gathering data for ns/openshift-config-managed...
2026-06-09T15:00:18.509216703Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-09T15:00:18.783250991Z Gathering data for ns/openshift-kube-apiserver...
2026-06-09T15:00:19.021202761Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-09T15:00:19.197206532Z Waiting for performance profile collector pods to become ready: 6
2026-06-09T15:00:19.287575588Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-09T15:00:19.530198903Z Gathering data for ns/openshift-kube-scheduler...
2026-06-09T15:00:19.759701967Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-09T15:00:19.877241780Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:20.016333051Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-09T15:00:20.304481494Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-09T15:00:20.310071145Z Waiting for performance profile collector pods to become ready: 7
2026-06-09T15:00:20.604472057Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-09T15:00:21.041316013Z Gathering data for ns/openshift-multus...
2026-06-09T15:00:21.407817441Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-09T15:00:21.618295098Z Collecting performance related data for node ip-10-0-136-76.ec2.internal
2026-06-09T15:00:21.625241155Z Collecting performance related data for node ip-10-0-142-176.ec2.internal
2026-06-09T15:00:21.632328586Z Collecting performance related data for node ip-10-0-129-81.ec2.internal
2026-06-09T15:00:22.439297339Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-09T15:00:23.624106475Z Gathering data for ns/openshift-host-network...
2026-06-09T15:00:24.011515445Z Gathering data for ns/openshift-network-diagnostics...
2026-06-09T15:00:24.573419255Z Gathering data for ns/openshift-network-node-identity...
2026-06-09T15:00:24.882700518Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:24.938147618Z Gathering data for ns/openshift-network-operator...
2026-06-09T15:00:25.279235578Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-09T15:00:25.516639558Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-09T15:00:26.005544182Z Gathering data for ns/openshift-apiserver-operator...
2026-06-09T15:00:26.288965451Z Gathering data for ns/openshift-apiserver...
2026-06-09T15:00:26.751222457Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-09T15:00:27.038343976Z Gathering data for ns/openshift-controller-manager...
2026-06-09T15:00:27.298029347Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-09T15:00:27.888755202Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-09T15:00:28.228784526Z Gathering data for ns/openshift-service-ca-operator...
2026-06-09T15:00:28.525957051Z Gathering data for ns/openshift-service-ca...
2026-06-09T15:00:28.865833484Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-09T15:00:29.529935306Z Collecting kubelet logs for node ip-10-0-136-76.ec2.internal
2026-06-09T15:00:29.531810915Z Collecting kubelet logs for node ip-10-0-142-176.ec2.internal
2026-06-09T15:00:29.534061061Z Collecting kubelet logs for node ip-10-0-129-81.ec2.internal
2026-06-09T15:00:29.769039189Z Wrote inspect data to must-gather.
2026-06-09T15:00:29.769154650Z error: inspection completed with the errors occurred while gathering data:
2026-06-09T15:00:29.769154650Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-09T15:00:29.836057508Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-7c8rv namespace
2026-06-09T15:00:29.841820080Z INFO: Node performance data collection complete.
2026-06-09T15:00:29.887830906Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-09T15:00:29.958857493Z error: the server doesn't have a resource type "clusters"
2026-06-09T15:00:30.097313103Z Caches written to disk