2026-09-16T19:14:58.850334406Z [disk usage checker] Started
2026-09-16T19:14:58.853726379Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:14:59.064006609Z Gathering data for ns/openshift-cluster-version...
2026-09-16T19:14:59.224472471Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T19:14:59.333953873Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:14:59.423149977Z Gathering data for ns/default...
2026-09-16T19:14:59.814864020Z Gathering data for ns/openshift...
2026-09-16T19:15:00.296570151Z Gathering data for ns/kube-system...
2026-09-16T19:15:01.078898812Z Gathering data for ns/openshift-etcd...
2026-09-16T19:15:01.447802108Z Wrote inspect data to must-gather.
2026-09-16T19:15:01.447837685Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T19:15:01.447837685Z     namespaces "assisted-installer" not found
2026-09-16T19:15:01.808177783Z Waiting on subprocesses to finish execution.
2026-09-16T19:15:01.810346934Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-16T19:15:01.816266154Z Executing Istio gather script
2026-09-16T19:15:01.817515876Z INFO: Gathering HAProxy config files
2026-09-16T19:15:01.823123212Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-16T19:15:01.831076980Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-16T19:15:01.854857906Z INFO: Collecting host service logs for crio
2026-09-16T19:15:01.855292046Z INFO: Collecting host service logs for kubelet
2026-09-16T19:15:01.855668141Z INFO: Collecting host service logs for rpm-ostreed
2026-09-16T19:15:01.856028651Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-16T19:15:01.856376309Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-16T19:15:01.856698738Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-16T19:15:01.857020643Z INFO: Collecting host service logs for NetworkManager
2026-09-16T19:15:01.857409094Z INFO: Collecting host service logs for openvswitch
2026-09-16T19:15:01.857734498Z INFO: Collecting host service logs for ovs-configuration
2026-09-16T19:15:01.858053566Z INFO: Collecting host service logs for ovsdb-server
2026-09-16T19:15:01.858397944Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-16T19:15:01.858788542Z INFO: Waiting for worker host service log collection to complete ...
2026-09-16T19:15:01.886578419Z INFO: Waiting for node performance related collection to complete ...
2026-09-16T19:15:02.111787368Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-16T19:15:02.233939317Z No resources found
2026-09-16T19:15:02.317807399Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-16T19:15:02.424721259Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-16T19:15:02.446119381Z INFO: Collecting Insights Archives from insights-operator-5844b856db-qdlfp
2026-09-16T19:15:02.446160425Z insights-runtime-extractor-6wdmf
2026-09-16T19:15:02.446179632Z insights-runtime-extractor-c2dhr
2026-09-16T19:15:02.446220579Z insights-runtime-extractor-ccshs
2026-09-16T19:15:02.452943313Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-16T19:15:02.453041906Z INFO: on-disk MachineConfig config collection complete.
2026-09-16T19:15:02.900527178Z error: the server doesn't have a resource type "performanceprofile"
2026-09-16T19:15:02.940494024Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-16T19:15:03.090467790Z ERROR: No running kube-apiserver pods found
2026-09-16T19:15:03.099900596Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-16T19:15:03.099900596Z See 'oc rsync -h' for help and examples
2026-09-16T19:15:03.299708713Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-16T19:15:03.504266519Z INFO: "metallb-operator" not detected. Skipping.
2026-09-16T19:15:03.804020424Z Wrote inspect data to must-gather.
2026-09-16T19:15:03.948648517Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:03.973579299Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.022331489Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-16T19:15:04.027892950Z No resources found in openshift-etcd namespace.
2026-09-16T19:15:04.034804417Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.074641309Z Gathering data for ns/openshift-monitoring...
2026-09-16T19:15:04.207982288Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:15:04.218688637Z No resources found
2026-09-16T19:15:04.259663081Z INFO: Worker host service log collection to complete.
2026-09-16T19:15:04.287514154Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.357156729Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.393422323Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-16T19:15:04.526961093Z error: the server doesn't have a resource type "machineconfigs"
2026-09-16T19:15:04.546248775Z tar: Removing leading `/' from member names
2026-09-16T19:15:04.554977018Z INFO: HAProxy config collection complete.
2026-09-16T19:15:04.560972605Z INFO: Getting rules from prometheus-k8s-0
2026-09-16T19:15:04.605831753Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.648614674Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.710848754Z INFO: 'previous-logs' folder not found on ip-10-0-128-157.ec2.internal, skipping...
2026-09-16T19:15:04.737486455Z Wrote inspect data to must-gather.
2026-09-16T19:15:04.758214416Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-16T19:15:05.012510696Z INFO: 'previous-logs' folder not found on ip-10-0-138-127.ec2.internal, skipping...
2026-09-16T19:15:05.021624011Z Wrote inspect data to must-gather.
2026-09-16T19:15:05.060308964Z Wrote inspect data to must-gather.
2026-09-16T19:15:05.136806452Z INFO: Getting status/config from prometheus-k8s-0
2026-09-16T19:15:05.237136555Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-16T19:15:05.365185354Z INFO: 'previous-logs' folder not found on ip-10-0-142-81.ec2.internal, skipping...
2026-09-16T19:15:05.365284855Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-16T19:15:05.365344235Z INFO: Machine Config Daemon termination log collection complete.
2026-09-16T19:15:05.410736204Z Wrote inspect data to must-gather.
2026-09-16T19:15:05.538785166Z Wrote inspect data to must-gather.
2026-09-16T19:15:05.668137969Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-16T19:15:05.798233469Z INFO: INTERCONNECT MODE
2026-09-16T19:15:05.798286866Z INFO: Gathering ovn-kubernetes DBs
2026-09-16T19:15:05.816163660Z Wrote inspect data to must-gather.
2026-09-16T19:15:05.957590691Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-16T19:15:05.993360470Z INFO: Gathering OVN_Northbound from ovnkube-node-csmcl...
2026-09-16T19:15:05.993650898Z INFO: Gathering OVN_Northbound from ovnkube-node-gz585...
2026-09-16T19:15:05.993960340Z INFO: Gathering OVN_Northbound from ovnkube-node-mz2g5...
2026-09-16T19:15:05.994285028Z INFO: Gathering OVN_Southbound from ovnkube-node-csmcl...
2026-09-16T19:15:05.994958831Z INFO: Gathering OVN_Southbound from ovnkube-node-gz585...
2026-09-16T19:15:05.995291982Z INFO: Gathering OVN_Southbound from ovnkube-node-mz2g5...
2026-09-16T19:15:06.203126207Z Wrote inspect data to must-gather.
2026-09-16T19:15:06.376127567Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.456312232Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-16T19:15:06.458671410Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.485247134Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.488211244Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.507095065Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.524092740Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-16T19:15:06.535934590Z tar: Removing leading `/' from member names
2026-09-16T19:15:06.544016386Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T19:15:06.612134488Z Wrote inspect data to must-gather.
2026-09-16T19:15:06.759324371Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-16T19:15:06.826255960Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-16T19:15:06.898115041Z Wrote inspect data to must-gather.
2026-09-16T19:15:06.996367403Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:07.067912973Z INFO: Waiting for network log collection to complete ...
2026-09-16T19:15:07.067996078Z INFO: Waiting for ovnk database copies to complete ...
2026-09-16T19:15:07.068537098Z INFO: Copying ovnk databases complete.
2026-09-16T19:15:07.070602669Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-09-16T19:15:07.074019646Z ovnk_database_store/
2026-09-16T19:15:07.074066712Z ovnk_database_store/ovnkube-node-gz585_sbdb
2026-09-16T19:15:07.102168726Z ovnk_database_store/ovnkube-node-mz2g5_sbdb
2026-09-16T19:15:07.122019104Z ovnk_database_store/ovnkube-node-csmcl_sbdb
2026-09-16T19:15:07.142157490Z W0916 19:15:07.142108    1480 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-16T19:15:07.144901145Z ovnk_database_store/ovnkube-node-mz2g5_nbdb
2026-09-16T19:15:07.147145636Z Wrote inspect data to must-gather.
2026-09-16T19:15:07.151698761Z ovnk_database_store/ovnkube-node-gz585_nbdb
2026-09-16T19:15:07.166214240Z ovnk_database_store/ovnkube-node-csmcl_nbdb
2026-09-16T19:15:07.180961260Z INFO: Network log collection complete.
2026-09-16T19:15:07.244848911Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:15:07.273507667Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:07.277624787Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-16T19:15:07.319151845Z INFO: Getting status from alertmanager-main-0
2026-09-16T19:15:07.457381086Z Gathering data for ns/openshift-network-console...
2026-09-16T19:15:07.495112259Z Wrote inspect data to must-gather.
2026-09-16T19:15:07.510878092Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:07.735094992Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:07.737527930Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-16T19:15:07.803734984Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-16T19:15:07.809685068Z INFO: Fallback to identify the container image from release info
2026-09-16T19:15:07.978648566Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:07.991680893Z Gathering data for ns/openshift-console-operator...
2026-09-16T19:15:08.181424504Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:08.184392060Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-16T19:15:08.391359139Z Gathering data for ns/openshift-console...
2026-09-16T19:15:08.411420041Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:08.583564856Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-16T19:15:08.634211515Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:08.636568670Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-16T19:15:08.730806873Z daemonset.apps/perf-node-gather-daemonset created
2026-09-16T19:15:08.862238130Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:08.902894385Z Waiting for performance profile collector pods to become ready: 1
2026-09-16T19:15:08.979184804Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:09.029796737Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-16T19:15:09.078860990Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:09.082188463Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-16T19:15:09.294394404Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:09.518474908Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:09.522038121Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-16T19:15:09.633972041Z Gathering data for ns/openshift-dns-operator...
2026-09-16T19:15:09.721894314Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:09.918816808Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:15:10.018815848Z Gathering data for ns/openshift-dns...
2026-09-16T19:15:10.058259163Z Waiting for performance profile collector pods to become ready: 2
2026-09-16T19:15:10.637667116Z Done executing Istio gather script
2026-09-16T19:15:10.686114809Z Gathering data for ns/openshift-image-registry...
2026-09-16T19:15:11.190729414Z Gathering data for ns/openshift-ingress-operator...
2026-09-16T19:15:11.210421123Z Waiting for performance profile collector pods to become ready: 3
2026-09-16T19:15:11.560085601Z Gathering data for ns/openshift-ingress...
2026-09-16T19:15:11.920935631Z Gathering data for ns/openshift-ingress-canary...
2026-09-16T19:15:12.339738166Z Gathering data for ns/openshift-insights...
2026-09-16T19:15:12.356330490Z Waiting for performance profile collector pods to become ready: 4
2026-09-16T19:15:13.482173402Z Waiting for performance profile collector pods to become ready: 5
2026-09-16T19:15:13.985868323Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:14.607653825Z Waiting for performance profile collector pods to become ready: 6
2026-09-16T19:15:15.005240766Z Gathering data for ns/kserve...
2026-09-16T19:15:15.746870654Z Waiting for performance profile collector pods to become ready: 7
2026-09-16T19:15:16.805883407Z Gathering data for ns/openshift-config...
2026-09-16T19:15:16.898631756Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-16T19:15:17.134424435Z Gathering data for ns/openshift-config-managed...
2026-09-16T19:15:17.269243727Z Collecting performance related data for node ip-10-0-128-157.ec2.internal
2026-09-16T19:15:17.276488090Z Collecting performance related data for node ip-10-0-142-81.ec2.internal
2026-09-16T19:15:17.283927186Z Collecting performance related data for node ip-10-0-138-127.ec2.internal
2026-09-16T19:15:17.768347298Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-16T19:15:18.179292484Z Gathering data for ns/openshift-kube-apiserver...
2026-09-16T19:15:18.599535445Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-16T19:15:18.994780418Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:19.000623752Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-16T19:15:19.379647606Z Gathering data for ns/openshift-kube-scheduler...
2026-09-16T19:15:19.696318544Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-16T19:15:20.100880902Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-16T19:15:20.536761602Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-16T19:15:20.926702600Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-16T19:15:21.545101427Z Gathering data for ns/openshift-multus...
2026-09-16T19:15:23.308714923Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-16T19:15:23.999948378Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:24.820679038Z Gathering data for ns/openshift-host-network...
2026-09-16T19:15:25.170550803Z Gathering data for ns/openshift-network-diagnostics...
2026-09-16T19:15:25.768572149Z Gathering data for ns/openshift-network-node-identity...
2026-09-16T19:15:25.968231025Z Collecting kubelet logs for node ip-10-0-128-157.ec2.internal
2026-09-16T19:15:25.970562334Z Collecting kubelet logs for node ip-10-0-142-81.ec2.internal
2026-09-16T19:15:25.975903157Z Collecting kubelet logs for node ip-10-0-138-127.ec2.internal
2026-09-16T19:15:26.303918474Z Gathering data for ns/openshift-network-operator...
2026-09-16T19:15:26.401875745Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ng22b namespace
2026-09-16T19:15:26.411295038Z INFO: Node performance data collection complete.
2026-09-16T19:15:26.751927225Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-16T19:15:27.082404436Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-16T19:15:27.602971047Z Gathering data for ns/openshift-apiserver-operator...
2026-09-16T19:15:27.914886171Z Gathering data for ns/openshift-apiserver...
2026-09-16T19:15:28.409425517Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-16T19:15:28.703138501Z Gathering data for ns/openshift-controller-manager...
2026-09-16T19:15:29.004607251Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-09-16T19:15:29.109677112Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-16T19:15:29.821525711Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-16T19:15:30.179533027Z Gathering data for ns/openshift-service-ca-operator...
2026-09-16T19:15:30.538162299Z Gathering data for ns/openshift-service-ca...
2026-09-16T19:15:30.960230678Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-16T19:15:31.953082702Z Wrote inspect data to must-gather.
2026-09-16T19:15:31.953127850Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T19:15:31.953127850Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-16T19:15:32.087459689Z error: the server doesn't have a resource type "clusters"
2026-09-16T19:15:32.226454233Z Caches written to disk