2026-07-27T21:54:25.843680270Z [disk usage checker] Started
2026-07-27T21:54:25.847293338Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:26.029680429Z Gathering data for ns/openshift-cluster-version...
2026-07-27T21:54:26.161635318Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T21:54:26.245806649Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:54:26.323739010Z Gathering data for ns/default...
2026-07-27T21:54:26.608401593Z Gathering data for ns/openshift...
2026-07-27T21:54:27.038396685Z Gathering data for ns/kube-system...
2026-07-27T21:54:27.731696656Z Gathering data for ns/openshift-etcd...
2026-07-27T21:54:28.022440736Z Wrote inspect data to must-gather.
2026-07-27T21:54:28.022472526Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T21:54:28.022472526Z     namespaces "assisted-installer" not found
2026-07-27T21:54:28.438572254Z Waiting on subprocesses to finish execution.
2026-07-27T21:54:28.446065251Z INFO: Collecting host service logs for crio
2026-07-27T21:54:28.446298012Z INFO: Collecting host service logs for kubelet
2026-07-27T21:54:28.446569509Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T21:54:28.446880001Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T21:54:28.447162377Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T21:54:28.447456365Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T21:54:28.447767812Z INFO: Collecting host service logs for NetworkManager
2026-07-27T21:54:28.448071432Z INFO: Collecting host service logs for openvswitch
2026-07-27T21:54:28.448377357Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T21:54:28.448676665Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T21:54:28.449102710Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T21:54:28.449463216Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T21:54:28.453045721Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T21:54:28.456018294Z INFO: Gathering HAProxy config files
2026-07-27T21:54:28.460933785Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T21:54:28.466951243Z Executing Istio gather script
2026-07-27T21:54:28.469879869Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T21:54:28.535935068Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T21:54:29.438476551Z No resources found in openshift-etcd namespace.
2026-07-27T21:54:29.656200786Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T21:54:29.674623626Z No resources found
2026-07-27T21:54:29.686464005Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-kwqx9
2026-07-27T21:54:29.686464005Z insights-runtime-extractor-6np85
2026-07-27T21:54:29.686464005Z insights-runtime-extractor-fkz7m
2026-07-27T21:54:29.686464005Z insights-runtime-extractor-wkkhn
2026-07-27T21:54:29.708642888Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T21:54:29.752978139Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T21:54:29.775693225Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T21:54:29.803965208Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T21:54:29.806561746Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T21:54:29.808839237Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T21:54:29.873446846Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T21:54:29.937384172Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T21:54:29.957492271Z Wrote inspect data to must-gather.
2026-07-27T21:54:29.966902286Z INFO: Worker host service log collection to complete.
2026-07-27T21:54:30.047148116Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T21:54:30.047148116Z See 'oc rsync -h' for help and examples
2026-07-27T21:54:30.248661463Z Wrote inspect data to must-gather.
2026-07-27T21:54:30.275393791Z ERROR: No running kube-apiserver pods found
2026-07-27T21:54:30.658493103Z tar: Removing leading `/' from member names
2026-07-27T21:54:30.674157609Z INFO: HAProxy config collection complete.
2026-07-27T21:54:30.705068173Z Wrote inspect data to must-gather.
2026-07-27T21:54:30.711011491Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:54:30.712148611Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T21:54:30.831051888Z Wrote inspect data to must-gather.
2026-07-27T21:54:30.912916540Z Wrote inspect data to must-gather.
2026-07-27T21:54:30.935124372Z No resources found
2026-07-27T21:54:30.944903661Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:30.966900783Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.043025482Z INFO: 'previous-logs' folder not found on ip-10-0-128-201.ec2.internal, skipping...
2026-07-27T21:54:31.132371014Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T21:54:31.160818036Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.224015170Z Gathering data for ns/openshift-monitoring...
2026-07-27T21:54:31.264659705Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T21:54:31.356370487Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.405688687Z INFO: 'previous-logs' folder not found on ip-10-0-135-121.ec2.internal, skipping...
2026-07-27T21:54:31.464904025Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T21:54:31.466503551Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T21:54:31.659003545Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.711360417Z INFO: 'previous-logs' folder not found on ip-10-0-138-50.ec2.internal, skipping...
2026-07-27T21:54:31.711453141Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T21:54:31.711498623Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T21:54:31.733900975Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.870505838Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T21:54:31.914888459Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T21:54:31.934146762Z Wrote inspect data to must-gather.
2026-07-27T21:54:31.941986891Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T21:54:32.136455364Z Wrote inspect data to must-gather.
2026-07-27T21:54:32.248873741Z INFO: INTERCONNECT MODE
2026-07-27T21:54:32.248900325Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T21:54:32.378015448Z Wrote inspect data to must-gather.
2026-07-27T21:54:32.385629256Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T21:54:32.424400951Z INFO: Gathering OVN_Northbound from ovnkube-node-926vh...
2026-07-27T21:54:32.424723281Z INFO: Gathering OVN_Northbound from ovnkube-node-l6mhn...
2026-07-27T21:54:32.425111033Z INFO: Gathering OVN_Northbound from ovnkube-node-xkcbq...
2026-07-27T21:54:32.425451782Z INFO: Gathering OVN_Southbound from ovnkube-node-926vh...
2026-07-27T21:54:32.425724441Z INFO: Gathering OVN_Southbound from ovnkube-node-l6mhn...
2026-07-27T21:54:32.426011761Z INFO: Gathering OVN_Southbound from ovnkube-node-xkcbq...
2026-07-27T21:54:32.743635356Z Wrote inspect data to must-gather.
2026-07-27T21:54:32.743693493Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T21:54:32.752278706Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.775303506Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.787447273Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.808781627Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.809379630Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.809698700Z tar: Removing leading `/' from member names
2026-07-27T21:54:32.853228528Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T21:54:32.983302254Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T21:54:32.991015065Z Wrote inspect data to must-gather.
2026-07-27T21:54:33.198843132Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:33.207546151Z Wrote inspect data to must-gather.
2026-07-27T21:54:33.250662262Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T21:54:33.288690627Z INFO: Waiting for network log collection to complete ...
2026-07-27T21:54:33.288876773Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T21:54:33.289307343Z INFO: Copying ovnk databases complete.
2026-07-27T21:54:33.291304915Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-07-27T21:54:33.295957596Z ovnk_database_store/
2026-07-27T21:54:33.295957596Z ovnk_database_store/ovnkube-node-xkcbq_nbdb
2026-07-27T21:54:33.307957985Z ovnk_database_store/ovnkube-node-l6mhn_nbdb
2026-07-27T21:54:33.320114975Z ovnk_database_store/ovnkube-node-xkcbq_sbdb
2026-07-27T21:54:33.339429187Z ovnk_database_store/ovnkube-node-926vh_sbdb
2026-07-27T21:54:33.362727853Z ovnk_database_store/ovnkube-node-926vh_nbdb
2026-07-27T21:54:33.374050835Z ovnk_database_store/ovnkube-node-l6mhn_sbdb
2026-07-27T21:54:33.401096618Z INFO: Network log collection complete.
2026-07-27T21:54:33.435947094Z W0727 21:54:33.435904    1471 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-27T21:54:33.439646398Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:33.441103992Z Wrote inspect data to must-gather.
2026-07-27T21:54:33.442517330Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T21:54:33.502059680Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:54:33.508932435Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T21:54:33.655835161Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:33.691997942Z Gathering data for ns/openshift-network-console...
2026-07-27T21:54:33.734249130Z Wrote inspect data to must-gather.
2026-07-27T21:54:33.881538058Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:33.885941354Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T21:54:33.957264827Z INFO: Getting status from alertmanager-main-0
2026-07-27T21:54:34.062013295Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T21:54:34.065076811Z INFO: Fallback to identify the container image from release info
2026-07-27T21:54:34.116134946Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:34.199588917Z Gathering data for ns/openshift-console-operator...
2026-07-27T21:54:34.326847679Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:34.328868388Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T21:54:34.533409577Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:34.570053589Z Gathering data for ns/openshift-console...
2026-07-27T21:54:34.738002687Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:34.741581599Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-27T21:54:34.779385537Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T21:54:34.908450206Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:34.926298007Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T21:54:35.079878697Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T21:54:35.122904445Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T21:54:35.126728227Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:35.128987303Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-27T21:54:35.304833065Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:35.522340636Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:35.524725084Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T21:54:35.640717701Z Gathering data for ns/openshift-dns-operator...
2026-07-27T21:54:35.753586650Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:35.934675453Z Gathering data for ns/openshift-dns...
2026-07-27T21:54:35.952836075Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:54:35.954674899Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:36.217265051Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T21:54:36.537485956Z Gathering data for ns/openshift-image-registry...
2026-07-27T21:54:36.601441601Z Done executing Istio gather script
2026-07-27T21:54:36.967546045Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T21:54:37.328081905Z Gathering data for ns/openshift-ingress...
2026-07-27T21:54:37.342075963Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T21:54:37.639044840Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T21:54:38.002518495Z Gathering data for ns/openshift-insights...
2026-07-27T21:54:38.459506366Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T21:54:39.575477580Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T21:54:40.195586524Z Gathering data for ns/kserve...
2026-07-27T21:54:40.703833094Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T21:54:40.959614119Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:41.809898045Z Gathering data for ns/openshift-config...
2026-07-27T21:54:41.823246201Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T21:54:42.044994887Z Gathering data for ns/openshift-config-managed...
2026-07-27T21:54:42.429795746Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T21:54:42.657003713Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T21:54:42.943827915Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T21:54:42.956074151Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T21:54:43.224843198Z Collecting performance related data for node ip-10-0-128-201.ec2.internal
2026-07-27T21:54:43.233254022Z Collecting performance related data for node ip-10-0-138-50.ec2.internal
2026-07-27T21:54:43.237700791Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T21:54:43.241881348Z Collecting performance related data for node ip-10-0-135-121.ec2.internal
2026-07-27T21:54:43.570365299Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T21:54:43.917569649Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T21:54:44.279818886Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T21:54:44.657603914Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T21:54:44.970279130Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T21:54:45.435524744Z Gathering data for ns/openshift-multus...
2026-07-27T21:54:45.964087992Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:46.750583666Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T21:54:47.985228737Z Gathering data for ns/openshift-host-network...
2026-07-27T21:54:48.313141491Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T21:54:48.790112803Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T21:54:49.155929118Z Gathering data for ns/openshift-network-operator...
2026-07-27T21:54:49.516060890Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T21:54:49.796805568Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T21:54:50.260586346Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T21:54:50.559476368Z Gathering data for ns/openshift-apiserver...
2026-07-27T21:54:50.932248106Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T21:54:50.970501510Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-27T21:54:51.157950039Z Collecting kubelet logs for node ip-10-0-128-201.ec2.internal
2026-07-27T21:54:51.159869029Z Collecting kubelet logs for node ip-10-0-138-50.ec2.internal
2026-07-27T21:54:51.161630977Z Collecting kubelet logs for node ip-10-0-135-121.ec2.internal
2026-07-27T21:54:51.166851073Z Gathering data for ns/openshift-controller-manager...
2026-07-27T21:54:51.473819817Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T21:54:51.540029397Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-6q2f4 namespace
2026-07-27T21:54:51.544216193Z INFO: Node performance data collection complete.
2026-07-27T21:54:52.096484926Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T21:54:52.356237699Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T21:54:52.649000526Z Gathering data for ns/openshift-service-ca...
2026-07-27T21:54:52.984014774Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T21:54:53.790397979Z Wrote inspect data to must-gather.
2026-07-27T21:54:53.790436166Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T21:54:53.790436166Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T21:54:53.908639434Z error: the server doesn't have a resource type "clusters"
2026-07-27T21:54:54.051213322Z Caches written to disk