2026-09-16T15:59:12.901635729Z [disk usage checker] Started
2026-09-16T15:59:12.905333030Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:13.109087362Z Gathering data for ns/openshift-cluster-version...
2026-09-16T15:59:13.266356519Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T15:59:13.363286122Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T15:59:13.463598358Z Gathering data for ns/default...
2026-09-16T15:59:13.784162272Z Gathering data for ns/openshift...
2026-09-16T15:59:14.210687606Z Gathering data for ns/kube-system...
2026-09-16T15:59:14.949263419Z Gathering data for ns/openshift-etcd...
2026-09-16T15:59:15.314777605Z Wrote inspect data to must-gather.
2026-09-16T15:59:15.314895246Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T15:59:15.314895246Z     namespaces "assisted-installer" not found
2026-09-16T15:59:15.639197086Z Waiting on subprocesses to finish execution.
2026-09-16T15:59:15.649730437Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-16T15:59:15.650577878Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-16T15:59:15.653130093Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-16T15:59:15.655001203Z Executing Istio gather script
2026-09-16T15:59:15.656694057Z INFO: Gathering HAProxy config files
2026-09-16T15:59:15.670876599Z INFO: Collecting host service logs for crio
2026-09-16T15:59:15.671240724Z INFO: Collecting host service logs for kubelet
2026-09-16T15:59:15.671597579Z INFO: Collecting host service logs for rpm-ostreed
2026-09-16T15:59:15.671944994Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-16T15:59:15.672267499Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-16T15:59:15.672598310Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-16T15:59:15.672937683Z INFO: Collecting host service logs for NetworkManager
2026-09-16T15:59:15.673262804Z INFO: Collecting host service logs for openvswitch
2026-09-16T15:59:15.673574403Z INFO: Collecting host service logs for ovs-configuration
2026-09-16T15:59:15.673896409Z INFO: Collecting host service logs for ovsdb-server
2026-09-16T15:59:15.674283275Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-16T15:59:15.674764956Z INFO: Waiting for worker host service log collection to complete ...
2026-09-16T15:59:15.753441429Z INFO: Waiting for node performance related collection to complete ...
2026-09-16T15:59:15.985732397Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-16T15:59:16.070859901Z INFO: Collecting Insights Archives from insights-operator-5844b856db-6gjgv
2026-09-16T15:59:16.070945606Z insights-runtime-extractor-5p4d5
2026-09-16T15:59:16.070971673Z insights-runtime-extractor-dk6n4
2026-09-16T15:59:16.070991945Z insights-runtime-extractor-rlq7p
2026-09-16T15:59:16.394777592Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-16T15:59:16.536417084Z No resources found in openshift-etcd namespace.
2026-09-16T15:59:16.719152552Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-16T15:59:16.775268695Z Wrote inspect data to must-gather.
2026-09-16T15:59:17.119487221Z error: the server doesn't have a resource type "performanceprofile"
2026-09-16T15:59:17.197819827Z ERROR: No running kube-apiserver pods found
2026-09-16T15:59:17.202739723Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-16T15:59:17.202739723Z See 'oc rsync -h' for help and examples
2026-09-16T15:59:17.239240156Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-16T15:59:17.267527260Z No resources found
2026-09-16T15:59:17.269311677Z INFO: "metallb-operator" not detected. Skipping.
2026-09-16T15:59:17.281647909Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-16T15:59:17.281712642Z INFO: on-disk MachineConfig config collection complete.
2026-09-16T15:59:17.341762089Z Wrote inspect data to must-gather.
2026-09-16T15:59:17.367820300Z INFO: Worker host service log collection to complete.
2026-09-16T15:59:17.381272728Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-16T15:59:17.390655435Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-16T15:59:17.618311483Z INFO: 'previous-logs' folder not found on ip-10-0-128-99.ec2.internal, skipping...
2026-09-16T15:59:17.807707569Z No resources found
2026-09-16T15:59:17.973729046Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.028186375Z tar: Removing leading `/' from member names
2026-09-16T15:59:18.033050606Z INFO: HAProxy config collection complete.
2026-09-16T15:59:18.093944680Z Gathering data for ns/openshift-monitoring...
2026-09-16T15:59:18.142003120Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:18.205508730Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-16T15:59:18.206563649Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.327126262Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.355381288Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T15:59:18.455981733Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.479441831Z error: the server doesn't have a resource type "machineconfigs"
2026-09-16T15:59:18.504364733Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.599735454Z INFO: 'previous-logs' folder not found on ip-10-0-136-96.ec2.internal, skipping...
2026-09-16T15:59:18.649134965Z INFO: Getting rules from prometheus-k8s-0
2026-09-16T15:59:18.685334991Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-16T15:59:18.847128255Z Wrote inspect data to must-gather.
2026-09-16T15:59:18.922321562Z INFO: 'previous-logs' folder not found on ip-10-0-140-96.ec2.internal, skipping...
2026-09-16T15:59:18.922355408Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-16T15:59:18.922355408Z INFO: Machine Config Daemon termination log collection complete.
2026-09-16T15:59:18.972205278Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.094748045Z INFO: Getting status/config from prometheus-k8s-0
2026-09-16T15:59:19.143074771Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.175533760Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-16T15:59:19.363165760Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-16T15:59:19.430596060Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.437515781Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.554799506Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-16T15:59:19.661096656Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.779253213Z INFO: INTERCONNECT MODE
2026-09-16T15:59:19.779303990Z INFO: Gathering ovn-kubernetes DBs
2026-09-16T15:59:19.813875306Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-16T15:59:19.863380468Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T15:59:19.931836596Z Wrote inspect data to must-gather.
2026-09-16T15:59:19.947447859Z INFO: Gathering OVN_Northbound from ovnkube-node-fk95v...
2026-09-16T15:59:19.947668962Z INFO: Gathering OVN_Northbound from ovnkube-node-hm7cm...
2026-09-16T15:59:19.947903458Z INFO: Gathering OVN_Northbound from ovnkube-node-p62s2...
2026-09-16T15:59:19.948263006Z INFO: Gathering OVN_Southbound from ovnkube-node-fk95v...
2026-09-16T15:59:19.948780868Z INFO: Gathering OVN_Southbound from ovnkube-node-hm7cm...
2026-09-16T15:59:19.949107930Z INFO: Gathering OVN_Southbound from ovnkube-node-p62s2...
2026-09-16T15:59:20.118094005Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-16T15:59:20.244174843Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.244829419Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.280047362Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.290518994Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.301466568Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.354584209Z Wrote inspect data to must-gather.
2026-09-16T15:59:20.364692636Z tar: Removing leading `/' from member names
2026-09-16T15:59:20.438536422Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-16T15:59:20.486356601Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-16T15:59:20.588961704Z W0916 15:59:20.588894    1428 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-16T15:59:20.595823808Z Wrote inspect data to must-gather.
2026-09-16T15:59:20.695038553Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:20.733189024Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T15:59:20.917889550Z INFO: Waiting for network log collection to complete ...
2026-09-16T15:59:20.917948090Z INFO: Waiting for ovnk database copies to complete ...
2026-09-16T15:59:20.918330121Z INFO: Copying ovnk databases complete.
2026-09-16T15:59:20.920301623Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-09-16T15:59:20.922882848Z ovnk_database_store/
2026-09-16T15:59:20.922882848Z ovnk_database_store/ovnkube-node-fk95v_sbdb
2026-09-16T15:59:20.944640791Z ovnk_database_store/ovnkube-node-fk95v_nbdb
2026-09-16T15:59:20.953080401Z ovnk_database_store/ovnkube-node-p62s2_nbdb
2026-09-16T15:59:20.959892190Z ovnk_database_store/ovnkube-node-hm7cm_nbdb
2026-09-16T15:59:20.966855192Z ovnk_database_store/ovnkube-node-hm7cm_sbdb
2026-09-16T15:59:20.972965566Z Gathering data for ns/openshift-network-console...
2026-09-16T15:59:20.972965566Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:20.975444756Z Wrote inspect data to must-gather.
2026-09-16T15:59:20.975524536Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-16T15:59:20.977719857Z INFO: Getting status from alertmanager-main-0
2026-09-16T15:59:20.986375063Z ovnk_database_store/ovnkube-node-p62s2_sbdb
2026-09-16T15:59:21.005838736Z INFO: Network log collection complete.
2026-09-16T15:59:21.205120956Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:21.297562914Z Wrote inspect data to must-gather.
2026-09-16T15:59:21.424607163Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:21.427531213Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-16T15:59:21.550204478Z Gathering data for ns/openshift-console-operator...
2026-09-16T15:59:21.602196497Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-16T15:59:21.605316217Z INFO: Fallback to identify the container image from release info
2026-09-16T15:59:21.660059708Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:21.868540278Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:21.871170941Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-16T15:59:21.989326969Z Gathering data for ns/openshift-console...
2026-09-16T15:59:22.087822285Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:22.189363990Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-16T15:59:22.304709529Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:22.307022448Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-16T15:59:22.336445278Z daemonset.apps/perf-node-gather-daemonset created
2026-09-16T15:59:22.487107199Z Waiting for performance profile collector pods to become ready: 1
2026-09-16T15:59:22.501648337Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:22.594383319Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-16T15:59:22.714602082Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:22.717217330Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-16T15:59:22.893957652Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:23.110639002Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:23.113820117Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-16T15:59:23.162084521Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:23.169542445Z Gathering data for ns/openshift-dns-operator...
2026-09-16T15:59:23.287156650Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:23.495541799Z Wrote inspect data to /must-gather/istio.
2026-09-16T15:59:23.498797809Z Gathering data for ns/openshift-dns...
2026-09-16T15:59:23.617868090Z Waiting for performance profile collector pods to become ready: 2
2026-09-16T15:59:24.175300400Z Done executing Istio gather script
2026-09-16T15:59:24.179777984Z Gathering data for ns/openshift-image-registry...
2026-09-16T15:59:24.691752185Z Gathering data for ns/openshift-ingress-operator...
2026-09-16T15:59:24.747859209Z Waiting for performance profile collector pods to become ready: 3
2026-09-16T15:59:25.086405160Z Gathering data for ns/openshift-ingress...
2026-09-16T15:59:25.429749843Z Gathering data for ns/openshift-ingress-canary...
2026-09-16T15:59:25.829115443Z Gathering data for ns/openshift-insights...
2026-09-16T15:59:25.872361190Z Waiting for performance profile collector pods to become ready: 4
2026-09-16T15:59:26.990664914Z Waiting for performance profile collector pods to become ready: 5
2026-09-16T15:59:28.110412919Z Waiting for performance profile collector pods to become ready: 6
2026-09-16T15:59:28.166737560Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:28.268532714Z Gathering data for ns/kserve...
2026-09-16T15:59:29.257990508Z Waiting for performance profile collector pods to become ready: 7
2026-09-16T15:59:29.636904393Z Gathering data for ns/openshift-config...
2026-09-16T15:59:29.938419107Z Gathering data for ns/openshift-config-managed...
2026-09-16T15:59:30.390840786Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-16T15:59:30.393568146Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-16T15:59:30.662355730Z Collecting performance related data for node ip-10-0-128-99.ec2.internal
2026-09-16T15:59:30.670787321Z Collecting performance related data for node ip-10-0-136-96.ec2.internal
2026-09-16T15:59:30.678741029Z Collecting performance related data for node ip-10-0-140-96.ec2.internal
2026-09-16T15:59:30.712029418Z Gathering data for ns/openshift-kube-apiserver...
2026-09-16T15:59:31.071768149Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-16T15:59:31.483840397Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-16T15:59:31.872696973Z Gathering data for ns/openshift-kube-scheduler...
2026-09-16T15:59:32.202781912Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-16T15:59:32.571581087Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-16T15:59:32.942223739Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-16T15:59:33.173159942Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:33.325982027Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-16T15:59:33.919975839Z Gathering data for ns/openshift-multus...
2026-09-16T15:59:35.436131147Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-16T15:59:36.787966435Z Gathering data for ns/openshift-host-network...
2026-09-16T15:59:37.106708871Z Gathering data for ns/openshift-network-diagnostics...
2026-09-16T15:59:37.663811617Z Gathering data for ns/openshift-network-node-identity...
2026-09-16T15:59:38.106617855Z Gathering data for ns/openshift-network-operator...
2026-09-16T15:59:38.177927630Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:38.595928245Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-16T15:59:38.882005745Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-16T15:59:38.984012899Z Collecting kubelet logs for node ip-10-0-128-99.ec2.internal
2026-09-16T15:59:38.986029590Z Collecting kubelet logs for node ip-10-0-136-96.ec2.internal
2026-09-16T15:59:38.988101862Z Collecting kubelet logs for node ip-10-0-140-96.ec2.internal
2026-09-16T15:59:39.357080105Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-l4hdc namespace
2026-09-16T15:59:39.363115393Z INFO: Node performance data collection complete.
2026-09-16T15:59:39.522510271Z Gathering data for ns/openshift-apiserver-operator...
2026-09-16T15:59:39.902171800Z Gathering data for ns/openshift-apiserver...
2026-09-16T15:59:40.330703017Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-16T15:59:40.604408402Z Gathering data for ns/openshift-controller-manager...
2026-09-16T15:59:40.899239020Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-16T15:59:41.547041169Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-16T15:59:41.842311180Z Gathering data for ns/openshift-service-ca-operator...
2026-09-16T15:59:42.163790528Z Gathering data for ns/openshift-service-ca...
2026-09-16T15:59:42.556148683Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-16T15:59:43.183782029Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-16T15:59:43.496941191Z Wrote inspect data to must-gather.
2026-09-16T15:59:43.496981303Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T15:59:43.496981303Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-16T15:59:43.630367451Z error: the server doesn't have a resource type "clusters"
2026-09-16T15:59:43.776603441Z Caches written to disk