2026-07-27T15:49:41.311951147Z [disk usage checker] Started
2026-07-27T15:49:41.315603861Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:49:41.515144618Z Gathering data for ns/openshift-cluster-version...
2026-07-27T15:49:41.653223731Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T15:49:41.756204539Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T15:49:41.837673188Z Gathering data for ns/default...
2026-07-27T15:49:42.151316697Z Gathering data for ns/openshift...
2026-07-27T15:49:42.599226882Z Gathering data for ns/kube-system...
2026-07-27T15:49:43.264157293Z Gathering data for ns/openshift-etcd...
2026-07-27T15:49:43.549046092Z Wrote inspect data to must-gather.
2026-07-27T15:49:43.549081895Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T15:49:43.549081895Z     namespaces "assisted-installer" not found
2026-07-27T15:49:43.907046483Z Waiting on subprocesses to finish execution.
2026-07-27T15:49:43.915318622Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T15:49:43.917302480Z INFO: Gathering HAProxy config files
2026-07-27T15:49:43.919333060Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T15:49:43.926599100Z INFO: Collecting host service logs for crio
2026-07-27T15:49:43.926767484Z Executing Istio gather script
2026-07-27T15:49:43.926987659Z INFO: Collecting host service logs for kubelet
2026-07-27T15:49:43.927274593Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T15:49:43.927581145Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T15:49:43.927929344Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T15:49:43.928212051Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T15:49:43.928562807Z INFO: Collecting host service logs for NetworkManager
2026-07-27T15:49:43.928883165Z INFO: Collecting host service logs for openvswitch
2026-07-27T15:49:43.929387818Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T15:49:43.929730022Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T15:49:43.935219599Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T15:49:43.936212293Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T15:49:43.937824428Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T15:49:44.139358722Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T15:49:44.665752278Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T15:49:44.700622946Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T15:49:44.700622946Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T15:49:44.741308051Z No resources found
2026-07-27T15:49:44.753489345Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T15:49:44.884478382Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-d4h2n
2026-07-27T15:49:44.884548403Z insights-runtime-extractor-ldrm4
2026-07-27T15:49:44.884569744Z insights-runtime-extractor-rw87b
2026-07-27T15:49:44.884589175Z insights-runtime-extractor-sczqn
2026-07-27T15:49:44.948304019Z No resources found in openshift-etcd namespace.
2026-07-27T15:49:45.039991026Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T15:49:45.087071485Z Wrote inspect data to must-gather.
2026-07-27T15:49:45.189013849Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T15:49:45.430216958Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T15:49:45.445648921Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T15:49:45.528449758Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T15:49:45.529502537Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T15:49:45.537357970Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T15:49:45.537357970Z See 'oc rsync -h' for help and examples
2026-07-27T15:49:45.566590413Z ERROR: No running kube-apiserver pods found
2026-07-27T15:49:45.617580423Z INFO: Worker host service log collection to complete.
2026-07-27T15:49:45.683026994Z Wrote inspect data to must-gather.
2026-07-27T15:49:45.784436434Z Wrote inspect data to must-gather.
2026-07-27T15:49:45.937591823Z No resources found
2026-07-27T15:49:45.992668186Z tar: Removing leading `/' from member names
2026-07-27T15:49:46.000893975Z INFO: HAProxy config collection complete.
2026-07-27T15:49:46.169006588Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T15:49:46.223817505Z Wrote inspect data to must-gather.
2026-07-27T15:49:46.262975623Z Gathering data for ns/openshift-monitoring...
2026-07-27T15:49:46.264761869Z INFO: 'previous-logs' folder not found on ip-10-0-141-171.ec2.internal, skipping...
2026-07-27T15:49:46.341121040Z Wrote inspect data to must-gather.
2026-07-27T15:49:46.386702339Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:49:46.393169740Z Wrote inspect data to must-gather.
2026-07-27T15:49:46.408181545Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T15:49:46.415452528Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T15:49:46.599261752Z Wrote inspect data to must-gather.
2026-07-27T15:49:46.653647727Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T15:49:46.724092964Z INFO: 'previous-logs' folder not found on ip-10-0-141-55.ec2.internal, skipping...
2026-07-27T15:49:46.751332159Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T15:49:46.805301689Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T15:49:46.908122730Z Wrote inspect data to must-gather.
2026-07-27T15:49:46.926685426Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.025756239Z INFO: 'previous-logs' folder not found on ip-10-0-142-155.ec2.internal, skipping...
2026-07-27T15:49:47.025840315Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T15:49:47.025875392Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T15:49:47.094004148Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T15:49:47.169378730Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T15:49:47.172158952Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.313312700Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.440197859Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.575412796Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.641263432Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T15:49:47.737878076Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T15:49:47.781833922Z INFO: INTERCONNECT MODE
2026-07-27T15:49:47.781919317Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T15:49:47.813590174Z Wrote inspect data to must-gather.
2026-07-27T15:49:47.852707182Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T15:49:47.900759956Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T15:49:47.921231305Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:47.949230193Z INFO: Gathering OVN_Northbound from ovnkube-node-6mjx6...
2026-07-27T15:49:47.949526439Z INFO: Gathering OVN_Northbound from ovnkube-node-j9hjj...
2026-07-27T15:49:47.949728207Z INFO: Gathering OVN_Northbound from ovnkube-node-p9pxs...
2026-07-27T15:49:47.950381532Z INFO: Gathering OVN_Southbound from ovnkube-node-6mjx6...
2026-07-27T15:49:47.950722486Z INFO: Gathering OVN_Southbound from ovnkube-node-j9hjj...
2026-07-27T15:49:47.951268568Z INFO: Gathering OVN_Southbound from ovnkube-node-p9pxs...
2026-07-27T15:49:48.108064412Z Wrote inspect data to must-gather.
2026-07-27T15:49:48.259104838Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.295196395Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.296102465Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.312009226Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:48.315545506Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T15:49:48.345923695Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.350573705Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.357560621Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T15:49:48.369008218Z tar: Removing leading `/' from member names
2026-07-27T15:49:48.432939841Z Wrote inspect data to must-gather.
2026-07-27T15:49:48.570335628Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:48.624140147Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T15:49:48.672773338Z W0727 15:49:48.672720    1438 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-27T15:49:48.679895863Z Wrote inspect data to must-gather.
2026-07-27T15:49:48.699717218Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T15:49:48.828003457Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:48.831611370Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T15:49:48.918259054Z Gathering data for ns/openshift-network-console...
2026-07-27T15:49:48.928966138Z INFO: Waiting for network log collection to complete ...
2026-07-27T15:49:48.929059868Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T15:49:48.929574378Z INFO: Copying ovnk databases complete.
2026-07-27T15:49:48.935490288Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-27T15:49:48.938413795Z ovnk_database_store/
2026-07-27T15:49:48.938413795Z ovnk_database_store/ovnkube-node-p9pxs_nbdb
2026-07-27T15:49:48.948294718Z ovnk_database_store/ovnkube-node-6mjx6_sbdb
2026-07-27T15:49:48.974866242Z ovnk_database_store/ovnkube-node-j9hjj_nbdb
2026-07-27T15:49:48.982264730Z ovnk_database_store/ovnkube-node-p9pxs_sbdb
2026-07-27T15:49:49.003603719Z ovnk_database_store/ovnkube-node-6mjx6_nbdb
2026-07-27T15:49:49.011838968Z ovnk_database_store/ovnkube-node-j9hjj_sbdb
2026-07-27T15:49:49.030684380Z Wrote inspect data to must-gather.
2026-07-27T15:49:49.034136007Z INFO: Network log collection complete.
2026-07-27T15:49:49.082735434Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:49.089123484Z INFO: Getting status from alertmanager-main-0
2026-07-27T15:49:49.299402505Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:49.305213829Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T15:49:49.321929973Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T15:49:49.324117773Z INFO: Fallback to identify the container image from release info
2026-07-27T15:49:49.404162240Z Gathering data for ns/openshift-console-operator...
2026-07-27T15:49:49.545684681Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:49.739536864Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:49.742839515Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T15:49:49.785464071Z Gathering data for ns/openshift-console...
2026-07-27T15:49:49.906784343Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:50.055130334Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T15:49:50.108379248Z Wrote inspect data to /must-gather/istio.
2026-07-27T15:49:50.189149423Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T15:49:50.313018799Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T15:49:50.335252298Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T15:49:50.840826528Z Done executing Istio gather script
2026-07-27T15:49:50.864398936Z Gathering data for ns/openshift-dns-operator...
2026-07-27T15:49:51.165582232Z Gathering data for ns/openshift-dns...
2026-07-27T15:49:51.394576869Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:49:51.458880090Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T15:49:51.735983111Z Gathering data for ns/openshift-image-registry...
2026-07-27T15:49:52.154530758Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T15:49:52.493016466Z Gathering data for ns/openshift-ingress...
2026-07-27T15:49:52.590115694Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T15:49:52.845886178Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T15:49:53.204520702Z Gathering data for ns/openshift-insights...
2026-07-27T15:49:53.702562516Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T15:49:54.824210947Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T15:49:55.011800657Z Gathering data for ns/openshift-jobset-operator...
2026-07-27T15:49:55.796507818Z Gathering data for ns/openshift-config...
2026-07-27T15:49:55.937718234Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T15:49:56.071833090Z Gathering data for ns/openshift-config-managed...
2026-07-27T15:49:56.400187488Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:49:56.513170828Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T15:49:56.809674391Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T15:49:57.060160741Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T15:49:57.063341154Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T15:49:57.352192401Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T15:49:57.675623518Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T15:49:57.964266636Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T15:49:58.178194613Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T15:49:58.262467854Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T15:49:58.445678241Z Collecting performance related data for node ip-10-0-141-55.ec2.internal
2026-07-27T15:49:58.453127888Z Collecting performance related data for node ip-10-0-141-171.ec2.internal
2026-07-27T15:49:58.460757239Z Collecting performance related data for node ip-10-0-142-155.ec2.internal
2026-07-27T15:49:58.676210502Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T15:49:59.018568264Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T15:49:59.654010815Z Gathering data for ns/openshift-multus...
2026-07-27T15:50:01.031667643Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T15:50:01.405596248Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:50:02.216300976Z Gathering data for ns/openshift-host-network...
2026-07-27T15:50:02.509005093Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T15:50:03.002028571Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T15:50:03.409783617Z Gathering data for ns/openshift-network-operator...
2026-07-27T15:50:03.803430171Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T15:50:04.078000615Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T15:50:04.571310359Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T15:50:04.857557836Z Gathering data for ns/openshift-apiserver...
2026-07-27T15:50:05.290867706Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T15:50:05.545821348Z Gathering data for ns/openshift-controller-manager...
2026-07-27T15:50:05.836143314Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T15:50:06.410446199Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-27T15:50:06.479558620Z Collecting kubelet logs for node ip-10-0-141-55.ec2.internal
2026-07-27T15:50:06.481601795Z Collecting kubelet logs for node ip-10-0-141-171.ec2.internal
2026-07-27T15:50:06.484164317Z Collecting kubelet logs for node ip-10-0-142-155.ec2.internal
2026-07-27T15:50:06.490857455Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T15:50:06.835390443Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-chpws namespace
2026-07-27T15:50:06.839597219Z INFO: Node performance data collection complete.
2026-07-27T15:50:06.845407804Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T15:50:07.167468383Z Gathering data for ns/openshift-service-ca...
2026-07-27T15:50:07.492437764Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T15:50:08.320590661Z Wrote inspect data to must-gather.
2026-07-27T15:50:08.320638922Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T15:50:08.320638922Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T15:50:08.446410537Z error: the server doesn't have a resource type "clusters"
2026-07-27T15:50:08.622183044Z Caches written to disk