2026-07-08T23:36:51.526182907Z [disk usage checker] Started
2026-07-08T23:36:51.529652481Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:36:51.723754134Z Gathering data for ns/openshift-cluster-version...
2026-07-08T23:36:51.853096296Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T23:36:51.930953271Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T23:36:52.009751091Z Gathering data for ns/default...
2026-07-08T23:36:52.297535410Z Gathering data for ns/openshift...
2026-07-08T23:36:52.703792273Z Gathering data for ns/kube-system...
2026-07-08T23:36:53.354607778Z Gathering data for ns/openshift-etcd...
2026-07-08T23:36:53.648447273Z Wrote inspect data to must-gather.
2026-07-08T23:36:53.648483619Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T23:36:53.648483619Z     namespaces "assisted-installer" not found
2026-07-08T23:36:54.088428624Z Waiting on subprocesses to finish execution.
2026-07-08T23:36:54.094213400Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-08T23:36:54.098651008Z Executing Istio gather script
2026-07-08T23:36:54.108362647Z INFO: Gathering HAProxy config files
2026-07-08T23:36:54.116521321Z INFO: Collecting host service logs for crio
2026-07-08T23:36:54.116800652Z INFO: Collecting host service logs for kubelet
2026-07-08T23:36:54.117141502Z INFO: Collecting host service logs for rpm-ostreed
2026-07-08T23:36:54.117434915Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-08T23:36:54.117749983Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-08T23:36:54.118064233Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-08T23:36:54.118358297Z INFO: Collecting host service logs for NetworkManager
2026-07-08T23:36:54.118672206Z INFO: Collecting host service logs for openvswitch
2026-07-08T23:36:54.119008675Z INFO: Collecting host service logs for ovs-configuration
2026-07-08T23:36:54.119320285Z INFO: Collecting host service logs for ovsdb-server
2026-07-08T23:36:54.119613403Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-08T23:36:54.119840771Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-08T23:36:54.119978169Z INFO: Waiting for worker host service log collection to complete ...
2026-07-08T23:36:54.128785239Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-08T23:36:54.168602557Z INFO: Waiting for node performance related collection to complete ...
2026-07-08T23:36:54.727708914Z No resources found
2026-07-08T23:36:55.012271007Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-hzgxc
2026-07-08T23:36:55.012323489Z insights-runtime-extractor-cd5zf
2026-07-08T23:36:55.012342884Z insights-runtime-extractor-jzc9x
2026-07-08T23:36:55.012361259Z insights-runtime-extractor-xk8fg
2026-07-08T23:36:55.064669232Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-08T23:36:55.097219873Z No resources found in openshift-etcd namespace.
2026-07-08T23:36:55.118971782Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-08T23:36:55.139393502Z Wrote inspect data to must-gather.
2026-07-08T23:36:55.256569924Z error: the server doesn't have a resource type "performanceprofile"
2026-07-08T23:36:55.270416094Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-08T23:36:55.352122537Z INFO: "metallb-operator" not detected. Skipping.
2026-07-08T23:36:55.438088448Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-08T23:36:55.471219339Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-08T23:36:55.471262914Z INFO: on-disk MachineConfig config collection complete.
2026-07-08T23:36:55.561781203Z Wrote inspect data to must-gather.
2026-07-08T23:36:55.849075432Z INFO: Worker host service log collection to complete.
2026-07-08T23:36:55.892538813Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-08T23:36:55.924908680Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-08T23:36:56.127390696Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T23:36:56.254162091Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-08T23:36:56.254162091Z See 'oc rsync -h' for help and examples
2026-07-08T23:36:56.289110414Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.293331663Z No resources found
2026-07-08T23:36:56.329653814Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-08T23:36:56.418223778Z tar: Removing leading `/' from member names
2026-07-08T23:36:56.423738194Z INFO: HAProxy config collection complete.
2026-07-08T23:36:56.425579064Z Gathering data for ns/openshift-monitoring...
2026-07-08T23:36:56.456577831Z ERROR: No running kube-apiserver pods found
2026-07-08T23:36:56.557944820Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.562189706Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:36:56.584073417Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.625554013Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.630984949Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.643901635Z INFO: 'previous-logs' folder not found on ip-10-0-135-235.ec2.internal, skipping...
2026-07-08T23:36:56.829109572Z error: the server doesn't have a resource type "machineconfigs"
2026-07-08T23:36:56.872598614Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-08T23:36:56.904578582Z Wrote inspect data to must-gather.
2026-07-08T23:36:56.957419797Z INFO: 'previous-logs' folder not found on ip-10-0-138-110.ec2.internal, skipping...
2026-07-08T23:36:56.968708929Z INFO: Getting rules from prometheus-k8s-0
2026-07-08T23:36:57.013683657Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-08T23:36:57.198443596Z Wrote inspect data to must-gather.
2026-07-08T23:36:57.239898526Z INFO: 'previous-logs' folder not found on ip-10-0-138-77.ec2.internal, skipping...
2026-07-08T23:36:57.239955435Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-08T23:36:57.239955435Z INFO: Machine Config Daemon termination log collection complete.
2026-07-08T23:36:57.251328408Z Wrote inspect data to must-gather.
2026-07-08T23:36:57.375485724Z INFO: Getting status/config from prometheus-k8s-0
2026-07-08T23:36:57.408438800Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-08T23:36:57.458298356Z Wrote inspect data to must-gather.
2026-07-08T23:36:57.624478845Z Wrote inspect data to must-gather.
2026-07-08T23:36:57.651373401Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-08T23:36:57.744584809Z INFO: INTERCONNECT MODE
2026-07-08T23:36:57.744584809Z INFO: Gathering ovn-kubernetes DBs
2026-07-08T23:36:57.817001575Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-08T23:36:57.870664514Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:57.880251388Z Wrote inspect data to must-gather.
2026-07-08T23:36:57.931465873Z INFO: Gathering OVN_Northbound from ovnkube-node-59fxh...
2026-07-08T23:36:57.931747199Z INFO: Gathering OVN_Northbound from ovnkube-node-5rtg5...
2026-07-08T23:36:57.931992742Z INFO: Gathering OVN_Northbound from ovnkube-node-sbh7p...
2026-07-08T23:36:57.932322418Z INFO: Gathering OVN_Southbound from ovnkube-node-59fxh...
2026-07-08T23:36:57.932593907Z INFO: Gathering OVN_Southbound from ovnkube-node-5rtg5...
2026-07-08T23:36:57.932880166Z INFO: Gathering OVN_Southbound from ovnkube-node-sbh7p...
2026-07-08T23:36:57.946093400Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T23:36:58.147167439Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-08T23:36:58.207979457Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:58.211731303Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-08T23:36:58.231154581Z Wrote inspect data to must-gather.
2026-07-08T23:36:58.247464669Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.282505216Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.328618517Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.346216414Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.348817757Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.352825815Z tar: Removing leading `/' from member names
2026-07-08T23:36:58.483979257Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-08T23:36:58.488052721Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:58.513200412Z Wrote inspect data to must-gather.
2026-07-08T23:36:58.746987899Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:58.749293244Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-08T23:36:58.764291914Z W0708 23:36:58.764224    1473 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-08T23:36:58.771337460Z Wrote inspect data to must-gather.
2026-07-08T23:36:58.774904272Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-08T23:36:58.789349429Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T23:36:58.840746312Z INFO: Waiting for network log collection to complete ...
2026-07-08T23:36:58.840904004Z INFO: Waiting for ovnk database copies to complete ...
2026-07-08T23:36:58.841438815Z INFO: Copying ovnk databases complete.
2026-07-08T23:36:58.843450468Z 2.4M	must-gather/network_logs/ovnk_database_store
2026-07-08T23:36:58.845562809Z ovnk_database_store/
2026-07-08T23:36:58.845562809Z ovnk_database_store/ovnkube-node-5rtg5_nbdb
2026-07-08T23:36:58.855104444Z ovnk_database_store/ovnkube-node-5rtg5_sbdb
2026-07-08T23:36:58.870013711Z ovnk_database_store/ovnkube-node-59fxh_sbdb
2026-07-08T23:36:58.889861363Z ovnk_database_store/ovnkube-node-59fxh_nbdb
2026-07-08T23:36:58.897079021Z ovnk_database_store/ovnkube-node-sbh7p_nbdb
2026-07-08T23:36:58.905187030Z ovnk_database_store/ovnkube-node-sbh7p_sbdb
2026-07-08T23:36:58.926311690Z INFO: Network log collection complete.
2026-07-08T23:36:58.990719199Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:58.999822547Z Gathering data for ns/openshift-network-console...
2026-07-08T23:36:59.077869413Z Wrote inspect data to must-gather.
2026-07-08T23:36:59.200376569Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:59.203267407Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-08T23:36:59.247900385Z INFO: Getting status from alertmanager-main-0
2026-07-08T23:36:59.391904486Z Wrote inspect data to must-gather.
2026-07-08T23:36:59.445729773Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:59.466281342Z Gathering data for ns/openshift-console-operator...
2026-07-08T23:36:59.653907795Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:59.657277039Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-08T23:36:59.668502472Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-08T23:36:59.671239991Z INFO: Fallback to identify the container image from release info
2026-07-08T23:36:59.828924298Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:36:59.848382349Z Gathering data for ns/openshift-console...
2026-07-08T23:37:00.009622130Z Wrote inspect data to /must-gather/istio.
2026-07-08T23:37:00.340895290Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-08T23:37:00.385359480Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-08T23:37:00.512491082Z daemonset.apps/perf-node-gather-daemonset created
2026-07-08T23:37:00.653910569Z Waiting for performance profile collector pods to become ready: 1
2026-07-08T23:37:00.701960428Z Done executing Istio gather script
2026-07-08T23:37:00.844141393Z Gathering data for ns/openshift-dns-operator...
2026-07-08T23:37:01.116429995Z Gathering data for ns/openshift-dns...
2026-07-08T23:37:01.580075422Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:37:01.663097506Z Gathering data for ns/openshift-image-registry...
2026-07-08T23:37:01.759308250Z Waiting for performance profile collector pods to become ready: 2
2026-07-08T23:37:02.083174805Z Gathering data for ns/openshift-ingress-operator...
2026-07-08T23:37:02.382802924Z Gathering data for ns/openshift-ingress...
2026-07-08T23:37:02.669964608Z Gathering data for ns/openshift-ingress-canary...
2026-07-08T23:37:02.872204788Z Waiting for performance profile collector pods to become ready: 3
2026-07-08T23:37:03.034638706Z Gathering data for ns/openshift-insights...
2026-07-08T23:37:03.976700889Z Waiting for performance profile collector pods to become ready: 4
2026-07-08T23:37:04.801515953Z Gathering data for ns/openshift-config...
2026-07-08T23:37:05.039451804Z Gathering data for ns/openshift-config-managed...
2026-07-08T23:37:05.110519908Z Waiting for performance profile collector pods to become ready: 5
2026-07-08T23:37:05.441418364Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-08T23:37:05.666524175Z Gathering data for ns/openshift-kube-apiserver...
2026-07-08T23:37:05.885900885Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-08T23:37:06.146957559Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-08T23:37:06.251863558Z Waiting for performance profile collector pods to become ready: 6
2026-07-08T23:37:06.505057293Z Gathering data for ns/openshift-kube-scheduler...
2026-07-08T23:37:06.584718003Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:37:06.726955672Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-08T23:37:06.989795962Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-08T23:37:07.286256721Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-08T23:37:07.378941824Z Waiting for performance profile collector pods to become ready: 7
2026-07-08T23:37:07.590586715Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-08T23:37:08.036303622Z Gathering data for ns/openshift-multus...
2026-07-08T23:37:08.483641121Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-08T23:37:08.705962014Z Collecting performance related data for node ip-10-0-138-110.ec2.internal
2026-07-08T23:37:08.712961140Z Collecting performance related data for node ip-10-0-138-77.ec2.internal
2026-07-08T23:37:08.720764123Z Collecting performance related data for node ip-10-0-135-235.ec2.internal
2026-07-08T23:37:09.510537549Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-08T23:37:10.706678529Z Gathering data for ns/openshift-host-network...
2026-07-08T23:37:11.009733551Z Gathering data for ns/openshift-network-diagnostics...
2026-07-08T23:37:11.544472380Z Gathering data for ns/openshift-network-node-identity...
2026-07-08T23:37:11.590421356Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:37:11.914483475Z Gathering data for ns/openshift-network-operator...
2026-07-08T23:37:12.265400841Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-08T23:37:12.501334241Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-08T23:37:12.950972940Z Gathering data for ns/openshift-apiserver-operator...
2026-07-08T23:37:13.226992898Z Gathering data for ns/openshift-apiserver...
2026-07-08T23:37:13.627576760Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-08T23:37:13.873429089Z Gathering data for ns/openshift-controller-manager...
2026-07-08T23:37:14.166991327Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-08T23:37:14.780379391Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-08T23:37:15.043633835Z Gathering data for ns/openshift-service-ca-operator...
2026-07-08T23:37:15.326327212Z Gathering data for ns/openshift-service-ca...
2026-07-08T23:37:15.646549084Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-08T23:37:16.483361877Z Wrote inspect data to must-gather.
2026-07-08T23:37:16.483408302Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T23:37:16.483408302Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-08T23:37:16.595762283Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-08T23:37:16.841267580Z Collecting kubelet logs for node ip-10-0-138-110.ec2.internal
2026-07-08T23:37:16.842983062Z Collecting kubelet logs for node ip-10-0-138-77.ec2.internal
2026-07-08T23:37:16.844648241Z Collecting kubelet logs for node ip-10-0-135-235.ec2.internal
2026-07-08T23:37:17.134712166Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x6l9j namespace
2026-07-08T23:37:17.139217466Z INFO: Node performance data collection complete.
2026-07-08T23:37:17.274973663Z error: the server doesn't have a resource type "clusters"
2026-07-08T23:37:17.397122498Z Caches written to disk