2026-07-28T14:07:34.330454697Z [disk usage checker] Started
2026-07-28T14:07:34.333899778Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:07:35.278123651Z Gathering data for ns/openshift-cluster-version...
2026-07-28T14:07:35.430232367Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T14:07:35.524219090Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T14:07:35.608434487Z Gathering data for ns/default...
2026-07-28T14:07:35.965982952Z Gathering data for ns/openshift...
2026-07-28T14:07:36.386999922Z Gathering data for ns/kube-system...
2026-07-28T14:07:37.260147353Z Gathering data for ns/openshift-etcd...
2026-07-28T14:07:37.622655443Z Wrote inspect data to must-gather.
2026-07-28T14:07:37.622825345Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T14:07:37.622825345Z     namespaces "assisted-installer" not found
2026-07-28T14:07:38.010923012Z Waiting on subprocesses to finish execution.
2026-07-28T14:07:38.018847483Z Executing Istio gather script
2026-07-28T14:07:38.019424559Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T14:07:38.024278530Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T14:07:38.041010858Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T14:07:38.041341035Z INFO: Gathering HAProxy config files
2026-07-28T14:07:38.046142989Z INFO: Collecting host service logs for crio
2026-07-28T14:07:38.046503368Z INFO: Collecting host service logs for kubelet
2026-07-28T14:07:38.047058993Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T14:07:38.047451123Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T14:07:38.047865549Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T14:07:38.048226311Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T14:07:38.048592788Z INFO: Collecting host service logs for NetworkManager
2026-07-28T14:07:38.048978589Z INFO: Collecting host service logs for openvswitch
2026-07-28T14:07:38.049356108Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T14:07:38.049670610Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T14:07:38.050029984Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T14:07:38.050397612Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T14:07:38.115615148Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T14:07:38.551363015Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-8k7sf
2026-07-28T14:07:38.551413678Z insights-runtime-extractor-5g56g
2026-07-28T14:07:38.551433905Z insights-runtime-extractor-kf94z
2026-07-28T14:07:38.551454460Z insights-runtime-extractor-lfppf
2026-07-28T14:07:38.672336178Z No resources found in openshift-etcd namespace.
2026-07-28T14:07:38.902114422Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T14:07:38.997600202Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T14:07:39.095091923Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T14:07:39.187148723Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T14:07:39.382923940Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T14:07:39.383054496Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T14:07:39.516333585Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T14:07:39.519247486Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:07:39.746690589Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T14:07:39.746690589Z See 'oc rsync -h' for help and examples
2026-07-28T14:07:39.764408083Z Wrote inspect data to must-gather.
2026-07-28T14:07:39.953686799Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T14:07:40.005546992Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.104037740Z No resources found
2026-07-28T14:07:40.167071800Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.254184861Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T14:07:40.282705305Z INFO: Worker host service log collection to complete.
2026-07-28T14:07:40.365734056Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T14:07:40.392103270Z No resources found
2026-07-28T14:07:40.462091327Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.523238668Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.631981319Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.681393099Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T14:07:40.712907070Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T14:07:40.754190882Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.948205303Z INFO: 'previous-logs' folder not found on ip-10-0-129-190.ec2.internal, skipping...
2026-07-28T14:07:40.968985468Z Wrote inspect data to must-gather.
2026-07-28T14:07:40.969398249Z ERROR: No running kube-apiserver pods found
2026-07-28T14:07:40.988371211Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T14:07:41.042094395Z tar: Removing leading `/' from member names
2026-07-28T14:07:41.049502489Z INFO: HAProxy config collection complete.
2026-07-28T14:07:41.165164648Z Gathering data for ns/openshift-monitoring...
2026-07-28T14:07:41.173853115Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T14:07:41.237820957Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T14:07:41.313246074Z Wrote inspect data to must-gather.
2026-07-28T14:07:41.411154296Z INFO: 'previous-logs' folder not found on ip-10-0-137-6.ec2.internal, skipping...
2026-07-28T14:07:41.492890920Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T14:07:41.550374468Z Wrote inspect data to must-gather.
2026-07-28T14:07:41.638003247Z Wrote inspect data to must-gather.
2026-07-28T14:07:41.717786390Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T14:07:41.755841063Z INFO: 'previous-logs' folder not found on ip-10-0-138-35.ec2.internal, skipping...
2026-07-28T14:07:41.755959413Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T14:07:41.756012349Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T14:07:41.766837592Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T14:07:42.004538940Z INFO: INTERCONNECT MODE
2026-07-28T14:07:42.004580504Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T14:07:42.021864535Z Wrote inspect data to must-gather.
2026-07-28T14:07:42.197431739Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T14:07:42.201833210Z INFO: Gathering OVN_Northbound from ovnkube-node-hvt2n...
2026-07-28T14:07:42.202031980Z INFO: Gathering OVN_Northbound from ovnkube-node-k48sz...
2026-07-28T14:07:42.202239053Z INFO: Gathering OVN_Northbound from ovnkube-node-x9s78...
2026-07-28T14:07:42.202472673Z INFO: Gathering OVN_Southbound from ovnkube-node-hvt2n...
2026-07-28T14:07:42.202753183Z INFO: Gathering OVN_Southbound from ovnkube-node-k48sz...
2026-07-28T14:07:42.203228344Z INFO: Gathering OVN_Southbound from ovnkube-node-x9s78...
2026-07-28T14:07:42.333380982Z Wrote inspect data to must-gather.
2026-07-28T14:07:42.516758640Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.516891373Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.550424512Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.566916773Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.580069572Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.581112855Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T14:07:42.632509116Z tar: Removing leading `/' from member names
2026-07-28T14:07:42.665134976Z Wrote inspect data to must-gather.
2026-07-28T14:07:42.856376328Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T14:07:42.900486459Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T14:07:42.917860240Z Wrote inspect data to must-gather.
2026-07-28T14:07:43.122703828Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T14:07:43.124117938Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:43.146390966Z INFO: Waiting for network log collection to complete ...
2026-07-28T14:07:43.146440881Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T14:07:43.146895104Z INFO: Copying ovnk databases complete.
2026-07-28T14:07:43.148877603Z 4.7M	must-gather/network_logs/ovnk_database_store
2026-07-28T14:07:43.151467264Z ovnk_database_store/
2026-07-28T14:07:43.151467264Z ovnk_database_store/ovnkube-node-hvt2n_nbdb
2026-07-28T14:07:43.162250138Z ovnk_database_store/ovnkube-node-hvt2n_sbdb
2026-07-28T14:07:43.162622674Z Wrote inspect data to must-gather.
2026-07-28T14:07:43.173353150Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T14:07:43.188623927Z ovnk_database_store/ovnkube-node-k48sz_nbdb
2026-07-28T14:07:43.208510450Z ovnk_database_store/ovnkube-node-x9s78_sbdb
2026-07-28T14:07:43.238515385Z ovnk_database_store/ovnkube-node-k48sz_sbdb
2026-07-28T14:07:43.273777737Z ovnk_database_store/ovnkube-node-x9s78_nbdb
2026-07-28T14:07:43.288003390Z INFO: Network log collection complete.
2026-07-28T14:07:43.376221978Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:43.380750404Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T14:07:43.404505217Z Wrote inspect data to must-gather.
2026-07-28T14:07:43.599805589Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:43.638679796Z W0728 14:07:43.638629    1524 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T14:07:43.644662478Z Wrote inspect data to must-gather.
2026-07-28T14:07:43.650053463Z INFO: Getting status from alertmanager-main-0
2026-07-28T14:07:43.773189938Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T14:07:43.841133240Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:43.846575354Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T14:07:43.957137506Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T14:07:43.961433219Z INFO: Fallback to identify the container image from release info
2026-07-28T14:07:43.975255220Z Gathering data for ns/openshift-network-console...
2026-07-28T14:07:44.080558537Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:44.290212715Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:44.293483937Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T14:07:44.434908210Z Gathering data for ns/openshift-console-operator...
2026-07-28T14:07:44.515937673Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:44.588411196Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:07:44.724875380Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:44.729840101Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T14:07:44.860819652Z Gathering data for ns/openshift-console...
2026-07-28T14:07:44.912979091Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.125210604Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.128419096Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T14:07:45.318452859Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.428902953Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T14:07:45.537288283Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.540545011Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T14:07:45.713146226Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.925193453Z Wrote inspect data to /must-gather/istio.
2026-07-28T14:07:45.969992308Z Gathering data for ns/openshift-dns-operator...
2026-07-28T14:07:46.137094164Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T14:07:46.294928997Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T14:07:46.357682280Z Gathering data for ns/openshift-dns...
2026-07-28T14:07:46.441231283Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T14:07:46.707881814Z Done executing Istio gather script
2026-07-28T14:07:47.062469900Z Gathering data for ns/openshift-image-registry...
2026-07-28T14:07:47.572235017Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T14:07:47.699013907Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T14:07:48.070434314Z Gathering data for ns/openshift-ingress...
2026-07-28T14:07:48.442244787Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T14:07:48.702056085Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T14:07:48.877088176Z Gathering data for ns/openshift-insights...
2026-07-28T14:07:49.593803495Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:07:49.823677071Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T14:07:50.958876976Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T14:07:51.446926362Z Gathering data for ns/kserve...
2026-07-28T14:07:52.078045383Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T14:07:53.214406855Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T14:07:53.536416331Z Gathering data for ns/openshift-config...
2026-07-28T14:07:53.818231787Z Gathering data for ns/openshift-config-managed...
2026-07-28T14:07:54.296394888Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T14:07:54.354933219Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T14:07:54.594240452Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T14:07:54.598981970Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:07:54.641755366Z Collecting performance related data for node ip-10-0-137-6.ec2.internal
2026-07-28T14:07:54.653260666Z Collecting performance related data for node ip-10-0-138-35.ec2.internal
2026-07-28T14:07:54.662926395Z Collecting performance related data for node ip-10-0-129-190.ec2.internal
2026-07-28T14:07:54.931943166Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T14:07:55.280447827Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T14:07:55.695370846Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T14:07:56.012958080Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T14:07:56.438633917Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T14:07:56.806560365Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T14:07:57.224860834Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T14:07:57.871416233Z Gathering data for ns/openshift-multus...
2026-07-28T14:07:59.604082333Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:08:00.017272675Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T14:08:01.426561974Z Gathering data for ns/openshift-host-network...
2026-07-28T14:08:01.751269513Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T14:08:02.350922379Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T14:08:02.818416257Z Gathering data for ns/openshift-network-operator...
2026-07-28T14:08:03.241052629Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T14:08:03.294157386Z Collecting kubelet logs for node ip-10-0-137-6.ec2.internal
2026-07-28T14:08:03.297305441Z Collecting kubelet logs for node ip-10-0-138-35.ec2.internal
2026-07-28T14:08:03.299675722Z Collecting kubelet logs for node ip-10-0-129-190.ec2.internal
2026-07-28T14:08:03.611713987Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T14:08:03.752320304Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wskns namespace
2026-07-28T14:08:03.760247257Z INFO: Node performance data collection complete.
2026-07-28T14:08:04.187550217Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T14:08:04.550403074Z Gathering data for ns/openshift-apiserver...
2026-07-28T14:08:04.612212351Z [disk usage checker] Volume usage percentage: current = 33 ; allowed = 70
2026-07-28T14:08:04.991242109Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T14:08:05.293674387Z Gathering data for ns/openshift-controller-manager...
2026-07-28T14:08:05.610820066Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T14:08:06.325195520Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T14:08:06.634606383Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T14:08:06.978447346Z Gathering data for ns/openshift-service-ca...
2026-07-28T14:08:07.410909394Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T14:08:08.443101119Z Wrote inspect data to must-gather.
2026-07-28T14:08:08.443150668Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T14:08:08.443150668Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T14:08:08.584928201Z error: the server doesn't have a resource type "clusters"
2026-07-28T14:08:09.005496954Z Caches written to disk