2026-07-09T18:53:13.458357712Z [disk usage checker] Started
2026-07-09T18:53:13.462430172Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:13.658053821Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:53:13.804101325Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:53:13.904584374Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:53:13.994567568Z Gathering data for ns/default...
2026-07-09T18:53:14.325775466Z Gathering data for ns/openshift...
2026-07-09T18:53:14.777196861Z Gathering data for ns/kube-system...
2026-07-09T18:53:15.506616549Z Gathering data for ns/openshift-etcd...
2026-07-09T18:53:15.827880670Z Wrote inspect data to must-gather.
2026-07-09T18:53:15.827917214Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:53:15.827917214Z     namespaces "assisted-installer" not found
2026-07-09T18:53:16.211111446Z Waiting on subprocesses to finish execution.
2026-07-09T18:53:16.227351651Z INFO: Gathering HAProxy config files
2026-07-09T18:53:16.228590858Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:53:16.229884009Z Executing Istio gather script
2026-07-09T18:53:16.234687114Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:53:16.234687114Z INFO: Collecting host service logs for crio
2026-07-09T18:53:16.235038390Z INFO: Collecting host service logs for kubelet
2026-07-09T18:53:16.235337658Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:53:16.235697874Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:53:16.236016658Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:53:16.236379582Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:53:16.236728181Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:53:16.237073099Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:53:16.237405253Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:53:16.237780264Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:53:16.238143034Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:53:16.238540979Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:53:16.255790782Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:53:16.619744537Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:53:16.649227342Z No resources found
2026-07-09T18:53:17.075261926Z No resources found in openshift-etcd namespace.
2026-07-09T18:53:17.152864325Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:53:17.335444553Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:53:17.515827383Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:53:17.515920027Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:53:17.518943213Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:53:17.608374387Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-4z99q
2026-07-09T18:53:17.608416212Z insights-runtime-extractor-jnp7h
2026-07-09T18:53:17.608436585Z insights-runtime-extractor-lzb6m
2026-07-09T18:53:17.608455309Z insights-runtime-extractor-zstkv
2026-07-09T18:53:17.784798174Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:53:17.796978289Z Wrote inspect data to must-gather.
2026-07-09T18:53:18.086015988Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:53:18.212332856Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:53:18.297269865Z INFO: 'previous-logs' folder not found on ip-10-0-130-107.ec2.internal, skipping...
2026-07-09T18:53:18.369545688Z Wrote inspect data to must-gather.
2026-07-09T18:53:18.423491638Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:53:18.466686128Z INFO: Worker host service log collection to complete.
2026-07-09T18:53:18.476813120Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:53:18.663455673Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:18.683525222Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:53:18.789126163Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:53:18.789126163Z See 'oc rsync -h' for help and examples
2026-07-09T18:53:18.796710666Z Wrote inspect data to must-gather.
2026-07-09T18:53:18.886964964Z Wrote inspect data to must-gather.
2026-07-09T18:53:18.893895250Z No resources found
2026-07-09T18:53:18.910882260Z Wrote inspect data to must-gather.
2026-07-09T18:53:18.915854197Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:53:18.938756291Z ERROR: No running kube-apiserver pods found
2026-07-09T18:53:19.145030821Z tar: Removing leading `/' from member names
2026-07-09T18:53:19.154488991Z INFO: HAProxy config collection complete.
2026-07-09T18:53:19.227289624Z INFO: 'previous-logs' folder not found on ip-10-0-135-87.ec2.internal, skipping...
2026-07-09T18:53:19.236820714Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:53:19.251483573Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.261451063Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.328963417Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.480032378Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:53:19.492180824Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:53:19.559386527Z INFO: 'previous-logs' folder not found on ip-10-0-141-141.ec2.internal, skipping...
2026-07-09T18:53:19.559423052Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:53:19.559451006Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:53:19.615956440Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.676523165Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:53:19.789809271Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:53:19.889941047Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.933791788Z Wrote inspect data to must-gather.
2026-07-09T18:53:19.963302462Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:53:20.116802865Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:53:20.199262358Z INFO: INTERCONNECT MODE
2026-07-09T18:53:20.199292110Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:53:20.348280855Z Wrote inspect data to must-gather.
2026-07-09T18:53:20.376536481Z INFO: Gathering OVN_Northbound from ovnkube-node-hmxsn...
2026-07-09T18:53:20.376743271Z INFO: Gathering OVN_Northbound from ovnkube-node-p7npv...
2026-07-09T18:53:20.376943206Z INFO: Gathering OVN_Northbound from ovnkube-node-p82cd...
2026-07-09T18:53:20.377214301Z INFO: Gathering OVN_Southbound from ovnkube-node-hmxsn...
2026-07-09T18:53:20.377446738Z INFO: Gathering OVN_Southbound from ovnkube-node-p7npv...
2026-07-09T18:53:20.377766710Z INFO: Gathering OVN_Southbound from ovnkube-node-p82cd...
2026-07-09T18:53:20.387104515Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:53:20.729244973Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.743294596Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.754694816Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.762707188Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.768237687Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.768518082Z tar: Removing leading `/' from member names
2026-07-09T18:53:20.775370244Z Wrote inspect data to must-gather.
2026-07-09T18:53:20.835475074Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:53:21.064888482Z Wrote inspect data to must-gather.
2026-07-09T18:53:21.083656450Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:53:21.130354305Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:53:21.139632183Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:53:21.301438195Z Wrote inspect data to must-gather.
2026-07-09T18:53:21.308136671Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:53:21.308187913Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:53:21.308750623Z INFO: Copying ovnk databases complete.
2026-07-09T18:53:21.310384308Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:53:21.313010893Z ovnk_database_store/
2026-07-09T18:53:21.313010893Z ovnk_database_store/ovnkube-node-p7npv_sbdb
2026-07-09T18:53:21.340217878Z ovnk_database_store/ovnkube-node-p7npv_nbdb
2026-07-09T18:53:21.350374718Z ovnk_database_store/ovnkube-node-p82cd_sbdb
2026-07-09T18:53:21.376283290Z ovnk_database_store/ovnkube-node-hmxsn_sbdb
2026-07-09T18:53:21.378324343Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:21.403688629Z ovnk_database_store/ovnkube-node-hmxsn_nbdb
2026-07-09T18:53:21.410967615Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:53:21.415398147Z ovnk_database_store/ovnkube-node-p82cd_nbdb
2026-07-09T18:53:21.429482322Z INFO: Network log collection complete.
2026-07-09T18:53:21.540505105Z Wrote inspect data to must-gather.
2026-07-09T18:53:21.603191867Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:21.605198065Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:53:21.751608093Z W0709 18:53:21.751537    1489 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-09T18:53:21.759056750Z Wrote inspect data to must-gather.
2026-07-09T18:53:21.782030690Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:53:21.825538882Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:21.836814857Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:53:21.983355706Z Gathering data for ns/openshift-network-console...
2026-07-09T18:53:22.075234929Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:22.079283207Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:53:22.094473414Z Wrote inspect data to must-gather.
2026-07-09T18:53:22.287534176Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:22.380466905Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:53:22.383076408Z INFO: Fallback to identify the container image from release info
2026-07-09T18:53:22.461481876Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:53:22.509870716Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:22.513656576Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:53:22.729762526Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:22.842839876Z Gathering data for ns/openshift-console...
2026-07-09T18:53:22.917985833Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:22.921323293Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T18:53:23.092381173Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:23.250999245Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:53:23.312401367Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:23.315641762Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T18:53:23.405183133Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:53:23.427757307Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:53:23.509003704Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:23.562450869Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:53:23.672517293Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:23.711271512Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:23.713661528Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T18:53:23.916849386Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.063268596Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:53:24.140041789Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.143412553Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:53:24.299707317Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.418564025Z Gathering data for ns/openshift-dns...
2026-07-09T18:53:24.496316182Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.498713141Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:53:24.661318730Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.689002269Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:53:24.859699486Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:24.862266528Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:53:25.018621788Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:25.187494752Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:53:25.221368817Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:53:25.719136404Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:53:25.836013440Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:53:25.899430589Z Done executing Istio gather script
2026-07-09T18:53:26.146796124Z Gathering data for ns/openshift-ingress...
2026-07-09T18:53:26.504586055Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:53:26.932241166Z Gathering data for ns/openshift-insights...
2026-07-09T18:53:26.965438603Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:53:28.088203173Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:53:28.677204067Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:29.203385486Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:53:29.626308099Z Gathering data for ns/kserve...
2026-07-09T18:53:30.327131611Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:53:31.203770444Z Gathering data for ns/openshift-config...
2026-07-09T18:53:31.457162162Z Waiting for performance profile collector pods to become ready: 8
2026-07-09T18:53:31.493795261Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:53:32.002032726Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:53:32.312039912Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:53:32.580766550Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:53:32.644517423Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:53:32.836234945Z Collecting performance related data for node ip-10-0-135-87.ec2.internal
2026-07-09T18:53:32.845022075Z Collecting performance related data for node ip-10-0-141-141.ec2.internal
2026-07-09T18:53:32.852870493Z Collecting performance related data for node ip-10-0-130-107.ec2.internal
2026-07-09T18:53:33.012191145Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:53:33.422788182Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:53:33.684510649Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:33.795755529Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:53:34.216666650Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:53:34.639976626Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:53:35.018866417Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:53:35.620897312Z Gathering data for ns/openshift-multus...
2026-07-09T18:53:37.105706680Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:53:38.517943903Z Gathering data for ns/openshift-host-network...
2026-07-09T18:53:38.689232892Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:38.914217718Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:53:39.545543603Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:53:39.989913093Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:53:40.438352338Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:53:40.724138769Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:53:40.933825077Z Collecting kubelet logs for node ip-10-0-135-87.ec2.internal
2026-07-09T18:53:40.935527160Z Collecting kubelet logs for node ip-10-0-141-141.ec2.internal
2026-07-09T18:53:40.937225649Z Collecting kubelet logs for node ip-10-0-130-107.ec2.internal
2026-07-09T18:53:41.285762909Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:53:41.354218822Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-gkb77 namespace
2026-07-09T18:53:41.359500631Z INFO: Node performance data collection complete.
2026-07-09T18:53:41.630013572Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:53:42.031839013Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:53:42.304000394Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:53:42.620430343Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:53:43.263170007Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:53:43.563278425Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:53:43.693632460Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:53:43.884174187Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:53:44.274070993Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:53:45.194701088Z Wrote inspect data to must-gather.
2026-07-09T18:53:45.194744178Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:53:45.194744178Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:53:45.317404878Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:53:45.440616914Z Caches written to disk