2026-07-10T19:26:08.761796750Z [disk usage checker] Started
2026-07-10T19:26:08.765282861Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:09.073387732Z Gathering data for ns/openshift-cluster-version...
2026-07-10T19:26:09.220353852Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-10T19:26:09.304763677Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-10T19:26:09.388697518Z Gathering data for ns/default...
2026-07-10T19:26:09.688006488Z Gathering data for ns/openshift...
2026-07-10T19:26:10.075111788Z Gathering data for ns/kube-system...
2026-07-10T19:26:10.772450440Z Gathering data for ns/openshift-etcd...
2026-07-10T19:26:11.069571237Z Wrote inspect data to must-gather.
2026-07-10T19:26:11.069667015Z error: inspection completed with the errors occurred while gathering data:
2026-07-10T19:26:11.069667015Z     namespaces "assisted-installer" not found
2026-07-10T19:26:11.623330134Z Waiting on subprocesses to finish execution.
2026-07-10T19:26:11.632707054Z INFO: Collecting host service logs for crio
2026-07-10T19:26:11.633023957Z INFO: Collecting host service logs for kubelet
2026-07-10T19:26:11.633341326Z INFO: Collecting host service logs for rpm-ostreed
2026-07-10T19:26:11.633646719Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-10T19:26:11.633972122Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-10T19:26:11.634284148Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-10T19:26:11.634607070Z INFO: Collecting host service logs for NetworkManager
2026-07-10T19:26:11.634624297Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-10T19:26:11.634963521Z INFO: Collecting host service logs for openvswitch
2026-07-10T19:26:11.635270570Z INFO: Collecting host service logs for ovs-configuration
2026-07-10T19:26:11.635595804Z INFO: Collecting host service logs for ovsdb-server
2026-07-10T19:26:11.635965688Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-10T19:26:11.636400165Z INFO: Waiting for worker host service log collection to complete ...
2026-07-10T19:26:11.641691482Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-10T19:26:11.652948033Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-10T19:26:11.653900629Z Executing Istio gather script
2026-07-10T19:26:11.654020294Z INFO: Gathering HAProxy config files
2026-07-10T19:26:11.699693674Z INFO: Waiting for node performance related collection to complete ...
2026-07-10T19:26:12.082780920Z No resources found in openshift-etcd namespace.
2026-07-10T19:26:12.263688983Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-gl882
2026-07-10T19:26:12.263737049Z insights-runtime-extractor-m44mk
2026-07-10T19:26:12.263757099Z insights-runtime-extractor-tksk4
2026-07-10T19:26:12.263775108Z insights-runtime-extractor-w8d7k
2026-07-10T19:26:12.352453184Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-10T19:26:12.547332606Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-10T19:26:12.635971667Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-10T19:26:12.941408716Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-10T19:26:13.069601123Z No resources found
2026-07-10T19:26:13.114168241Z Wrote inspect data to must-gather.
2026-07-10T19:26:13.142726037Z INFO: "metallb-operator" not detected. Skipping.
2026-07-10T19:26:13.169177923Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-10T19:26:13.185376136Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-10T19:26:13.185480893Z INFO: on-disk MachineConfig config collection complete.
2026-07-10T19:26:13.240875971Z INFO: Worker host service log collection to complete.
2026-07-10T19:26:13.265366808Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-10T19:26:13.265366808Z See 'oc rsync -h' for help and examples
2026-07-10T19:26:13.538674691Z Wrote inspect data to must-gather.
2026-07-10T19:26:13.678402263Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-10T19:26:13.679612736Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-10T19:26:13.694223426Z error: the server doesn't have a resource type "performanceprofile"
2026-07-10T19:26:13.704416280Z No resources found
2026-07-10T19:26:13.779934884Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-10T19:26:13.921975626Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:14.031786961Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.127814469Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.145026823Z ERROR: No running kube-apiserver pods found
2026-07-10T19:26:14.159140331Z INFO: 'previous-logs' folder not found on ip-10-0-130-33.ec2.internal, skipping...
2026-07-10T19:26:14.242731200Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.285967539Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.346116877Z tar: Removing leading `/' from member names
2026-07-10T19:26:14.355673195Z INFO: HAProxy config collection complete.
2026-07-10T19:26:14.417773629Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.455204810Z INFO: Getting rules from prometheus-k8s-0
2026-07-10T19:26:14.567782805Z INFO: 'previous-logs' folder not found on ip-10-0-136-15.ec2.internal, skipping...
2026-07-10T19:26:14.583812381Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.662392078Z error: the server doesn't have a resource type "machineconfigs"
2026-07-10T19:26:14.777757749Z Gathering data for ns/openshift-monitoring...
2026-07-10T19:26:14.881566646Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-10T19:26:14.901104090Z INFO: 'previous-logs' folder not found on ip-10-0-143-15.ec2.internal, skipping...
2026-07-10T19:26:14.901204885Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-10T19:26:14.901253248Z INFO: Machine Config Daemon termination log collection complete.
2026-07-10T19:26:14.936360602Z Wrote inspect data to must-gather.
2026-07-10T19:26:14.961821888Z INFO: Getting status/config from prometheus-k8s-0
2026-07-10T19:26:15.144627477Z Wrote inspect data to must-gather.
2026-07-10T19:26:15.236258816Z Wrote inspect data to must-gather.
2026-07-10T19:26:15.320637029Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-10T19:26:15.341026229Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-10T19:26:15.430467465Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-10T19:26:15.577720588Z INFO: INTERCONNECT MODE
2026-07-10T19:26:15.577747953Z INFO: Gathering ovn-kubernetes DBs
2026-07-10T19:26:15.584791711Z Wrote inspect data to must-gather.
2026-07-10T19:26:15.674255432Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-10T19:26:15.745147665Z INFO: Gathering OVN_Northbound from ovnkube-node-bdfcs...
2026-07-10T19:26:15.745442403Z INFO: Gathering OVN_Northbound from ovnkube-node-j58jz...
2026-07-10T19:26:15.745694402Z INFO: Gathering OVN_Northbound from ovnkube-node-r5f8x...
2026-07-10T19:26:15.746103990Z INFO: Gathering OVN_Southbound from ovnkube-node-bdfcs...
2026-07-10T19:26:15.746391648Z INFO: Gathering OVN_Southbound from ovnkube-node-j58jz...
2026-07-10T19:26:15.746661994Z INFO: Gathering OVN_Southbound from ovnkube-node-r5f8x...
2026-07-10T19:26:15.843352747Z Wrote inspect data to must-gather.
2026-07-10T19:26:16.016010143Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.019600452Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-10T19:26:16.056024381Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.078448012Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.081902040Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.090258195Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.100351952Z tar: Removing leading `/' from member names
2026-07-10T19:26:16.215647458Z Wrote inspect data to must-gather.
2026-07-10T19:26:16.285045499Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-10T19:26:16.331555443Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-10T19:26:16.355838561Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-10T19:26:16.488770244Z Wrote inspect data to must-gather.
2026-07-10T19:26:16.599555046Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:16.666441691Z INFO: Waiting for network log collection to complete ...
2026-07-10T19:26:16.666479957Z INFO: Waiting for ovnk database copies to complete ...
2026-07-10T19:26:16.666990017Z INFO: Copying ovnk databases complete.
2026-07-10T19:26:16.668959670Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-10T19:26:16.671203260Z ovnk_database_store/
2026-07-10T19:26:16.671278222Z ovnk_database_store/ovnkube-node-r5f8x_nbdb
2026-07-10T19:26:16.680537544Z ovnk_database_store/ovnkube-node-r5f8x_sbdb
2026-07-10T19:26:16.708033667Z ovnk_database_store/ovnkube-node-bdfcs_sbdb
2026-07-10T19:26:16.729321457Z ovnk_database_store/ovnkube-node-bdfcs_nbdb
2026-07-10T19:26:16.734176172Z Wrote inspect data to must-gather.
2026-07-10T19:26:16.741482990Z ovnk_database_store/ovnkube-node-j58jz_nbdb
2026-07-10T19:26:16.749363904Z ovnk_database_store/ovnkube-node-j58jz_sbdb
2026-07-10T19:26:16.778734240Z INFO: Network log collection complete.
2026-07-10T19:26:16.803212344Z INFO: Getting status from alertmanager-main-0
2026-07-10T19:26:16.850982703Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:16.856322991Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-10T19:26:16.976231996Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-10T19:26:17.073880457Z Wrote inspect data to must-gather.
2026-07-10T19:26:17.076233599Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:17.165073507Z Gathering data for ns/openshift-network-console...
2026-07-10T19:26:17.300423373Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:17.304552023Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-10T19:26:17.399861323Z Wrote inspect data to must-gather.
2026-07-10T19:26:17.529232279Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:17.683384731Z Gathering data for ns/openshift-console-operator...
2026-07-10T19:26:17.732440063Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-10T19:26:17.735397339Z INFO: Fallback to identify the container image from release info
2026-07-10T19:26:17.769827379Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:17.774100489Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-10T19:26:17.999854198Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:18.079364419Z Gathering data for ns/openshift-console...
2026-07-10T19:26:18.207640350Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:18.211779086Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-10T19:26:18.347351389Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-10T19:26:18.380471400Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:18.494655080Z daemonset.apps/perf-node-gather-daemonset created
2026-07-10T19:26:18.613194426Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:18.617979120Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-10T19:26:18.640271752Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-10T19:26:18.658003483Z Waiting for performance profile collector pods to become ready: 1
2026-07-10T19:26:18.792482100Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:18.931965499Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:19.021051946Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:19.026144696Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-10T19:26:19.197017855Z Gathering data for ns/openshift-dns-operator...
2026-07-10T19:26:19.220291324Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:19.428109553Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:19.431789268Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-10T19:26:19.500075230Z Gathering data for ns/openshift-dns...
2026-07-10T19:26:19.616448566Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:19.798732868Z Waiting for performance profile collector pods to become ready: 2
2026-07-10T19:26:19.828929945Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:19.833032164Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-10T19:26:19.989010508Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:20.157478847Z Gathering data for ns/openshift-image-registry...
2026-07-10T19:26:20.188065676Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:20.191484005Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-10T19:26:20.364402256Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:20.561094460Z Wrote inspect data to /must-gather/istio.
2026-07-10T19:26:20.703337891Z Gathering data for ns/openshift-ingress-operator...
2026-07-10T19:26:20.937839732Z Waiting for performance profile collector pods to become ready: 3
2026-07-10T19:26:21.087064743Z Gathering data for ns/openshift-ingress...
2026-07-10T19:26:21.288087470Z Done executing Istio gather script
2026-07-10T19:26:21.422279645Z Gathering data for ns/openshift-ingress-canary...
2026-07-10T19:26:21.789260064Z Gathering data for ns/openshift-insights...
2026-07-10T19:26:22.067987745Z Waiting for performance profile collector pods to become ready: 4
2026-07-10T19:26:23.195877358Z Waiting for performance profile collector pods to become ready: 5
2026-07-10T19:26:23.936710000Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:24.015040526Z Gathering data for ns/kserve...
2026-07-10T19:26:24.311114158Z Waiting for performance profile collector pods to become ready: 6
2026-07-10T19:26:25.435830441Z Waiting for performance profile collector pods to become ready: 7
2026-07-10T19:26:25.626052864Z Gathering data for ns/openshift-config...
2026-07-10T19:26:25.883648181Z Gathering data for ns/openshift-config-managed...
2026-07-10T19:26:26.297593695Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-10T19:26:26.570590306Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-10T19:26:26.623096539Z Gathering data for ns/openshift-kube-apiserver...
2026-07-10T19:26:26.849041345Z Collecting performance related data for node ip-10-0-143-15.ec2.internal
2026-07-10T19:26:26.858576061Z Collecting performance related data for node ip-10-0-136-15.ec2.internal
2026-07-10T19:26:26.864897173Z Collecting performance related data for node ip-10-0-130-33.ec2.internal
2026-07-10T19:26:26.932804808Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-10T19:26:27.290281928Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-10T19:26:27.661299309Z Gathering data for ns/openshift-kube-scheduler...
2026-07-10T19:26:27.998668598Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-10T19:26:28.357040009Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-10T19:26:28.719336656Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-10T19:26:28.941434652Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:29.032500044Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-10T19:26:29.570789974Z Gathering data for ns/openshift-multus...
2026-07-10T19:26:30.909669159Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-10T19:26:32.156934925Z Gathering data for ns/openshift-host-network...
2026-07-10T19:26:32.505545812Z Gathering data for ns/openshift-network-diagnostics...
2026-07-10T19:26:33.020788100Z Gathering data for ns/openshift-network-node-identity...
2026-07-10T19:26:33.405408279Z Gathering data for ns/openshift-network-operator...
2026-07-10T19:26:33.766305859Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-10T19:26:33.946059126Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-10T19:26:34.039157527Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-10T19:26:34.510275785Z Gathering data for ns/openshift-apiserver-operator...
2026-07-10T19:26:34.802346887Z Gathering data for ns/openshift-apiserver...
2026-07-10T19:26:34.825197580Z Collecting kubelet logs for node ip-10-0-143-15.ec2.internal
2026-07-10T19:26:34.828011140Z Collecting kubelet logs for node ip-10-0-136-15.ec2.internal
2026-07-10T19:26:34.830050881Z Collecting kubelet logs for node ip-10-0-130-33.ec2.internal
2026-07-10T19:26:35.255547980Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-znhzn namespace
2026-07-10T19:26:35.264822455Z INFO: Node performance data collection complete.
2026-07-10T19:26:35.302554508Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-10T19:26:35.589143805Z Gathering data for ns/openshift-controller-manager...
2026-07-10T19:26:35.859980331Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-10T19:26:36.481592260Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-10T19:26:36.749008396Z Gathering data for ns/openshift-service-ca-operator...
2026-07-10T19:26:37.066586126Z Gathering data for ns/openshift-service-ca...
2026-07-10T19:26:37.420274856Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-10T19:26:38.315427162Z Wrote inspect data to must-gather.
2026-07-10T19:26:38.315474068Z error: inspection completed with the errors occurred while gathering data:
2026-07-10T19:26:38.315474068Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-10T19:26:38.444223951Z error: the server doesn't have a resource type "clusters"
2026-07-10T19:26:38.588821833Z Caches written to disk