2026-07-02T14:33:45.839545038Z [disk usage checker] Started
2026-07-02T14:33:45.843214119Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:33:46.049060420Z Gathering data for ns/openshift-cluster-version...
2026-07-02T14:33:46.183980450Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:33:46.283085417Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:33:46.367626245Z Gathering data for ns/default...
2026-07-02T14:33:46.706255515Z Gathering data for ns/openshift...
2026-07-02T14:33:47.195462444Z Gathering data for ns/kube-system...
2026-07-02T14:33:48.020185688Z Gathering data for ns/openshift-etcd...
2026-07-02T14:33:48.350797444Z Wrote inspect data to must-gather.
2026-07-02T14:33:48.350845929Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:33:48.350845929Z     namespaces "assisted-installer" not found
2026-07-02T14:33:48.728937590Z Waiting on subprocesses to finish execution.
2026-07-02T14:33:48.738617211Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T14:33:48.743342203Z Executing Istio gather script
2026-07-02T14:33:48.743939577Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T14:33:48.750735011Z INFO: Gathering HAProxy config files
2026-07-02T14:33:48.754141346Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T14:33:48.754173472Z INFO: Collecting host service logs for crio
2026-07-02T14:33:48.754474714Z INFO: Collecting host service logs for kubelet
2026-07-02T14:33:48.754795994Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T14:33:48.755101974Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T14:33:48.755378482Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T14:33:48.755695302Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T14:33:48.756029517Z INFO: Collecting host service logs for NetworkManager
2026-07-02T14:33:48.756329197Z INFO: Collecting host service logs for openvswitch
2026-07-02T14:33:48.756628581Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T14:33:48.756983810Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T14:33:48.757304285Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T14:33:48.757701434Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T14:33:48.798591801Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T14:33:49.350864683Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T14:33:49.350908532Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T14:33:49.410921536Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-s6bjn
2026-07-02T14:33:49.410972819Z insights-runtime-extractor-lh2qk
2026-07-02T14:33:49.410992690Z insights-runtime-extractor-n8mlp
2026-07-02T14:33:49.411012299Z insights-runtime-extractor-nv5c5
2026-07-02T14:33:49.648743030Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T14:33:49.697558908Z No resources found
2026-07-02T14:33:49.886786341Z No resources found in openshift-etcd namespace.
2026-07-02T14:33:49.894302041Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T14:33:50.058385037Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T14:33:50.186788830Z INFO: Worker host service log collection to complete.
2026-07-02T14:33:50.208123276Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T14:33:50.318590990Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T14:33:50.333742596Z Wrote inspect data to must-gather.
2026-07-02T14:33:50.415758720Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T14:33:50.640135408Z Wrote inspect data to must-gather.
2026-07-02T14:33:50.791371057Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:33:50.810579296Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T14:33:50.856029892Z ERROR: No running kube-apiserver pods found
2026-07-02T14:33:50.880487232Z Wrote inspect data to must-gather.
2026-07-02T14:33:50.937801674Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:33:50.960372820Z Gathering data for ns/openshift-monitoring...
2026-07-02T14:33:51.100064106Z No resources found
2026-07-02T14:33:51.197739275Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T14:33:51.284891393Z INFO: 'previous-logs' folder not found on ip-10-0-133-209.ec2.internal, skipping...
2026-07-02T14:33:51.355259524Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T14:33:51.401131304Z Wrote inspect data to must-gather.
2026-07-02T14:33:51.403263018Z Wrote inspect data to must-gather.
2026-07-02T14:33:51.436982783Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T14:33:51.436982783Z See 'oc rsync -h' for help and examples
2026-07-02T14:33:51.500792245Z Wrote inspect data to must-gather.
2026-07-02T14:33:51.508875256Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T14:33:51.550309641Z Wrote inspect data to must-gather.
2026-07-02T14:33:51.738483256Z tar: Removing leading `/' from member names
2026-07-02T14:33:51.745161296Z INFO: HAProxy config collection complete.
2026-07-02T14:33:51.750511587Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T14:33:51.768675574Z INFO: 'previous-logs' folder not found on ip-10-0-137-40.ec2.internal, skipping...
2026-07-02T14:33:51.772895888Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T14:33:51.809473979Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.018686077Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.091292066Z INFO: 'previous-logs' folder not found on ip-10-0-138-214.ec2.internal, skipping...
2026-07-02T14:33:52.091390161Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T14:33:52.091408015Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T14:33:52.108056981Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.239163632Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T14:33:52.248957919Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T14:33:52.306423589Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T14:33:52.392422832Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.476886644Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.694740812Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T14:33:52.710403958Z INFO: INTERCONNECT MODE
2026-07-02T14:33:52.710438183Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T14:33:52.722592427Z Wrote inspect data to must-gather.
2026-07-02T14:33:52.931955497Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:33:52.952545815Z INFO: Gathering OVN_Northbound from ovnkube-node-8f2ll...
2026-07-02T14:33:52.952777781Z INFO: Gathering OVN_Northbound from ovnkube-node-lmvr2...
2026-07-02T14:33:52.953052014Z INFO: Gathering OVN_Northbound from ovnkube-node-x9gx2...
2026-07-02T14:33:52.953339264Z INFO: Gathering OVN_Southbound from ovnkube-node-8f2ll...
2026-07-02T14:33:52.953728998Z INFO: Gathering OVN_Southbound from ovnkube-node-lmvr2...
2026-07-02T14:33:52.954104308Z INFO: Gathering OVN_Southbound from ovnkube-node-x9gx2...
2026-07-02T14:33:52.975440073Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T14:33:52.979054054Z Wrote inspect data to must-gather.
2026-07-02T14:33:53.262179208Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.279784402Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.326958754Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.342721265Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.373392171Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.380058219Z Wrote inspect data to must-gather.
2026-07-02T14:33:53.404274350Z tar: Removing leading `/' from member names
2026-07-02T14:33:53.444023003Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T14:33:53.476591754Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T14:33:53.648809200Z W0702 14:33:53.648762    1406 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T14:33:53.656517155Z Wrote inspect data to must-gather.
2026-07-02T14:33:53.715372524Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:33:53.720217737Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:53.734018918Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T14:33:53.953522944Z INFO: Waiting for network log collection to complete ...
2026-07-02T14:33:53.953578327Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T14:33:53.954083041Z INFO: Copying ovnk databases complete.
2026-07-02T14:33:53.955776008Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-02T14:33:53.957609639Z Gathering data for ns/openshift-network-console...
2026-07-02T14:33:53.958052919Z ovnk_database_store/
2026-07-02T14:33:53.958067757Z ovnk_database_store/ovnkube-node-8f2ll_nbdb
2026-07-02T14:33:53.968481708Z ovnk_database_store/ovnkube-node-lmvr2_sbdb
2026-07-02T14:33:53.991351635Z ovnk_database_store/ovnkube-node-lmvr2_nbdb
2026-07-02T14:33:54.001051200Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:54.004381121Z ovnk_database_store/ovnkube-node-x9gx2_nbdb
2026-07-02T14:33:54.004381121Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T14:33:54.014688658Z ovnk_database_store/ovnkube-node-x9gx2_sbdb
2026-07-02T14:33:54.043175568Z ovnk_database_store/ovnkube-node-8f2ll_sbdb
2026-07-02T14:33:54.067392035Z Wrote inspect data to must-gather.
2026-07-02T14:33:54.071301878Z INFO: Network log collection complete.
2026-07-02T14:33:54.188346638Z INFO: Getting status from alertmanager-main-0
2026-07-02T14:33:54.274679604Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:54.388784319Z Wrote inspect data to must-gather.
2026-07-02T14:33:54.500178286Z Gathering data for ns/openshift-console-operator...
2026-07-02T14:33:54.514060595Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:54.516913442Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T14:33:54.692221322Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T14:33:54.695029285Z INFO: Fallback to identify the container image from release info
2026-07-02T14:33:54.729403480Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:54.938986378Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:54.939247866Z Gathering data for ns/openshift-console...
2026-07-02T14:33:54.941553359Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T14:33:55.159174439Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:55.347051563Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:55.350408549Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T14:33:55.541109627Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T14:33:55.555465679Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:55.696447850Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-02T14:33:55.777735297Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:55.782600577Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T14:33:55.840683998Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T14:33:55.958531846Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:33:55.987668668Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:56.008711085Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T14:33:56.178030411Z Gathering data for ns/openshift-dns-operator...
2026-07-02T14:33:56.207084710Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:56.210308835Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T14:33:56.412996772Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:56.550081768Z Gathering data for ns/openshift-dns...
2026-07-02T14:33:56.631776099Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:56.634091864Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T14:33:56.799723026Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:56.998384797Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:57.000585386Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T14:33:57.140326860Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T14:33:57.179058041Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:57.388032437Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:57.390384468Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T14:33:57.419034304Z Gathering data for ns/openshift-image-registry...
2026-07-02T14:33:57.562427565Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:57.765435456Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:33:58.075075096Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T14:33:58.275143986Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T14:33:58.479005158Z Gathering data for ns/openshift-ingress...
2026-07-02T14:33:58.518470397Z Done executing Istio gather script
2026-07-02T14:33:58.836932532Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T14:33:59.262443787Z Gathering data for ns/openshift-insights...
2026-07-02T14:33:59.388642296Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T14:34:00.504964361Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T14:34:00.963959190Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:34:01.547711592Z Gathering data for ns/kserve...
2026-07-02T14:34:01.626982459Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T14:34:02.748810462Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T14:34:03.085267781Z Gathering data for ns/openshift-config...
2026-07-02T14:34:03.400284312Z Gathering data for ns/openshift-config-managed...
2026-07-02T14:34:03.897775232Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T14:34:03.903503908Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T14:34:04.183119693Z Collecting performance related data for node ip-10-0-138-214.ec2.internal
2026-07-02T14:34:04.196623912Z Collecting performance related data for node ip-10-0-137-40.ec2.internal
2026-07-02T14:34:04.201378761Z Collecting performance related data for node ip-10-0-133-209.ec2.internal
2026-07-02T14:34:04.251517480Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T14:34:04.611726919Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T14:34:04.992574974Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T14:34:05.373042798Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T14:34:05.697656424Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T14:34:05.969375766Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:34:06.075857421Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T14:34:06.404419860Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T14:34:06.773757753Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T14:34:07.342590501Z Gathering data for ns/openshift-multus...
2026-07-02T14:34:08.760544990Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T14:34:10.091761393Z Gathering data for ns/openshift-host-network...
2026-07-02T14:34:10.403011810Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T14:34:10.925190766Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T14:34:10.979187445Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:34:11.344427168Z Gathering data for ns/openshift-network-operator...
2026-07-02T14:34:11.744505580Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T14:34:12.017555626Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T14:34:12.520592380Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T14:34:12.559350987Z Collecting kubelet logs for node ip-10-0-138-214.ec2.internal
2026-07-02T14:34:12.562108996Z Collecting kubelet logs for node ip-10-0-137-40.ec2.internal
2026-07-02T14:34:12.564749528Z Collecting kubelet logs for node ip-10-0-133-209.ec2.internal
2026-07-02T14:34:12.912420674Z Gathering data for ns/openshift-apiserver...
2026-07-02T14:34:12.954137985Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-p8hbs namespace
2026-07-02T14:34:12.959508697Z INFO: Node performance data collection complete.
2026-07-02T14:34:13.368367674Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T14:34:13.640274266Z Gathering data for ns/openshift-controller-manager...
2026-07-02T14:34:13.917224665Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T14:34:14.576077096Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T14:34:14.869216920Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T14:34:15.209199467Z Gathering data for ns/openshift-service-ca...
2026-07-02T14:34:15.578377072Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T14:34:15.984364773Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-02T14:34:16.460412640Z Wrote inspect data to must-gather.
2026-07-02T14:34:16.460463083Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:34:16.460463083Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T14:34:16.588400021Z error: the server doesn't have a resource type "clusters"
2026-07-02T14:34:16.744626977Z Caches written to disk