2026-07-31T16:54:26.767460690Z [disk usage checker] Started 2026-07-31T16:54:26.771213974Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:26.962123115Z Gathering data for ns/openshift-cluster-version... 2026-07-31T16:54:27.085394489Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-31T16:54:27.163757954Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-31T16:54:27.235403289Z Gathering data for ns/default... 2026-07-31T16:54:27.504389576Z Gathering data for ns/openshift... 2026-07-31T16:54:27.936088215Z Gathering data for ns/kube-system... 2026-07-31T16:54:28.604766534Z Gathering data for ns/openshift-etcd... 2026-07-31T16:54:28.880722759Z Wrote inspect data to must-gather. 2026-07-31T16:54:28.880749403Z error: inspection completed with the errors occurred while gathering data: 2026-07-31T16:54:28.880749403Z namespaces "assisted-installer" not found 2026-07-31T16:54:29.232494680Z Waiting on subprocesses to finish execution. 2026-07-31T16:54:29.237867541Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-31T16:54:29.241358946Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-31T16:54:29.245869867Z INFO: Gathering HAProxy config files 2026-07-31T16:54:29.249675721Z Executing Istio gather script 2026-07-31T16:54:29.263992384Z INFO: Collecting host service logs for crio 2026-07-31T16:54:29.264327473Z INFO: Collecting host service logs for kubelet 2026-07-31T16:54:29.264653924Z INFO: Collecting host service logs for rpm-ostreed 2026-07-31T16:54:29.264959213Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-31T16:54:29.265324226Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-31T16:54:29.265641326Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-31T16:54:29.265988417Z INFO: Collecting host service logs for NetworkManager 2026-07-31T16:54:29.266326866Z INFO: Collecting host service logs for openvswitch 2026-07-31T16:54:29.266646694Z INFO: Collecting host service logs for ovs-configuration 2026-07-31T16:54:29.267007590Z INFO: Collecting host service logs for ovsdb-server 2026-07-31T16:54:29.267387200Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-31T16:54:29.267779870Z INFO: Waiting for worker host service log collection to complete ... 2026-07-31T16:54:29.277130422Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-31T16:54:29.527428549Z INFO: Waiting for node performance related collection to complete ... 2026-07-31T16:54:29.592818517Z INFO: Collecting Insights Archives from insights-operator-77696685b-84jgp 2026-07-31T16:54:29.592818517Z insights-runtime-extractor-dcnrq 2026-07-31T16:54:29.592818517Z insights-runtime-extractor-kx8kc 2026-07-31T16:54:29.592818517Z insights-runtime-extractor-sjsrs 2026-07-31T16:54:29.659426034Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-31T16:54:29.659508487Z INFO: on-disk MachineConfig config collection complete. 2026-07-31T16:54:30.070408794Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-31T16:54:30.222108320Z Wrote inspect data to must-gather. 2026-07-31T16:54:30.278793845Z error: the server doesn't have a resource type "performanceprofile" 2026-07-31T16:54:30.309939422Z No resources found in openshift-etcd namespace. 2026-07-31T16:54:30.381013020Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-31T16:54:30.391148441Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-31T16:54:30.405191014Z INFO: "metallb-operator" not detected. Skipping. 2026-07-31T16:54:30.425769049Z No resources found 2026-07-31T16:54:30.504798966Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-31T16:54:30.504798966Z See 'oc rsync -h' for help and examples 2026-07-31T16:54:30.554945227Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-31T16:54:30.575365787Z INFO: Worker host service log collection to complete. 2026-07-31T16:54:30.993630118Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-31T16:54:31.221433424Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-31T16:54:31.357304201Z No resources found 2026-07-31T16:54:31.505337100Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.512546379Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.517890872Z ERROR: No running kube-apiserver pods found 2026-07-31T16:54:31.526315425Z Gathering data for ns/openshift-monitoring... 2026-07-31T16:54:31.592319007Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-31T16:54:31.631374743Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.683603341Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.684847970Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.748936903Z INFO: 'previous-logs' folder not found on ip-10-0-133-33.ec2.internal, skipping... 2026-07-31T16:54:31.766134800Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-31T16:54:31.768961010Z Wrote inspect data to must-gather. 2026-07-31T16:54:31.778861261Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:31.840086693Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-31T16:54:31.874485209Z error: the server doesn't have a resource type "machineconfigs" 2026-07-31T16:54:32.012101601Z tar: Removing leading `/' from member names 2026-07-31T16:54:32.022039753Z INFO: HAProxy config collection complete. 2026-07-31T16:54:32.034624314Z INFO: 'previous-logs' folder not found on ip-10-0-139-78.ec2.internal, skipping... 2026-07-31T16:54:32.049999515Z Wrote inspect data to must-gather. 2026-07-31T16:54:32.058892293Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-31T16:54:32.059875827Z INFO: Getting rules from prometheus-k8s-0 2026-07-31T16:54:32.295515791Z Wrote inspect data to must-gather. 2026-07-31T16:54:32.296931427Z Wrote inspect data to must-gather. 2026-07-31T16:54:32.316424159Z INFO: 'previous-logs' folder not found on ip-10-0-140-146.ec2.internal, skipping... 2026-07-31T16:54:32.316518348Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-31T16:54:32.316571634Z INFO: Machine Config Daemon termination log collection complete. 2026-07-31T16:54:32.456312216Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-31T16:54:32.471157119Z INFO: Getting status/config from prometheus-k8s-0 2026-07-31T16:54:32.552956232Z Wrote inspect data to must-gather. 2026-07-31T16:54:32.643049966Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-31T16:54:32.664023828Z Wrote inspect data to must-gather. 2026-07-31T16:54:32.829826565Z INFO: INTERCONNECT MODE 2026-07-31T16:54:32.829914458Z INFO: Gathering ovn-kubernetes DBs 2026-07-31T16:54:32.838100069Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:32.855820250Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-31T16:54:32.880115433Z Wrote inspect data to must-gather. 2026-07-31T16:54:33.004501076Z INFO: Gathering OVN_Northbound from ovnkube-node-5ql44... 2026-07-31T16:54:33.004807506Z INFO: Gathering OVN_Northbound from ovnkube-node-8dlk2... 2026-07-31T16:54:33.005089741Z INFO: Gathering OVN_Northbound from ovnkube-node-phngx... 2026-07-31T16:54:33.006578565Z INFO: Gathering OVN_Southbound from ovnkube-node-5ql44... 2026-07-31T16:54:33.006872762Z INFO: Gathering OVN_Southbound from ovnkube-node-8dlk2... 2026-07-31T16:54:33.007545873Z INFO: Gathering OVN_Southbound from ovnkube-node-phngx... 2026-07-31T16:54:33.081147080Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:33.086576879Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-31T16:54:33.101088665Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-31T16:54:33.180458760Z Wrote inspect data to must-gather. 2026-07-31T16:54:33.295210393Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-31T16:54:33.300782268Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.325213362Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.334868982Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.351224151Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.371136637Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.371674436Z tar: Removing leading `/' from member names 2026-07-31T16:54:33.406758348Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:33.449471089Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-31T16:54:33.475958333Z Wrote inspect data to must-gather. 2026-07-31T16:54:33.652075372Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:33.655934757Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-31T16:54:33.715415999Z Wrote inspect data to must-gather. 2026-07-31T16:54:33.728203968Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-31T16:54:33.850691313Z INFO: Waiting for network log collection to complete ... 2026-07-31T16:54:33.850803533Z INFO: Waiting for ovnk database copies to complete ... 2026-07-31T16:54:33.851220421Z INFO: Copying ovnk databases complete. 2026-07-31T16:54:33.855503537Z 2.5M must-gather/network_logs/ovnk_database_store 2026-07-31T16:54:33.859889552Z ovnk_database_store/ 2026-07-31T16:54:33.860101092Z ovnk_database_store/ovnkube-node-8dlk2_nbdb 2026-07-31T16:54:33.869022068Z ovnk_database_store/ovnkube-node-8dlk2_sbdb 2026-07-31T16:54:33.872571585Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:33.888611243Z ovnk_database_store/ovnkube-node-5ql44_nbdb 2026-07-31T16:54:33.894464717Z ovnk_database_store/ovnkube-node-5ql44_sbdb 2026-07-31T16:54:33.910816824Z ovnk_database_store/ovnkube-node-phngx_nbdb 2026-07-31T16:54:33.917134873Z ovnk_database_store/ovnkube-node-phngx_sbdb 2026-07-31T16:54:33.931429010Z W0731 16:54:33.931398 1520 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-31T16:54:33.936042305Z Wrote inspect data to must-gather. 2026-07-31T16:54:33.938389627Z INFO: Network log collection complete. 2026-07-31T16:54:33.992569964Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-31T16:54:34.080182610Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:34.083386506Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-31T16:54:34.129678665Z INFO: Getting status from alertmanager-main-0 2026-07-31T16:54:34.189249215Z Gathering data for ns/openshift-network-console... 2026-07-31T16:54:34.245865264Z Wrote inspect data to must-gather. 2026-07-31T16:54:34.314984696Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:34.493304088Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-31T16:54:34.496398478Z INFO: Fallback to identify the container image from release info 2026-07-31T16:54:34.515082949Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:34.517649122Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-31T16:54:34.634093839Z Gathering data for ns/openshift-console-operator... 2026-07-31T16:54:34.674834505Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:34.848593581Z Wrote inspect data to /must-gather/istio. 2026-07-31T16:54:35.005664620Z Gathering data for ns/openshift-console... 2026-07-31T16:54:35.040885578Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675 2026-07-31T16:54:35.172615673Z daemonset.apps/perf-node-gather-daemonset created 2026-07-31T16:54:35.295426330Z Waiting for performance profile collector pods to become ready: 1 2026-07-31T16:54:35.509140175Z Done executing Istio gather script 2026-07-31T16:54:35.513014957Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-31T16:54:35.987951619Z Gathering data for ns/openshift-dns-operator... 2026-07-31T16:54:36.235883708Z Gathering data for ns/openshift-dns... 2026-07-31T16:54:36.390736783Z Waiting for performance profile collector pods to become ready: 2 2026-07-31T16:54:36.785307320Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:36.801324512Z Gathering data for ns/openshift-image-registry... 2026-07-31T16:54:37.218695540Z Gathering data for ns/openshift-ingress-operator... 2026-07-31T16:54:37.508706315Z Waiting for performance profile collector pods to become ready: 3 2026-07-31T16:54:37.518075871Z Gathering data for ns/openshift-ingress... 2026-07-31T16:54:37.823360106Z Gathering data for ns/openshift-ingress-canary... 2026-07-31T16:54:38.180614998Z Gathering data for ns/openshift-insights... 2026-07-31T16:54:38.608994997Z Waiting for performance profile collector pods to become ready: 4 2026-07-31T16:54:39.712878967Z Waiting for performance profile collector pods to become ready: 5 2026-07-31T16:54:39.963961607Z Gathering data for ns/openshift-config... 2026-07-31T16:54:40.197533722Z Gathering data for ns/openshift-config-managed... 2026-07-31T16:54:40.570148157Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-31T16:54:40.829249578Z Waiting for performance profile collector pods to become ready: 6 2026-07-31T16:54:40.829901580Z Gathering data for ns/openshift-kube-apiserver... 2026-07-31T16:54:41.069083460Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-31T16:54:41.329052552Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-31T16:54:41.585048751Z Gathering data for ns/openshift-kube-scheduler... 2026-07-31T16:54:41.791136126Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:41.816993460Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-31T16:54:41.940004045Z Waiting for performance profile collector pods to become ready: 7 2026-07-31T16:54:42.110032465Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-31T16:54:42.401558627Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-31T16:54:42.676872558Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-31T16:54:43.062671691Z Waiting for performance profile collector pods to become ready: 8 2026-07-31T16:54:43.155581821Z Gathering data for ns/openshift-multus... 2026-07-31T16:54:44.177071566Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-31T16:54:44.398166810Z Collecting performance related data for node ip-10-0-133-33.ec2.internal 2026-07-31T16:54:44.405011202Z Collecting performance related data for node ip-10-0-140-146.ec2.internal 2026-07-31T16:54:44.412551616Z Collecting performance related data for node ip-10-0-139-78.ec2.internal 2026-07-31T16:54:44.898134250Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-31T16:54:46.323501811Z Gathering data for ns/openshift-host-network... 2026-07-31T16:54:46.743038521Z Gathering data for ns/openshift-network-diagnostics... 2026-07-31T16:54:46.795728486Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:47.478719960Z Gathering data for ns/openshift-network-node-identity... 2026-07-31T16:54:47.936095884Z Gathering data for ns/openshift-network-operator... 2026-07-31T16:54:48.385078669Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-31T16:54:48.726100123Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-31T16:54:49.294304296Z Gathering data for ns/openshift-apiserver-operator... 2026-07-31T16:54:49.614711883Z Gathering data for ns/openshift-apiserver... 2026-07-31T16:54:50.046121394Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-31T16:54:50.337166886Z Gathering data for ns/openshift-controller-manager... 2026-07-31T16:54:50.636679537Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-31T16:54:51.385583209Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-31T16:54:51.709417840Z Gathering data for ns/openshift-service-ca-operator... 2026-07-31T16:54:51.801476780Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-31T16:54:52.038388269Z Gathering data for ns/openshift-service-ca... 2026-07-31T16:54:52.166217999Z Collecting kubelet logs for node ip-10-0-133-33.ec2.internal 2026-07-31T16:54:52.167960520Z Collecting kubelet logs for node ip-10-0-140-146.ec2.internal 2026-07-31T16:54:52.169847605Z Collecting kubelet logs for node ip-10-0-139-78.ec2.internal 2026-07-31T16:54:52.489698328Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-31T16:54:52.503742098Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-l4cln namespace 2026-07-31T16:54:52.508736157Z INFO: Node performance data collection complete. 2026-07-31T16:54:53.324884263Z Wrote inspect data to must-gather. 2026-07-31T16:54:53.324928749Z error: inspection completed with the errors occurred while gathering data: 2026-07-31T16:54:53.324928749Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-31T16:54:53.440302048Z error: the server doesn't have a resource type "clusters" 2026-07-31T16:54:53.571047854Z Caches written to disk