2026-07-28T13:58:42.429424137Z [disk usage checker] Started
2026-07-28T13:58:42.433149769Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:58:42.639776541Z Gathering data for ns/openshift-cluster-version...
2026-07-28T13:58:42.794076344Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T13:58:42.893595905Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:58:42.983306700Z Gathering data for ns/default...
2026-07-28T13:58:43.314462864Z Gathering data for ns/openshift...
2026-07-28T13:58:43.754574725Z Gathering data for ns/kube-system...
2026-07-28T13:58:44.515160211Z Gathering data for ns/openshift-etcd...
2026-07-28T13:58:44.895609484Z Wrote inspect data to must-gather.
2026-07-28T13:58:44.895641440Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T13:58:44.895641440Z     namespaces "assisted-installer" not found
2026-07-28T13:58:45.044769170Z Waiting on subprocesses to finish execution.
2026-07-28T13:58:45.057667666Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T13:58:45.065598626Z Executing Istio gather script
2026-07-28T13:58:45.065983713Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T13:58:45.067122333Z INFO: Gathering HAProxy config files
2026-07-28T13:58:45.068098493Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T13:58:45.092447491Z INFO: Collecting host service logs for crio
2026-07-28T13:58:45.092778993Z INFO: Collecting host service logs for kubelet
2026-07-28T13:58:45.093135148Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T13:58:45.093424191Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T13:58:45.093736259Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T13:58:45.094069048Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T13:58:45.094349477Z INFO: Collecting host service logs for NetworkManager
2026-07-28T13:58:45.094633190Z INFO: Collecting host service logs for openvswitch
2026-07-28T13:58:45.094952127Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T13:58:45.095264095Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T13:58:45.095566468Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T13:58:45.096061053Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T13:58:45.378459177Z No resources found
2026-07-28T13:58:45.551642042Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T13:58:45.639739504Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T13:58:45.923493107Z No resources found in openshift-etcd namespace.
2026-07-28T13:58:45.924073744Z Wrote inspect data to must-gather.
2026-07-28T13:58:46.040006971Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T13:58:46.121991908Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T13:58:46.130094828Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T13:58:46.130133036Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T13:58:46.249303352Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T13:58:46.310404298Z Wrote inspect data to must-gather.
2026-07-28T13:58:46.451411227Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T13:58:46.531139892Z INFO: Worker host service log collection to complete.
2026-07-28T13:58:46.582248175Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-2vl87
2026-07-28T13:58:46.582298472Z insights-runtime-extractor-56xrw
2026-07-28T13:58:46.582318879Z insights-runtime-extractor-qk8tc
2026-07-28T13:58:46.582340218Z insights-runtime-extractor-swfrj
2026-07-28T13:58:46.589396592Z ERROR: No running kube-apiserver pods found
2026-07-28T13:58:46.634427487Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T13:58:46.689165285Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T13:58:46.761138265Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T13:58:46.761138265Z See 'oc rsync -h' for help and examples
2026-07-28T13:58:46.806999204Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T13:58:46.899298287Z Gathering data for ns/openshift-monitoring...
2026-07-28T13:58:47.230329288Z No resources found
2026-07-28T13:58:47.274348194Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:58:47.277548921Z Wrote inspect data to must-gather.
2026-07-28T13:58:47.465774146Z Wrote inspect data to must-gather.
2026-07-28T13:58:47.479271694Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:58:47.582318229Z Wrote inspect data to must-gather.
2026-07-28T13:58:47.621926301Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T13:58:47.637779169Z INFO: 'previous-logs' folder not found on ip-10-0-133-136.ec2.internal, skipping...
2026-07-28T13:58:47.652318408Z tar: Removing leading `/' from member names
2026-07-28T13:58:47.656443863Z INFO: HAProxy config collection complete.
2026-07-28T13:58:47.663006442Z Wrote inspect data to must-gather.
2026-07-28T13:58:47.667474427Z Wrote inspect data to must-gather.
2026-07-28T13:58:47.725430198Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T13:58:47.853105542Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T13:58:47.921150038Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T13:58:47.943936702Z INFO: 'previous-logs' folder not found on ip-10-0-141-63.ec2.internal, skipping...
2026-07-28T13:58:47.963621732Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.020961166Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T13:58:48.161543391Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T13:58:48.229011184Z INFO: 'previous-logs' folder not found on ip-10-0-142-189.ec2.internal, skipping...
2026-07-28T13:58:48.229049954Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T13:58:48.229075623Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T13:58:48.241613898Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.263087601Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.427525588Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T13:58:48.484319972Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.576778411Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T13:58:48.649296165Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.771448682Z INFO: INTERCONNECT MODE
2026-07-28T13:58:48.771486428Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T13:58:48.812068727Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T13:58:48.838134359Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T13:58:48.894125974Z Wrote inspect data to must-gather.
2026-07-28T13:58:48.912345060Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T13:58:48.950618707Z INFO: Gathering OVN_Northbound from ovnkube-node-4nb4j...
2026-07-28T13:58:48.950914640Z INFO: Gathering OVN_Northbound from ovnkube-node-n8wdk...
2026-07-28T13:58:48.951161159Z INFO: Gathering OVN_Northbound from ovnkube-node-t6wks...
2026-07-28T13:58:48.951450033Z INFO: Gathering OVN_Southbound from ovnkube-node-4nb4j...
2026-07-28T13:58:48.951700343Z INFO: Gathering OVN_Southbound from ovnkube-node-n8wdk...
2026-07-28T13:58:48.951954869Z INFO: Gathering OVN_Southbound from ovnkube-node-t6wks...
2026-07-28T13:58:49.046355614Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:49.120670173Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T13:58:49.205264620Z Wrote inspect data to must-gather.
2026-07-28T13:58:49.296999913Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.314023746Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.321919186Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.344294047Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.366133626Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.372424559Z tar: Removing leading `/' from member names
2026-07-28T13:58:49.442658294Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:49.446395797Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T13:58:49.500181632Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T13:58:49.511321846Z Wrote inspect data to must-gather.
2026-07-28T13:58:49.686745725Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:49.740832243Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T13:58:49.837202563Z INFO: Waiting for network log collection to complete ...
2026-07-28T13:58:49.837313570Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T13:58:49.837878470Z INFO: Copying ovnk databases complete.
2026-07-28T13:58:49.841170066Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-28T13:58:49.846867916Z ovnk_database_store/
2026-07-28T13:58:49.846919450Z ovnk_database_store/ovnkube-node-n8wdk_sbdb
2026-07-28T13:58:49.850158390Z Wrote inspect data to must-gather.
2026-07-28T13:58:49.868159811Z ovnk_database_store/ovnkube-node-n8wdk_nbdb
2026-07-28T13:58:49.875357974Z ovnk_database_store/ovnkube-node-4nb4j_sbdb
2026-07-28T13:58:49.892022521Z ovnk_database_store/ovnkube-node-4nb4j_nbdb
2026-07-28T13:58:49.898001378Z ovnk_database_store/ovnkube-node-t6wks_sbdb
2026-07-28T13:58:49.918929338Z ovnk_database_store/ovnkube-node-t6wks_nbdb
2026-07-28T13:58:49.923445509Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:49.927022829Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T13:58:49.928960750Z INFO: Network log collection complete.
2026-07-28T13:58:49.944584996Z INFO: Getting status from alertmanager-main-0
2026-07-28T13:58:49.992633942Z Gathering data for ns/openshift-network-console...
2026-07-28T13:58:50.149244932Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:50.174393842Z Wrote inspect data to must-gather.
2026-07-28T13:58:50.363876207Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:50.366025182Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T13:58:50.490074194Z Wrote inspect data to must-gather.
2026-07-28T13:58:50.496298080Z Gathering data for ns/openshift-console-operator...
2026-07-28T13:58:50.586730839Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:50.750456104Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T13:58:50.753052739Z INFO: Fallback to identify the container image from release info
2026-07-28T13:58:50.803230415Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:50.806738409Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T13:58:50.957082531Z Gathering data for ns/openshift-console...
2026-07-28T13:58:50.971041743Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:51.152855622Z Wrote inspect data to /must-gather/istio.
2026-07-28T13:58:51.505101916Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T13:58:51.691203289Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T13:58:51.820840626Z Done executing Istio gather script
2026-07-28T13:58:51.825441566Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T13:58:51.943949749Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T13:58:52.092180874Z Gathering data for ns/openshift-dns-operator...
2026-07-28T13:58:52.456128279Z Gathering data for ns/openshift-dns...
2026-07-28T13:58:52.491645772Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:58:53.060910152Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T13:58:53.141430676Z Gathering data for ns/openshift-image-registry...
2026-07-28T13:58:53.645306106Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T13:58:54.067696896Z Gathering data for ns/openshift-ingress...
2026-07-28T13:58:54.184437896Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T13:58:54.453425143Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T13:58:54.884704327Z Gathering data for ns/openshift-insights...
2026-07-28T13:58:55.300958862Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T13:58:56.427599368Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T13:58:56.820918084Z Gathering data for ns/openshift-jobset-operator...
2026-07-28T13:58:57.496190084Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:58:57.547434854Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T13:58:57.642149286Z Gathering data for ns/openshift-config...
2026-07-28T13:58:57.980308111Z Gathering data for ns/openshift-config-managed...
2026-07-28T13:58:58.529501816Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T13:58:58.669050300Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T13:58:58.896652548Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T13:58:59.228673455Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T13:58:59.660132910Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T13:58:59.796111222Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T13:59:00.046199612Z Collecting performance related data for node ip-10-0-142-189.ec2.internal
2026-07-28T13:59:00.056037268Z Collecting performance related data for node ip-10-0-133-136.ec2.internal
2026-07-28T13:59:00.063913507Z Collecting performance related data for node ip-10-0-141-63.ec2.internal
2026-07-28T13:59:00.074432569Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T13:59:00.484593636Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T13:59:00.948495743Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T13:59:01.391862092Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T13:59:01.798553614Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T13:59:02.406399434Z Gathering data for ns/openshift-multus...
2026-07-28T13:59:02.500994362Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:59:04.032456605Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T13:59:05.464294544Z Gathering data for ns/openshift-host-network...
2026-07-28T13:59:05.815298247Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T13:59:06.386505870Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T13:59:06.828362740Z Gathering data for ns/openshift-network-operator...
2026-07-28T13:59:07.262626268Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T13:59:07.506411681Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:59:07.570116864Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T13:59:07.905689017Z Collecting kubelet logs for node ip-10-0-142-189.ec2.internal
2026-07-28T13:59:07.908329867Z Collecting kubelet logs for node ip-10-0-133-136.ec2.internal
2026-07-28T13:59:07.910422572Z Collecting kubelet logs for node ip-10-0-141-63.ec2.internal
2026-07-28T13:59:08.185356792Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T13:59:08.266160920Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8f9qv namespace
2026-07-28T13:59:08.271996595Z INFO: Node performance data collection complete.
2026-07-28T13:59:08.536676954Z Gathering data for ns/openshift-apiserver...
2026-07-28T13:59:09.019672457Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T13:59:09.304771583Z Gathering data for ns/openshift-controller-manager...
2026-07-28T13:59:09.612246489Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T13:59:10.341616230Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T13:59:10.674468906Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T13:59:11.010505408Z Gathering data for ns/openshift-service-ca...
2026-07-28T13:59:11.398472100Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T13:59:12.332495737Z Wrote inspect data to must-gather.
2026-07-28T13:59:12.332542043Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T13:59:12.332542043Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T13:59:12.459372895Z error: the server doesn't have a resource type "clusters"
2026-07-28T13:59:12.511230267Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T13:59:12.593106577Z Caches written to disk