2026-07-28T11:40:22.949096006Z [disk usage checker] Started
2026-07-28T11:40:22.952455268Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:23.277933773Z Gathering data for ns/openshift-cluster-version...
2026-07-28T11:40:23.418947418Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T11:40:23.511940118Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T11:40:23.600983087Z Gathering data for ns/default...
2026-07-28T11:40:23.925902685Z Gathering data for ns/openshift...
2026-07-28T11:40:24.345517510Z Gathering data for ns/kube-system...
2026-07-28T11:40:25.172010635Z Gathering data for ns/openshift-etcd...
2026-07-28T11:40:25.523725258Z Wrote inspect data to must-gather.
2026-07-28T11:40:25.523758385Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T11:40:25.523758385Z     namespaces "assisted-installer" not found
2026-07-28T11:40:25.975855817Z Waiting on subprocesses to finish execution.
2026-07-28T11:40:25.983076372Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T11:40:25.985139717Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T11:40:25.987806839Z Executing Istio gather script
2026-07-28T11:40:25.993494735Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T11:40:25.993625221Z INFO: Collecting host service logs for crio
2026-07-28T11:40:25.993658330Z INFO: Gathering HAProxy config files
2026-07-28T11:40:25.993968058Z INFO: Collecting host service logs for kubelet
2026-07-28T11:40:25.994459576Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T11:40:25.994778277Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T11:40:25.995083007Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T11:40:25.995360930Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T11:40:25.995650737Z INFO: Collecting host service logs for NetworkManager
2026-07-28T11:40:25.996033187Z INFO: Collecting host service logs for openvswitch
2026-07-28T11:40:25.996460085Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T11:40:25.996796069Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T11:40:25.997133584Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T11:40:25.997531809Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T11:40:26.203821644Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T11:40:26.318455362Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T11:40:26.366349696Z No resources found in openshift-etcd namespace.
2026-07-28T11:40:26.644535458Z Wrote inspect data to must-gather.
2026-07-28T11:40:26.817876977Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T11:40:27.054741589Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T11:40:27.097116586Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T11:40:27.133871545Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T11:40:27.133943755Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T11:40:27.237663345Z No resources found
2026-07-28T11:40:27.472502823Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T11:40:27.517609578Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-gl5c5
2026-07-28T11:40:27.517676099Z insights-runtime-extractor-46tjh
2026-07-28T11:40:27.517782519Z insights-runtime-extractor-fr5z5
2026-07-28T11:40:27.517819539Z insights-runtime-extractor-p9hm6
2026-07-28T11:40:27.683892491Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T11:40:27.815529706Z INFO: Worker host service log collection to complete.
2026-07-28T11:40:27.819338419Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T11:40:27.831002093Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.029314555Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:28.314645028Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T11:40:28.326182171Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T11:40:28.383636225Z No resources found
2026-07-28T11:40:28.418056727Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.439850830Z tar: Removing leading `/' from member names
2026-07-28T11:40:28.449504718Z INFO: HAProxy config collection complete.
2026-07-28T11:40:28.521125610Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.527897854Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.546378114Z INFO: 'previous-logs' folder not found on ip-10-0-133-108.ec2.internal, skipping...
2026-07-28T11:40:28.566889623Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.647840787Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T11:40:28.710539104Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T11:40:28.710539104Z See 'oc rsync -h' for help and examples
2026-07-28T11:40:28.753508179Z Wrote inspect data to must-gather.
2026-07-28T11:40:28.779053746Z ERROR: No running kube-apiserver pods found
2026-07-28T11:40:28.992642879Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T11:40:29.098432998Z INFO: 'previous-logs' folder not found on ip-10-0-136-186.ec2.internal, skipping...
2026-07-28T11:40:29.108169841Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T11:40:29.112621296Z Gathering data for ns/openshift-monitoring...
2026-07-28T11:40:29.140009930Z Wrote inspect data to must-gather.
2026-07-28T11:40:29.197878626Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T11:40:29.415489511Z INFO: 'previous-logs' folder not found on ip-10-0-136-194.ec2.internal, skipping...
2026-07-28T11:40:29.415526774Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T11:40:29.415540809Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T11:40:29.460737630Z Wrote inspect data to must-gather.
2026-07-28T11:40:29.481234844Z Wrote inspect data to must-gather.
2026-07-28T11:40:29.594240174Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T11:40:29.632936989Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T11:40:29.673163267Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T11:40:29.737130399Z Wrote inspect data to must-gather.
2026-07-28T11:40:29.914675386Z Wrote inspect data to must-gather.
2026-07-28T11:40:30.068387580Z INFO: INTERCONNECT MODE
2026-07-28T11:40:30.068480866Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T11:40:30.085611503Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T11:40:30.181708049Z Wrote inspect data to must-gather.
2026-07-28T11:40:30.237658637Z INFO: Gathering OVN_Northbound from ovnkube-node-jn2ms...
2026-07-28T11:40:30.237897047Z INFO: Gathering OVN_Northbound from ovnkube-node-kvnkg...
2026-07-28T11:40:30.238094998Z INFO: Gathering OVN_Northbound from ovnkube-node-pdrhp...
2026-07-28T11:40:30.238643884Z INFO: Gathering OVN_Southbound from ovnkube-node-jn2ms...
2026-07-28T11:40:30.238949070Z INFO: Gathering OVN_Southbound from ovnkube-node-kvnkg...
2026-07-28T11:40:30.239237079Z INFO: Gathering OVN_Southbound from ovnkube-node-pdrhp...
2026-07-28T11:40:30.447487340Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T11:40:30.553162789Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.559071924Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.574925873Z Wrote inspect data to must-gather.
2026-07-28T11:40:30.580206305Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.598136114Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.615934590Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.627841238Z tar: Removing leading `/' from member names
2026-07-28T11:40:30.786065548Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T11:40:30.873015184Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T11:40:30.899107489Z Wrote inspect data to must-gather.
2026-07-28T11:40:30.968594371Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T11:40:31.121708119Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T11:40:31.161676192Z Wrote inspect data to must-gather.
2026-07-28T11:40:31.219641536Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:31.231155517Z INFO: Waiting for network log collection to complete ...
2026-07-28T11:40:31.231239658Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T11:40:31.231733932Z INFO: Copying ovnk databases complete.
2026-07-28T11:40:31.233756441Z 4.6M	must-gather/network_logs/ovnk_database_store
2026-07-28T11:40:31.236384668Z ovnk_database_store/
2026-07-28T11:40:31.236445333Z ovnk_database_store/ovnkube-node-pdrhp_nbdb
2026-07-28T11:40:31.249871430Z ovnk_database_store/ovnkube-node-pdrhp_sbdb
2026-07-28T11:40:31.275256740Z ovnk_database_store/ovnkube-node-jn2ms_nbdb
2026-07-28T11:40:31.285208360Z ovnk_database_store/ovnkube-node-kvnkg_nbdb
2026-07-28T11:40:31.298011162Z ovnk_database_store/ovnkube-node-jn2ms_sbdb
2026-07-28T11:40:31.322015703Z ovnk_database_store/ovnkube-node-kvnkg_sbdb
2026-07-28T11:40:31.356965785Z INFO: Network log collection complete.
2026-07-28T11:40:31.424855019Z Wrote inspect data to must-gather.
2026-07-28T11:40:31.460852862Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:31.465654273Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T11:40:31.590198341Z INFO: Getting status from alertmanager-main-0
2026-07-28T11:40:31.592864162Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T11:40:31.697653097Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:31.793127672Z Wrote inspect data to must-gather.
2026-07-28T11:40:31.800326037Z Gathering data for ns/openshift-network-console...
2026-07-28T11:40:31.933082370Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:31.936654638Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T11:40:32.100380811Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T11:40:32.104013606Z INFO: Fallback to identify the container image from release info
2026-07-28T11:40:32.160191565Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:32.262212334Z Gathering data for ns/openshift-console-operator...
2026-07-28T11:40:32.385553759Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:32.389433235Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T11:40:32.616234756Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:32.649917623Z Gathering data for ns/openshift-console...
2026-07-28T11:40:32.799113616Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T11:40:32.840832954Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:32.845901485Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T11:40:32.941420688Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T11:40:33.040670085Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:33.072870060Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:33.105790540Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T11:40:33.252490554Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:33.256256608Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T11:40:33.258128381Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T11:40:33.459135326Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:33.688665450Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:33.693076092Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T11:40:33.879900366Z Gathering data for ns/openshift-dns-operator...
2026-07-28T11:40:33.890555400Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:34.097029164Z Wrote inspect data to /must-gather/istio.
2026-07-28T11:40:34.246425397Z Gathering data for ns/openshift-dns...
2026-07-28T11:40:34.257318801Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T11:40:34.847465296Z Done executing Istio gather script
2026-07-28T11:40:34.947290141Z Gathering data for ns/openshift-image-registry...
2026-07-28T11:40:35.387097999Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T11:40:35.475534359Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T11:40:35.800339039Z Gathering data for ns/openshift-ingress...
2026-07-28T11:40:36.142167171Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T11:40:36.541613688Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T11:40:36.552206235Z Gathering data for ns/openshift-insights...
2026-07-28T11:40:37.654999510Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T11:40:38.077877434Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:38.779128839Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T11:40:38.899559559Z Gathering data for ns/kserve...
2026-07-28T11:40:39.904296832Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T11:40:40.957995948Z Gathering data for ns/openshift-config...
2026-07-28T11:40:41.041175732Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T11:40:41.255417043Z Gathering data for ns/openshift-config-managed...
2026-07-28T11:40:41.382070733Z Collecting performance related data for node ip-10-0-136-186.ec2.internal
2026-07-28T11:40:41.388908445Z Collecting performance related data for node ip-10-0-133-108.ec2.internal
2026-07-28T11:40:41.398900069Z Collecting performance related data for node ip-10-0-136-194.ec2.internal
2026-07-28T11:40:41.753110030Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T11:40:42.102718130Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T11:40:42.435849114Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T11:40:42.796747412Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T11:40:43.083749817Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:43.151084855Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T11:40:43.419367898Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T11:40:43.710592527Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T11:40:44.056879323Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T11:40:44.416799780Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T11:40:44.940010326Z Gathering data for ns/openshift-multus...
2026-07-28T11:40:46.401265534Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T11:40:47.851497396Z Gathering data for ns/openshift-host-network...
2026-07-28T11:40:48.088528190Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:48.152470792Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T11:40:48.726903224Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T11:40:49.160671051Z Gathering data for ns/openshift-network-operator...
2026-07-28T11:40:49.531848300Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T11:40:49.813391127Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T11:40:50.296280918Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T11:40:50.301496520Z Collecting kubelet logs for node ip-10-0-136-186.ec2.internal
2026-07-28T11:40:50.303208989Z Collecting kubelet logs for node ip-10-0-133-108.ec2.internal
2026-07-28T11:40:50.305316112Z Collecting kubelet logs for node ip-10-0-136-194.ec2.internal
2026-07-28T11:40:50.658159045Z Gathering data for ns/openshift-apiserver...
2026-07-28T11:40:50.808779817Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x6kqb namespace
2026-07-28T11:40:50.814664202Z INFO: Node performance data collection complete.
2026-07-28T11:40:51.116095701Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T11:40:51.393320028Z Gathering data for ns/openshift-controller-manager...
2026-07-28T11:40:51.666856927Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T11:40:52.311806870Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T11:40:52.588950219Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T11:40:52.912515823Z Gathering data for ns/openshift-service-ca...
2026-07-28T11:40:53.093425048Z [disk usage checker] Volume usage percentage: current = 49 ; allowed = 70
2026-07-28T11:40:53.281049774Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T11:40:54.231864341Z Wrote inspect data to must-gather.
2026-07-28T11:40:54.231912918Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T11:40:54.231912918Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T11:40:54.367378763Z error: the server doesn't have a resource type "clusters"
2026-07-28T11:40:54.528010307Z Caches written to disk