2026-07-28T18:48:12.541100270Z [disk usage checker] Started
2026-07-28T18:48:12.544378176Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-28T18:48:12.763842174Z Gathering data for ns/openshift-cluster-version...
2026-07-28T18:48:12.918552651Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T18:48:13.018276275Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:48:13.103265770Z Gathering data for ns/default...
2026-07-28T18:48:13.459772564Z Gathering data for ns/openshift...
2026-07-28T18:48:13.892519145Z Gathering data for ns/kube-system...
2026-07-28T18:48:14.655054337Z Gathering data for ns/openshift-etcd...
2026-07-28T18:48:15.059474730Z Wrote inspect data to must-gather.
2026-07-28T18:48:15.059518285Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T18:48:15.059518285Z     namespaces "assisted-installer" not found
2026-07-28T18:48:15.482384915Z Waiting on subprocesses to finish execution.
2026-07-28T18:48:15.491642022Z INFO: Gathering HAProxy config files
2026-07-28T18:48:15.495220871Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T18:48:15.500964211Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T18:48:15.501788937Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T18:48:15.504609907Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T18:48:15.505486200Z Executing Istio gather script
2026-07-28T18:48:15.538374389Z INFO: Collecting host service logs for crio
2026-07-28T18:48:15.538759419Z INFO: Collecting host service logs for kubelet
2026-07-28T18:48:15.539155087Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T18:48:15.539544191Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T18:48:15.539909099Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T18:48:15.540266217Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T18:48:15.540661375Z INFO: Collecting host service logs for NetworkManager
2026-07-28T18:48:15.541017963Z INFO: Collecting host service logs for openvswitch
2026-07-28T18:48:15.541453611Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T18:48:15.541821746Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T18:48:15.542189466Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T18:48:15.542663151Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T18:48:16.123790041Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T18:48:16.123829403Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T18:48:16.442181414Z Wrote inspect data to must-gather.
2026-07-28T18:48:16.515333588Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T18:48:16.701403141Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-vx5gq
2026-07-28T18:48:16.701403141Z insights-runtime-extractor-gdrjx
2026-07-28T18:48:16.701403141Z insights-runtime-extractor-kqb5c
2026-07-28T18:48:16.701403141Z insights-runtime-extractor-slzgv
2026-07-28T18:48:16.855822485Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T18:48:16.875017322Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T18:48:16.926679342Z No resources found
2026-07-28T18:48:16.992998998Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T18:48:17.115945518Z No resources found in openshift-etcd namespace.
2026-07-28T18:48:17.135685098Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T18:48:17.156113894Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T18:48:17.280685363Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T18:48:17.280685363Z See 'oc rsync -h' for help and examples
2026-07-28T18:48:17.283091409Z ERROR: No running kube-apiserver pods found
2026-07-28T18:48:17.338421454Z INFO: Worker host service log collection to complete.
2026-07-28T18:48:17.511067536Z No resources found
2026-07-28T18:48:17.567743085Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-28T18:48:17.611995095Z Wrote inspect data to must-gather.
2026-07-28T18:48:17.678468551Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T18:48:18.085561216Z INFO: 'previous-logs' folder not found on ip-10-0-132-11.ec2.internal, skipping...
2026-07-28T18:48:18.127437229Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T18:48:18.129335543Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.156388097Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.162628808Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.228396441Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T18:48:18.362910587Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.368391475Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:48:18.529721459Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.565769097Z Gathering data for ns/openshift-monitoring...
2026-07-28T18:48:18.575434220Z tar: Removing leading `/' from member names
2026-07-28T18:48:18.581416326Z INFO: HAProxy config collection complete.
2026-07-28T18:48:18.645983219Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T18:48:18.685792751Z Wrote inspect data to must-gather.
2026-07-28T18:48:18.696167252Z INFO: 'previous-logs' folder not found on ip-10-0-138-92.ec2.internal, skipping...
2026-07-28T18:48:18.745724385Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T18:48:18.867509158Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T18:48:18.958553518Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T18:48:18.998641421Z INFO: 'previous-logs' folder not found on ip-10-0-142-197.ec2.internal, skipping...
2026-07-28T18:48:18.998678901Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T18:48:18.998693554Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T18:48:19.003813943Z Wrote inspect data to must-gather.
2026-07-28T18:48:19.122358791Z Wrote inspect data to must-gather.
2026-07-28T18:48:19.224155587Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T18:48:19.267002144Z Wrote inspect data to must-gather.
2026-07-28T18:48:19.319419590Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T18:48:19.566363311Z Wrote inspect data to must-gather.
2026-07-28T18:48:19.603697795Z INFO: INTERCONNECT MODE
2026-07-28T18:48:19.603745245Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T18:48:19.694443935Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T18:48:19.808840256Z INFO: Gathering OVN_Northbound from ovnkube-node-6spk5...
2026-07-28T18:48:19.809113418Z INFO: Gathering OVN_Northbound from ovnkube-node-mdz9h...
2026-07-28T18:48:19.809159606Z Wrote inspect data to must-gather.
2026-07-28T18:48:19.809356706Z INFO: Gathering OVN_Northbound from ovnkube-node-rj4lx...
2026-07-28T18:48:19.809628809Z INFO: Gathering OVN_Southbound from ovnkube-node-6spk5...
2026-07-28T18:48:19.809918850Z INFO: Gathering OVN_Southbound from ovnkube-node-mdz9h...
2026-07-28T18:48:19.810191505Z INFO: Gathering OVN_Southbound from ovnkube-node-rj4lx...
2026-07-28T18:48:20.057135062Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T18:48:20.189876264Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.210851570Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.213487675Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.223750540Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.235592854Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.273891550Z Wrote inspect data to must-gather.
2026-07-28T18:48:20.302438424Z tar: Removing leading `/' from member names
2026-07-28T18:48:20.362251579Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T18:48:20.408616655Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T18:48:20.537411874Z Wrote inspect data to must-gather.
2026-07-28T18:48:20.711538252Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T18:48:20.713077827Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T18:48:20.773178706Z INFO: Waiting for network log collection to complete ...
2026-07-28T18:48:20.773301135Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T18:48:20.773848567Z INFO: Copying ovnk databases complete.
2026-07-28T18:48:20.776076211Z 4.6M	must-gather/network_logs/ovnk_database_store
2026-07-28T18:48:20.778573972Z ovnk_database_store/
2026-07-28T18:48:20.778648969Z ovnk_database_store/ovnkube-node-6spk5_nbdb
2026-07-28T18:48:20.790154551Z ovnk_database_store/ovnkube-node-rj4lx_nbdb
2026-07-28T18:48:20.798579688Z ovnk_database_store/ovnkube-node-6spk5_sbdb
2026-07-28T18:48:20.801569642Z Wrote inspect data to must-gather.
2026-07-28T18:48:20.825752344Z ovnk_database_store/ovnkube-node-mdz9h_nbdb
2026-07-28T18:48:20.838859349Z ovnk_database_store/ovnkube-node-mdz9h_sbdb
2026-07-28T18:48:20.868833687Z ovnk_database_store/ovnkube-node-rj4lx_sbdb
2026-07-28T18:48:20.895631630Z INFO: Network log collection complete.
2026-07-28T18:48:20.937664312Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:21.043524339Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:48:21.149796942Z Wrote inspect data to must-gather.
2026-07-28T18:48:21.166281427Z INFO: Getting status from alertmanager-main-0
2026-07-28T18:48:21.199076162Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:21.209091397Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T18:48:21.247418774Z Gathering data for ns/openshift-network-console...
2026-07-28T18:48:21.435211677Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:21.455803636Z Wrote inspect data to must-gather.
2026-07-28T18:48:21.670929269Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:21.673914974Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T18:48:21.740385080Z Gathering data for ns/openshift-console-operator...
2026-07-28T18:48:21.770294422Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T18:48:21.773889325Z INFO: Fallback to identify the container image from release info
2026-07-28T18:48:21.894615836Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:22.097406915Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:22.100982284Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T18:48:22.157265642Z Gathering data for ns/openshift-console...
2026-07-28T18:48:22.340629545Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:22.425900824Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T18:48:22.566380299Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:22.570437333Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T18:48:22.585419984Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T18:48:22.589773698Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-28T18:48:22.744661506Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T18:48:22.744745590Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T18:48:22.780803689Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.001020041Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.004393005Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T18:48:23.214942359Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.355660505Z Gathering data for ns/openshift-dns-operator...
2026-07-28T18:48:23.448709051Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.451777988Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T18:48:23.636964867Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.691386331Z Gathering data for ns/openshift-dns...
2026-07-28T18:48:23.844005512Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:48:23.890875260Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T18:48:24.363441733Z Gathering data for ns/openshift-image-registry...
2026-07-28T18:48:24.578003690Z Done executing Istio gather script
2026-07-28T18:48:24.881858668Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T18:48:25.021395619Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T18:48:25.228787107Z Gathering data for ns/openshift-ingress...
2026-07-28T18:48:25.578884212Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T18:48:26.009789005Z Gathering data for ns/openshift-insights...
2026-07-28T18:48:26.139296694Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T18:48:27.255252680Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T18:48:27.596184375Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-28T18:48:28.385732251Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T18:48:28.456828075Z Gathering data for ns/kserve...
2026-07-28T18:48:29.517267286Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T18:48:30.654701666Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T18:48:30.668169996Z Gathering data for ns/openshift-config...
2026-07-28T18:48:30.949153089Z Collecting performance related data for node ip-10-0-138-92.ec2.internal
2026-07-28T18:48:30.956126278Z Collecting performance related data for node ip-10-0-132-11.ec2.internal
2026-07-28T18:48:30.968042212Z Collecting performance related data for node ip-10-0-142-197.ec2.internal
2026-07-28T18:48:30.990875407Z Gathering data for ns/openshift-config-managed...
2026-07-28T18:48:31.526737162Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T18:48:31.900848942Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T18:48:32.263276519Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T18:48:32.603475692Z [disk usage checker] Volume usage percentage: current = 26 ; allowed = 70
2026-07-28T18:48:32.686906059Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T18:48:33.004568082Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T18:48:33.283501285Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T18:48:33.679607554Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T18:48:34.076727823Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T18:48:34.439856581Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T18:48:34.968246847Z Gathering data for ns/openshift-multus...
2026-07-28T18:48:36.519365455Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T18:48:37.609257994Z [disk usage checker] Volume usage percentage: current = 26 ; allowed = 70
2026-07-28T18:48:37.800952850Z Gathering data for ns/openshift-host-network...
2026-07-28T18:48:38.127728020Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T18:48:38.719801478Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T18:48:39.172864650Z Gathering data for ns/openshift-network-operator...
2026-07-28T18:48:39.394688861Z Collecting kubelet logs for node ip-10-0-138-92.ec2.internal
2026-07-28T18:48:39.396452931Z Collecting kubelet logs for node ip-10-0-132-11.ec2.internal
2026-07-28T18:48:39.398135852Z Collecting kubelet logs for node ip-10-0-142-197.ec2.internal
2026-07-28T18:48:39.634498388Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T18:48:39.876282932Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-2g68f namespace
2026-07-28T18:48:39.889030506Z INFO: Node performance data collection complete.
2026-07-28T18:48:40.028040725Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T18:48:40.538048079Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T18:48:40.848893912Z Gathering data for ns/openshift-apiserver...
2026-07-28T18:48:41.296071051Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T18:48:41.569404082Z Gathering data for ns/openshift-controller-manager...
2026-07-28T18:48:41.864181406Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T18:48:42.535413763Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T18:48:42.614051430Z [disk usage checker] Volume usage percentage: current = 26 ; allowed = 70
2026-07-28T18:48:42.837208769Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T18:48:43.176405622Z Gathering data for ns/openshift-service-ca...
2026-07-28T18:48:43.601380531Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T18:48:44.511203905Z Wrote inspect data to must-gather.
2026-07-28T18:48:44.511250860Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T18:48:44.511250860Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T18:48:44.644133679Z error: the server doesn't have a resource type "clusters"
2026-07-28T18:48:44.801163089Z Caches written to disk