2026-07-28T08:00:38.359592529Z [disk usage checker] Started
2026-07-28T08:00:38.362955160Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:00:38.591269300Z Gathering data for ns/openshift-cluster-version...
2026-07-28T08:00:38.741221788Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T08:00:38.839375019Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T08:00:38.926608446Z Gathering data for ns/default...
2026-07-28T08:00:39.222151604Z Gathering data for ns/openshift...
2026-07-28T08:00:39.643063363Z Gathering data for ns/kube-system...
2026-07-28T08:00:40.376289098Z Gathering data for ns/openshift-etcd...
2026-07-28T08:00:40.701860411Z Wrote inspect data to must-gather.
2026-07-28T08:00:40.701896456Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T08:00:40.701896456Z     namespaces "assisted-installer" not found
2026-07-28T08:00:41.079130984Z Waiting on subprocesses to finish execution.
2026-07-28T08:00:41.095990790Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T08:00:41.107677085Z Executing Istio gather script
2026-07-28T08:00:41.113073034Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T08:00:41.113362199Z INFO: Gathering HAProxy config files
2026-07-28T08:00:41.117616194Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T08:00:41.147640495Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T08:00:41.210645742Z INFO: Collecting host service logs for crio
2026-07-28T08:00:41.211128646Z INFO: Collecting host service logs for kubelet
2026-07-28T08:00:41.211509109Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T08:00:41.211832583Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T08:00:41.212146234Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T08:00:41.212478926Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T08:00:41.212802620Z INFO: Collecting host service logs for NetworkManager
2026-07-28T08:00:41.213108496Z INFO: Collecting host service logs for openvswitch
2026-07-28T08:00:41.213442004Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T08:00:41.213766463Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T08:00:41.214081714Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T08:00:41.214488849Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T08:00:41.386027884Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T08:00:41.386134938Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T08:00:41.489526724Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T08:00:41.533444686Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-kt56p
2026-07-28T08:00:41.533518641Z insights-runtime-extractor-8qbdp
2026-07-28T08:00:41.533548891Z insights-runtime-extractor-bjvdl
2026-07-28T08:00:41.533577288Z insights-runtime-extractor-ll4hl
2026-07-28T08:00:41.687527519Z No resources found
2026-07-28T08:00:41.817122545Z No resources found in openshift-etcd namespace.
2026-07-28T08:00:41.937402766Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T08:00:42.010592482Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T08:00:42.127015056Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T08:00:42.174220144Z Wrote inspect data to must-gather.
2026-07-28T08:00:42.228740040Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T08:00:42.282852300Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T08:00:42.282852300Z See 'oc rsync -h' for help and examples
2026-07-28T08:00:42.573354879Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T08:00:42.945717413Z INFO: Worker host service log collection to complete.
2026-07-28T08:00:43.201394879Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.208398716Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T08:00:43.307464107Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T08:00:43.394778900Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T08:00:43.423676780Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:00:43.439939392Z No resources found
2026-07-28T08:00:43.494377383Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.530109857Z ERROR: No running kube-apiserver pods found
2026-07-28T08:00:43.541458918Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T08:00:43.569989700Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.636065144Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.658732534Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.698398046Z INFO: 'previous-logs' folder not found on ip-10-0-131-149.ec2.internal, skipping...
2026-07-28T08:00:43.744215214Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.854507011Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T08:00:43.884953300Z Wrote inspect data to must-gather.
2026-07-28T08:00:43.960955378Z tar: Removing leading `/' from member names
2026-07-28T08:00:43.965619804Z INFO: HAProxy config collection complete.
2026-07-28T08:00:44.059796640Z INFO: 'previous-logs' folder not found on ip-10-0-132-33.ec2.internal, skipping...
2026-07-28T08:00:44.075159900Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T08:00:44.079098945Z Gathering data for ns/openshift-monitoring...
2026-07-28T08:00:44.112998566Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T08:00:44.189276141Z Wrote inspect data to must-gather.
2026-07-28T08:00:44.338984289Z Wrote inspect data to must-gather.
2026-07-28T08:00:44.361977010Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T08:00:44.375725373Z INFO: 'previous-logs' folder not found on ip-10-0-138-41.ec2.internal, skipping...
2026-07-28T08:00:44.375838910Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T08:00:44.375891926Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T08:00:44.482448277Z Wrote inspect data to must-gather.
2026-07-28T08:00:44.506993812Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T08:00:44.564149577Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T08:00:44.732821433Z Wrote inspect data to must-gather.
2026-07-28T08:00:44.787667111Z INFO: INTERCONNECT MODE
2026-07-28T08:00:44.787710183Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T08:00:44.940140699Z Wrote inspect data to must-gather.
2026-07-28T08:00:44.953280691Z INFO: Gathering OVN_Northbound from ovnkube-node-tzl5t...
2026-07-28T08:00:44.953642077Z INFO: Gathering OVN_Northbound from ovnkube-node-wkcpj...
2026-07-28T08:00:44.953904202Z INFO: Gathering OVN_Northbound from ovnkube-node-wmf27...
2026-07-28T08:00:44.954180198Z INFO: Gathering OVN_Southbound from ovnkube-node-tzl5t...
2026-07-28T08:00:44.954435589Z INFO: Gathering OVN_Southbound from ovnkube-node-wkcpj...
2026-07-28T08:00:44.954742573Z INFO: Gathering OVN_Southbound from ovnkube-node-wmf27...
2026-07-28T08:00:45.068424273Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T08:00:45.200197819Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T08:00:45.279992969Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.280191698Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.288994593Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.306222859Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.306292211Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.310331353Z tar: Removing leading `/' from member names
2026-07-28T08:00:45.325268075Z Wrote inspect data to must-gather.
2026-07-28T08:00:45.416715804Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:45.441800587Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T08:00:45.587616394Z Wrote inspect data to must-gather.
2026-07-28T08:00:45.695905321Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T08:00:45.724882804Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:45.727427248Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T08:00:45.729538522Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T08:00:45.811013636Z INFO: Waiting for network log collection to complete ...
2026-07-28T08:00:45.811061461Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T08:00:45.811564289Z INFO: Copying ovnk databases complete.
2026-07-28T08:00:45.813843062Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-28T08:00:45.816910304Z ovnk_database_store/
2026-07-28T08:00:45.816969368Z ovnk_database_store/ovnkube-node-wkcpj_nbdb
2026-07-28T08:00:45.826491402Z ovnk_database_store/ovnkube-node-wkcpj_sbdb
2026-07-28T08:00:45.831276800Z Wrote inspect data to must-gather.
2026-07-28T08:00:45.847630736Z ovnk_database_store/ovnkube-node-wmf27_sbdb
2026-07-28T08:00:45.870003100Z ovnk_database_store/ovnkube-node-wmf27_nbdb
2026-07-28T08:00:45.876986851Z ovnk_database_store/ovnkube-node-tzl5t_nbdb
2026-07-28T08:00:45.885981449Z ovnk_database_store/ovnkube-node-tzl5t_sbdb
2026-07-28T08:00:45.904710809Z INFO: Network log collection complete.
2026-07-28T08:00:45.965330535Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:45.990025681Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T08:00:46.055906283Z Wrote inspect data to must-gather.
2026-07-28T08:00:46.192616484Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:46.195405575Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T08:00:46.280267132Z W0728 08:00:46.280200    1540 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T08:00:46.285381531Z Wrote inspect data to must-gather.
2026-07-28T08:00:46.363571008Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T08:00:46.386921757Z INFO: Getting status from alertmanager-main-0
2026-07-28T08:00:46.425395351Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:46.560938651Z Gathering data for ns/openshift-network-console...
2026-07-28T08:00:46.571334883Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T08:00:46.573553241Z INFO: Fallback to identify the container image from release info
2026-07-28T08:00:46.663559939Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:46.667031728Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T08:00:46.885814886Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:47.009079293Z Gathering data for ns/openshift-console-operator...
2026-07-28T08:00:47.088087425Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:47.091651296Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T08:00:47.259353341Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:47.450672115Z Wrote inspect data to /must-gather/istio.
2026-07-28T08:00:47.497660542Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T08:00:47.561280469Z Gathering data for ns/openshift-console...
2026-07-28T08:00:47.634919760Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T08:00:47.780510533Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T08:00:48.190239462Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T08:00:48.193908823Z Done executing Istio gather script
2026-07-28T08:00:48.442117867Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:00:48.710815373Z Gathering data for ns/openshift-dns-operator...
2026-07-28T08:00:48.897571650Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T08:00:49.013990671Z Gathering data for ns/openshift-dns...
2026-07-28T08:00:49.628412460Z Gathering data for ns/openshift-image-registry...
2026-07-28T08:00:50.017865181Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T08:00:50.110862927Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T08:00:50.484987734Z Gathering data for ns/openshift-ingress...
2026-07-28T08:00:50.828653037Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T08:00:51.145055824Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T08:00:51.238832266Z Gathering data for ns/openshift-insights...
2026-07-28T08:00:52.265244591Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T08:00:53.114969105Z Gathering data for ns/openshift-jobset-operator...
2026-07-28T08:00:53.393322104Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T08:00:53.448091125Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:00:53.923714300Z Gathering data for ns/openshift-config...
2026-07-28T08:00:54.188956578Z Gathering data for ns/openshift-config-managed...
2026-07-28T08:00:54.529273115Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T08:00:54.635260405Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T08:00:54.899102286Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T08:00:55.197323423Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T08:00:55.479687708Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T08:00:55.641512966Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T08:00:55.842131724Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T08:00:55.920478143Z Collecting performance related data for node ip-10-0-132-33.ec2.internal
2026-07-28T08:00:55.927703774Z Collecting performance related data for node ip-10-0-138-41.ec2.internal
2026-07-28T08:00:55.935645654Z Collecting performance related data for node ip-10-0-131-149.ec2.internal
2026-07-28T08:00:56.157611781Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T08:00:56.501181172Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T08:00:57.004576605Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T08:00:57.415809132Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T08:00:58.034509216Z Gathering data for ns/openshift-multus...
2026-07-28T08:00:58.453276812Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:00:59.688313257Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T08:01:01.031650413Z Gathering data for ns/openshift-host-network...
2026-07-28T08:01:01.355628007Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T08:01:01.926043655Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T08:01:02.342131190Z Gathering data for ns/openshift-network-operator...
2026-07-28T08:01:02.732689398Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T08:01:02.990001059Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T08:01:03.459863975Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-28T08:01:03.481233640Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T08:01:03.824043156Z Gathering data for ns/openshift-apiserver...
2026-07-28T08:01:04.213433407Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T08:01:04.496344949Z Gathering data for ns/openshift-controller-manager...
2026-07-28T08:01:04.798199259Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T08:01:04.937838781Z Collecting kubelet logs for node ip-10-0-132-33.ec2.internal
2026-07-28T08:01:04.939832899Z Collecting kubelet logs for node ip-10-0-138-41.ec2.internal
2026-07-28T08:01:04.941713463Z Collecting kubelet logs for node ip-10-0-131-149.ec2.internal
2026-07-28T08:01:05.293866568Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kg4t5 namespace
2026-07-28T08:01:05.299940828Z INFO: Node performance data collection complete.
2026-07-28T08:01:05.532015351Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T08:01:05.840669726Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T08:01:06.164307782Z Gathering data for ns/openshift-service-ca...
2026-07-28T08:01:06.516077215Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T08:01:07.462999925Z Wrote inspect data to must-gather.
2026-07-28T08:01:07.463045849Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T08:01:07.463045849Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T08:01:07.596371047Z error: the server doesn't have a resource type "clusters"
2026-07-28T08:01:07.731985865Z Caches written to disk