2026-07-07T15:22:59.833918027Z [disk usage checker] Started
2026-07-07T15:22:59.837428280Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:00.037883938Z Gathering data for ns/openshift-cluster-version...
2026-07-07T15:23:00.176908014Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:23:00.285604238Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:23:00.369775937Z Gathering data for ns/default...
2026-07-07T15:23:00.702127797Z Gathering data for ns/openshift...
2026-07-07T15:23:01.224951881Z Gathering data for ns/kube-system...
2026-07-07T15:23:02.017127721Z Gathering data for ns/openshift-etcd...
2026-07-07T15:23:02.345728404Z Wrote inspect data to must-gather.
2026-07-07T15:23:02.345767976Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:23:02.345767976Z     namespaces "assisted-installer" not found
2026-07-07T15:23:02.691995993Z Waiting on subprocesses to finish execution.
2026-07-07T15:23:02.697005978Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T15:23:02.701891367Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T15:23:02.706873864Z Executing Istio gather script
2026-07-07T15:23:02.725417638Z INFO: Gathering HAProxy config files
2026-07-07T15:23:02.725608270Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T15:23:02.732156208Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T15:23:02.735234869Z INFO: Collecting host service logs for crio
2026-07-07T15:23:02.735594164Z INFO: Collecting host service logs for kubelet
2026-07-07T15:23:02.735918323Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T15:23:02.736244837Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T15:23:02.736635824Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T15:23:02.736994316Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T15:23:02.737364885Z INFO: Collecting host service logs for NetworkManager
2026-07-07T15:23:02.737707281Z INFO: Collecting host service logs for openvswitch
2026-07-07T15:23:02.738041573Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T15:23:02.738398960Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T15:23:02.738743768Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T15:23:02.739136418Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T15:23:03.018149029Z No resources found in openshift-etcd namespace.
2026-07-07T15:23:03.117052592Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T15:23:03.339022921Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T15:23:03.595281193Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T15:23:03.650092099Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T15:23:03.650174729Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T15:23:03.931939828Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T15:23:04.063865524Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T15:23:04.089887757Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T15:23:04.146750158Z Wrote inspect data to must-gather.
2026-07-07T15:23:04.293477304Z No resources found
2026-07-07T15:23:04.302765423Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T15:23:04.304543069Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-c5q6l
2026-07-07T15:23:04.304587295Z insights-runtime-extractor-fm8rl
2026-07-07T15:23:04.304607423Z insights-runtime-extractor-jn94p
2026-07-07T15:23:04.304628295Z insights-runtime-extractor-sdnbn
2026-07-07T15:23:04.373243735Z INFO: Worker host service log collection to complete.
2026-07-07T15:23:04.683473489Z Wrote inspect data to must-gather.
2026-07-07T15:23:04.721338444Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T15:23:04.790592753Z tar: Removing leading `/' from member names
2026-07-07T15:23:04.798016723Z INFO: HAProxy config collection complete.
2026-07-07T15:23:04.876119945Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T15:23:04.876119945Z See 'oc rsync -h' for help and examples
2026-07-07T15:23:04.990695594Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:05.116364394Z ERROR: No running kube-apiserver pods found
2026-07-07T15:23:05.154561141Z INFO: 'previous-logs' folder not found on ip-10-0-130-141.ec2.internal, skipping...
2026-07-07T15:23:05.358242789Z Wrote inspect data to must-gather.
2026-07-07T15:23:05.436444238Z No resources found
2026-07-07T15:23:05.482813726Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:23:05.509508083Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T15:23:05.559032574Z Wrote inspect data to must-gather.
2026-07-07T15:23:05.562269545Z Wrote inspect data to must-gather.
2026-07-07T15:23:05.626980962Z Wrote inspect data to must-gather.
2026-07-07T15:23:05.634825811Z Wrote inspect data to must-gather.
2026-07-07T15:23:05.695050970Z INFO: 'previous-logs' folder not found on ip-10-0-138-241.ec2.internal, skipping...
2026-07-07T15:23:05.814489442Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T15:23:05.850047328Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T15:23:05.886507294Z Gathering data for ns/openshift-monitoring...
2026-07-07T15:23:05.966526381Z Wrote inspect data to must-gather.
2026-07-07T15:23:06.063006360Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T15:23:06.080776191Z INFO: 'previous-logs' folder not found on ip-10-0-143-168.ec2.internal, skipping...
2026-07-07T15:23:06.080921807Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T15:23:06.080983925Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T15:23:06.082114739Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T15:23:06.243465866Z Wrote inspect data to must-gather.
2026-07-07T15:23:06.329596739Z Wrote inspect data to must-gather.
2026-07-07T15:23:06.331220314Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T15:23:06.502656426Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T15:23:06.507666482Z Wrote inspect data to must-gather.
2026-07-07T15:23:06.735596713Z Wrote inspect data to must-gather.
2026-07-07T15:23:06.777605279Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T15:23:06.814854915Z INFO: INTERCONNECT MODE
2026-07-07T15:23:06.814905290Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T15:23:06.972474438Z Wrote inspect data to must-gather.
2026-07-07T15:23:07.003511852Z INFO: Gathering OVN_Northbound from ovnkube-node-nq8wm...
2026-07-07T15:23:07.003828236Z INFO: Gathering OVN_Northbound from ovnkube-node-q5z8m...
2026-07-07T15:23:07.004121029Z INFO: Gathering OVN_Northbound from ovnkube-node-wfsf9...
2026-07-07T15:23:07.004528625Z INFO: Gathering OVN_Southbound from ovnkube-node-nq8wm...
2026-07-07T15:23:07.004890514Z INFO: Gathering OVN_Southbound from ovnkube-node-q5z8m...
2026-07-07T15:23:07.005181350Z INFO: Gathering OVN_Southbound from ovnkube-node-wfsf9...
2026-07-07T15:23:07.047716868Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T15:23:07.234538274Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T15:23:07.361201780Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.382584443Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.384235006Z Wrote inspect data to must-gather.
2026-07-07T15:23:07.398173572Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.427859590Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.429016814Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.431028057Z tar: Removing leading `/' from member names
2026-07-07T15:23:07.484259256Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T15:23:07.517405183Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:07.552080880Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:23:07.676235748Z Wrote inspect data to must-gather.
2026-07-07T15:23:07.778865301Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T15:23:07.779926101Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:07.782895902Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T15:23:07.963915172Z Wrote inspect data to must-gather.
2026-07-07T15:23:08.030184751Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:08.043382237Z INFO: Waiting for network log collection to complete ...
2026-07-07T15:23:08.043423060Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T15:23:08.043826130Z INFO: Copying ovnk databases complete.
2026-07-07T15:23:08.045339271Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-07T15:23:08.047552363Z ovnk_database_store/
2026-07-07T15:23:08.047552363Z ovnk_database_store/ovnkube-node-nq8wm_nbdb
2026-07-07T15:23:08.058795327Z ovnk_database_store/ovnkube-node-q5z8m_nbdb
2026-07-07T15:23:08.070714507Z ovnk_database_store/ovnkube-node-nq8wm_sbdb
2026-07-07T15:23:08.097302168Z ovnk_database_store/ovnkube-node-wfsf9_sbdb
2026-07-07T15:23:08.117482378Z ovnk_database_store/ovnkube-node-q5z8m_sbdb
2026-07-07T15:23:08.139818395Z ovnk_database_store/ovnkube-node-wfsf9_nbdb
2026-07-07T15:23:08.150975346Z INFO: Network log collection complete.
2026-07-07T15:23:08.208574900Z W0707 15:23:08.208521    1487 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-07T15:23:08.215238686Z Wrote inspect data to must-gather.
2026-07-07T15:23:08.241710438Z INFO: Getting status from alertmanager-main-0
2026-07-07T15:23:08.283229313Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:23:08.293599641Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:08.297111542Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T15:23:08.483890532Z Gathering data for ns/openshift-network-console...
2026-07-07T15:23:08.527114999Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:08.530106860Z Wrote inspect data to must-gather.
2026-07-07T15:23:08.754452397Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:08.758654993Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T15:23:08.835823310Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T15:23:08.838969679Z INFO: Fallback to identify the container image from release info
2026-07-07T15:23:08.986219199Z Gathering data for ns/openshift-console-operator...
2026-07-07T15:23:09.007285350Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:09.210653801Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:09.214653127Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T15:23:09.401407971Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:09.408309948Z Gathering data for ns/openshift-console...
2026-07-07T15:23:09.475971556Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T15:23:09.629881555Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:09.632187746Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T15:23:09.652124900Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T15:23:09.798599382Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:09.801966496Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T15:23:10.007787064Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:10.010597538Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T15:23:10.033701151Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:10.127141924Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T15:23:10.222292403Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:10.430843777Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:10.434473313Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T15:23:10.620632224Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:10.699378125Z Gathering data for ns/openshift-dns-operator...
2026-07-07T15:23:10.835375259Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:10.838603649Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T15:23:10.951702612Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T15:23:11.024629804Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:11.036715674Z Gathering data for ns/openshift-dns...
2026-07-07T15:23:11.198636548Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:11.201230489Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T15:23:11.373276863Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:11.563916090Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:23:11.719674696Z Gathering data for ns/openshift-image-registry...
2026-07-07T15:23:12.079939745Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T15:23:12.255100906Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T15:23:12.292436505Z Done executing Istio gather script
2026-07-07T15:23:12.628594962Z Gathering data for ns/openshift-ingress...
2026-07-07T15:23:12.987102433Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T15:23:13.213408528Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T15:23:13.394694972Z Gathering data for ns/openshift-insights...
2026-07-07T15:23:14.331836858Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T15:23:15.038297051Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:15.459828824Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T15:23:15.927868535Z Gathering data for ns/kserve...
2026-07-07T15:23:16.596776368Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T15:23:17.479035419Z Gathering data for ns/openshift-config...
2026-07-07T15:23:17.724589760Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T15:23:17.817030193Z Gathering data for ns/openshift-config-managed...
2026-07-07T15:23:18.003799777Z Collecting performance related data for node ip-10-0-130-141.ec2.internal
2026-07-07T15:23:18.011035849Z Collecting performance related data for node ip-10-0-138-241.ec2.internal
2026-07-07T15:23:18.018987198Z Collecting performance related data for node ip-10-0-143-168.ec2.internal
2026-07-07T15:23:18.337779044Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T15:23:18.737913092Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T15:23:19.147844238Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T15:23:19.597770949Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T15:23:19.979299787Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T15:23:20.044565321Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:20.275801840Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T15:23:20.647466194Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T15:23:21.044638224Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T15:23:21.411259691Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T15:23:21.960655728Z Gathering data for ns/openshift-multus...
2026-07-07T15:23:23.395805768Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T15:23:24.895870705Z Gathering data for ns/openshift-host-network...
2026-07-07T15:23:25.049128609Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:25.278824694Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T15:23:25.898302000Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T15:23:26.425954077Z Gathering data for ns/openshift-network-operator...
2026-07-07T15:23:26.627564445Z Collecting kubelet logs for node ip-10-0-130-141.ec2.internal
2026-07-07T15:23:26.629278646Z Collecting kubelet logs for node ip-10-0-138-241.ec2.internal
2026-07-07T15:23:26.631030500Z Collecting kubelet logs for node ip-10-0-143-168.ec2.internal
2026-07-07T15:23:26.930834594Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T15:23:27.030158561Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tv5km namespace
2026-07-07T15:23:27.035021913Z INFO: Node performance data collection complete.
2026-07-07T15:23:27.395205830Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T15:23:28.035577233Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T15:23:28.392715567Z Gathering data for ns/openshift-apiserver...
2026-07-07T15:23:28.818685388Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T15:23:29.102566601Z Gathering data for ns/openshift-controller-manager...
2026-07-07T15:23:29.402141055Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T15:23:30.054482539Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T15:23:30.095304675Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T15:23:30.492290579Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T15:23:30.818643888Z Gathering data for ns/openshift-service-ca...
2026-07-07T15:23:31.196261448Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T15:23:32.094214324Z Wrote inspect data to must-gather.
2026-07-07T15:23:32.094260794Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:23:32.094260794Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T15:23:32.224261958Z error: the server doesn't have a resource type "clusters"
2026-07-07T15:23:32.457449665Z Caches written to disk