2026-09-17T16:07:28.251022813Z [disk usage checker] Started
2026-09-17T16:07:28.254500084Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:28.453637052Z Gathering data for ns/openshift-cluster-version...
2026-09-17T16:07:28.601156741Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-17T16:07:28.703519001Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T16:07:28.800338606Z Gathering data for ns/default...
2026-09-17T16:07:29.169678575Z Gathering data for ns/openshift...
2026-09-17T16:07:29.634783852Z Gathering data for ns/kube-system...
2026-09-17T16:07:30.390705782Z Gathering data for ns/openshift-etcd...
2026-09-17T16:07:30.731328430Z Wrote inspect data to must-gather.
2026-09-17T16:07:30.731419954Z error: inspection completed with the errors occurred while gathering data:
2026-09-17T16:07:30.731419954Z     namespaces "assisted-installer" not found
2026-09-17T16:07:31.009271322Z Waiting on subprocesses to finish execution.
2026-09-17T16:07:31.015738265Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-17T16:07:31.017760138Z INFO: Gathering HAProxy config files
2026-09-17T16:07:31.020474659Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-17T16:07:31.029702295Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-17T16:07:31.033102525Z Executing Istio gather script
2026-09-17T16:07:31.045363555Z INFO: Collecting host service logs for crio
2026-09-17T16:07:31.045680208Z INFO: Collecting host service logs for kubelet
2026-09-17T16:07:31.046027005Z INFO: Collecting host service logs for rpm-ostreed
2026-09-17T16:07:31.046377766Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-17T16:07:31.046693453Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-17T16:07:31.047008214Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-17T16:07:31.047318842Z INFO: Collecting host service logs for NetworkManager
2026-09-17T16:07:31.047618474Z INFO: Collecting host service logs for openvswitch
2026-09-17T16:07:31.047921429Z INFO: Collecting host service logs for ovs-configuration
2026-09-17T16:07:31.048237387Z INFO: Collecting host service logs for ovsdb-server
2026-09-17T16:07:31.048536492Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-17T16:07:31.048889126Z INFO: Waiting for worker host service log collection to complete ...
2026-09-17T16:07:31.084484503Z INFO: Waiting for node performance related collection to complete ...
2026-09-17T16:07:31.480297039Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-17T16:07:31.551357446Z No resources found in openshift-etcd namespace.
2026-09-17T16:07:31.832010864Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-17T16:07:31.832097648Z INFO: on-disk MachineConfig config collection complete.
2026-09-17T16:07:31.954187920Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-17T16:07:32.253426508Z Wrote inspect data to must-gather.
2026-09-17T16:07:32.295357127Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-17T16:07:32.391864564Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-17T16:07:32.408831212Z error: the server doesn't have a resource type "performanceprofile"
2026-09-17T16:07:32.477121727Z INFO: Collecting Insights Archives from insights-operator-5844b856db-gvzhp
2026-09-17T16:07:32.477197593Z insights-runtime-extractor-mrj2w
2026-09-17T16:07:32.477220379Z insights-runtime-extractor-wvvp2
2026-09-17T16:07:32.477240387Z insights-runtime-extractor-xthph
2026-09-17T16:07:32.621410132Z INFO: "metallb-operator" not detected. Skipping.
2026-09-17T16:07:32.647599629Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-17T16:07:32.960964172Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-17T16:07:33.223661731Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.303179401Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-17T16:07:33.303179401Z See 'oc rsync -h' for help and examples
2026-09-17T16:07:33.351422474Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-17T16:07:33.355307807Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.425922405Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:33.540975400Z tar: Removing leading `/' from member names
2026-09-17T16:07:33.550160448Z INFO: HAProxy config collection complete.
2026-09-17T16:07:33.592485760Z INFO: Worker host service log collection to complete.
2026-09-17T16:07:33.612888578Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.616285776Z ERROR: No running kube-apiserver pods found
2026-09-17T16:07:33.619199918Z INFO: 'previous-logs' folder not found on ip-10-0-129-204.ec2.internal, skipping...
2026-09-17T16:07:33.619698197Z No resources found
2026-09-17T16:07:33.632572560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T16:07:33.647762233Z Gathering data for ns/openshift-monitoring...
2026-09-17T16:07:33.667338730Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.674858795Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.677369981Z No resources found
2026-09-17T16:07:33.804261456Z Wrote inspect data to must-gather.
2026-09-17T16:07:33.871615616Z error: the server doesn't have a resource type "machineconfigs"
2026-09-17T16:07:33.970261204Z INFO: 'previous-logs' folder not found on ip-10-0-133-29.ec2.internal, skipping...
2026-09-17T16:07:34.017650377Z INFO: Getting rules from prometheus-k8s-0
2026-09-17T16:07:34.082032157Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-17T16:07:34.131664805Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.261245168Z INFO: 'previous-logs' folder not found on ip-10-0-143-66.ec2.internal, skipping...
2026-09-17T16:07:34.261312382Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-17T16:07:34.261377132Z INFO: Machine Config Daemon termination log collection complete.
2026-09-17T16:07:34.331267463Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.393204137Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.495157757Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-17T16:07:34.539836072Z INFO: Getting status/config from prometheus-k8s-0
2026-09-17T16:07:34.670474884Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.750684516Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.823983747Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-17T16:07:34.978403484Z Wrote inspect data to must-gather.
2026-09-17T16:07:34.990026547Z INFO: INTERCONNECT MODE
2026-09-17T16:07:34.990055263Z INFO: Gathering ovn-kubernetes DBs
2026-09-17T16:07:34.999185069Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-17T16:07:35.170355622Z INFO: Gathering OVN_Northbound from ovnkube-node-926t8...
2026-09-17T16:07:35.170573613Z INFO: Gathering OVN_Northbound from ovnkube-node-j5hdl...
2026-09-17T16:07:35.170795120Z INFO: Gathering OVN_Northbound from ovnkube-node-w5sh4...
2026-09-17T16:07:35.171139328Z INFO: Gathering OVN_Southbound from ovnkube-node-926t8...
2026-09-17T16:07:35.171589111Z INFO: Gathering OVN_Southbound from ovnkube-node-j5hdl...
2026-09-17T16:07:35.171877051Z INFO: Gathering OVN_Southbound from ovnkube-node-w5sh4...
2026-09-17T16:07:35.240211632Z Wrote inspect data to must-gather.
2026-09-17T16:07:35.253177456Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-17T16:07:35.518630138Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.520199379Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.543601331Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.566135564Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.578394217Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.579043732Z tar: Removing leading `/' from member names
2026-09-17T16:07:35.611295515Z Wrote inspect data to must-gather.
2026-09-17T16:07:35.662671573Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-17T16:07:35.668458545Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-17T16:07:35.855570327Z Wrote inspect data to must-gather.
2026-09-17T16:07:35.966108187Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-17T16:07:36.088079643Z INFO: Waiting for network log collection to complete ...
2026-09-17T16:07:36.088130661Z INFO: Waiting for ovnk database copies to complete ...
2026-09-17T16:07:36.088615894Z INFO: Copying ovnk databases complete.
2026-09-17T16:07:36.089698138Z Wrote inspect data to must-gather.
2026-09-17T16:07:36.090868572Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-09-17T16:07:36.094771143Z ovnk_database_store/
2026-09-17T16:07:36.094816063Z ovnk_database_store/ovnkube-node-w5sh4_sbdb
2026-09-17T16:07:36.120236917Z ovnk_database_store/ovnkube-node-w5sh4_nbdb
2026-09-17T16:07:36.129222664Z ovnk_database_store/ovnkube-node-j5hdl_sbdb
2026-09-17T16:07:36.151443952Z ovnk_database_store/ovnkube-node-926t8_nbdb
2026-09-17T16:07:36.160488498Z ovnk_database_store/ovnkube-node-926t8_sbdb
2026-09-17T16:07:36.170719096Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-17T16:07:36.181932953Z ovnk_database_store/ovnkube-node-j5hdl_nbdb
2026-09-17T16:07:36.193015487Z INFO: Network log collection complete.
2026-09-17T16:07:36.305011538Z W0917 16:07:36.304968    1489 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-17T16:07:36.310636781Z Wrote inspect data to must-gather.
2026-09-17T16:07:36.344412192Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T16:07:36.366710014Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:36.396619840Z INFO: Getting status from alertmanager-main-0
2026-09-17T16:07:36.562213982Z Gathering data for ns/openshift-network-console...
2026-09-17T16:07:36.608018163Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:36.612401282Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-17T16:07:36.620109510Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-17T16:07:36.622911197Z INFO: Fallback to identify the container image from release info
2026-09-17T16:07:36.804875935Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.011896476Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.014153854Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-17T16:07:37.085395481Z Gathering data for ns/openshift-console-operator...
2026-09-17T16:07:37.226088274Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.320113381Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-17T16:07:37.425643997Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.429116489Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-17T16:07:37.479279610Z daemonset.apps/perf-node-gather-daemonset created
2026-09-17T16:07:37.558928579Z Gathering data for ns/openshift-console...
2026-09-17T16:07:37.624451673Z Waiting for performance profile collector pods to become ready: 1
2026-09-17T16:07:37.665168297Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.854146996Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:37.857103354Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-17T16:07:38.061729762Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:38.162068820Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-17T16:07:38.281406868Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:38.284091315Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-17T16:07:38.434104223Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:38.442721915Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:38.649037605Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:38.651891414Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-17T16:07:38.765618579Z Waiting for performance profile collector pods to become ready: 2
2026-09-17T16:07:38.765790787Z Gathering data for ns/openshift-dns-operator...
2026-09-17T16:07:38.844757905Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:39.022744152Z Wrote inspect data to /must-gather/istio.
2026-09-17T16:07:39.172249480Z Gathering data for ns/openshift-dns...
2026-09-17T16:07:39.660518566Z Done executing Istio gather script
2026-09-17T16:07:39.897493597Z Waiting for performance profile collector pods to become ready: 3
2026-09-17T16:07:39.937516819Z Gathering data for ns/openshift-image-registry...
2026-09-17T16:07:40.473892253Z Gathering data for ns/openshift-ingress-operator...
2026-09-17T16:07:40.834707698Z Gathering data for ns/openshift-ingress...
2026-09-17T16:07:41.015419640Z Waiting for performance profile collector pods to become ready: 4
2026-09-17T16:07:41.192172642Z Gathering data for ns/openshift-ingress-canary...
2026-09-17T16:07:41.638931887Z Gathering data for ns/openshift-insights...
2026-09-17T16:07:42.130725237Z Waiting for performance profile collector pods to become ready: 5
2026-09-17T16:07:43.242710071Z Waiting for performance profile collector pods to become ready: 6
2026-09-17T16:07:43.438928013Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:44.147820110Z Gathering data for ns/kserve...
2026-09-17T16:07:44.360601035Z Waiting for performance profile collector pods to become ready: 7
2026-09-17T16:07:45.493560063Z Waiting for performance profile collector pods to become ready: 8
2026-09-17T16:07:45.901787365Z Gathering data for ns/openshift-config...
2026-09-17T16:07:46.208268066Z Gathering data for ns/openshift-config-managed...
2026-09-17T16:07:46.624368974Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-17T16:07:46.696553729Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-17T16:07:46.883083583Z Collecting performance related data for node ip-10-0-143-66.ec2.internal
2026-09-17T16:07:46.892085708Z Collecting performance related data for node ip-10-0-129-204.ec2.internal
2026-09-17T16:07:46.901067215Z Collecting performance related data for node ip-10-0-133-29.ec2.internal
2026-09-17T16:07:47.052583700Z Gathering data for ns/openshift-kube-apiserver...
2026-09-17T16:07:47.396067543Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-17T16:07:47.823176446Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-17T16:07:48.297063949Z Gathering data for ns/openshift-kube-scheduler...
2026-09-17T16:07:48.444444694Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:48.659151997Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-17T16:07:48.984452859Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-17T16:07:49.452075863Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-17T16:07:49.918928069Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-17T16:07:50.511757563Z Gathering data for ns/openshift-multus...
2026-09-17T16:07:52.207871378Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-17T16:07:53.450414069Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:53.661919615Z Gathering data for ns/openshift-host-network...
2026-09-17T16:07:54.024747897Z Gathering data for ns/openshift-network-diagnostics...
2026-09-17T16:07:54.694400057Z Gathering data for ns/openshift-network-node-identity...
2026-09-17T16:07:55.035273077Z Collecting kubelet logs for node ip-10-0-143-66.ec2.internal
2026-09-17T16:07:55.037188740Z Collecting kubelet logs for node ip-10-0-129-204.ec2.internal
2026-09-17T16:07:55.039346098Z Collecting kubelet logs for node ip-10-0-133-29.ec2.internal
2026-09-17T16:07:55.206977208Z Gathering data for ns/openshift-network-operator...
2026-09-17T16:07:55.427161347Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-nxdvm namespace
2026-09-17T16:07:55.432526849Z INFO: Node performance data collection complete.
2026-09-17T16:07:55.703061048Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-17T16:07:56.012500966Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-17T16:07:56.543172946Z Gathering data for ns/openshift-apiserver-operator...
2026-09-17T16:07:56.876232669Z Gathering data for ns/openshift-apiserver...
2026-09-17T16:07:57.291768250Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-17T16:07:57.642062567Z Gathering data for ns/openshift-controller-manager...
2026-09-17T16:07:57.982897089Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-17T16:07:58.455554927Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-09-17T16:07:58.670264665Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-17T16:07:58.996103220Z Gathering data for ns/openshift-service-ca-operator...
2026-09-17T16:07:59.339444377Z Gathering data for ns/openshift-service-ca...
2026-09-17T16:07:59.732539994Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-17T16:08:00.689901293Z Wrote inspect data to must-gather.
2026-09-17T16:08:00.689943476Z error: inspection completed with the errors occurred while gathering data:
2026-09-17T16:08:00.689943476Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-17T16:08:00.818898135Z error: the server doesn't have a resource type "clusters"
2026-09-17T16:08:01.012338667Z Caches written to disk