2026-07-02T14:10:03.732662840Z [disk usage checker] Started
2026-07-02T14:10:03.736577338Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:03.955533485Z Gathering data for ns/openshift-cluster-version...
2026-07-02T14:10:04.178554919Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:10:04.362677892Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:10:04.459915919Z Gathering data for ns/default...
2026-07-02T14:10:04.820970014Z Gathering data for ns/openshift...
2026-07-02T14:10:05.259298995Z Gathering data for ns/kube-system...
2026-07-02T14:10:06.036927743Z Gathering data for ns/openshift-etcd...
2026-07-02T14:10:06.380014145Z Wrote inspect data to must-gather.
2026-07-02T14:10:06.380049527Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:10:06.380049527Z     namespaces "assisted-installer" not found
2026-07-02T14:10:06.580324563Z Waiting on subprocesses to finish execution.
2026-07-02T14:10:06.587931451Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T14:10:06.588013507Z Executing Istio gather script
2026-07-02T14:10:06.598361003Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T14:10:06.599109218Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T14:10:06.615856655Z INFO: Collecting host service logs for crio
2026-07-02T14:10:06.616275114Z INFO: Collecting host service logs for kubelet
2026-07-02T14:10:06.616676326Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T14:10:06.617046879Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T14:10:06.617406726Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T14:10:06.618718399Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T14:10:06.619130050Z INFO: Collecting host service logs for NetworkManager
2026-07-02T14:10:06.619527084Z INFO: Collecting host service logs for openvswitch
2026-07-02T14:10:06.619587791Z INFO: Gathering HAProxy config files
2026-07-02T14:10:06.619909385Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T14:10:06.620307455Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T14:10:06.620702142Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T14:10:06.621151634Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T14:10:06.772694988Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T14:10:07.073179805Z No resources found
2026-07-02T14:10:07.202693992Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T14:10:07.305089097Z No resources found in openshift-etcd namespace.
2026-07-02T14:10:07.383386222Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T14:10:07.590550513Z Wrote inspect data to must-gather.
2026-07-02T14:10:07.797503704Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T14:10:07.797591019Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T14:10:07.807249785Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T14:10:07.909298636Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T14:10:08.129080090Z ERROR: No running kube-apiserver pods found
2026-07-02T14:10:08.298391116Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-5w28p
2026-07-02T14:10:08.298442845Z insights-runtime-extractor-dgrlg
2026-07-02T14:10:08.298461979Z insights-runtime-extractor-jshgg
2026-07-02T14:10:08.298495898Z insights-runtime-extractor-nj7wv
2026-07-02T14:10:08.552539512Z INFO: Worker host service log collection to complete.
2026-07-02T14:10:08.592335703Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T14:10:08.675305965Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T14:10:08.850169089Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T14:10:08.983334715Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T14:10:08.983334715Z See 'oc rsync -h' for help and examples
2026-07-02T14:10:09.016688888Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.040965992Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:09.070941638Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.114578237Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:10:09.151049715Z Gathering data for ns/openshift-monitoring...
2026-07-02T14:10:09.183604215Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.188289428Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T14:10:09.347672113Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.364269820Z No resources found
2026-07-02T14:10:09.377057405Z INFO: 'previous-logs' folder not found on ip-10-0-133-96.ec2.internal, skipping...
2026-07-02T14:10:09.399647904Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T14:10:09.404811095Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.498510959Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.527980985Z tar: Removing leading `/' from member names
2026-07-02T14:10:09.538854779Z INFO: HAProxy config collection complete.
2026-07-02T14:10:09.650303209Z Wrote inspect data to must-gather.
2026-07-02T14:10:09.721994864Z INFO: 'previous-logs' folder not found on ip-10-0-136-2.ec2.internal, skipping...
2026-07-02T14:10:09.726848290Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T14:10:09.924834501Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T14:10:09.933075676Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T14:10:09.974594178Z Wrote inspect data to must-gather.
2026-07-02T14:10:10.034649906Z INFO: 'previous-logs' folder not found on ip-10-0-140-174.ec2.internal, skipping...
2026-07-02T14:10:10.034687834Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T14:10:10.034716160Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T14:10:10.183148153Z Wrote inspect data to must-gather.
2026-07-02T14:10:10.193989468Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T14:10:10.266352921Z Wrote inspect data to must-gather.
2026-07-02T14:10:10.351739265Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T14:10:10.379395959Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T14:10:10.596393150Z INFO: INTERCONNECT MODE
2026-07-02T14:10:10.596419367Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T14:10:10.598368003Z Wrote inspect data to must-gather.
2026-07-02T14:10:10.787271092Z INFO: Gathering OVN_Northbound from ovnkube-node-7l9rh...
2026-07-02T14:10:10.787476885Z INFO: Gathering OVN_Northbound from ovnkube-node-j86jl...
2026-07-02T14:10:10.787737754Z INFO: Gathering OVN_Northbound from ovnkube-node-zf9mn...
2026-07-02T14:10:10.788091020Z INFO: Gathering OVN_Southbound from ovnkube-node-7l9rh...
2026-07-02T14:10:10.788343246Z INFO: Gathering OVN_Southbound from ovnkube-node-j86jl...
2026-07-02T14:10:10.788634817Z INFO: Gathering OVN_Southbound from ovnkube-node-zf9mn...
2026-07-02T14:10:10.838946385Z Wrote inspect data to must-gather.
2026-07-02T14:10:10.878613564Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T14:10:11.121321682Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.123546399Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.165660976Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.177762029Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.198551521Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.210247870Z tar: Removing leading `/' from member names
2026-07-02T14:10:11.235040714Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:10:11.272446304Z Wrote inspect data to must-gather.
2026-07-02T14:10:11.344005687Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T14:10:11.457927715Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T14:10:11.539630020Z Wrote inspect data to must-gather.
2026-07-02T14:10:11.635421016Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T14:10:11.703923892Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:11.732087765Z INFO: Waiting for network log collection to complete ...
2026-07-02T14:10:11.732201359Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T14:10:11.732734324Z INFO: Copying ovnk databases complete.
2026-07-02T14:10:11.734569654Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-02T14:10:11.739100430Z ovnk_database_store/
2026-07-02T14:10:11.739120570Z ovnk_database_store/ovnkube-node-zf9mn_nbdb
2026-07-02T14:10:11.749626643Z ovnk_database_store/ovnkube-node-j86jl_sbdb
2026-07-02T14:10:11.775931025Z ovnk_database_store/ovnkube-node-7l9rh_nbdb
2026-07-02T14:10:11.783654640Z ovnk_database_store/ovnkube-node-7l9rh_sbdb
2026-07-02T14:10:11.784209134Z W0702 14:10:11.784183    1455 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T14:10:11.792053495Z Wrote inspect data to must-gather.
2026-07-02T14:10:11.804831488Z ovnk_database_store/ovnkube-node-j86jl_nbdb
2026-07-02T14:10:11.814871609Z ovnk_database_store/ovnkube-node-zf9mn_sbdb
2026-07-02T14:10:11.839397064Z INFO: Network log collection complete.
2026-07-02T14:10:11.883941573Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:10:11.935767807Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T14:10:11.940014571Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:11.943938629Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T14:10:12.090243919Z Gathering data for ns/openshift-network-console...
2026-07-02T14:10:12.107659456Z Wrote inspect data to must-gather.
2026-07-02T14:10:12.169792490Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:12.374338089Z INFO: Getting status from alertmanager-main-0
2026-07-02T14:10:12.421505768Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:12.424639114Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T14:10:12.439558757Z Wrote inspect data to must-gather.
2026-07-02T14:10:12.615442470Z Gathering data for ns/openshift-console-operator...
2026-07-02T14:10:12.648428096Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:12.749596578Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T14:10:12.752469949Z INFO: Fallback to identify the container image from release info
2026-07-02T14:10:12.867785807Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:12.870385705Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T14:10:13.041998347Z Gathering data for ns/openshift-console...
2026-07-02T14:10:13.097650028Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:13.300276016Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:13.302711250Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T14:10:13.477900417Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:13.588047581Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T14:10:13.694591836Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:13.697196204Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T14:10:13.861193725Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.053201455Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:14.074844146Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.077906244Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T14:10:14.142171291Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-02T14:10:14.171695686Z Gathering data for ns/openshift-dns-operator...
2026-07-02T14:10:14.278463194Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.299498262Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T14:10:14.435172039Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T14:10:14.482628761Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.485187195Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T14:10:14.557896508Z Gathering data for ns/openshift-dns...
2026-07-02T14:10:14.657011234Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.870328213Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:14.874528225Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T14:10:15.051388086Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:15.255633323Z Gathering data for ns/openshift-image-registry...
2026-07-02T14:10:15.259150365Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:15.261395212Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T14:10:15.434506179Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:15.571977600Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T14:10:15.636862157Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:10:15.793268739Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T14:10:16.166342477Z Gathering data for ns/openshift-ingress...
2026-07-02T14:10:16.346865554Z Done executing Istio gather script
2026-07-02T14:10:16.522311928Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T14:10:16.695660068Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T14:10:16.963395155Z Gathering data for ns/openshift-insights...
2026-07-02T14:10:17.810849322Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T14:10:18.927659717Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T14:10:19.058442526Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:19.342456634Z Gathering data for ns/kserve...
2026-07-02T14:10:20.057044290Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T14:10:20.966604001Z Gathering data for ns/openshift-config...
2026-07-02T14:10:21.178130927Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T14:10:21.266937892Z Gathering data for ns/openshift-config-managed...
2026-07-02T14:10:21.745466105Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T14:10:22.018692358Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T14:10:22.307371655Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T14:10:22.309633983Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T14:10:22.584083026Z Collecting performance related data for node ip-10-0-140-174.ec2.internal
2026-07-02T14:10:22.593424927Z Collecting performance related data for node ip-10-0-136-2.ec2.internal
2026-07-02T14:10:22.602858183Z Collecting performance related data for node ip-10-0-133-96.ec2.internal
2026-07-02T14:10:22.647860123Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T14:10:23.054635968Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T14:10:23.373841541Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T14:10:23.799523332Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T14:10:24.064096428Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:24.254756762Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T14:10:24.599158307Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T14:10:25.163182455Z Gathering data for ns/openshift-multus...
2026-07-02T14:10:26.705149051Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T14:10:28.180952734Z Gathering data for ns/openshift-host-network...
2026-07-02T14:10:28.521869784Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T14:10:29.069421713Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:29.107148320Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T14:10:29.539571724Z Gathering data for ns/openshift-network-operator...
2026-07-02T14:10:29.984607141Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T14:10:30.264140518Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T14:10:30.810715677Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T14:10:30.978780956Z Collecting kubelet logs for node ip-10-0-140-174.ec2.internal
2026-07-02T14:10:30.980551912Z Collecting kubelet logs for node ip-10-0-136-2.ec2.internal
2026-07-02T14:10:30.982708346Z Collecting kubelet logs for node ip-10-0-133-96.ec2.internal
2026-07-02T14:10:31.284943682Z Gathering data for ns/openshift-apiserver...
2026-07-02T14:10:31.429933987Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-f6j6j namespace
2026-07-02T14:10:31.435514081Z INFO: Node performance data collection complete.
2026-07-02T14:10:31.775724508Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T14:10:32.112247831Z Gathering data for ns/openshift-controller-manager...
2026-07-02T14:10:32.432424155Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T14:10:33.159758308Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T14:10:33.522192315Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T14:10:33.868938669Z Gathering data for ns/openshift-service-ca...
2026-07-02T14:10:34.076996008Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-02T14:10:34.234634540Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T14:10:35.155955528Z Wrote inspect data to must-gather.
2026-07-02T14:10:35.155998730Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:10:35.155998730Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T14:10:35.281804509Z error: the server doesn't have a resource type "clusters"
2026-07-02T14:10:35.424272251Z Caches written to disk