2026-06-08T16:43:16.859622827Z [disk usage checker] Started
2026-06-08T16:43:16.863066591Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:17.075271596Z Gathering data for ns/openshift-cluster-version...
2026-06-08T16:43:17.222036366Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-08T16:43:17.315347674Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T16:43:17.399380278Z Gathering data for ns/default...
2026-06-08T16:43:17.694282893Z Gathering data for ns/openshift...
2026-06-08T16:43:18.103772107Z Gathering data for ns/kube-system...
2026-06-08T16:43:18.835469637Z Gathering data for ns/openshift-etcd...
2026-06-08T16:43:19.157735394Z Wrote inspect data to must-gather.
2026-06-08T16:43:19.157832672Z error: inspection completed with the errors occurred while gathering data:
2026-06-08T16:43:19.157832672Z     namespaces "assisted-installer" not found
2026-06-08T16:43:19.451925838Z Waiting on subprocesses to finish execution.
2026-06-08T16:43:19.457778159Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-08T16:43:19.463268013Z Executing Istio gather script
2026-06-08T16:43:19.466055755Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-08T16:43:19.466668932Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-08T16:43:19.473987773Z INFO: Gathering HAProxy config files
2026-06-08T16:43:19.488303528Z INFO: Waiting for node performance related collection to complete ...
2026-06-08T16:43:19.495069627Z INFO: Collecting host service logs for crio
2026-06-08T16:43:19.495490533Z INFO: Collecting host service logs for kubelet
2026-06-08T16:43:19.495858741Z INFO: Collecting host service logs for rpm-ostreed
2026-06-08T16:43:19.496263695Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-08T16:43:19.496613730Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-08T16:43:19.496975365Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-08T16:43:19.497386358Z INFO: Collecting host service logs for NetworkManager
2026-06-08T16:43:19.497759490Z INFO: Collecting host service logs for openvswitch
2026-06-08T16:43:19.498060812Z INFO: Collecting host service logs for ovs-configuration
2026-06-08T16:43:19.498434128Z INFO: Collecting host service logs for ovsdb-server
2026-06-08T16:43:19.498782989Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-08T16:43:19.499252208Z INFO: Waiting for worker host service log collection to complete ...
2026-06-08T16:43:20.132677695Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-08T16:43:20.132773372Z INFO: on-disk MachineConfig config collection complete.
2026-06-08T16:43:20.450506929Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-08T16:43:20.474714401Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-08T16:43:20.516456584Z No resources found in openshift-etcd namespace.
2026-06-08T16:43:20.545182016Z error: the server doesn't have a resource type "performanceprofile"
2026-06-08T16:43:20.682616153Z INFO: "metallb-operator" not detected. Skipping.
2026-06-08T16:43:20.701713455Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-08T16:43:20.718109527Z ERROR: No running kube-apiserver pods found
2026-06-08T16:43:20.806422409Z Wrote inspect data to must-gather.
2026-06-08T16:43:20.863638185Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-08T16:43:20.867387014Z INFO: Collecting Insights Archives from insights-operator-896549976-5qk5j
2026-06-08T16:43:20.867427855Z insights-runtime-extractor-b7w94
2026-06-08T16:43:20.867448163Z insights-runtime-extractor-fcpsf
2026-06-08T16:43:20.867468743Z insights-runtime-extractor-j97v8
2026-06-08T16:43:20.955440302Z Wrote inspect data to must-gather.
2026-06-08T16:43:21.025886779Z No resources found
2026-06-08T16:43:21.096149562Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-08T16:43:21.118218690Z INFO: Worker host service log collection to complete.
2026-06-08T16:43:21.197265463Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-08T16:43:21.197265463Z See 'oc rsync -h' for help and examples
2026-06-08T16:43:21.402180833Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-08T16:43:21.505070962Z Wrote inspect data to must-gather.
2026-06-08T16:43:21.580988515Z Gathering data for ns/openshift-monitoring...
2026-06-08T16:43:21.727724516Z No resources found
2026-06-08T16:43:21.842043499Z INFO: 'previous-logs' folder not found on ip-10-0-128-139.ec2.internal, skipping...
2026-06-08T16:43:21.866279927Z Wrote inspect data to must-gather.
2026-06-08T16:43:21.885836766Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-08T16:43:21.922616530Z Wrote inspect data to must-gather.
2026-06-08T16:43:21.937628015Z Wrote inspect data to must-gather.
2026-06-08T16:43:21.946548958Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:21.989355973Z Wrote inspect data to must-gather.
2026-06-08T16:43:22.082148467Z error: the server doesn't have a resource type "machineconfigs"
2026-06-08T16:43:22.092314175Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T16:43:22.127613047Z tar: Removing leading `/' from member names
2026-06-08T16:43:22.133369060Z INFO: HAProxy config collection complete.
2026-06-08T16:43:22.287974166Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-08T16:43:22.322826613Z INFO: Getting rules from prometheus-k8s-0
2026-06-08T16:43:22.342252364Z INFO: 'previous-logs' folder not found on ip-10-0-130-36.ec2.internal, skipping...
2026-06-08T16:43:22.353320084Z Wrote inspect data to must-gather.
2026-06-08T16:43:22.556866228Z Wrote inspect data to must-gather.
2026-06-08T16:43:22.579847097Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-08T16:43:22.644662410Z Wrote inspect data to must-gather.
2026-06-08T16:43:22.653383537Z INFO: 'previous-logs' folder not found on ip-10-0-135-41.ec2.internal, skipping...
2026-06-08T16:43:22.653438937Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-08T16:43:22.653449923Z INFO: Machine Config Daemon termination log collection complete.
2026-06-08T16:43:22.731560772Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-08T16:43:22.744715609Z INFO: Getting status/config from prometheus-k8s-0
2026-06-08T16:43:22.906117255Z Wrote inspect data to must-gather.
2026-06-08T16:43:22.960806116Z Wrote inspect data to must-gather.
2026-06-08T16:43:23.181728986Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-08T16:43:23.190052552Z Wrote inspect data to must-gather.
2026-06-08T16:43:23.202752023Z INFO: INTERCONNECT MODE
2026-06-08T16:43:23.202808080Z INFO: Gathering ovn-kubernetes DBs
2026-06-08T16:43:23.355733823Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-08T16:43:23.366606411Z INFO: Gathering OVN_Northbound from ovnkube-node-pn7t8...
2026-06-08T16:43:23.366855882Z INFO: Gathering OVN_Northbound from ovnkube-node-rhtts...
2026-06-08T16:43:23.367114992Z INFO: Gathering OVN_Northbound from ovnkube-node-tlrhd...
2026-06-08T16:43:23.367348370Z INFO: Gathering OVN_Southbound from ovnkube-node-pn7t8...
2026-06-08T16:43:23.367591309Z INFO: Gathering OVN_Southbound from ovnkube-node-rhtts...
2026-06-08T16:43:23.367834784Z INFO: Gathering OVN_Southbound from ovnkube-node-tlrhd...
2026-06-08T16:43:23.462262360Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-08T16:43:23.474928597Z Wrote inspect data to must-gather.
2026-06-08T16:43:23.547956084Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-08T16:43:23.694991691Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.715424399Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.725547048Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.744441957Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.812475443Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.826899921Z tar: Removing leading `/' from member names
2026-06-08T16:43:23.839062206Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:23.845676425Z Wrote inspect data to must-gather.
2026-06-08T16:43:23.850690973Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-08T16:43:24.085208108Z W0608 16:43:24.085159    1414 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-08T16:43:24.092678485Z Wrote inspect data to must-gather.
2026-06-08T16:43:24.112476974Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:24.114977079Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-08T16:43:24.160511230Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-08T16:43:24.179738221Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T16:43:24.302243435Z INFO: Waiting for network log collection to complete ...
2026-06-08T16:43:24.302279272Z INFO: Waiting for ovnk database copies to complete ...
2026-06-08T16:43:24.302659864Z INFO: Copying ovnk databases complete.
2026-06-08T16:43:24.304232946Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-06-08T16:43:24.307447702Z ovnk_database_store/
2026-06-08T16:43:24.307518111Z ovnk_database_store/ovnkube-node-rhtts_nbdb
2026-06-08T16:43:24.317919468Z ovnk_database_store/ovnkube-node-pn7t8_sbdb
2026-06-08T16:43:24.333696548Z ovnk_database_store/ovnkube-node-tlrhd_sbdb
2026-06-08T16:43:24.338434499Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:24.352512144Z ovnk_database_store/ovnkube-node-pn7t8_nbdb
2026-06-08T16:43:24.361233754Z ovnk_database_store/ovnkube-node-tlrhd_nbdb
2026-06-08T16:43:24.371153867Z ovnk_database_store/ovnkube-node-rhtts_sbdb
2026-06-08T16:43:24.391018620Z INFO: Network log collection complete.
2026-06-08T16:43:24.430821510Z Gathering data for ns/openshift-network-console...
2026-06-08T16:43:24.449386631Z Wrote inspect data to must-gather.
2026-06-08T16:43:24.577015226Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:24.579600777Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-08T16:43:24.579989500Z INFO: Getting status from alertmanager-main-0
2026-06-08T16:43:24.774861600Z Wrote inspect data to must-gather.
2026-06-08T16:43:24.784609010Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:24.948360406Z Gathering data for ns/openshift-console-operator...
2026-06-08T16:43:25.000702769Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:25.004760812Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-08T16:43:25.066469266Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-08T16:43:25.068576186Z INFO: Fallback to identify the container image from release info
2026-06-08T16:43:25.248897821Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:25.369585398Z Gathering data for ns/openshift-console...
2026-06-08T16:43:25.434560744Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:25.437292630Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-08T16:43:25.594452515Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:25.788615594Z Wrote inspect data to /must-gather/istio.
2026-06-08T16:43:25.872514322Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-08T16:43:25.916819338Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-08T16:43:26.011550138Z daemonset.apps/perf-node-gather-daemonset created
2026-06-08T16:43:26.140107509Z Waiting for performance profile collector pods to become ready: 1
2026-06-08T16:43:26.513041184Z Done executing Istio gather script
2026-06-08T16:43:26.519737040Z Gathering data for ns/openshift-dns-operator...
2026-06-08T16:43:26.807424641Z Gathering data for ns/openshift-dns...
2026-06-08T16:43:26.956038692Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:27.249294298Z Waiting for performance profile collector pods to become ready: 2
2026-06-08T16:43:27.428111840Z Gathering data for ns/openshift-image-registry...
2026-06-08T16:43:27.891554421Z Gathering data for ns/openshift-ingress-operator...
2026-06-08T16:43:28.217408547Z Gathering data for ns/openshift-ingress...
2026-06-08T16:43:28.370764720Z Waiting for performance profile collector pods to become ready: 3
2026-06-08T16:43:28.562094156Z Gathering data for ns/openshift-ingress-canary...
2026-06-08T16:43:28.963596737Z Gathering data for ns/openshift-insights...
2026-06-08T16:43:29.481852929Z Waiting for performance profile collector pods to become ready: 4
2026-06-08T16:43:30.597482251Z Waiting for performance profile collector pods to become ready: 5
2026-06-08T16:43:30.866297623Z Gathering data for ns/openshift-jobset-operator...
2026-06-08T16:43:31.653157116Z Gathering data for ns/openshift-config...
2026-06-08T16:43:31.721559773Z Waiting for performance profile collector pods to become ready: 6
2026-06-08T16:43:31.924643126Z Gathering data for ns/openshift-config-managed...
2026-06-08T16:43:31.961817496Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:32.342401707Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-08T16:43:32.604525371Z Gathering data for ns/openshift-kube-apiserver...
2026-06-08T16:43:32.845226080Z Waiting for performance profile collector pods to become ready: 7
2026-06-08T16:43:32.884716626Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-08T16:43:33.167497802Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-08T16:43:33.453731285Z Gathering data for ns/openshift-kube-scheduler...
2026-06-08T16:43:33.755521117Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-08T16:43:33.959588667Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-08T16:43:34.053543060Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-08T16:43:34.212750169Z Collecting performance related data for node ip-10-0-135-41.ec2.internal
2026-06-08T16:43:34.219264195Z Collecting performance related data for node ip-10-0-128-139.ec2.internal
2026-06-08T16:43:34.226351239Z Collecting performance related data for node ip-10-0-130-36.ec2.internal
2026-06-08T16:43:34.442194303Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-08T16:43:34.821732125Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-08T16:43:35.458112263Z Gathering data for ns/openshift-multus...
2026-06-08T16:43:36.967612369Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:37.047402815Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-08T16:43:38.497196293Z Gathering data for ns/openshift-host-network...
2026-06-08T16:43:38.862849513Z Gathering data for ns/openshift-network-diagnostics...
2026-06-08T16:43:39.411223521Z Gathering data for ns/openshift-network-node-identity...
2026-06-08T16:43:39.844265206Z Gathering data for ns/openshift-network-operator...
2026-06-08T16:43:40.239426384Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-08T16:43:40.504504290Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-08T16:43:41.009540916Z Gathering data for ns/openshift-apiserver-operator...
2026-06-08T16:43:41.301899348Z Gathering data for ns/openshift-apiserver...
2026-06-08T16:43:41.705934328Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-08T16:43:41.974240648Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T16:43:42.009721871Z Gathering data for ns/openshift-controller-manager...
2026-06-08T16:43:42.239774668Z Collecting kubelet logs for node ip-10-0-135-41.ec2.internal
2026-06-08T16:43:42.241587244Z Collecting kubelet logs for node ip-10-0-128-139.ec2.internal
2026-06-08T16:43:42.243531975Z Collecting kubelet logs for node ip-10-0-130-36.ec2.internal
2026-06-08T16:43:42.304405804Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-08T16:43:42.577169133Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kgrnn namespace
2026-06-08T16:43:42.584070670Z INFO: Node performance data collection complete.
2026-06-08T16:43:42.987900334Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-08T16:43:43.291862277Z Gathering data for ns/openshift-service-ca-operator...
2026-06-08T16:43:43.592009091Z Gathering data for ns/openshift-service-ca...
2026-06-08T16:43:43.937744471Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-08T16:43:44.809441367Z Wrote inspect data to must-gather.
2026-06-08T16:43:44.809490579Z error: inspection completed with the errors occurred while gathering data:
2026-06-08T16:43:44.809490579Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-08T16:43:44.941720922Z error: the server doesn't have a resource type "clusters"
2026-06-08T16:43:45.062204947Z Caches written to disk