2026-06-05T10:35:46.540236264Z [disk usage checker] Started
2026-06-05T10:35:46.544554603Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:35:46.732709328Z Gathering data for ns/openshift-cluster-version...
2026-06-05T10:35:46.856603064Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-05T10:35:46.938688945Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-05T10:35:47.015499972Z Gathering data for ns/default...
2026-06-05T10:35:47.304210150Z Gathering data for ns/openshift...
2026-06-05T10:35:47.698360184Z Gathering data for ns/kube-system...
2026-06-05T10:35:48.303661196Z Gathering data for ns/openshift-etcd...
2026-06-05T10:35:48.584704471Z Wrote inspect data to must-gather.
2026-06-05T10:35:48.584741171Z error: inspection completed with the errors occurred while gathering data:
2026-06-05T10:35:48.584741171Z     namespaces "assisted-installer" not found
2026-06-05T10:35:49.264279323Z Waiting on subprocesses to finish execution.
2026-06-05T10:35:49.274496139Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-05T10:35:49.274611409Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-05T10:35:49.275277389Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-05T10:35:49.279764093Z INFO: Gathering HAProxy config files
2026-06-05T10:35:49.286804275Z INFO: Collecting host service logs for crio
2026-06-05T10:35:49.287085950Z Executing Istio gather script
2026-06-05T10:35:49.287213143Z INFO: Collecting host service logs for kubelet
2026-06-05T10:35:49.287611949Z INFO: Collecting host service logs for rpm-ostreed
2026-06-05T10:35:49.287969307Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-05T10:35:49.288471576Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-05T10:35:49.288833247Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-05T10:35:49.289174250Z INFO: Collecting host service logs for NetworkManager
2026-06-05T10:35:49.289550840Z INFO: Collecting host service logs for openvswitch
2026-06-05T10:35:49.289895047Z INFO: Collecting host service logs for ovs-configuration
2026-06-05T10:35:49.290244980Z INFO: Collecting host service logs for ovsdb-server
2026-06-05T10:35:49.290698159Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-05T10:35:49.291109468Z INFO: Waiting for worker host service log collection to complete ...
2026-06-05T10:35:49.376312380Z INFO: Waiting for node performance related collection to complete ...
2026-06-05T10:35:49.683151887Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-05T10:35:49.934117582Z No resources found
2026-06-05T10:35:50.064146931Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-05T10:35:50.064146931Z INFO: on-disk MachineConfig config collection complete.
2026-06-05T10:35:50.136558329Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-05T10:35:50.312820559Z error: the server doesn't have a resource type "performanceprofile"
2026-06-05T10:35:50.361199243Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-05T10:35:50.437518591Z No resources found in openshift-etcd namespace.
2026-06-05T10:35:50.612465578Z INFO: Worker host service log collection to complete.
2026-06-05T10:35:50.622948185Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-05T10:35:50.654889167Z INFO: "metallb-operator" not detected. Skipping.
2026-06-05T10:35:50.702431618Z INFO: Collecting Insights Archives from insights-operator-896549976-vxb7n
2026-06-05T10:35:50.702496850Z insights-runtime-extractor-c6rm8
2026-06-05T10:35:50.702517356Z insights-runtime-extractor-g944m
2026-06-05T10:35:50.702536970Z insights-runtime-extractor-q9kxs
2026-06-05T10:35:50.717560827Z ERROR: No running kube-apiserver pods found
2026-06-05T10:35:51.133404121Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.185032552Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-05T10:35:51.224862234Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-05T10:35:51.350434868Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-05T10:35:51.379841193Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.515819945Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-05T10:35:51.581931837Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.657240968Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:35:51.750604950Z No resources found
2026-06-05T10:35:51.790290575Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.812376153Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.831058634Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-05T10:35:51.831058634Z See 'oc rsync -h' for help and examples
2026-06-05T10:35:51.842459720Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.877186343Z INFO: 'previous-logs' folder not found on ip-10-0-140-176.ec2.internal, skipping...
2026-06-05T10:35:51.988356842Z Gathering data for ns/openshift-monitoring...
2026-06-05T10:35:51.996602485Z Wrote inspect data to must-gather.
2026-06-05T10:35:51.996921096Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-05T10:35:52.052712640Z INFO: HAProxy config collection complete.
2026-06-05T10:35:52.067038606Z error: the server doesn't have a resource type "machineconfigs"
2026-06-05T10:35:52.178175305Z Wrote inspect data to must-gather.
2026-06-05T10:35:52.232461184Z INFO: 'previous-logs' folder not found on ip-10-0-141-131.ec2.internal, skipping...
2026-06-05T10:35:52.259560691Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-05T10:35:52.281356134Z INFO: Getting rules from prometheus-k8s-0
2026-06-05T10:35:52.484779937Z Wrote inspect data to must-gather.
2026-06-05T10:35:52.516404648Z Wrote inspect data to must-gather.
2026-06-05T10:35:52.536583887Z INFO: 'previous-logs' folder not found on ip-10-0-141-39.ec2.internal, skipping...
2026-06-05T10:35:52.536615064Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-05T10:35:52.536647878Z INFO: Machine Config Daemon termination log collection complete.
2026-06-05T10:35:52.702582157Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-05T10:35:52.713893917Z INFO: Getting status/config from prometheus-k8s-0
2026-06-05T10:35:52.750238473Z Wrote inspect data to must-gather.
2026-06-05T10:35:52.929201622Z Wrote inspect data to must-gather.
2026-06-05T10:35:53.051988689Z INFO: INTERCONNECT MODE
2026-06-05T10:35:53.052094670Z INFO: Gathering ovn-kubernetes DBs
2026-06-05T10:35:53.141207069Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-05T10:35:53.160873871Z Wrote inspect data to must-gather.
2026-06-05T10:35:53.256963102Z INFO: Gathering OVN_Northbound from ovnkube-node-8vwjd...
2026-06-05T10:35:53.257255959Z INFO: Gathering OVN_Northbound from ovnkube-node-hxfv9...
2026-06-05T10:35:53.257570066Z INFO: Gathering OVN_Northbound from ovnkube-node-s6n98...
2026-06-05T10:35:53.257834429Z INFO: Gathering OVN_Southbound from ovnkube-node-8vwjd...
2026-06-05T10:35:53.258069583Z INFO: Gathering OVN_Southbound from ovnkube-node-hxfv9...
2026-06-05T10:35:53.258316525Z INFO: Gathering OVN_Southbound from ovnkube-node-s6n98...
2026-06-05T10:35:53.354060017Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-05T10:35:53.454928434Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-05T10:35:53.460142572Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-05T10:35:53.537697535Z Wrote inspect data to must-gather.
2026-06-05T10:35:53.620202760Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.622623912Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.634411559Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.640475712Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.654514516Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.687370642Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:53.692518392Z tar: Removing leading `/' from member names
2026-06-05T10:35:53.823432494Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-05T10:35:53.833846925Z Wrote inspect data to must-gather.
2026-06-05T10:35:53.962634692Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:53.965137481Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-05T10:35:54.080481666Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-05T10:35:54.097403146Z W0605 10:35:54.097357    1454 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-05T10:35:54.104865887Z Wrote inspect data to must-gather.
2026-06-05T10:35:54.204275082Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:54.213146580Z INFO: Waiting for network log collection to complete ...
2026-06-05T10:35:54.213280815Z INFO: Waiting for ovnk database copies to complete ...
2026-06-05T10:35:54.213841762Z INFO: Copying ovnk databases complete.
2026-06-05T10:35:54.215900372Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-06-05T10:35:54.218799859Z ovnk_database_store/
2026-06-05T10:35:54.218851421Z ovnk_database_store/ovnkube-node-hxfv9_sbdb
2026-06-05T10:35:54.239487415Z ovnk_database_store/ovnkube-node-s6n98_sbdb
2026-06-05T10:35:54.249484134Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-05T10:35:54.258028648Z ovnk_database_store/ovnkube-node-hxfv9_nbdb
2026-06-05T10:35:54.268518541Z ovnk_database_store/ovnkube-node-8vwjd_nbdb
2026-06-05T10:35:54.279405304Z ovnk_database_store/ovnkube-node-s6n98_nbdb
2026-06-05T10:35:54.287072381Z ovnk_database_store/ovnkube-node-8vwjd_sbdb
2026-06-05T10:35:54.310102707Z INFO: Network log collection complete.
2026-06-05T10:35:54.452938074Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:54.456146732Z Wrote inspect data to must-gather.
2026-06-05T10:35:54.457041396Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-05T10:35:54.465366458Z Gathering data for ns/openshift-network-console...
2026-06-05T10:35:54.596632112Z INFO: Getting status from alertmanager-main-0
2026-06-05T10:35:54.671439390Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:54.788380512Z Wrote inspect data to must-gather.
2026-06-05T10:35:54.913829475Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:54.917351600Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-05T10:35:54.932700663Z Gathering data for ns/openshift-console-operator...
2026-06-05T10:35:55.094143257Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-05T10:35:55.096621445Z INFO: Fallback to identify the container image from release info
2026-06-05T10:35:55.161382071Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:55.328489390Z Gathering data for ns/openshift-console...
2026-06-05T10:35:55.368050537Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:55.370738894Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-05T10:35:55.531313545Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:55.733965096Z Wrote inspect data to /must-gather/istio.
2026-06-05T10:35:55.819071131Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-05T10:35:55.833735573Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-05T10:35:55.962056988Z daemonset.apps/perf-node-gather-daemonset created
2026-06-05T10:35:56.098609267Z Waiting for performance profile collector pods to become ready: 1
2026-06-05T10:35:56.372540528Z Gathering data for ns/openshift-dns-operator...
2026-06-05T10:35:56.518757975Z Done executing Istio gather script
2026-06-05T10:35:56.673730567Z Gathering data for ns/openshift-dns...
2026-06-05T10:35:56.677343561Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:35:57.195495976Z Gathering data for ns/openshift-image-registry...
2026-06-05T10:35:57.219356226Z Waiting for performance profile collector pods to become ready: 2
2026-06-05T10:35:57.608972398Z Gathering data for ns/openshift-ingress-operator...
2026-06-05T10:35:57.894449971Z Gathering data for ns/openshift-ingress...
2026-06-05T10:35:58.193375204Z Gathering data for ns/openshift-ingress-canary...
2026-06-05T10:35:58.344420458Z Waiting for performance profile collector pods to become ready: 3
2026-06-05T10:35:58.546696816Z Gathering data for ns/openshift-insights...
2026-06-05T10:35:59.449591247Z Waiting for performance profile collector pods to become ready: 4
2026-06-05T10:36:00.357011326Z Gathering data for ns/openshift-jobset-operator...
2026-06-05T10:36:00.564748205Z Waiting for performance profile collector pods to become ready: 5
2026-06-05T10:36:01.131159602Z Gathering data for ns/openshift-config...
2026-06-05T10:36:01.362473717Z Gathering data for ns/openshift-config-managed...
2026-06-05T10:36:01.672492620Z Waiting for performance profile collector pods to become ready: 6
2026-06-05T10:36:01.683257343Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:36:01.819957622Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-05T10:36:02.066236126Z Gathering data for ns/openshift-kube-apiserver...
2026-06-05T10:36:02.307049502Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-05T10:36:02.610228023Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-05T10:36:02.788832752Z Waiting for performance profile collector pods to become ready: 7
2026-06-05T10:36:02.878373776Z Gathering data for ns/openshift-kube-scheduler...
2026-06-05T10:36:03.116997921Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-05T10:36:03.426398255Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-05T10:36:03.718044940Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-05T10:36:03.901320599Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-05T10:36:04.024192349Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-05T10:36:04.173621253Z Collecting performance related data for node ip-10-0-140-176.ec2.internal
2026-06-05T10:36:04.182290838Z Collecting performance related data for node ip-10-0-141-39.ec2.internal
2026-06-05T10:36:04.192415464Z Collecting performance related data for node ip-10-0-141-131.ec2.internal
2026-06-05T10:36:04.612169812Z Gathering data for ns/openshift-multus...
2026-06-05T10:36:06.044812294Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-05T10:36:06.688490005Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:36:07.190279463Z Gathering data for ns/openshift-host-network...
2026-06-05T10:36:07.492905065Z Gathering data for ns/openshift-network-diagnostics...
2026-06-05T10:36:07.994879531Z Gathering data for ns/openshift-network-node-identity...
2026-06-05T10:36:08.383355061Z Gathering data for ns/openshift-network-operator...
2026-06-05T10:36:08.740236493Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-05T10:36:08.989980073Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-05T10:36:09.459455085Z Gathering data for ns/openshift-apiserver-operator...
2026-06-05T10:36:09.778075655Z Gathering data for ns/openshift-apiserver...
2026-06-05T10:36:10.145142688Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-05T10:36:10.382395678Z Gathering data for ns/openshift-controller-manager...
2026-06-05T10:36:10.635352852Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-05T10:36:11.288974467Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-05T10:36:11.543652525Z Gathering data for ns/openshift-service-ca-operator...
2026-06-05T10:36:11.695169812Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-05T10:36:11.819821562Z Gathering data for ns/openshift-service-ca...
2026-06-05T10:36:12.162891491Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-05T10:36:12.750080298Z Collecting kubelet logs for node ip-10-0-140-176.ec2.internal
2026-06-05T10:36:12.751960098Z Collecting kubelet logs for node ip-10-0-141-39.ec2.internal
2026-06-05T10:36:12.753973160Z Collecting kubelet logs for node ip-10-0-141-131.ec2.internal
2026-06-05T10:36:13.083853506Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-27v7q namespace
2026-06-05T10:36:13.088790152Z INFO: Node performance data collection complete.
2026-06-05T10:36:13.091680560Z Wrote inspect data to must-gather.
2026-06-05T10:36:13.091714089Z error: inspection completed with the errors occurred while gathering data:
2026-06-05T10:36:13.091714089Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-05T10:36:13.225797644Z error: the server doesn't have a resource type "clusters"
2026-06-05T10:36:13.366716545Z Caches written to disk