2026-07-07T17:51:55.791722629Z [disk usage checker] Started
2026-07-07T17:51:55.795073531Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:51:56.004019414Z Gathering data for ns/openshift-cluster-version...
2026-07-07T17:51:56.130367142Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T17:51:56.211297748Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T17:51:56.287292712Z Gathering data for ns/default...
2026-07-07T17:51:56.568113945Z Gathering data for ns/openshift...
2026-07-07T17:51:56.953754252Z Gathering data for ns/kube-system...
2026-07-07T17:51:57.636359399Z Gathering data for ns/openshift-etcd...
2026-07-07T17:51:57.932432462Z Wrote inspect data to must-gather.
2026-07-07T17:51:57.932469758Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T17:51:57.932469758Z     namespaces "assisted-installer" not found
2026-07-07T17:51:58.359600071Z Waiting on subprocesses to finish execution.
2026-07-07T17:51:58.364065628Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T17:51:58.366932055Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T17:51:58.373032570Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T17:51:58.378545864Z INFO: Collecting host service logs for crio
2026-07-07T17:51:58.378925237Z INFO: Collecting host service logs for kubelet
2026-07-07T17:51:58.379267720Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T17:51:58.379365599Z Executing Istio gather script
2026-07-07T17:51:58.379600011Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T17:51:58.379894814Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T17:51:58.380040898Z INFO: Gathering HAProxy config files
2026-07-07T17:51:58.380213108Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T17:51:58.380624971Z INFO: Collecting host service logs for NetworkManager
2026-07-07T17:51:58.380899135Z INFO: Collecting host service logs for openvswitch
2026-07-07T17:51:58.381188487Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T17:51:58.381490509Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T17:51:58.381997479Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T17:51:58.382336253Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T17:51:58.478473579Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T17:51:59.021843516Z No resources found
2026-07-07T17:51:59.085281555Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T17:51:59.148029940Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T17:51:59.370167186Z Wrote inspect data to must-gather.
2026-07-07T17:51:59.378191139Z No resources found in openshift-etcd namespace.
2026-07-07T17:51:59.492618516Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T17:51:59.514187043Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T17:51:59.568321768Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T17:51:59.568350987Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T17:51:59.644729919Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-cbt8k
2026-07-07T17:51:59.644844529Z insights-runtime-extractor-5t2tl
2026-07-07T17:51:59.644870666Z insights-runtime-extractor-w8kd5
2026-07-07T17:51:59.644903201Z insights-runtime-extractor-zg6rr
2026-07-07T17:51:59.818053635Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T17:51:59.837527073Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T17:51:59.893115829Z INFO: Worker host service log collection to complete.
2026-07-07T17:51:59.897325352Z ERROR: No running kube-apiserver pods found
2026-07-07T17:52:00.090369650Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T17:52:00.090369650Z See 'oc rsync -h' for help and examples
2026-07-07T17:52:00.282704127Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T17:52:00.415655865Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.474313418Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.507248739Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T17:52:00.557016960Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T17:52:00.615041886Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.616185442Z No resources found
2026-07-07T17:52:00.659985619Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.668768247Z Gathering data for ns/openshift-monitoring...
2026-07-07T17:52:00.710856678Z INFO: 'previous-logs' folder not found on ip-10-0-136-186.ec2.internal, skipping...
2026-07-07T17:52:00.824882564Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T17:52:00.827121259Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T17:52:00.829614356Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:52:00.837857381Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.937368202Z Wrote inspect data to must-gather.
2026-07-07T17:52:00.974578233Z tar: Removing leading `/' from member names
2026-07-07T17:52:00.980926213Z INFO: HAProxy config collection complete.
2026-07-07T17:52:01.031083164Z Wrote inspect data to must-gather.
2026-07-07T17:52:01.054719005Z INFO: 'previous-logs' folder not found on ip-10-0-138-123.ec2.internal, skipping...
2026-07-07T17:52:01.187803966Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T17:52:01.215203095Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T17:52:01.225715192Z Wrote inspect data to must-gather.
2026-07-07T17:52:01.330880368Z INFO: 'previous-logs' folder not found on ip-10-0-141-139.ec2.internal, skipping...
2026-07-07T17:52:01.330924662Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T17:52:01.330952346Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T17:52:01.395878558Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T17:52:01.465719593Z Wrote inspect data to must-gather.
2026-07-07T17:52:01.594147287Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T17:52:01.614535745Z Wrote inspect data to must-gather.
2026-07-07T17:52:01.760357058Z INFO: INTERCONNECT MODE
2026-07-07T17:52:01.760400436Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T17:52:01.782710930Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T17:52:01.833839350Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T17:52:01.915804398Z INFO: Gathering OVN_Northbound from ovnkube-node-fl9n6...
2026-07-07T17:52:01.916146732Z INFO: Gathering OVN_Northbound from ovnkube-node-n4mp6...
2026-07-07T17:52:01.916441677Z INFO: Gathering OVN_Northbound from ovnkube-node-pclps...
2026-07-07T17:52:01.916795217Z INFO: Gathering OVN_Southbound from ovnkube-node-fl9n6...
2026-07-07T17:52:01.917079728Z INFO: Gathering OVN_Southbound from ovnkube-node-n4mp6...
2026-07-07T17:52:01.917337547Z INFO: Gathering OVN_Southbound from ovnkube-node-pclps...
2026-07-07T17:52:02.037860847Z Wrote inspect data to must-gather.
2026-07-07T17:52:02.086951991Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:02.109238482Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T17:52:02.197760523Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.230763569Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.241207315Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.296722833Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.306734962Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.309785871Z tar: Removing leading `/' from member names
2026-07-07T17:52:02.383129701Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:02.385753552Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T17:52:02.396422534Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T17:52:02.398511166Z Wrote inspect data to must-gather.
2026-07-07T17:52:02.470445499Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T17:52:02.624413264Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:02.632264173Z Wrote inspect data to must-gather.
2026-07-07T17:52:02.657919057Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T17:52:02.735181642Z INFO: Waiting for network log collection to complete ...
2026-07-07T17:52:02.735226004Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T17:52:02.735744163Z INFO: Copying ovnk databases complete.
2026-07-07T17:52:02.737330279Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-07T17:52:02.739913774Z ovnk_database_store/
2026-07-07T17:52:02.739913774Z ovnk_database_store/ovnkube-node-fl9n6_nbdb
2026-07-07T17:52:02.750455777Z ovnk_database_store/ovnkube-node-fl9n6_sbdb
2026-07-07T17:52:02.767042416Z ovnk_database_store/ovnkube-node-pclps_nbdb
2026-07-07T17:52:02.773805698Z ovnk_database_store/ovnkube-node-pclps_sbdb
2026-07-07T17:52:02.793035557Z ovnk_database_store/ovnkube-node-n4mp6_nbdb
2026-07-07T17:52:02.798989848Z ovnk_database_store/ovnkube-node-n4mp6_sbdb
2026-07-07T17:52:02.816193256Z INFO: Network log collection complete.
2026-07-07T17:52:02.840963619Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:02.843533692Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T17:52:02.864626058Z Wrote inspect data to must-gather.
2026-07-07T17:52:02.918845809Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T17:52:03.028351189Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:03.063763285Z W0707 17:52:03.063710    1487 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-07T17:52:03.070570017Z Wrote inspect data to must-gather.
2026-07-07T17:52:03.136417832Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T17:52:03.244146535Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:03.247093204Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T17:52:03.354080435Z INFO: Getting status from alertmanager-main-0
2026-07-07T17:52:03.355070468Z Gathering data for ns/openshift-network-console...
2026-07-07T17:52:03.376458753Z Wrote inspect data to must-gather.
2026-07-07T17:52:03.501922128Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:03.673397041Z Wrote inspect data to must-gather.
2026-07-07T17:52:03.701266702Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:03.704465900Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T17:52:03.825390618Z Gathering data for ns/openshift-console-operator...
2026-07-07T17:52:03.870702127Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:03.944198552Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T17:52:03.946484529Z INFO: Fallback to identify the container image from release info
2026-07-07T17:52:04.067923025Z Wrote inspect data to /must-gather/istio.
2026-07-07T17:52:04.229071941Z Gathering data for ns/openshift-console...
2026-07-07T17:52:04.590878192Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T17:52:04.717983829Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T17:52:04.731892043Z Done executing Istio gather script
2026-07-07T17:52:04.753324933Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T17:52:04.839917969Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T17:52:05.271751032Z Gathering data for ns/openshift-dns-operator...
2026-07-07T17:52:05.539030990Z Gathering data for ns/openshift-dns...
2026-07-07T17:52:05.841027218Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:52:05.944949435Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T17:52:06.133436931Z Gathering data for ns/openshift-image-registry...
2026-07-07T17:52:06.584690636Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T17:52:06.888581361Z Gathering data for ns/openshift-ingress...
2026-07-07T17:52:07.061378756Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T17:52:07.210030690Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T17:52:07.586954591Z Gathering data for ns/openshift-insights...
2026-07-07T17:52:08.177917370Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T17:52:09.290577722Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T17:52:09.504108686Z Gathering data for ns/openshift-config...
2026-07-07T17:52:09.768042156Z Gathering data for ns/openshift-config-managed...
2026-07-07T17:52:10.182710537Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T17:52:10.405739392Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T17:52:10.465222971Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T17:52:10.817066731Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T17:52:10.846728454Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:52:11.151650524Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T17:52:11.506742301Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T17:52:11.528003057Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T17:52:11.793878218Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T17:52:12.130522591Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T17:52:12.471701819Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T17:52:12.640655878Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T17:52:12.791566249Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T17:52:12.900185962Z Collecting performance related data for node ip-10-0-141-139.ec2.internal
2026-07-07T17:52:12.906841097Z Collecting performance related data for node ip-10-0-136-186.ec2.internal
2026-07-07T17:52:12.913973367Z Collecting performance related data for node ip-10-0-138-123.ec2.internal
2026-07-07T17:52:13.365978392Z Gathering data for ns/openshift-multus...
2026-07-07T17:52:14.948541753Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T17:52:15.852708725Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:52:16.274127613Z Gathering data for ns/openshift-host-network...
2026-07-07T17:52:16.572452597Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T17:52:17.132725144Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T17:52:17.641465097Z Gathering data for ns/openshift-network-operator...
2026-07-07T17:52:18.214280596Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T17:52:18.534683315Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T17:52:19.036252585Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T17:52:19.323944101Z Gathering data for ns/openshift-apiserver...
2026-07-07T17:52:19.711356937Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T17:52:19.975211312Z Gathering data for ns/openshift-controller-manager...
2026-07-07T17:52:20.250233832Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T17:52:20.599665284Z Collecting kubelet logs for node ip-10-0-141-139.ec2.internal
2026-07-07T17:52:20.601332703Z Collecting kubelet logs for node ip-10-0-136-186.ec2.internal
2026-07-07T17:52:20.603098023Z Collecting kubelet logs for node ip-10-0-138-123.ec2.internal
2026-07-07T17:52:20.857322531Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-07T17:52:20.920159913Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-cwlwx namespace
2026-07-07T17:52:20.927086572Z INFO: Node performance data collection complete.
2026-07-07T17:52:20.930404084Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T17:52:21.235576432Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T17:52:21.545538110Z Gathering data for ns/openshift-service-ca...
2026-07-07T17:52:21.901101195Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T17:52:22.767588232Z Wrote inspect data to must-gather.
2026-07-07T17:52:22.767632748Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T17:52:22.767632748Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T17:52:22.892176153Z error: the server doesn't have a resource type "clusters"
2026-07-07T17:52:23.013812524Z Caches written to disk