2026-07-28T16:52:46.049663856Z [disk usage checker] Started
2026-07-28T16:52:46.053026457Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:52:46.246410579Z Gathering data for ns/openshift-cluster-version...
2026-07-28T16:52:46.395927820Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:52:46.491421645Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:52:46.575460087Z Gathering data for ns/default...
2026-07-28T16:52:46.875653540Z Gathering data for ns/openshift...
2026-07-28T16:52:47.323248224Z Gathering data for ns/kube-system...
2026-07-28T16:52:48.048582480Z Gathering data for ns/openshift-etcd...
2026-07-28T16:52:48.356646037Z Wrote inspect data to must-gather.
2026-07-28T16:52:48.356677334Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:52:48.356677334Z     namespaces "assisted-installer" not found
2026-07-28T16:52:48.805722339Z Waiting on subprocesses to finish execution.
2026-07-28T16:52:48.812052501Z INFO: Gathering HAProxy config files
2026-07-28T16:52:48.817197200Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T16:52:48.822664030Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T16:52:48.825261875Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T16:52:48.835693931Z Executing Istio gather script
2026-07-28T16:52:48.852490530Z INFO: Collecting host service logs for crio
2026-07-28T16:52:48.852799108Z INFO: Collecting host service logs for kubelet
2026-07-28T16:52:48.853099490Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T16:52:48.853416476Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T16:52:48.853725877Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T16:52:48.854022871Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T16:52:48.854320284Z INFO: Collecting host service logs for NetworkManager
2026-07-28T16:52:48.854636984Z INFO: Collecting host service logs for openvswitch
2026-07-28T16:52:48.854931418Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T16:52:48.855227608Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T16:52:48.855543112Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T16:52:48.855901524Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T16:52:48.924012147Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T16:52:49.378311631Z No resources found
2026-07-28T16:52:49.474183089Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T16:52:49.668503495Z No resources found in openshift-etcd namespace.
2026-07-28T16:52:49.670537527Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T16:52:49.811599120Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T16:52:49.904441004Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T16:52:49.904522596Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T16:52:50.139871597Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T16:52:50.176429600Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-qk5nf
2026-07-28T16:52:50.176488933Z insights-runtime-extractor-95bv8
2026-07-28T16:52:50.176521708Z insights-runtime-extractor-98fhb
2026-07-28T16:52:50.176551664Z insights-runtime-extractor-bfj8q
2026-07-28T16:52:50.190082077Z Wrote inspect data to must-gather.
2026-07-28T16:52:50.209141034Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T16:52:50.392535974Z INFO: Worker host service log collection to complete.
2026-07-28T16:52:50.394918548Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T16:52:50.416391964Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T16:52:50.474798400Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T16:52:50.496061298Z Wrote inspect data to must-gather.
2026-07-28T16:52:50.642991794Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T16:52:50.642991794Z See 'oc rsync -h' for help and examples
2026-07-28T16:52:50.869964760Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:52:50.893025519Z ERROR: No running kube-apiserver pods found
2026-07-28T16:52:51.071868872Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:52:51.124918047Z Wrote inspect data to must-gather.
2026-07-28T16:52:51.220576513Z Wrote inspect data to must-gather.
2026-07-28T16:52:51.251363085Z INFO: 'previous-logs' folder not found on ip-10-0-132-134.ec2.internal, skipping...
2026-07-28T16:52:51.351471053Z tar: Removing leading `/' from member names
2026-07-28T16:52:51.365427184Z INFO: HAProxy config collection complete.
2026-07-28T16:52:51.399428903Z No resources found
2026-07-28T16:52:51.408727148Z Gathering data for ns/openshift-monitoring...
2026-07-28T16:52:51.444999940Z Wrote inspect data to must-gather.
2026-07-28T16:52:51.452972068Z Wrote inspect data to must-gather.
2026-07-28T16:52:51.465704197Z Wrote inspect data to must-gather.
2026-07-28T16:52:51.468191664Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T16:52:51.484585542Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T16:52:51.604468760Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T16:52:51.764653668Z INFO: 'previous-logs' folder not found on ip-10-0-132-37.ec2.internal, skipping...
2026-07-28T16:52:51.819961444Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T16:52:51.821643442Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.002754413Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T16:52:52.077961713Z INFO: 'previous-logs' folder not found on ip-10-0-133-120.ec2.internal, skipping...
2026-07-28T16:52:52.078075183Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T16:52:52.078126737Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T16:52:52.124737573Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.126235645Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.296003482Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T16:52:52.320921712Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T16:52:52.405847309Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.488866012Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T16:52:52.543319090Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.719300093Z INFO: INTERCONNECT MODE
2026-07-28T16:52:52.719333206Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T16:52:52.735757811Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T16:52:52.794161891Z Wrote inspect data to must-gather.
2026-07-28T16:52:52.893917997Z INFO: Gathering OVN_Northbound from ovnkube-node-bqh6d...
2026-07-28T16:52:52.894194802Z INFO: Gathering OVN_Northbound from ovnkube-node-lngdw...
2026-07-28T16:52:52.894456318Z INFO: Gathering OVN_Northbound from ovnkube-node-pszj4...
2026-07-28T16:52:52.894757580Z INFO: Gathering OVN_Southbound from ovnkube-node-bqh6d...
2026-07-28T16:52:52.895035245Z INFO: Gathering OVN_Southbound from ovnkube-node-lngdw...
2026-07-28T16:52:52.895358385Z INFO: Gathering OVN_Southbound from ovnkube-node-pszj4...
2026-07-28T16:52:53.047829833Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T16:52:53.104301672Z Wrote inspect data to must-gather.
2026-07-28T16:52:53.291594619Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.298597280Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.299332020Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.305105236Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.309264806Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.328753746Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:52:53.336832484Z tar: Removing leading `/' from member names
2026-07-28T16:52:53.426172935Z Wrote inspect data to must-gather.
2026-07-28T16:52:53.440094452Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T16:52:53.612247941Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T16:52:53.691101606Z Wrote inspect data to must-gather.
2026-07-28T16:52:53.808330481Z INFO: Waiting for network log collection to complete ...
2026-07-28T16:52:53.808412567Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T16:52:53.808933629Z INFO: Copying ovnk databases complete.
2026-07-28T16:52:53.811708294Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-28T16:52:53.814872296Z ovnk_database_store/
2026-07-28T16:52:53.814918526Z ovnk_database_store/ovnkube-node-bqh6d_sbdb
2026-07-28T16:52:53.823264121Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:53.840256045Z ovnk_database_store/ovnkube-node-pszj4_nbdb
2026-07-28T16:52:53.851257815Z ovnk_database_store/ovnkube-node-pszj4_sbdb
2026-07-28T16:52:53.879018888Z ovnk_database_store/ovnkube-node-bqh6d_nbdb
2026-07-28T16:52:53.886613780Z ovnk_database_store/ovnkube-node-lngdw_nbdb
2026-07-28T16:52:53.894576874Z ovnk_database_store/ovnkube-node-lngdw_sbdb
2026-07-28T16:52:53.915944482Z INFO: Network log collection complete.
2026-07-28T16:52:53.945936303Z W0728 16:52:53.945897    1490 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T16:52:53.949198330Z INFO: Getting status from alertmanager-main-0
2026-07-28T16:52:53.951059685Z Wrote inspect data to must-gather.
2026-07-28T16:52:54.047900361Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:52:54.065696696Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:54.071444564Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T16:52:54.248692090Z Gathering data for ns/openshift-network-console...
2026-07-28T16:52:54.262770454Z Wrote inspect data to must-gather.
2026-07-28T16:52:54.288825127Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:54.516489923Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:54.521242849Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T16:52:54.558929527Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T16:52:54.562282307Z INFO: Fallback to identify the container image from release info
2026-07-28T16:52:54.745475846Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:54.751548135Z Gathering data for ns/openshift-console-operator...
2026-07-28T16:52:54.944720335Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:54.947570209Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T16:52:55.133271176Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T16:52:55.154173570Z Gathering data for ns/openshift-console...
2026-07-28T16:52:55.178034232Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:55.281401294Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T16:52:55.388363746Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:55.391609849Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T16:52:55.441727830Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T16:52:55.566203791Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:55.803286768Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:55.806465543Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T16:52:55.821111833Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T16:52:55.988198422Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:56.138826782Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:52:56.191935366Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:56.194589926Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T16:52:56.367925760Z Gathering data for ns/openshift-dns-operator...
2026-07-28T16:52:56.388653974Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:56.562538371Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T16:52:56.584716278Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:52:56.720655104Z Gathering data for ns/openshift-dns...
2026-07-28T16:52:57.254466785Z Done executing Istio gather script
2026-07-28T16:52:57.377254526Z Gathering data for ns/openshift-image-registry...
2026-07-28T16:52:57.684484123Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T16:52:57.897327679Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T16:52:58.229703136Z Gathering data for ns/openshift-ingress...
2026-07-28T16:52:58.576251970Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T16:52:58.804489335Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T16:52:58.980399039Z Gathering data for ns/openshift-insights...
2026-07-28T16:52:59.914740335Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T16:53:01.043570957Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T16:53:01.144074302Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:53:01.281598202Z Gathering data for ns/kserve...
2026-07-28T16:53:02.158283590Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T16:53:02.974926871Z Gathering data for ns/openshift-config...
2026-07-28T16:53:03.290528937Z Gathering data for ns/openshift-config-managed...
2026-07-28T16:53:03.290798489Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T16:53:03.558801719Z Collecting performance related data for node ip-10-0-132-37.ec2.internal
2026-07-28T16:53:03.565423397Z Collecting performance related data for node ip-10-0-132-134.ec2.internal
2026-07-28T16:53:03.573314310Z Collecting performance related data for node ip-10-0-133-120.ec2.internal
2026-07-28T16:53:03.818956303Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T16:53:04.187168597Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T16:53:04.548631253Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T16:53:04.915102659Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T16:53:05.244894082Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T16:53:05.509296570Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T16:53:05.828026164Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T16:53:06.148684843Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:53:06.222014111Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T16:53:06.603247587Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T16:53:07.142600954Z Gathering data for ns/openshift-multus...
2026-07-28T16:53:08.958921309Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T16:53:10.510052751Z Gathering data for ns/openshift-host-network...
2026-07-28T16:53:10.846057144Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T16:53:11.153936582Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:53:11.471919690Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T16:53:11.701020985Z Collecting kubelet logs for node ip-10-0-132-37.ec2.internal
2026-07-28T16:53:11.703286398Z Collecting kubelet logs for node ip-10-0-132-134.ec2.internal
2026-07-28T16:53:11.705419583Z Collecting kubelet logs for node ip-10-0-133-120.ec2.internal
2026-07-28T16:53:11.975103409Z Gathering data for ns/openshift-network-operator...
2026-07-28T16:53:12.120922440Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kvj4s namespace
2026-07-28T16:53:12.127950683Z INFO: Node performance data collection complete.
2026-07-28T16:53:12.422563350Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T16:53:12.689182059Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T16:53:13.176083350Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T16:53:13.486717463Z Gathering data for ns/openshift-apiserver...
2026-07-28T16:53:13.882026423Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T16:53:14.190223935Z Gathering data for ns/openshift-controller-manager...
2026-07-28T16:53:14.473222118Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T16:53:15.089201519Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T16:53:15.380360724Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T16:53:15.686259912Z Gathering data for ns/openshift-service-ca...
2026-07-28T16:53:16.055793883Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T16:53:16.158946569Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:53:16.924891939Z Wrote inspect data to must-gather.
2026-07-28T16:53:16.924937070Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:53:16.924937070Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T16:53:17.058208376Z error: the server doesn't have a resource type "clusters"
2026-07-28T16:53:17.196800175Z Caches written to disk