2026-07-24T18:11:22.773086681Z [disk usage checker] Started
2026-07-24T18:11:22.776573894Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:22.998316636Z Gathering data for ns/openshift-cluster-version...
2026-07-24T18:11:23.152141936Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-24T18:11:23.255066730Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T18:11:23.359922468Z Gathering data for ns/default...
2026-07-24T18:11:23.688844626Z Gathering data for ns/openshift...
2026-07-24T18:11:24.187073740Z Gathering data for ns/kube-system...
2026-07-24T18:11:24.984772996Z Gathering data for ns/openshift-etcd...
2026-07-24T18:11:25.330846147Z Wrote inspect data to must-gather.
2026-07-24T18:11:25.331014679Z error: inspection completed with the errors occurred while gathering data:
2026-07-24T18:11:25.331014679Z     namespaces "assisted-installer" not found
2026-07-24T18:11:25.879097903Z Waiting on subprocesses to finish execution.
2026-07-24T18:11:25.887489308Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-24T18:11:25.889429115Z INFO: Gathering HAProxy config files
2026-07-24T18:11:25.892529313Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-24T18:11:25.899578580Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-24T18:11:25.908278587Z Executing Istio gather script
2026-07-24T18:11:25.917277158Z INFO: Waiting for node performance related collection to complete ...
2026-07-24T18:11:25.960236764Z INFO: Collecting host service logs for crio
2026-07-24T18:11:25.960542742Z INFO: Collecting host service logs for kubelet
2026-07-24T18:11:25.960846855Z INFO: Collecting host service logs for rpm-ostreed
2026-07-24T18:11:25.961141742Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-24T18:11:25.961443680Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-24T18:11:25.961726125Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-24T18:11:25.962009711Z INFO: Collecting host service logs for NetworkManager
2026-07-24T18:11:25.962339193Z INFO: Collecting host service logs for openvswitch
2026-07-24T18:11:25.962629156Z INFO: Collecting host service logs for ovs-configuration
2026-07-24T18:11:25.962942829Z INFO: Collecting host service logs for ovsdb-server
2026-07-24T18:11:25.963256832Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-24T18:11:25.963623969Z INFO: Waiting for worker host service log collection to complete ...
2026-07-24T18:11:26.561677035Z INFO: "metallb-operator" not detected. Skipping.
2026-07-24T18:11:26.608250976Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-24T18:11:26.693449981Z Wrote inspect data to must-gather.
2026-07-24T18:11:26.815431702Z No resources found in openshift-etcd namespace.
2026-07-24T18:11:26.825202824Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-24T18:11:26.989661394Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-24T18:11:27.184715475Z No resources found
2026-07-24T18:11:27.233142343Z ERROR: No running kube-apiserver pods found
2026-07-24T18:11:27.357357533Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-24T18:11:27.370431559Z error: the server doesn't have a resource type "performanceprofile"
2026-07-24T18:11:27.377326260Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-24T18:11:27.377429141Z INFO: on-disk MachineConfig config collection complete.
2026-07-24T18:11:27.472235327Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-wf2c6
2026-07-24T18:11:27.472311020Z insights-runtime-extractor-ddjjh
2026-07-24T18:11:27.472332444Z insights-runtime-extractor-kwd24
2026-07-24T18:11:27.472351678Z insights-runtime-extractor-tn85q
2026-07-24T18:11:27.731195974Z Wrote inspect data to must-gather.
2026-07-24T18:11:27.919971901Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:27.996437472Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-24T18:11:28.005437768Z INFO: Worker host service log collection to complete.
2026-07-24T18:11:28.081786061Z Wrote inspect data to must-gather.
2026-07-24T18:11:28.332591889Z Wrote inspect data to must-gather.
2026-07-24T18:11:28.334450898Z Gathering data for ns/openshift-monitoring...
2026-07-24T18:11:28.394940847Z No resources found
2026-07-24T18:11:28.428241194Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T18:11:28.460638344Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-24T18:11:28.511171067Z Wrote inspect data to must-gather.
2026-07-24T18:11:28.575315475Z INFO: 'previous-logs' folder not found on ip-10-0-133-116.ec2.internal, skipping...
2026-07-24T18:11:28.649806859Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-24T18:11:28.649806859Z See 'oc rsync -h' for help and examples
2026-07-24T18:11:28.722926709Z Wrote inspect data to must-gather.
2026-07-24T18:11:28.733304250Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-24T18:11:28.769839613Z error: the server doesn't have a resource type "machineconfigs"
2026-07-24T18:11:28.792656623Z tar: Removing leading `/' from member names
2026-07-24T18:11:28.797743151Z INFO: HAProxy config collection complete.
2026-07-24T18:11:28.931423718Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.049301290Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.051715753Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-24T18:11:29.067056225Z INFO: 'previous-logs' folder not found on ip-10-0-138-131.ec2.internal, skipping...
2026-07-24T18:11:29.335491400Z INFO: Getting rules from prometheus-k8s-0
2026-07-24T18:11:29.355382495Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.379860354Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.406888554Z INFO: 'previous-logs' folder not found on ip-10-0-143-155.ec2.internal, skipping...
2026-07-24T18:11:29.407011225Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-24T18:11:29.407084964Z INFO: Machine Config Daemon termination log collection complete.
2026-07-24T18:11:29.552810138Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-24T18:11:29.680701433Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.732212164Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-24T18:11:29.839832316Z Wrote inspect data to must-gather.
2026-07-24T18:11:29.849006749Z INFO: Getting status/config from prometheus-k8s-0
2026-07-24T18:11:30.042929943Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-24T18:11:30.069294062Z INFO: INTERCONNECT MODE
2026-07-24T18:11:30.069294062Z INFO: Gathering ovn-kubernetes DBs
2026-07-24T18:11:30.109589696Z Wrote inspect data to must-gather.
2026-07-24T18:11:30.282395575Z INFO: Gathering OVN_Northbound from ovnkube-node-4fdxn...
2026-07-24T18:11:30.282615960Z INFO: Gathering OVN_Northbound from ovnkube-node-h8l8j...
2026-07-24T18:11:30.282864231Z INFO: Gathering OVN_Northbound from ovnkube-node-q72xm...
2026-07-24T18:11:30.283226917Z INFO: Gathering OVN_Southbound from ovnkube-node-4fdxn...
2026-07-24T18:11:30.284882217Z INFO: Gathering OVN_Southbound from ovnkube-node-h8l8j...
2026-07-24T18:11:30.285220190Z INFO: Gathering OVN_Southbound from ovnkube-node-q72xm...
2026-07-24T18:11:30.433238105Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-24T18:11:30.439263854Z Wrote inspect data to must-gather.
2026-07-24T18:11:30.643668784Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.654651804Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.663567444Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.713981331Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.715141649Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.739691185Z tar: Removing leading `/' from member names
2026-07-24T18:11:30.841097055Z W0724 18:11:30.840988    1354 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-24T18:11:30.848320685Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-24T18:11:30.848945395Z Wrote inspect data to must-gather.
2026-07-24T18:11:30.921501160Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T18:11:31.080266690Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-24T18:11:31.162735192Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-24T18:11:31.182388018Z Gathering data for ns/openshift-network-console...
2026-07-24T18:11:31.258765405Z Wrote inspect data to must-gather.
2026-07-24T18:11:31.366006805Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:31.366750635Z INFO: Waiting for network log collection to complete ...
2026-07-24T18:11:31.366803797Z INFO: Waiting for ovnk database copies to complete ...
2026-07-24T18:11:31.367368179Z INFO: Copying ovnk databases complete.
2026-07-24T18:11:31.369293098Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-07-24T18:11:31.371799688Z ovnk_database_store/
2026-07-24T18:11:31.371799688Z ovnk_database_store/ovnkube-node-h8l8j_nbdb
2026-07-24T18:11:31.381486606Z ovnk_database_store/ovnkube-node-q72xm_nbdb
2026-07-24T18:11:31.388323432Z ovnk_database_store/ovnkube-node-4fdxn_sbdb
2026-07-24T18:11:31.404807942Z ovnk_database_store/ovnkube-node-h8l8j_sbdb
2026-07-24T18:11:31.420512661Z ovnk_database_store/ovnkube-node-q72xm_sbdb
2026-07-24T18:11:31.438386130Z ovnk_database_store/ovnkube-node-4fdxn_nbdb
2026-07-24T18:11:31.448463006Z INFO: Network log collection complete.
2026-07-24T18:11:31.480790717Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-24T18:11:31.631569400Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:31.636563866Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-24T18:11:31.644250188Z Wrote inspect data to must-gather.
2026-07-24T18:11:31.773391409Z Gathering data for ns/openshift-console-operator...
2026-07-24T18:11:31.882677392Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:31.965708977Z INFO: Getting status from alertmanager-main-0
2026-07-24T18:11:31.993615055Z Wrote inspect data to must-gather.
2026-07-24T18:11:32.140131278Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:32.144386287Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-24T18:11:32.281432567Z Gathering data for ns/openshift-console...
2026-07-24T18:11:32.343969636Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-24T18:11:32.346954998Z INFO: Fallback to identify the container image from release info
2026-07-24T18:11:32.378659235Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:32.617452407Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:32.620704575Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-24T18:11:32.866099833Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-24T18:11:32.872619463Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:32.930069497Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:33.031766027Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-24T18:11:33.111202154Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:33.115458960Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-24T18:11:33.188575249Z daemonset.apps/perf-node-gather-daemonset created
2026-07-24T18:11:33.339459887Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:33.361573927Z Waiting for performance profile collector pods to become ready: 1
2026-07-24T18:11:33.508918850Z Gathering data for ns/openshift-dns-operator...
2026-07-24T18:11:33.589129367Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:33.592913473Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-24T18:11:33.811536854Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:33.887229254Z Gathering data for ns/openshift-dns...
2026-07-24T18:11:34.023136448Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:34.026511899Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-24T18:11:34.217088619Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:34.438200046Z Wrote inspect data to /must-gather/istio.
2026-07-24T18:11:34.538493950Z Waiting for performance profile collector pods to become ready: 2
2026-07-24T18:11:34.663163818Z Gathering data for ns/openshift-image-registry...
2026-07-24T18:11:35.252864510Z Done executing Istio gather script
2026-07-24T18:11:35.255216741Z Gathering data for ns/openshift-ingress-operator...
2026-07-24T18:11:35.613225673Z Gathering data for ns/openshift-ingress...
2026-07-24T18:11:35.693112453Z Waiting for performance profile collector pods to become ready: 3
2026-07-24T18:11:35.981924158Z Gathering data for ns/openshift-ingress-canary...
2026-07-24T18:11:36.403929868Z Gathering data for ns/openshift-insights...
2026-07-24T18:11:36.830981474Z Waiting for performance profile collector pods to become ready: 4
2026-07-24T18:11:37.937280462Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:37.972916574Z Waiting for performance profile collector pods to become ready: 5
2026-07-24T18:11:38.964281088Z Gathering data for ns/kserve...
2026-07-24T18:11:39.115836804Z Waiting for performance profile collector pods to become ready: 6
2026-07-24T18:11:40.252650161Z Waiting for performance profile collector pods to become ready: 7
2026-07-24T18:11:40.364183757Z Gathering data for ns/openshift-config...
2026-07-24T18:11:40.655818361Z Gathering data for ns/openshift-config-managed...
2026-07-24T18:11:41.133418367Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-24T18:11:41.403668382Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-24T18:11:41.436384894Z Gathering data for ns/openshift-kube-apiserver...
2026-07-24T18:11:41.718892415Z Collecting performance related data for node ip-10-0-143-155.ec2.internal
2026-07-24T18:11:41.729785061Z Collecting performance related data for node ip-10-0-138-131.ec2.internal
2026-07-24T18:11:41.739011611Z Collecting performance related data for node ip-10-0-133-116.ec2.internal
2026-07-24T18:11:41.741790405Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-24T18:11:42.115182838Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-24T18:11:42.517331323Z Gathering data for ns/openshift-kube-scheduler...
2026-07-24T18:11:42.860679002Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-24T18:11:42.945674654Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:43.269918380Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-24T18:11:43.664304773Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-24T18:11:44.005274120Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-24T18:11:44.615349631Z Gathering data for ns/openshift-multus...
2026-07-24T18:11:46.348261724Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-24T18:11:47.956238237Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:48.011291308Z Gathering data for ns/openshift-host-network...
2026-07-24T18:11:48.347337521Z Gathering data for ns/openshift-network-diagnostics...
2026-07-24T18:11:48.918236161Z Gathering data for ns/openshift-network-node-identity...
2026-07-24T18:11:49.390076493Z Gathering data for ns/openshift-network-operator...
2026-07-24T18:11:49.788866133Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-24T18:11:50.116733686Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-24T18:11:50.677750875Z Gathering data for ns/openshift-apiserver-operator...
2026-07-24T18:11:50.722223444Z Collecting kubelet logs for node ip-10-0-143-155.ec2.internal
2026-07-24T18:11:50.725189175Z Collecting kubelet logs for node ip-10-0-138-131.ec2.internal
2026-07-24T18:11:50.728768535Z Collecting kubelet logs for node ip-10-0-133-116.ec2.internal
2026-07-24T18:11:51.055436665Z Gathering data for ns/openshift-apiserver...
2026-07-24T18:11:51.150849766Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-dms24 namespace
2026-07-24T18:11:51.156221413Z INFO: Node performance data collection complete.
2026-07-24T18:11:51.539885487Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-24T18:11:51.820574739Z Gathering data for ns/openshift-controller-manager...
2026-07-24T18:11:52.146250883Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-24T18:11:52.810735841Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-24T18:11:52.961208526Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-24T18:11:53.131026893Z Gathering data for ns/openshift-service-ca-operator...
2026-07-24T18:11:53.465084412Z Gathering data for ns/openshift-service-ca...
2026-07-24T18:11:53.849354037Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-24T18:11:54.779362188Z Wrote inspect data to must-gather.
2026-07-24T18:11:54.779422891Z error: inspection completed with the errors occurred while gathering data:
2026-07-24T18:11:54.779422891Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-24T18:11:54.926452029Z error: the server doesn't have a resource type "clusters"
2026-07-24T18:11:55.094459596Z Caches written to disk