2026-07-07T15:25:26.544885592Z [disk usage checker] Started
2026-07-07T15:25:26.549023637Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:26.767285702Z Gathering data for ns/openshift-cluster-version...
2026-07-07T15:25:26.920389405Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:25:27.038852381Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:25:27.134672237Z Gathering data for ns/default...
2026-07-07T15:25:27.578665988Z Gathering data for ns/openshift...
2026-07-07T15:25:28.083879731Z Gathering data for ns/kube-system...
2026-07-07T15:25:29.006842208Z Gathering data for ns/openshift-etcd...
2026-07-07T15:25:29.396608367Z Wrote inspect data to must-gather.
2026-07-07T15:25:29.396644531Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:25:29.396644531Z     namespaces "assisted-installer" not found
2026-07-07T15:25:29.762529127Z Waiting on subprocesses to finish execution.
2026-07-07T15:25:29.771940065Z INFO: Gathering HAProxy config files
2026-07-07T15:25:29.772450494Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T15:25:29.775575988Z INFO: Collecting host service logs for crio
2026-07-07T15:25:29.775979364Z INFO: Collecting host service logs for kubelet
2026-07-07T15:25:29.776408419Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T15:25:29.776865397Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T15:25:29.777361101Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T15:25:29.777744257Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T15:25:29.778124631Z INFO: Collecting host service logs for NetworkManager
2026-07-07T15:25:29.778418978Z Executing Istio gather script
2026-07-07T15:25:29.778485742Z INFO: Collecting host service logs for openvswitch
2026-07-07T15:25:29.778792592Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T15:25:29.779229465Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T15:25:29.779598717Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T15:25:29.779876718Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T15:25:29.779993365Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T15:25:29.783661787Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T15:25:29.812767835Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T15:25:30.378511095Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T15:25:30.569609148Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T15:25:30.710432546Z No resources found in openshift-etcd namespace.
2026-07-07T15:25:30.832663146Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T15:25:30.832663146Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T15:25:31.035085833Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-kqd56
2026-07-07T15:25:31.035085833Z insights-runtime-extractor-2jd5s
2026-07-07T15:25:31.035085833Z insights-runtime-extractor-csrnj
2026-07-07T15:25:31.035085833Z insights-runtime-extractor-t24sv
2026-07-07T15:25:31.160607780Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T15:25:31.380741257Z No resources found
2026-07-07T15:25:31.629985703Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T15:25:31.781025361Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:31.873228495Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T15:25:31.925164346Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.030094110Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.048896739Z INFO: Worker host service log collection to complete.
2026-07-07T15:25:32.073715068Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T15:25:32.134620926Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T15:25:32.134620926Z See 'oc rsync -h' for help and examples
2026-07-07T15:25:32.338633052Z No resources found
2026-07-07T15:25:32.378728324Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.392117993Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T15:25:32.411535619Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:25:32.468601423Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T15:25:32.503066367Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T15:25:32.611217465Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.678002261Z ERROR: No running kube-apiserver pods found
2026-07-07T15:25:32.842277939Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.863872769Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.865135925Z Wrote inspect data to must-gather.
2026-07-07T15:25:32.914831244Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T15:25:32.926154231Z INFO: 'previous-logs' folder not found on ip-10-0-130-13.ec2.internal, skipping...
2026-07-07T15:25:33.026100930Z tar: Removing leading `/' from member names
2026-07-07T15:25:33.035582916Z INFO: HAProxy config collection complete.
2026-07-07T15:25:33.165585092Z Gathering data for ns/openshift-monitoring...
2026-07-07T15:25:33.180488388Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T15:25:33.251453458Z Wrote inspect data to must-gather.
2026-07-07T15:25:33.294281941Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T15:25:33.296195007Z INFO: 'previous-logs' folder not found on ip-10-0-132-127.ec2.internal, skipping...
2026-07-07T15:25:33.503236046Z Wrote inspect data to must-gather.
2026-07-07T15:25:33.587095476Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T15:25:33.632825569Z Wrote inspect data to must-gather.
2026-07-07T15:25:33.639479253Z INFO: 'previous-logs' folder not found on ip-10-0-140-158.ec2.internal, skipping...
2026-07-07T15:25:33.639596960Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T15:25:33.639654703Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T15:25:33.719097009Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T15:25:33.865653524Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T15:25:33.968100377Z Wrote inspect data to must-gather.
2026-07-07T15:25:33.991618654Z Wrote inspect data to must-gather.
2026-07-07T15:25:34.228558655Z Wrote inspect data to must-gather.
2026-07-07T15:25:34.326517728Z INFO: INTERCONNECT MODE
2026-07-07T15:25:34.326556232Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T15:25:34.367378328Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T15:25:34.465274058Z Wrote inspect data to must-gather.
2026-07-07T15:25:34.529821887Z INFO: Gathering OVN_Northbound from ovnkube-node-lw2hn...
2026-07-07T15:25:34.530126546Z INFO: Gathering OVN_Northbound from ovnkube-node-rswbd...
2026-07-07T15:25:34.530361065Z INFO: Gathering OVN_Northbound from ovnkube-node-x9bhb...
2026-07-07T15:25:34.530661917Z INFO: Gathering OVN_Southbound from ovnkube-node-lw2hn...
2026-07-07T15:25:34.530919271Z INFO: Gathering OVN_Southbound from ovnkube-node-rswbd...
2026-07-07T15:25:34.531235860Z INFO: Gathering OVN_Southbound from ovnkube-node-x9bhb...
2026-07-07T15:25:34.687783281Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T15:25:34.753705177Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T15:25:34.853091523Z Wrote inspect data to must-gather.
2026-07-07T15:25:34.914240707Z tar: Removing leading `/' from member names
2026-07-07T15:25:34.924603939Z tar: Removing leading `/' from member names
2026-07-07T15:25:34.943767550Z tar: Removing leading `/' from member names
2026-07-07T15:25:34.950095912Z tar: Removing leading `/' from member names
2026-07-07T15:25:34.950274842Z tar: Removing leading `/' from member names
2026-07-07T15:25:34.951529315Z tar: Removing leading `/' from member names
2026-07-07T15:25:35.056269821Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:35.090991733Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T15:25:35.167096038Z Wrote inspect data to must-gather.
2026-07-07T15:25:35.257349021Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:25:35.361041057Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:35.364386923Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T15:25:35.425307438Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T15:25:35.450618446Z Wrote inspect data to must-gather.
2026-07-07T15:25:35.530033737Z INFO: Waiting for network log collection to complete ...
2026-07-07T15:25:35.530526481Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T15:25:35.530709205Z INFO: Copying ovnk databases complete.
2026-07-07T15:25:35.533131708Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-07T15:25:35.536039051Z ovnk_database_store/
2026-07-07T15:25:35.536039051Z ovnk_database_store/ovnkube-node-rswbd_sbdb
2026-07-07T15:25:35.558563537Z ovnk_database_store/ovnkube-node-lw2hn_sbdb
2026-07-07T15:25:35.581775225Z ovnk_database_store/ovnkube-node-lw2hn_nbdb
2026-07-07T15:25:35.592472020Z ovnk_database_store/ovnkube-node-rswbd_nbdb
2026-07-07T15:25:35.602663887Z ovnk_database_store/ovnkube-node-x9bhb_sbdb
2026-07-07T15:25:35.625122903Z ovnk_database_store/ovnkube-node-x9bhb_nbdb
2026-07-07T15:25:35.638370326Z INFO: Network log collection complete.
2026-07-07T15:25:35.642165884Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:35.745495913Z Wrote inspect data to must-gather.
2026-07-07T15:25:35.886512553Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:35.889650294Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T15:25:35.906375984Z INFO: Getting status from alertmanager-main-0
2026-07-07T15:25:36.058422307Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:25:36.103810446Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T15:25:36.105950072Z INFO: Fallback to identify the container image from release info
2026-07-07T15:25:36.220101744Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:36.313125339Z Gathering data for ns/openshift-network-console...
2026-07-07T15:25:36.450656790Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:36.453783301Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T15:25:36.699636344Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:36.803864630Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T15:25:36.850242929Z Gathering data for ns/openshift-console-operator...
2026-07-07T15:25:36.906992738Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:36.980112295Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:36.984315124Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T15:25:36.999130845Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T15:25:37.148893813Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T15:25:37.184479073Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:37.320156761Z Gathering data for ns/openshift-console...
2026-07-07T15:25:37.404171221Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:37.407069496Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T15:25:37.638713254Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:37.867951056Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:37.871131855Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T15:25:37.988154214Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T15:25:38.066773416Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:38.307156033Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:38.310834274Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T15:25:38.323396850Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T15:25:38.513034030Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:38.641655455Z Gathering data for ns/openshift-dns-operator...
2026-07-07T15:25:38.751696309Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:38.753963824Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T15:25:38.939851752Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:39.028546409Z Gathering data for ns/openshift-dns...
2026-07-07T15:25:39.150698355Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:39.153238154Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T15:25:39.342836915Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:39.472956416Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T15:25:39.574349450Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:25:39.909182158Z Gathering data for ns/openshift-image-registry...
2026-07-07T15:25:40.351112393Z Done executing Istio gather script
2026-07-07T15:25:40.471219956Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T15:25:40.619293036Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T15:25:40.879416872Z Gathering data for ns/openshift-ingress...
2026-07-07T15:25:41.283916655Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T15:25:41.744319558Z Gathering data for ns/openshift-insights...
2026-07-07T15:25:41.777652986Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T15:25:41.912847331Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:42.906392208Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T15:25:44.048740287Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T15:25:44.164585816Z Gathering data for ns/kserve...
2026-07-07T15:25:45.203805466Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T15:25:45.499587003Z Collecting performance related data for node ip-10-0-132-127.ec2.internal
2026-07-07T15:25:45.507439101Z Collecting performance related data for node ip-10-0-140-158.ec2.internal
2026-07-07T15:25:45.515836975Z Collecting performance related data for node ip-10-0-130-13.ec2.internal
2026-07-07T15:25:46.251032737Z Gathering data for ns/openshift-config...
2026-07-07T15:25:46.658789373Z Gathering data for ns/openshift-config-managed...
2026-07-07T15:25:46.921584916Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:47.270955043Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T15:25:47.660175860Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T15:25:48.040738125Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T15:25:48.461343480Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T15:25:48.890899222Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T15:25:49.211353711Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T15:25:49.614109568Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T15:25:49.996078052Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T15:25:50.372425619Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T15:25:50.952863947Z Gathering data for ns/openshift-multus...
2026-07-07T15:25:51.927945313Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:52.479522033Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T15:25:53.897506814Z Gathering data for ns/openshift-host-network...
2026-07-07T15:25:54.273067123Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T15:25:54.614811665Z Collecting kubelet logs for node ip-10-0-132-127.ec2.internal
2026-07-07T15:25:54.616667815Z Collecting kubelet logs for node ip-10-0-140-158.ec2.internal
2026-07-07T15:25:54.618532141Z Collecting kubelet logs for node ip-10-0-130-13.ec2.internal
2026-07-07T15:25:54.982365830Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T15:25:55.060827498Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-2kxcl namespace
2026-07-07T15:25:55.067188719Z INFO: Node performance data collection complete.
2026-07-07T15:25:55.519326738Z Gathering data for ns/openshift-network-operator...
2026-07-07T15:25:55.952708622Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T15:25:56.305533535Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T15:25:56.852387623Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T15:25:56.932744664Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:25:57.239208852Z Gathering data for ns/openshift-apiserver...
2026-07-07T15:25:57.733018324Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T15:25:58.162505497Z Gathering data for ns/openshift-controller-manager...
2026-07-07T15:25:58.507153568Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T15:25:59.336663386Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T15:25:59.762449738Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T15:26:00.183583357Z Gathering data for ns/openshift-service-ca...
2026-07-07T15:26:00.713591021Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T15:26:01.937512153Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70
2026-07-07T15:26:02.013662988Z Wrote inspect data to must-gather.
2026-07-07T15:26:02.013717792Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:26:02.013717792Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T15:26:02.165996037Z error: the server doesn't have a resource type "clusters"
2026-07-07T15:26:02.544972069Z Caches written to disk