2026-07-29T11:17:20.576200840Z [disk usage checker] Started
2026-07-29T11:17:20.579470786Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:20.790632132Z Gathering data for ns/openshift-cluster-version...
2026-07-29T11:17:20.938322463Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T11:17:21.042796666Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T11:17:21.130375770Z Gathering data for ns/default...
2026-07-29T11:17:21.479370363Z Gathering data for ns/openshift...
2026-07-29T11:17:21.960658251Z Gathering data for ns/kube-system...
2026-07-29T11:17:22.730997512Z Gathering data for ns/openshift-etcd...
2026-07-29T11:17:23.105054742Z Wrote inspect data to must-gather.
2026-07-29T11:17:23.105085772Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T11:17:23.105085772Z     namespaces "assisted-installer" not found
2026-07-29T11:17:23.525893054Z Waiting on subprocesses to finish execution.
2026-07-29T11:17:23.530674709Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-29T11:17:23.532966195Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-29T11:17:23.534137515Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-29T11:17:23.534166616Z INFO: Gathering HAProxy config files
2026-07-29T11:17:23.550891850Z INFO: Collecting host service logs for crio
2026-07-29T11:17:23.551201033Z INFO: Collecting host service logs for kubelet
2026-07-29T11:17:23.551522060Z INFO: Collecting host service logs for rpm-ostreed
2026-07-29T11:17:23.552179066Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-29T11:17:23.552528448Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-29T11:17:23.552884624Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-29T11:17:23.553251343Z INFO: Collecting host service logs for NetworkManager
2026-07-29T11:17:23.553658498Z INFO: Collecting host service logs for openvswitch
2026-07-29T11:17:23.554043350Z INFO: Collecting host service logs for ovs-configuration
2026-07-29T11:17:23.554446760Z INFO: Collecting host service logs for ovsdb-server
2026-07-29T11:17:23.554828861Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-29T11:17:23.555375931Z INFO: Waiting for worker host service log collection to complete ...
2026-07-29T11:17:23.573467961Z Executing Istio gather script
2026-07-29T11:17:23.635058108Z INFO: Waiting for node performance related collection to complete ...
2026-07-29T11:17:24.144593543Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-565pv
2026-07-29T11:17:24.144668094Z insights-runtime-extractor-86nmd
2026-07-29T11:17:24.144689007Z insights-runtime-extractor-d47bh
2026-07-29T11:17:24.144709234Z insights-runtime-extractor-rwhkx
2026-07-29T11:17:24.346205637Z No resources found in openshift-etcd namespace.
2026-07-29T11:17:24.350969571Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-29T11:17:24.351054103Z INFO: on-disk MachineConfig config collection complete.
2026-07-29T11:17:24.366059393Z INFO: "metallb-operator" not detected. Skipping.
2026-07-29T11:17:24.554611865Z No resources found
2026-07-29T11:17:24.567728198Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-29T11:17:24.817304582Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-29T11:17:24.844900283Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-29T11:17:24.889116606Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-29T11:17:25.026184002Z error: the server doesn't have a resource type "performanceprofile"
2026-07-29T11:17:25.065283963Z Wrote inspect data to must-gather.
2026-07-29T11:17:25.092363314Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-29T11:17:25.092363314Z See 'oc rsync -h' for help and examples
2026-07-29T11:17:25.183942853Z INFO: Worker host service log collection to complete.
2026-07-29T11:17:25.332109712Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-29T11:17:25.360490688Z Wrote inspect data to must-gather.
2026-07-29T11:17:25.439506495Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-29T11:17:25.645341773Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:25.813603854Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-29T11:17:25.827702983Z ERROR: No running kube-apiserver pods found
2026-07-29T11:17:25.847453931Z No resources found
2026-07-29T11:17:25.905378450Z Gathering data for ns/openshift-monitoring...
2026-07-29T11:17:26.093067569Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.106509485Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.113324040Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.178001670Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T11:17:26.219463049Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.366177827Z tar: Removing leading `/' from member names
2026-07-29T11:17:26.371593612Z INFO: 'previous-logs' folder not found on ip-10-0-129-138.ec2.internal, skipping...
2026-07-29T11:17:26.372471811Z INFO: HAProxy config collection complete.
2026-07-29T11:17:26.530632315Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.541636645Z INFO: Getting rules from prometheus-k8s-0
2026-07-29T11:17:26.547015203Z error: the server doesn't have a resource type "machineconfigs"
2026-07-29T11:17:26.614732629Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.747868853Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-29T11:17:26.769016015Z INFO: 'previous-logs' folder not found on ip-10-0-137-188.ec2.internal, skipping...
2026-07-29T11:17:26.943671084Z Wrote inspect data to must-gather.
2026-07-29T11:17:26.948652711Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-29T11:17:27.028979184Z INFO: Getting status/config from prometheus-k8s-0
2026-07-29T11:17:27.046227015Z Wrote inspect data to must-gather.
2026-07-29T11:17:27.087190612Z INFO: 'previous-logs' folder not found on ip-10-0-143-36.ec2.internal, skipping...
2026-07-29T11:17:27.087227338Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-29T11:17:27.087254998Z INFO: Machine Config Daemon termination log collection complete.
2026-07-29T11:17:27.223866356Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-29T11:17:27.226365501Z Wrote inspect data to must-gather.
2026-07-29T11:17:27.452314882Z Wrote inspect data to must-gather.
2026-07-29T11:17:27.487819744Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-29T11:17:27.543189897Z INFO: INTERCONNECT MODE
2026-07-29T11:17:27.543235874Z INFO: Gathering ovn-kubernetes DBs
2026-07-29T11:17:27.719485342Z Wrote inspect data to must-gather.
2026-07-29T11:17:27.745416307Z INFO: Gathering OVN_Northbound from ovnkube-node-2hvgf...
2026-07-29T11:17:27.745648682Z INFO: Gathering OVN_Northbound from ovnkube-node-4gwff...
2026-07-29T11:17:27.745985181Z INFO: Gathering OVN_Northbound from ovnkube-node-64gmd...
2026-07-29T11:17:27.746197199Z INFO: Gathering OVN_Southbound from ovnkube-node-2hvgf...
2026-07-29T11:17:27.746547087Z INFO: Gathering OVN_Southbound from ovnkube-node-4gwff...
2026-07-29T11:17:27.746879263Z INFO: Gathering OVN_Southbound from ovnkube-node-64gmd...
2026-07-29T11:17:27.747152118Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-29T11:17:27.900270776Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T11:17:28.093657868Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.122502281Z Wrote inspect data to must-gather.
2026-07-29T11:17:28.134021224Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.147231584Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.160260496Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.171666119Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.201125947Z tar: Removing leading `/' from member names
2026-07-29T11:17:28.259289879Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-29T11:17:28.403134717Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-29T11:17:28.440771329Z Wrote inspect data to must-gather.
2026-07-29T11:17:28.569302796Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-29T11:17:28.638572093Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:28.667124670Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T11:17:28.799319484Z Wrote inspect data to must-gather.
2026-07-29T11:17:28.804870817Z INFO: Waiting for network log collection to complete ...
2026-07-29T11:17:28.804951774Z INFO: Waiting for ovnk database copies to complete ...
2026-07-29T11:17:28.805384304Z INFO: Copying ovnk databases complete.
2026-07-29T11:17:28.807404511Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-07-29T11:17:28.810510714Z ovnk_database_store/
2026-07-29T11:17:28.810510714Z ovnk_database_store/ovnkube-node-2hvgf_nbdb
2026-07-29T11:17:28.819175561Z ovnk_database_store/ovnkube-node-4gwff_sbdb
2026-07-29T11:17:28.844688164Z ovnk_database_store/ovnkube-node-4gwff_nbdb
2026-07-29T11:17:28.855966393Z ovnk_database_store/ovnkube-node-2hvgf_sbdb
2026-07-29T11:17:28.882398066Z ovnk_database_store/ovnkube-node-64gmd_nbdb
2026-07-29T11:17:28.899858385Z ovnk_database_store/ovnkube-node-64gmd_sbdb
2026-07-29T11:17:28.911119668Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:28.915342094Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-29T11:17:28.919132233Z Gathering data for ns/openshift-network-console...
2026-07-29T11:17:28.920226384Z INFO: Network log collection complete.
2026-07-29T11:17:29.045894731Z INFO: Getting status from alertmanager-main-0
2026-07-29T11:17:29.144624938Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:29.149230918Z Wrote inspect data to must-gather.
2026-07-29T11:17:29.387681774Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:29.391231063Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-29T11:17:29.467893030Z Wrote inspect data to must-gather.
2026-07-29T11:17:29.492844018Z Gathering data for ns/openshift-console-operator...
2026-07-29T11:17:29.633230160Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:29.775278321Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-29T11:17:29.778204834Z INFO: Fallback to identify the container image from release info
2026-07-29T11:17:29.844207828Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:29.847657495Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-29T11:17:29.980243081Z Gathering data for ns/openshift-console...
2026-07-29T11:17:30.085709422Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:30.294530132Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:30.297597951Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-29T11:17:30.377370967Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-29T11:17:30.508578954Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:30.529531621Z daemonset.apps/perf-node-gather-daemonset created
2026-07-29T11:17:30.622530457Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-29T11:17:30.680810170Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:30.695377993Z Waiting for performance profile collector pods to become ready: 1
2026-07-29T11:17:30.759499824Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:30.762012316Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-29T11:17:30.965667331Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:31.200188126Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:31.203843192Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-29T11:17:31.267434387Z Gathering data for ns/openshift-dns-operator...
2026-07-29T11:17:31.393653724Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:31.604003029Z Wrote inspect data to /must-gather/istio.
2026-07-29T11:17:31.632667118Z Gathering data for ns/openshift-dns...
2026-07-29T11:17:31.827722584Z Waiting for performance profile collector pods to become ready: 2
2026-07-29T11:17:32.333160116Z Gathering data for ns/openshift-image-registry...
2026-07-29T11:17:32.334086389Z Done executing Istio gather script
2026-07-29T11:17:32.872659015Z Gathering data for ns/openshift-ingress-operator...
2026-07-29T11:17:32.968275338Z Waiting for performance profile collector pods to become ready: 3
2026-07-29T11:17:33.230056306Z Gathering data for ns/openshift-ingress...
2026-07-29T11:17:33.577105494Z Gathering data for ns/openshift-ingress-canary...
2026-07-29T11:17:34.007870724Z Gathering data for ns/openshift-insights...
2026-07-29T11:17:34.097424433Z Waiting for performance profile collector pods to become ready: 4
2026-07-29T11:17:35.217460623Z Waiting for performance profile collector pods to become ready: 5
2026-07-29T11:17:35.687136254Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:36.343547526Z Waiting for performance profile collector pods to become ready: 6
2026-07-29T11:17:36.506351442Z Gathering data for ns/kserve...
2026-07-29T11:17:37.485870651Z Waiting for performance profile collector pods to become ready: 7
2026-07-29T11:17:38.238837571Z Gathering data for ns/openshift-config...
2026-07-29T11:17:38.540155337Z Gathering data for ns/openshift-config-managed...
2026-07-29T11:17:38.632295035Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-29T11:17:38.910892417Z Collecting performance related data for node ip-10-0-137-188.ec2.internal
2026-07-29T11:17:38.925155230Z Collecting performance related data for node ip-10-0-143-36.ec2.internal
2026-07-29T11:17:38.934705705Z Collecting performance related data for node ip-10-0-129-138.ec2.internal
2026-07-29T11:17:39.038070754Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-29T11:17:39.421120705Z Gathering data for ns/openshift-kube-apiserver...
2026-07-29T11:17:39.798527421Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-29T11:17:40.204535868Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-29T11:17:40.618333041Z Gathering data for ns/openshift-kube-scheduler...
2026-07-29T11:17:40.693156477Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:40.928298960Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-29T11:17:41.291595757Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-29T11:17:41.699026853Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-29T11:17:42.083772965Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-29T11:17:42.650585485Z Gathering data for ns/openshift-multus...
2026-07-29T11:17:44.251771538Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-29T11:17:45.685474772Z Gathering data for ns/openshift-host-network...
2026-07-29T11:17:45.701425534Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:46.009191262Z Gathering data for ns/openshift-network-diagnostics...
2026-07-29T11:17:46.587115293Z Gathering data for ns/openshift-network-node-identity...
2026-07-29T11:17:47.051671787Z Gathering data for ns/openshift-network-operator...
2026-07-29T11:17:47.319710517Z Collecting kubelet logs for node ip-10-0-137-188.ec2.internal
2026-07-29T11:17:47.323559460Z Collecting kubelet logs for node ip-10-0-143-36.ec2.internal
2026-07-29T11:17:47.325297025Z Collecting kubelet logs for node ip-10-0-129-138.ec2.internal
2026-07-29T11:17:47.501147335Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-29T11:17:47.726571759Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8kqzn namespace
2026-07-29T11:17:47.732871506Z INFO: Node performance data collection complete.
2026-07-29T11:17:47.838659841Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-29T11:17:48.389888224Z Gathering data for ns/openshift-apiserver-operator...
2026-07-29T11:17:48.702344600Z Gathering data for ns/openshift-apiserver...
2026-07-29T11:17:49.161117415Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-29T11:17:49.485630136Z Gathering data for ns/openshift-controller-manager...
2026-07-29T11:17:49.778863068Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-29T11:17:50.517093323Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-29T11:17:50.707276865Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-29T11:17:50.888023747Z Gathering data for ns/openshift-service-ca-operator...
2026-07-29T11:17:51.254771768Z Gathering data for ns/openshift-service-ca...
2026-07-29T11:17:51.689339880Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-29T11:17:52.654000063Z Wrote inspect data to must-gather.
2026-07-29T11:17:52.654048737Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T11:17:52.654048737Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-29T11:17:52.792178982Z error: the server doesn't have a resource type "clusters"
2026-07-29T11:17:52.933139424Z Caches written to disk