2026-07-30T18:32:05.282136221Z [disk usage checker] Started
2026-07-30T18:32:05.285503667Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:06.787646308Z Gathering data for ns/openshift-cluster-version...
2026-07-30T18:32:06.921741406Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T18:32:07.003596755Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T18:32:07.086436580Z Gathering data for ns/default...
2026-07-30T18:32:07.402724023Z Gathering data for ns/openshift...
2026-07-30T18:32:07.801683220Z Gathering data for ns/kube-system...
2026-07-30T18:32:08.569378438Z Gathering data for ns/openshift-etcd...
2026-07-30T18:32:09.007028164Z Wrote inspect data to must-gather.
2026-07-30T18:32:09.007057843Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T18:32:09.007057843Z     namespaces "assisted-installer" not found
2026-07-30T18:32:09.380135908Z Waiting on subprocesses to finish execution.
2026-07-30T18:32:09.388169397Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-30T18:32:09.391925678Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-30T18:32:09.398139035Z Executing Istio gather script
2026-07-30T18:32:09.401419391Z INFO: Collecting host service logs for crio
2026-07-30T18:32:09.401750441Z INFO: Collecting host service logs for kubelet
2026-07-30T18:32:09.402079942Z INFO: Collecting host service logs for rpm-ostreed
2026-07-30T18:32:09.402356686Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-30T18:32:09.402653986Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-30T18:32:09.402935224Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-30T18:32:09.403344156Z INFO: Collecting host service logs for NetworkManager
2026-07-30T18:32:09.403663910Z INFO: Collecting host service logs for openvswitch
2026-07-30T18:32:09.403755278Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-30T18:32:09.403982925Z INFO: Collecting host service logs for ovs-configuration
2026-07-30T18:32:09.404365211Z INFO: Collecting host service logs for ovsdb-server
2026-07-30T18:32:09.404726462Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-30T18:32:09.405134264Z INFO: Waiting for worker host service log collection to complete ...
2026-07-30T18:32:09.425315348Z INFO: Gathering HAProxy config files
2026-07-30T18:32:09.427541615Z INFO: Waiting for node performance related collection to complete ...
2026-07-30T18:32:10.254750994Z No resources found in openshift-etcd namespace.
2026-07-30T18:32:10.342411526Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-30T18:32:10.350367016Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:10.423957475Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-30T18:32:10.501886337Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-4qm6z
2026-07-30T18:32:10.501933326Z insights-runtime-extractor-9rcbw
2026-07-30T18:32:10.501954592Z insights-runtime-extractor-sm7cs
2026-07-30T18:32:10.501974120Z insights-runtime-extractor-tkbtl
2026-07-30T18:32:10.544851423Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-30T18:32:10.544932205Z INFO: on-disk MachineConfig config collection complete.
2026-07-30T18:32:10.575425115Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-30T18:32:10.728271171Z Wrote inspect data to must-gather.
2026-07-30T18:32:10.780070756Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-30T18:32:10.842432732Z Wrote inspect data to must-gather.
2026-07-30T18:32:10.843059005Z error: the server doesn't have a resource type "performanceprofile"
2026-07-30T18:32:10.937687067Z No resources found
2026-07-30T18:32:10.947178577Z INFO: "metallb-operator" not detected. Skipping.
2026-07-30T18:32:10.969103895Z INFO: Worker host service log collection to complete.
2026-07-30T18:32:11.033949896Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-30T18:32:11.033949896Z See 'oc rsync -h' for help and examples
2026-07-30T18:32:11.272792561Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-30T18:32:11.484240747Z ERROR: No running kube-apiserver pods found
2026-07-30T18:32:11.519109673Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-30T18:32:11.780310866Z Gathering data for ns/openshift-monitoring...
2026-07-30T18:32:11.859765398Z Wrote inspect data to must-gather.
2026-07-30T18:32:11.883315654Z Wrote inspect data to must-gather.
2026-07-30T18:32:11.940078174Z Wrote inspect data to must-gather.
2026-07-30T18:32:11.943142557Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T18:32:11.995446687Z Wrote inspect data to must-gather.
2026-07-30T18:32:12.023134436Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-30T18:32:12.088524018Z INFO: 'previous-logs' folder not found on ip-10-0-142-223.ec2.internal, skipping...
2026-07-30T18:32:12.089319998Z Wrote inspect data to must-gather.
2026-07-30T18:32:12.146370805Z No resources found
2026-07-30T18:32:12.166379022Z tar: Removing leading `/' from member names
2026-07-30T18:32:12.173993450Z INFO: HAProxy config collection complete.
2026-07-30T18:32:12.311845369Z error: the server doesn't have a resource type "machineconfigs"
2026-07-30T18:32:12.431770558Z INFO: Getting rules from prometheus-k8s-0
2026-07-30T18:32:12.444214305Z Wrote inspect data to must-gather.
2026-07-30T18:32:12.479235648Z INFO: 'previous-logs' folder not found on ip-10-0-142-31.ec2.internal, skipping...
2026-07-30T18:32:12.530812434Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-30T18:32:12.755740218Z Wrote inspect data to must-gather.
2026-07-30T18:32:12.813799838Z Wrote inspect data to must-gather.
2026-07-30T18:32:12.817946007Z INFO: 'previous-logs' folder not found on ip-10-0-143-190.ec2.internal, skipping...
2026-07-30T18:32:12.817986523Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-30T18:32:12.818034276Z INFO: Machine Config Daemon termination log collection complete.
2026-07-30T18:32:12.903470833Z INFO: Getting status/config from prometheus-k8s-0
2026-07-30T18:32:13.010278409Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-30T18:32:13.063803819Z Wrote inspect data to must-gather.
2026-07-30T18:32:13.128306755Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-30T18:32:13.252683143Z Wrote inspect data to must-gather.
2026-07-30T18:32:13.364193244Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-30T18:32:13.412658341Z INFO: INTERCONNECT MODE
2026-07-30T18:32:13.412688692Z INFO: Gathering ovn-kubernetes DBs
2026-07-30T18:32:13.513872991Z Wrote inspect data to must-gather.
2026-07-30T18:32:13.545351421Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-30T18:32:13.582836351Z INFO: Gathering OVN_Northbound from ovnkube-node-l9f99...
2026-07-30T18:32:13.583074093Z INFO: Gathering OVN_Northbound from ovnkube-node-p6fp7...
2026-07-30T18:32:13.583299084Z INFO: Gathering OVN_Northbound from ovnkube-node-t87pw...
2026-07-30T18:32:13.583591884Z INFO: Gathering OVN_Southbound from ovnkube-node-l9f99...
2026-07-30T18:32:13.583880119Z INFO: Gathering OVN_Southbound from ovnkube-node-p6fp7...
2026-07-30T18:32:13.584192323Z INFO: Gathering OVN_Southbound from ovnkube-node-t87pw...
2026-07-30T18:32:13.632442824Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-30T18:32:13.878410643Z tar: Removing leading `/' from member names
2026-07-30T18:32:13.931558733Z Wrote inspect data to must-gather.
2026-07-30T18:32:13.968651479Z tar: Removing leading `/' from member names
2026-07-30T18:32:14.029028582Z tar: Removing leading `/' from member names
2026-07-30T18:32:14.037477041Z tar: Removing leading `/' from member names
2026-07-30T18:32:14.056485812Z tar: Removing leading `/' from member names
2026-07-30T18:32:14.097071118Z tar: Removing leading `/' from member names
2026-07-30T18:32:14.135460336Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-30T18:32:14.253248450Z Wrote inspect data to must-gather.
2026-07-30T18:32:14.298432552Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-30T18:32:14.392901301Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-30T18:32:14.484020644Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-30T18:32:14.546653990Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:14.621407967Z Wrote inspect data to must-gather.
2026-07-30T18:32:14.627614593Z INFO: Waiting for network log collection to complete ...
2026-07-30T18:32:14.627659270Z INFO: Waiting for ovnk database copies to complete ...
2026-07-30T18:32:14.628190813Z INFO: Copying ovnk databases complete.
2026-07-30T18:32:14.630253403Z 4.7M	must-gather/network_logs/ovnk_database_store
2026-07-30T18:32:14.632652357Z ovnk_database_store/
2026-07-30T18:32:14.632652357Z ovnk_database_store/ovnkube-node-l9f99_sbdb
2026-07-30T18:32:14.669562893Z ovnk_database_store/ovnkube-node-t87pw_nbdb
2026-07-30T18:32:14.683464789Z ovnk_database_store/ovnkube-node-p6fp7_nbdb
2026-07-30T18:32:14.694788477Z ovnk_database_store/ovnkube-node-l9f99_nbdb
2026-07-30T18:32:14.707532192Z ovnk_database_store/ovnkube-node-t87pw_sbdb
2026-07-30T18:32:14.720684401Z Gathering data for ns/openshift-network-console...
2026-07-30T18:32:14.740458580Z ovnk_database_store/ovnkube-node-p6fp7_sbdb
2026-07-30T18:32:14.767289593Z INFO: Network log collection complete.
2026-07-30T18:32:14.816437782Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:14.821812581Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-30T18:32:14.860085983Z INFO: Getting status from alertmanager-main-0
2026-07-30T18:32:14.974851526Z Wrote inspect data to must-gather.
2026-07-30T18:32:15.043594668Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:15.239042919Z Gathering data for ns/openshift-console-operator...
2026-07-30T18:32:15.265719706Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:15.269371850Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-30T18:32:15.297993415Z Wrote inspect data to must-gather.
2026-07-30T18:32:15.368848891Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:15.486844330Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:15.579523208Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-30T18:32:15.585235331Z INFO: Fallback to identify the container image from release info
2026-07-30T18:32:15.662892412Z Gathering data for ns/openshift-console...
2026-07-30T18:32:15.699542216Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:15.702476883Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-30T18:32:15.920240412Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.125712403Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.129525184Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-30T18:32:16.206442727Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-30T18:32:16.310271376Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.498709875Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.502655982Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-30T18:32:16.695280984Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.758360303Z Gathering data for ns/openshift-dns-operator...
2026-07-30T18:32:16.758732371Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-30T18:32:16.913656533Z daemonset.apps/perf-node-gather-daemonset created
2026-07-30T18:32:16.920121793Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:16.925537120Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-30T18:32:17.079916070Z Waiting for performance profile collector pods to become ready: 1
2026-07-30T18:32:17.109686966Z Gathering data for ns/openshift-dns...
2026-07-30T18:32:17.118403304Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:17.286902736Z Wrote inspect data to /must-gather/istio.
2026-07-30T18:32:17.779519988Z Gathering data for ns/openshift-image-registry...
2026-07-30T18:32:18.040948387Z Done executing Istio gather script
2026-07-30T18:32:18.208192459Z Waiting for performance profile collector pods to become ready: 2
2026-07-30T18:32:18.311193254Z Gathering data for ns/openshift-ingress-operator...
2026-07-30T18:32:18.624670980Z Gathering data for ns/openshift-ingress...
2026-07-30T18:32:18.944951188Z Gathering data for ns/openshift-ingress-canary...
2026-07-30T18:32:19.344322641Z Waiting for performance profile collector pods to become ready: 3
2026-07-30T18:32:19.357152901Z Gathering data for ns/openshift-insights...
2026-07-30T18:32:20.374222346Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:20.465070116Z Waiting for performance profile collector pods to become ready: 4
2026-07-30T18:32:21.595684673Z Waiting for performance profile collector pods to become ready: 5
2026-07-30T18:32:21.740600900Z Gathering data for ns/kserve...
2026-07-30T18:32:22.708778275Z Waiting for performance profile collector pods to become ready: 6
2026-07-30T18:32:23.834029594Z Waiting for performance profile collector pods to become ready: 7
2026-07-30T18:32:23.874980635Z Gathering data for ns/openshift-config...
2026-07-30T18:32:24.146528405Z Gathering data for ns/openshift-config-managed...
2026-07-30T18:32:24.550598192Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-30T18:32:24.806044964Z Gathering data for ns/openshift-kube-apiserver...
2026-07-30T18:32:24.966371541Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-30T18:32:25.092288947Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-30T18:32:25.242663388Z Collecting performance related data for node ip-10-0-142-223.ec2.internal
2026-07-30T18:32:25.250849869Z Collecting performance related data for node ip-10-0-143-190.ec2.internal
2026-07-30T18:32:25.258972227Z Collecting performance related data for node ip-10-0-142-31.ec2.internal
2026-07-30T18:32:25.381800211Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:25.446585002Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-30T18:32:25.782492283Z Gathering data for ns/openshift-kube-scheduler...
2026-07-30T18:32:26.151965121Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-30T18:32:26.525981731Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-30T18:32:26.897233628Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-30T18:32:27.212517149Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-30T18:32:27.729642595Z Gathering data for ns/openshift-multus...
2026-07-30T18:32:29.111737519Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-30T18:32:30.387660471Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:30.431185910Z Gathering data for ns/openshift-host-network...
2026-07-30T18:32:30.746483315Z Gathering data for ns/openshift-network-diagnostics...
2026-07-30T18:32:31.255097981Z Gathering data for ns/openshift-network-node-identity...
2026-07-30T18:32:31.684288643Z Gathering data for ns/openshift-network-operator...
2026-07-30T18:32:32.055877752Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-30T18:32:32.372116845Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-30T18:32:32.875517397Z Gathering data for ns/openshift-apiserver-operator...
2026-07-30T18:32:33.208663575Z Gathering data for ns/openshift-apiserver...
2026-07-30T18:32:33.596230970Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-30T18:32:33.612072012Z Collecting kubelet logs for node ip-10-0-142-223.ec2.internal
2026-07-30T18:32:33.614228981Z Collecting kubelet logs for node ip-10-0-143-190.ec2.internal
2026-07-30T18:32:33.616040055Z Collecting kubelet logs for node ip-10-0-142-31.ec2.internal
2026-07-30T18:32:33.930406813Z Gathering data for ns/openshift-controller-manager...
2026-07-30T18:32:34.059262374Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mgfj9 namespace
2026-07-30T18:32:34.064602663Z INFO: Node performance data collection complete.
2026-07-30T18:32:34.221989848Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-30T18:32:34.886762294Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-30T18:32:35.201783588Z Gathering data for ns/openshift-service-ca-operator...
2026-07-30T18:32:35.393522655Z [disk usage checker] Volume usage percentage: current = 39 ; allowed = 70
2026-07-30T18:32:35.508311297Z Gathering data for ns/openshift-service-ca...
2026-07-30T18:32:35.851908727Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-30T18:32:36.715836840Z Wrote inspect data to must-gather.
2026-07-30T18:32:36.715883154Z error: inspection completed with the errors occurred while gathering data:
2026-07-30T18:32:36.715883154Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-30T18:32:36.847848280Z error: the server doesn't have a resource type "clusters"
2026-07-30T18:32:37.000178935Z Caches written to disk