2026-07-24T12:46:35.949476720Z [disk usage checker] Started
2026-07-24T12:46:35.953015912Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:46:36.138041405Z Gathering data for ns/openshift-cluster-version...
2026-07-24T12:46:36.265751016Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-24T12:46:36.364889182Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T12:46:36.449501840Z Gathering data for ns/default...
2026-07-24T12:46:36.755612105Z Gathering data for ns/openshift...
2026-07-24T12:46:37.152150496Z Gathering data for ns/kube-system...
2026-07-24T12:46:37.904156339Z Gathering data for ns/openshift-etcd...
2026-07-24T12:46:38.229465682Z Wrote inspect data to must-gather.
2026-07-24T12:46:38.229501269Z error: inspection completed with the errors occurred while gathering data:
2026-07-24T12:46:38.229501269Z     namespaces "assisted-installer" not found
2026-07-24T12:46:38.403805771Z Waiting on subprocesses to finish execution.
2026-07-24T12:46:38.412315947Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-24T12:46:38.417123075Z Executing Istio gather script
2026-07-24T12:46:38.418425165Z INFO: Collecting host service logs for crio
2026-07-24T12:46:38.418780387Z INFO: Collecting host service logs for kubelet
2026-07-24T12:46:38.419163885Z INFO: Collecting host service logs for rpm-ostreed
2026-07-24T12:46:38.419512482Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-24T12:46:38.419841618Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-24T12:46:38.420163116Z INFO: Gathering HAProxy config files
2026-07-24T12:46:38.420255624Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-24T12:46:38.420558236Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-24T12:46:38.420582366Z INFO: Collecting host service logs for NetworkManager
2026-07-24T12:46:38.420894133Z INFO: Collecting host service logs for openvswitch
2026-07-24T12:46:38.421495062Z INFO: Collecting host service logs for ovs-configuration
2026-07-24T12:46:38.421800891Z INFO: Collecting host service logs for ovsdb-server
2026-07-24T12:46:38.422120260Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-24T12:46:38.422476973Z INFO: Waiting for worker host service log collection to complete ...
2026-07-24T12:46:38.435630469Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-24T12:46:38.446435434Z INFO: Waiting for node performance related collection to complete ...
2026-07-24T12:46:38.762953908Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-24T12:46:39.001418398Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-qbsjh
2026-07-24T12:46:39.001472184Z insights-runtime-extractor-4bl7v
2026-07-24T12:46:39.001490300Z insights-runtime-extractor-n6248
2026-07-24T12:46:39.001507483Z insights-runtime-extractor-vvg4t
2026-07-24T12:46:39.014739471Z No resources found in openshift-etcd namespace.
2026-07-24T12:46:39.087445273Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-24T12:46:39.470469231Z INFO: "metallb-operator" not detected. Skipping.
2026-07-24T12:46:39.491381038Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-24T12:46:39.496443788Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-24T12:46:39.551708157Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-24T12:46:39.551708157Z See 'oc rsync -h' for help and examples
2026-07-24T12:46:39.651009654Z error: the server doesn't have a resource type "performanceprofile"
2026-07-24T12:46:39.662391204Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-24T12:46:39.779952914Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-24T12:46:39.780047126Z INFO: on-disk MachineConfig config collection complete.
2026-07-24T12:46:39.846836344Z No resources found
2026-07-24T12:46:39.931711853Z INFO: Worker host service log collection to complete.
2026-07-24T12:46:39.967907602Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.023323025Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-24T12:46:40.176814484Z Gathering data for ns/openshift-monitoring...
2026-07-24T12:46:40.524516302Z ERROR: No running kube-apiserver pods found
2026-07-24T12:46:40.588415644Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.655767770Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.686027791Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.720418249Z tar: Removing leading `/' from member names
2026-07-24T12:46:40.726491502Z INFO: HAProxy config collection complete.
2026-07-24T12:46:40.774966169Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.782859616Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T12:46:40.814794865Z Wrote inspect data to must-gather.
2026-07-24T12:46:40.831810659Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-24T12:46:40.895266913Z No resources found
2026-07-24T12:46:40.975814506Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:46:41.015683028Z INFO: 'previous-logs' folder not found on ip-10-0-134-28.ec2.internal, skipping...
2026-07-24T12:46:41.048254220Z Wrote inspect data to must-gather.
2026-07-24T12:46:41.071839609Z error: the server doesn't have a resource type "machineconfigs"
2026-07-24T12:46:41.180473026Z Wrote inspect data to must-gather.
2026-07-24T12:46:41.189141363Z INFO: Getting rules from prometheus-k8s-0
2026-07-24T12:46:41.254682167Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-24T12:46:41.319267974Z INFO: 'previous-logs' folder not found on ip-10-0-136-40.ec2.internal, skipping...
2026-07-24T12:46:41.478842729Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-24T12:46:41.499641855Z Wrote inspect data to must-gather.
2026-07-24T12:46:41.530132324Z Wrote inspect data to must-gather.
2026-07-24T12:46:41.642930529Z INFO: 'previous-logs' folder not found on ip-10-0-142-169.ec2.internal, skipping...
2026-07-24T12:46:41.642959948Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-24T12:46:41.643015119Z INFO: Machine Config Daemon termination log collection complete.
2026-07-24T12:46:41.656189508Z INFO: Getting status/config from prometheus-k8s-0
2026-07-24T12:46:41.697614271Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-24T12:46:41.758014446Z Wrote inspect data to must-gather.
2026-07-24T12:46:41.934312250Z Wrote inspect data to must-gather.
2026-07-24T12:46:42.017699011Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-24T12:46:42.058271530Z INFO: INTERCONNECT MODE
2026-07-24T12:46:42.058315545Z INFO: Gathering ovn-kubernetes DBs
2026-07-24T12:46:42.072218717Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-24T12:46:42.158202084Z Wrote inspect data to must-gather.
2026-07-24T12:46:42.219171682Z INFO: Gathering OVN_Northbound from ovnkube-node-dvdzl...
2026-07-24T12:46:42.219414133Z INFO: Gathering OVN_Northbound from ovnkube-node-tn6ch...
2026-07-24T12:46:42.219682466Z INFO: Gathering OVN_Northbound from ovnkube-node-vjqkm...
2026-07-24T12:46:42.219924332Z INFO: Gathering OVN_Southbound from ovnkube-node-dvdzl...
2026-07-24T12:46:42.220238676Z INFO: Gathering OVN_Southbound from ovnkube-node-tn6ch...
2026-07-24T12:46:42.220514891Z INFO: Gathering OVN_Southbound from ovnkube-node-vjqkm...
2026-07-24T12:46:42.252205187Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-24T12:46:42.372547423Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-24T12:46:42.506767500Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.539190964Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.556234700Z Wrote inspect data to must-gather.
2026-07-24T12:46:42.583083112Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.609601050Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.629461970Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:42.636704507Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.686038719Z tar: Removing leading `/' from member names
2026-07-24T12:46:42.750681838Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-24T12:46:42.787213791Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-24T12:46:42.894631092Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:42.900649364Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-24T12:46:42.935177909Z Wrote inspect data to must-gather.
2026-07-24T12:46:43.037555520Z Gathering data for ns/openshift-network-console...
2026-07-24T12:46:43.052757233Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-24T12:46:43.132143366Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:43.147602866Z INFO: Waiting for network log collection to complete ...
2026-07-24T12:46:43.147719302Z INFO: Waiting for ovnk database copies to complete ...
2026-07-24T12:46:43.148255612Z INFO: Copying ovnk databases complete.
2026-07-24T12:46:43.150146131Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-24T12:46:43.156499592Z ovnk_database_store/
2026-07-24T12:46:43.156548997Z ovnk_database_store/ovnkube-node-tn6ch_nbdb
2026-07-24T12:46:43.170165813Z ovnk_database_store/ovnkube-node-vjqkm_sbdb
2026-07-24T12:46:43.187741117Z ovnk_database_store/ovnkube-node-dvdzl_nbdb
2026-07-24T12:46:43.194054519Z ovnk_database_store/ovnkube-node-vjqkm_nbdb
2026-07-24T12:46:43.201777462Z ovnk_database_store/ovnkube-node-dvdzl_sbdb
2026-07-24T12:46:43.216727317Z ovnk_database_store/ovnkube-node-tn6ch_sbdb
2026-07-24T12:46:43.234659601Z INFO: Network log collection complete.
2026-07-24T12:46:43.259476362Z Wrote inspect data to must-gather.
2026-07-24T12:46:43.347209204Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:43.350303970Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-24T12:46:43.474845652Z INFO: Getting status from alertmanager-main-0
2026-07-24T12:46:43.565562594Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:43.574351459Z Wrote inspect data to must-gather.
2026-07-24T12:46:43.614593030Z Gathering data for ns/openshift-console-operator...
2026-07-24T12:46:43.778881106Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:43.782768270Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-24T12:46:43.854754752Z Wrote inspect data to must-gather.
2026-07-24T12:46:43.997908046Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:44.052479554Z Gathering data for ns/openshift-console...
2026-07-24T12:46:44.095506122Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-24T12:46:44.098382975Z INFO: Fallback to identify the container image from release info
2026-07-24T12:46:44.200964772Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:44.203102062Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-24T12:46:44.346439576Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:44.533694222Z Wrote inspect data to /must-gather/istio.
2026-07-24T12:46:44.569860392Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-24T12:46:44.826237456Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-24T12:46:44.956292461Z daemonset.apps/perf-node-gather-daemonset created
2026-07-24T12:46:45.100183507Z Waiting for performance profile collector pods to become ready: 1
2026-07-24T12:46:45.189493956Z Done executing Istio gather script
2026-07-24T12:46:45.193824453Z Gathering data for ns/openshift-dns-operator...
2026-07-24T12:46:45.510856238Z Gathering data for ns/openshift-dns...
2026-07-24T12:46:45.983291050Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:46:46.193354034Z Gathering data for ns/openshift-image-registry...
2026-07-24T12:46:46.212979090Z Waiting for performance profile collector pods to become ready: 2
2026-07-24T12:46:46.649676603Z Gathering data for ns/openshift-ingress-operator...
2026-07-24T12:46:46.971311895Z Gathering data for ns/openshift-ingress...
2026-07-24T12:46:47.330749800Z Waiting for performance profile collector pods to become ready: 3
2026-07-24T12:46:47.393398753Z Gathering data for ns/openshift-ingress-canary...
2026-07-24T12:46:47.804510991Z Gathering data for ns/openshift-insights...
2026-07-24T12:46:48.442211240Z Waiting for performance profile collector pods to become ready: 4
2026-07-24T12:46:49.547300855Z Waiting for performance profile collector pods to become ready: 5
2026-07-24T12:46:49.856004859Z Gathering data for ns/openshift-jobset-operator...
2026-07-24T12:46:50.647584629Z Gathering data for ns/openshift-config...
2026-07-24T12:46:50.656549703Z Waiting for performance profile collector pods to become ready: 6
2026-07-24T12:46:50.954397636Z Gathering data for ns/openshift-config-managed...
2026-07-24T12:46:50.989219434Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:46:51.378359011Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-24T12:46:51.696980476Z Gathering data for ns/openshift-kube-apiserver...
2026-07-24T12:46:51.783835007Z Waiting for performance profile collector pods to become ready: 7
2026-07-24T12:46:51.978846410Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-24T12:46:52.319738282Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-24T12:46:52.687573856Z Gathering data for ns/openshift-kube-scheduler...
2026-07-24T12:46:52.896125983Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-24T12:46:52.957509847Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-24T12:46:53.138666182Z Collecting performance related data for node ip-10-0-136-40.ec2.internal
2026-07-24T12:46:53.146675423Z Collecting performance related data for node ip-10-0-142-169.ec2.internal
2026-07-24T12:46:53.156136472Z Collecting performance related data for node ip-10-0-134-28.ec2.internal
2026-07-24T12:46:53.313445876Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-24T12:46:53.685800261Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-24T12:46:54.009181656Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-24T12:46:54.554419427Z Gathering data for ns/openshift-multus...
2026-07-24T12:46:55.994083003Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:46:56.133529660Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-24T12:46:57.467218747Z Gathering data for ns/openshift-host-network...
2026-07-24T12:46:57.842001696Z Gathering data for ns/openshift-network-diagnostics...
2026-07-24T12:46:58.436672625Z Gathering data for ns/openshift-network-node-identity...
2026-07-24T12:46:58.840690502Z Gathering data for ns/openshift-network-operator...
2026-07-24T12:46:59.228788603Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-24T12:46:59.554538591Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-24T12:47:00.061502194Z Gathering data for ns/openshift-apiserver-operator...
2026-07-24T12:47:00.397785078Z Gathering data for ns/openshift-apiserver...
2026-07-24T12:47:00.833289743Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-24T12:47:00.999195230Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-24T12:47:01.153376183Z Gathering data for ns/openshift-controller-manager...
2026-07-24T12:47:01.338749211Z Collecting kubelet logs for node ip-10-0-136-40.ec2.internal
2026-07-24T12:47:01.341007143Z Collecting kubelet logs for node ip-10-0-142-169.ec2.internal
2026-07-24T12:47:01.343229131Z Collecting kubelet logs for node ip-10-0-134-28.ec2.internal
2026-07-24T12:47:01.458023246Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-24T12:47:01.671321005Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qdxdl namespace
2026-07-24T12:47:01.677454639Z INFO: Node performance data collection complete.
2026-07-24T12:47:02.132424895Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-24T12:47:02.404375726Z Gathering data for ns/openshift-service-ca-operator...
2026-07-24T12:47:02.715868679Z Gathering data for ns/openshift-service-ca...
2026-07-24T12:47:03.073085706Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-24T12:47:04.008508100Z Wrote inspect data to must-gather.
2026-07-24T12:47:04.008553123Z error: inspection completed with the errors occurred while gathering data:
2026-07-24T12:47:04.008553123Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-24T12:47:04.125733044Z error: the server doesn't have a resource type "clusters"
2026-07-24T12:47:04.258776608Z Caches written to disk