2026-07-09T15:50:24.128609028Z [disk usage checker] Started
2026-07-09T15:50:24.131917494Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:24.334977641Z Gathering data for ns/openshift-cluster-version...
2026-07-09T15:50:24.479759184Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T15:50:24.582597300Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T15:50:24.667195082Z Gathering data for ns/default...
2026-07-09T15:50:25.019870089Z Gathering data for ns/openshift...
2026-07-09T15:50:25.504644456Z Gathering data for ns/kube-system...
2026-07-09T15:50:26.303928564Z Gathering data for ns/openshift-etcd...
2026-07-09T15:50:26.662564781Z Wrote inspect data to must-gather.
2026-07-09T15:50:26.662599512Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T15:50:26.662599512Z     namespaces "assisted-installer" not found
2026-07-09T15:50:26.894234865Z Waiting on subprocesses to finish execution.
2026-07-09T15:50:26.898725929Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T15:50:26.901781895Z INFO: Gathering HAProxy config files
2026-07-09T15:50:26.902089025Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T15:50:26.908401698Z INFO: Collecting host service logs for crio
2026-07-09T15:50:26.908557578Z INFO: Collecting host service logs for kubelet
2026-07-09T15:50:26.908941343Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T15:50:26.909284561Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T15:50:26.909801907Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T15:50:26.910147746Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T15:50:26.910501913Z INFO: Collecting host service logs for NetworkManager
2026-07-09T15:50:26.910866648Z INFO: Collecting host service logs for openvswitch
2026-07-09T15:50:26.911255801Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T15:50:26.911728346Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T15:50:26.912059071Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T15:50:26.912437044Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T15:50:26.917349420Z Executing Istio gather script
2026-07-09T15:50:26.920749897Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T15:50:26.996693031Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T15:50:27.556271906Z No resources found in openshift-etcd namespace.
2026-07-09T15:50:27.656484741Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T15:50:27.883863918Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T15:50:27.936876800Z Wrote inspect data to must-gather.
2026-07-09T15:50:27.939838486Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-d7c8c
2026-07-09T15:50:27.939838486Z insights-runtime-extractor-h5ng6
2026-07-09T15:50:27.939838486Z insights-runtime-extractor-nx9nr
2026-07-09T15:50:27.939838486Z insights-runtime-extractor-wcbmw
2026-07-09T15:50:27.949104958Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T15:50:27.949182472Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T15:50:27.983712655Z No resources found
2026-07-09T15:50:28.001072771Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T15:50:28.024483699Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T15:50:28.339272819Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T15:50:28.421045474Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T15:50:28.451785703Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T15:50:28.451785703Z See 'oc rsync -h' for help and examples
2026-07-09T15:50:28.501973226Z INFO: Worker host service log collection to complete.
2026-07-09T15:50:28.545600789Z ERROR: No running kube-apiserver pods found
2026-07-09T15:50:28.964130728Z Wrote inspect data to must-gather.
2026-07-09T15:50:28.996402110Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T15:50:29.058092152Z Wrote inspect data to must-gather.
2026-07-09T15:50:29.136165115Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T15:50:29.136524955Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T15:50:29.147284818Z Wrote inspect data to must-gather.
2026-07-09T15:50:29.175244371Z Wrote inspect data to must-gather.
2026-07-09T15:50:29.207281557Z INFO: 'previous-logs' folder not found on ip-10-0-130-131.ec2.internal, skipping...
2026-07-09T15:50:29.207844661Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:29.507206109Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T15:50:29.548159694Z No resources found
2026-07-09T15:50:29.579191985Z Wrote inspect data to must-gather.
2026-07-09T15:50:29.607366799Z Wrote inspect data to must-gather.
2026-07-09T15:50:29.617119759Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T15:50:29.638819209Z tar: Removing leading `/' from member names
2026-07-09T15:50:29.642371632Z INFO: HAProxy config collection complete.
2026-07-09T15:50:29.676625677Z Gathering data for ns/openshift-monitoring...
2026-07-09T15:50:29.786472775Z INFO: 'previous-logs' folder not found on ip-10-0-137-28.ec2.internal, skipping...
2026-07-09T15:50:29.817384110Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T15:50:29.817840728Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T15:50:29.910689947Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.080780919Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.095711020Z INFO: 'previous-logs' folder not found on ip-10-0-138-133.ec2.internal, skipping...
2026-07-09T15:50:30.095809499Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T15:50:30.095860096Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T15:50:30.170450707Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T15:50:30.220965156Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.268403900Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T15:50:30.273505977Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T15:50:30.473049098Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.487630947Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.707318598Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T15:50:30.716355598Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.777242033Z INFO: INTERCONNECT MODE
2026-07-09T15:50:30.777299737Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T15:50:30.930728306Z Wrote inspect data to must-gather.
2026-07-09T15:50:30.942868212Z INFO: Gathering OVN_Northbound from ovnkube-node-hznv5...
2026-07-09T15:50:30.943080989Z INFO: Gathering OVN_Northbound from ovnkube-node-xdmbt...
2026-07-09T15:50:30.943298757Z INFO: Gathering OVN_Northbound from ovnkube-node-zp7dp...
2026-07-09T15:50:30.943592549Z INFO: Gathering OVN_Southbound from ovnkube-node-hznv5...
2026-07-09T15:50:30.944077187Z INFO: Gathering OVN_Southbound from ovnkube-node-xdmbt...
2026-07-09T15:50:30.944378976Z INFO: Gathering OVN_Southbound from ovnkube-node-zp7dp...
2026-07-09T15:50:30.987953732Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T15:50:31.231246858Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.254393896Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.260115117Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.280688619Z Wrote inspect data to must-gather.
2026-07-09T15:50:31.296536864Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.301031000Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.332716947Z tar: Removing leading `/' from member names
2026-07-09T15:50:31.403887161Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T15:50:31.423158387Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T15:50:31.512263670Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T15:50:31.557750676Z Wrote inspect data to must-gather.
2026-07-09T15:50:31.646983004Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:31.702539375Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T15:50:31.811866264Z Wrote inspect data to must-gather.
2026-07-09T15:50:31.858211122Z INFO: Waiting for network log collection to complete ...
2026-07-09T15:50:31.858326260Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T15:50:31.858889562Z INFO: Copying ovnk databases complete.
2026-07-09T15:50:31.860917699Z 3.0M	must-gather/network_logs/ovnk_database_store
2026-07-09T15:50:31.863139343Z ovnk_database_store/
2026-07-09T15:50:31.863139343Z ovnk_database_store/ovnkube-node-hznv5_nbdb
2026-07-09T15:50:31.873162372Z ovnk_database_store/ovnkube-node-zp7dp_nbdb
2026-07-09T15:50:31.883085516Z ovnk_database_store/ovnkube-node-zp7dp_sbdb
2026-07-09T15:50:31.900807950Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:31.903826869Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T15:50:31.908159143Z ovnk_database_store/ovnkube-node-xdmbt_sbdb
2026-07-09T15:50:31.924765175Z ovnk_database_store/ovnkube-node-hznv5_sbdb
2026-07-09T15:50:31.941673643Z ovnk_database_store/ovnkube-node-xdmbt_nbdb
2026-07-09T15:50:31.952269968Z INFO: Network log collection complete.
2026-07-09T15:50:32.060401387Z Wrote inspect data to must-gather.
2026-07-09T15:50:32.125384456Z INFO: Getting status from alertmanager-main-0
2026-07-09T15:50:32.125791496Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:32.229460561Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T15:50:32.355276352Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:32.358551557Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T15:50:32.370251252Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T15:50:32.372625824Z INFO: Fallback to identify the container image from release info
2026-07-09T15:50:32.414007439Z Gathering data for ns/openshift-network-console...
2026-07-09T15:50:32.563208904Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:32.768335127Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:32.771720238Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T15:50:32.883221925Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T15:50:32.886031164Z Gathering data for ns/openshift-console-operator...
2026-07-09T15:50:33.034573605Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:33.041062828Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T15:50:33.174846727Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T15:50:33.247949353Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:33.250694741Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T15:50:33.311219476Z Gathering data for ns/openshift-console...
2026-07-09T15:50:33.432601035Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:33.638846303Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:33.641342309Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T15:50:33.833416905Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:33.878044895Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T15:50:34.029197111Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:34.031496632Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T15:50:34.216096358Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:34.223186223Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:34.327120900Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T15:50:34.432690436Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:34.435277348Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T15:50:34.435646701Z Gathering data for ns/openshift-dns-operator...
2026-07-09T15:50:34.611272797Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:34.787165535Z Gathering data for ns/openshift-dns...
2026-07-09T15:50:34.819457815Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:34.821817953Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T15:50:34.971874304Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:35.188951963Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:35.192052843Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T15:50:35.364534862Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:35.480627215Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T15:50:35.572087736Z Wrote inspect data to /must-gather/istio.
2026-07-09T15:50:35.623266373Z Gathering data for ns/openshift-image-registry...
2026-07-09T15:50:36.205392005Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T15:50:36.276525439Z Done executing Istio gather script
2026-07-09T15:50:36.616369665Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T15:50:36.630316768Z Gathering data for ns/openshift-ingress...
2026-07-09T15:50:36.983614035Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T15:50:37.365234291Z Gathering data for ns/openshift-insights...
2026-07-09T15:50:37.745566353Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T15:50:38.877060833Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T15:50:39.228252412Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:39.608232237Z Gathering data for ns/kserve...
2026-07-09T15:50:40.008368290Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T15:50:40.863884544Z Gathering data for ns/openshift-config...
2026-07-09T15:50:41.147116358Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T15:50:41.168052555Z Gathering data for ns/openshift-config-managed...
2026-07-09T15:50:41.408881833Z Collecting performance related data for node ip-10-0-130-131.ec2.internal
2026-07-09T15:50:41.416108051Z Collecting performance related data for node ip-10-0-137-28.ec2.internal
2026-07-09T15:50:41.424193519Z Collecting performance related data for node ip-10-0-138-133.ec2.internal
2026-07-09T15:50:41.697533636Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T15:50:42.072562693Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T15:50:42.464389684Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T15:50:42.920581372Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T15:50:43.271771024Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T15:50:43.578706033Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T15:50:43.893061821Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T15:50:44.235910291Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:44.296419860Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T15:50:44.679142581Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T15:50:45.222129528Z Gathering data for ns/openshift-multus...
2026-07-09T15:50:46.915918058Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T15:50:48.271078473Z Gathering data for ns/openshift-host-network...
2026-07-09T15:50:48.570419555Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T15:50:49.126053025Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T15:50:49.241067231Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:49.566258442Z Gathering data for ns/openshift-network-operator...
2026-07-09T15:50:49.729502991Z Collecting kubelet logs for node ip-10-0-130-131.ec2.internal
2026-07-09T15:50:49.732833991Z Collecting kubelet logs for node ip-10-0-137-28.ec2.internal
2026-07-09T15:50:49.734756573Z Collecting kubelet logs for node ip-10-0-138-133.ec2.internal
2026-07-09T15:50:50.031065567Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T15:50:50.102039003Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-lkw82 namespace
2026-07-09T15:50:50.107885937Z INFO: Node performance data collection complete.
2026-07-09T15:50:50.345689089Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T15:50:50.849490809Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T15:50:51.177234824Z Gathering data for ns/openshift-apiserver...
2026-07-09T15:50:51.635471779Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T15:50:51.929762044Z Gathering data for ns/openshift-controller-manager...
2026-07-09T15:50:52.221308880Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T15:50:52.890304941Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T15:50:53.227862919Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T15:50:53.567700579Z Gathering data for ns/openshift-service-ca...
2026-07-09T15:50:54.004778766Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T15:50:54.246808813Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-09T15:50:54.933604569Z Wrote inspect data to must-gather.
2026-07-09T15:50:54.933643535Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T15:50:54.933643535Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T15:50:55.077223413Z error: the server doesn't have a resource type "clusters"
2026-07-09T15:50:55.224677979Z Caches written to disk