2026-07-27T19:01:31.297133539Z [disk usage checker] Started
2026-07-27T19:01:31.300582977Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:31.707019560Z Gathering data for ns/openshift-cluster-version...
2026-07-27T19:01:31.837921073Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T19:01:31.922181008Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T19:01:32.003400688Z Gathering data for ns/default...
2026-07-27T19:01:32.275137062Z Gathering data for ns/openshift...
2026-07-27T19:01:32.670334592Z Gathering data for ns/kube-system...
2026-07-27T19:01:33.415759919Z Gathering data for ns/openshift-etcd...
2026-07-27T19:01:33.698543782Z Wrote inspect data to must-gather.
2026-07-27T19:01:33.698577208Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T19:01:33.698577208Z     namespaces "assisted-installer" not found
2026-07-27T19:01:34.358470037Z Waiting on subprocesses to finish execution.
2026-07-27T19:01:34.364896441Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T19:01:34.365323525Z Executing Istio gather script
2026-07-27T19:01:34.367538366Z INFO: Gathering HAProxy config files
2026-07-27T19:01:34.368017824Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T19:01:34.373928411Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T19:01:34.377057495Z INFO: Collecting host service logs for crio
2026-07-27T19:01:34.377357005Z INFO: Collecting host service logs for kubelet
2026-07-27T19:01:34.377668318Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T19:01:34.377989409Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T19:01:34.378364745Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T19:01:34.378720714Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T19:01:34.379051108Z INFO: Collecting host service logs for NetworkManager
2026-07-27T19:01:34.379491084Z INFO: Collecting host service logs for openvswitch
2026-07-27T19:01:34.379895708Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T19:01:34.380287603Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T19:01:34.380618163Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T19:01:34.381000297Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T19:01:34.488420062Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T19:01:34.786320480Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T19:01:34.834125158Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-lcdg4
2026-07-27T19:01:34.834125158Z insights-runtime-extractor-8ctdw
2026-07-27T19:01:34.834125158Z insights-runtime-extractor-cdct8
2026-07-27T19:01:34.834125158Z insights-runtime-extractor-ptt5w
2026-07-27T19:01:34.840829221Z No resources found in openshift-etcd namespace.
2026-07-27T19:01:35.079721150Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T19:01:35.201349014Z Wrote inspect data to must-gather.
2026-07-27T19:01:35.356213871Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T19:01:35.479764509Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T19:01:35.754435834Z No resources found
2026-07-27T19:01:35.905773380Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T19:01:35.930812697Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T19:01:35.930849761Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T19:01:35.935127987Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T19:01:35.935127987Z See 'oc rsync -h' for help and examples
2026-07-27T19:01:35.944560844Z INFO: Worker host service log collection to complete.
2026-07-27T19:01:35.960316832Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T19:01:35.988452778Z ERROR: No running kube-apiserver pods found
2026-07-27T19:01:36.051159254Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T19:01:36.052357155Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T19:01:36.342857486Z Wrote inspect data to must-gather.
2026-07-27T19:01:36.386225906Z INFO: 'previous-logs' folder not found on ip-10-0-129-58.ec2.internal, skipping...
2026-07-27T19:01:36.608662534Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:36.731903114Z Wrote inspect data to must-gather.
2026-07-27T19:01:36.817550235Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T19:01:36.879239285Z Wrote inspect data to must-gather.
2026-07-27T19:01:36.947755556Z Gathering data for ns/openshift-monitoring...
2026-07-27T19:01:37.010117929Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T19:01:37.011900772Z No resources found
2026-07-27T19:01:37.033537645Z tar: Removing leading `/' from member names
2026-07-27T19:01:37.044132415Z INFO: HAProxy config collection complete.
2026-07-27T19:01:37.163524572Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.250546338Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T19:01:37.258317115Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.272723978Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.346119795Z INFO: 'previous-logs' folder not found on ip-10-0-139-106.ec2.internal, skipping...
2026-07-27T19:01:37.362177124Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T19:01:37.469434771Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T19:01:37.563553788Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.651825612Z INFO: 'previous-logs' folder not found on ip-10-0-140-0.ec2.internal, skipping...
2026-07-27T19:01:37.651898957Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T19:01:37.651997913Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T19:01:37.738756571Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.797566961Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T19:01:37.810760172Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T19:01:37.864555886Z Wrote inspect data to must-gather.
2026-07-27T19:01:37.917754717Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T19:01:38.130825648Z Wrote inspect data to must-gather.
2026-07-27T19:01:38.146722684Z Wrote inspect data to must-gather.
2026-07-27T19:01:38.226350300Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T19:01:38.418148006Z Wrote inspect data to must-gather.
2026-07-27T19:01:38.488915345Z INFO: INTERCONNECT MODE
2026-07-27T19:01:38.488915345Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T19:01:38.491964618Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T19:01:38.543198950Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T19:01:38.664885348Z Wrote inspect data to must-gather.
2026-07-27T19:01:38.689401098Z INFO: Gathering OVN_Northbound from ovnkube-node-6nhms...
2026-07-27T19:01:38.689642700Z INFO: Gathering OVN_Northbound from ovnkube-node-fm8gt...
2026-07-27T19:01:38.689902230Z INFO: Gathering OVN_Northbound from ovnkube-node-vc2tz...
2026-07-27T19:01:38.690215174Z INFO: Gathering OVN_Southbound from ovnkube-node-6nhms...
2026-07-27T19:01:38.690536888Z INFO: Gathering OVN_Southbound from ovnkube-node-fm8gt...
2026-07-27T19:01:38.690842003Z INFO: Gathering OVN_Southbound from ovnkube-node-vc2tz...
2026-07-27T19:01:38.764814452Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T19:01:39.038224535Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.045195381Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.062530836Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.094126525Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.098914349Z Wrote inspect data to must-gather.
2026-07-27T19:01:39.103366381Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.120493874Z tar: Removing leading `/' from member names
2026-07-27T19:01:39.237313030Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T19:01:39.239454493Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T19:01:39.334467381Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T19:01:39.458098756Z Wrote inspect data to must-gather.
2026-07-27T19:01:39.493180850Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:39.577362509Z Gathering data for ns/openshift-network-console...
2026-07-27T19:01:39.730552054Z INFO: Waiting for network log collection to complete ...
2026-07-27T19:01:39.730677978Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T19:01:39.731207446Z INFO: Copying ovnk databases complete.
2026-07-27T19:01:39.733343046Z 4.6M	must-gather/network_logs/ovnk_database_store
2026-07-27T19:01:39.736099633Z ovnk_database_store/
2026-07-27T19:01:39.736150985Z ovnk_database_store/ovnkube-node-6nhms_nbdb
2026-07-27T19:01:39.746796404Z INFO: Getting status from alertmanager-main-0
2026-07-27T19:01:39.747597721Z ovnk_database_store/ovnkube-node-fm8gt_sbdb
2026-07-27T19:01:39.777128856Z ovnk_database_store/ovnkube-node-vc2tz_sbdb
2026-07-27T19:01:39.785137650Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:39.789665926Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T19:01:39.811768721Z ovnk_database_store/ovnkube-node-fm8gt_nbdb
2026-07-27T19:01:39.824089065Z ovnk_database_store/ovnkube-node-6nhms_sbdb
2026-07-27T19:01:39.837146530Z Wrote inspect data to must-gather.
2026-07-27T19:01:39.845075872Z ovnk_database_store/ovnkube-node-vc2tz_nbdb
2026-07-27T19:01:39.861721374Z INFO: Network log collection complete.
2026-07-27T19:01:40.027983607Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:40.131319595Z Gathering data for ns/openshift-console-operator...
2026-07-27T19:01:40.165217366Z Wrote inspect data to must-gather.
2026-07-27T19:01:40.255427840Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:40.258789660Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T19:01:40.464204267Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T19:01:40.467954166Z INFO: Fallback to identify the container image from release info
2026-07-27T19:01:40.470509496Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:40.546739496Z Gathering data for ns/openshift-console...
2026-07-27T19:01:40.671924269Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:40.675765115Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T19:01:40.891499986Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.106422014Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.110949459Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-27T19:01:41.111578298Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T19:01:41.159721795Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T19:01:41.303915221Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.304579242Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T19:01:41.472086952Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T19:01:41.529204654Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.533429913Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-27T19:01:41.632287470Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:41.652789146Z Gathering data for ns/openshift-dns-operator...
2026-07-27T19:01:41.753026264Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.963473847Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:41.968152162Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T19:01:42.018125580Z Gathering data for ns/openshift-dns...
2026-07-27T19:01:42.128183524Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:42.327201173Z Wrote inspect data to /must-gather/istio.
2026-07-27T19:01:42.625327629Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T19:01:42.643044851Z Gathering data for ns/openshift-image-registry...
2026-07-27T19:01:43.047853980Z Done executing Istio gather script
2026-07-27T19:01:43.145995829Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T19:01:43.449297447Z Gathering data for ns/openshift-ingress...
2026-07-27T19:01:43.773746800Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T19:01:43.791711424Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T19:01:44.145928481Z Gathering data for ns/openshift-insights...
2026-07-27T19:01:44.889425854Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T19:01:46.015407119Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T19:01:46.464610726Z Gathering data for ns/kserve...
2026-07-27T19:01:46.637021751Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:47.139048954Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T19:01:48.257480606Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T19:01:48.887552403Z Gathering data for ns/openshift-config...
2026-07-27T19:01:49.174350122Z Gathering data for ns/openshift-config-managed...
2026-07-27T19:01:49.379856242Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T19:01:49.588333595Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T19:01:49.663776372Z Collecting performance related data for node ip-10-0-140-0.ec2.internal
2026-07-27T19:01:49.674483822Z Collecting performance related data for node ip-10-0-129-58.ec2.internal
2026-07-27T19:01:49.680634617Z Collecting performance related data for node ip-10-0-139-106.ec2.internal
2026-07-27T19:01:49.921618632Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T19:01:50.211878982Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T19:01:50.577624392Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T19:01:50.954603032Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T19:01:51.294824741Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T19:01:51.609592169Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T19:01:51.641896563Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:51.944123081Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T19:01:52.298447088Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T19:01:52.915427694Z Gathering data for ns/openshift-multus...
2026-07-27T19:01:54.389968984Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T19:01:55.658051672Z Gathering data for ns/openshift-host-network...
2026-07-27T19:01:55.961083767Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T19:01:56.483045656Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T19:01:56.647231238Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:01:56.870494674Z Gathering data for ns/openshift-network-operator...
2026-07-27T19:01:57.262515076Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T19:01:57.522197740Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T19:01:58.016376693Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T19:01:58.316063206Z Gathering data for ns/openshift-apiserver...
2026-07-27T19:01:58.399596741Z Collecting kubelet logs for node ip-10-0-140-0.ec2.internal
2026-07-27T19:01:58.401460768Z Collecting kubelet logs for node ip-10-0-129-58.ec2.internal
2026-07-27T19:01:58.403290068Z Collecting kubelet logs for node ip-10-0-139-106.ec2.internal
2026-07-27T19:01:58.785676025Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T19:01:58.914237325Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-grcz9 namespace
2026-07-27T19:01:58.919162431Z INFO: Node performance data collection complete.
2026-07-27T19:01:59.132755660Z Gathering data for ns/openshift-controller-manager...
2026-07-27T19:01:59.402071066Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T19:02:00.044063596Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T19:02:00.303720115Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T19:02:00.600753996Z Gathering data for ns/openshift-service-ca...
2026-07-27T19:02:00.973742483Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T19:02:01.652081770Z [disk usage checker] Volume usage percentage: current = 43 ; allowed = 70
2026-07-27T19:02:01.947187643Z Wrote inspect data to must-gather.
2026-07-27T19:02:01.947230638Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T19:02:01.947230638Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T19:02:02.080082225Z error: the server doesn't have a resource type "clusters"
2026-07-27T19:02:02.230192790Z Caches written to disk