2026-07-27T21:40:23.852267231Z [disk usage checker] Started
2026-07-27T21:40:23.856778663Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:24.056553635Z Gathering data for ns/openshift-cluster-version...
2026-07-27T21:40:24.197768856Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T21:40:24.278328446Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:40:24.362557945Z Gathering data for ns/default...
2026-07-27T21:40:24.663071842Z Gathering data for ns/openshift...
2026-07-27T21:40:25.067828303Z Gathering data for ns/kube-system...
2026-07-27T21:40:25.767319360Z Gathering data for ns/openshift-etcd...
2026-07-27T21:40:26.063706968Z Wrote inspect data to must-gather.
2026-07-27T21:40:26.063740418Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T21:40:26.063740418Z     namespaces "assisted-installer" not found
2026-07-27T21:40:26.647527071Z Waiting on subprocesses to finish execution.
2026-07-27T21:40:26.653779439Z INFO: Gathering HAProxy config files
2026-07-27T21:40:26.657737611Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-27T21:40:26.658253848Z Executing Istio gather script
2026-07-27T21:40:26.660534461Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-27T21:40:26.662495888Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-27T21:40:26.677850038Z INFO: Collecting host service logs for crio
2026-07-27T21:40:26.678157449Z INFO: Collecting host service logs for kubelet
2026-07-27T21:40:26.678476596Z INFO: Collecting host service logs for rpm-ostreed
2026-07-27T21:40:26.678878804Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-27T21:40:26.679258887Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-27T21:40:26.679698861Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-27T21:40:26.680084010Z INFO: Collecting host service logs for NetworkManager
2026-07-27T21:40:26.680416197Z INFO: Collecting host service logs for openvswitch
2026-07-27T21:40:26.680819484Z INFO: Collecting host service logs for ovs-configuration
2026-07-27T21:40:26.681180456Z INFO: Collecting host service logs for ovsdb-server
2026-07-27T21:40:26.681538045Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-27T21:40:26.681942499Z INFO: Waiting for worker host service log collection to complete ...
2026-07-27T21:40:26.762138738Z INFO: Waiting for node performance related collection to complete ...
2026-07-27T21:40:27.263033185Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-27T21:40:27.343336264Z error: the server doesn't have a resource type "performanceprofile"
2026-07-27T21:40:27.544338406Z INFO: "metallb-operator" not detected. Skipping.
2026-07-27T21:40:27.575956449Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-27T21:40:27.710874004Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-27T21:40:27.710907753Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-27T21:40:27.944977635Z No resources found
2026-07-27T21:40:28.019284276Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-wtrtk
2026-07-27T21:40:28.019331884Z insights-runtime-extractor-8mxbp
2026-07-27T21:40:28.019351706Z insights-runtime-extractor-c66x5
2026-07-27T21:40:28.019371338Z insights-runtime-extractor-dwcjn
2026-07-27T21:40:28.114131079Z No resources found in openshift-etcd namespace.
2026-07-27T21:40:28.131466744Z Wrote inspect data to must-gather.
2026-07-27T21:40:28.320559141Z INFO: Worker host service log collection to complete.
2026-07-27T21:40:28.323178246Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-27T21:40:28.323202408Z INFO: on-disk MachineConfig config collection complete.
2026-07-27T21:40:28.427586377Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-27T21:40:28.599370975Z Wrote inspect data to must-gather.
2026-07-27T21:40:28.616096861Z Wrote inspect data to must-gather.
2026-07-27T21:40:28.654795270Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-27T21:40:28.760124270Z INFO: 'previous-logs' folder not found on ip-10-0-131-253.ec2.internal, skipping...
2026-07-27T21:40:28.762704802Z Gathering data for ns/openshift-monitoring...
2026-07-27T21:40:28.770730353Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-27T21:40:28.770730353Z See 'oc rsync -h' for help and examples
2026-07-27T21:40:28.871658846Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-27T21:40:28.874623308Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:29.029650509Z ERROR: No running kube-apiserver pods found
2026-07-27T21:40:29.041816986Z No resources found
2026-07-27T21:40:29.177923939Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.204636119Z tar: Removing leading `/' from member names
2026-07-27T21:40:29.210979010Z INFO: HAProxy config collection complete.
2026-07-27T21:40:29.216103649Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:40:29.349088898Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.350560245Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.355949942Z error: the server doesn't have a resource type "machineconfigs"
2026-07-27T21:40:29.436475429Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.541631498Z INFO: 'previous-logs' folder not found on ip-10-0-134-38.ec2.internal, skipping...
2026-07-27T21:40:29.566468770Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-27T21:40:29.682395577Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.698714525Z INFO: Getting rules from prometheus-k8s-0
2026-07-27T21:40:29.861448032Z Wrote inspect data to must-gather.
2026-07-27T21:40:29.908064958Z INFO: 'previous-logs' folder not found on ip-10-0-139-153.ec2.internal, skipping...
2026-07-27T21:40:29.908089379Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-27T21:40:29.908113920Z INFO: Machine Config Daemon termination log collection complete.
2026-07-27T21:40:30.008710814Z Wrote inspect data to must-gather.
2026-07-27T21:40:30.036411114Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-27T21:40:30.229546551Z INFO: Getting status/config from prometheus-k8s-0
2026-07-27T21:40:30.295390611Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-27T21:40:30.304453351Z Wrote inspect data to must-gather.
2026-07-27T21:40:30.316358411Z Wrote inspect data to must-gather.
2026-07-27T21:40:30.447093111Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-27T21:40:30.571256013Z Wrote inspect data to must-gather.
2026-07-27T21:40:30.658088406Z INFO: INTERCONNECT MODE
2026-07-27T21:40:30.658159351Z INFO: Gathering ovn-kubernetes DBs
2026-07-27T21:40:30.716903889Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-27T21:40:30.849727246Z INFO: Gathering OVN_Northbound from ovnkube-node-jnlqv...
2026-07-27T21:40:30.849992225Z INFO: Gathering OVN_Northbound from ovnkube-node-qx29w...
2026-07-27T21:40:30.850214495Z INFO: Gathering OVN_Northbound from ovnkube-node-rk54c...
2026-07-27T21:40:30.850487170Z INFO: Gathering OVN_Southbound from ovnkube-node-jnlqv...
2026-07-27T21:40:30.850714773Z INFO: Gathering OVN_Southbound from ovnkube-node-qx29w...
2026-07-27T21:40:30.850968548Z INFO: Gathering OVN_Southbound from ovnkube-node-rk54c...
2026-07-27T21:40:30.851071198Z Wrote inspect data to must-gather.
2026-07-27T21:40:31.088066095Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-27T21:40:31.143995917Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.184766143Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.228166221Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.228209067Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.256033749Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-27T21:40:31.256591183Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.273771933Z tar: Removing leading `/' from member names
2026-07-27T21:40:31.381152225Z Wrote inspect data to must-gather.
2026-07-27T21:40:31.463078215Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-27T21:40:31.516289473Z Gathering data for ns/openshift-network-console...
2026-07-27T21:40:31.652170186Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-27T21:40:31.743370222Z Wrote inspect data to must-gather.
2026-07-27T21:40:31.766969102Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-27T21:40:31.800718754Z INFO: Waiting for network log collection to complete ...
2026-07-27T21:40:31.800837263Z INFO: Waiting for ovnk database copies to complete ...
2026-07-27T21:40:31.801357218Z INFO: Copying ovnk databases complete.
2026-07-27T21:40:31.803682277Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-27T21:40:31.806913394Z ovnk_database_store/
2026-07-27T21:40:31.806961479Z ovnk_database_store/ovnkube-node-qx29w_nbdb
2026-07-27T21:40:31.819052524Z ovnk_database_store/ovnkube-node-rk54c_sbdb
2026-07-27T21:40:31.843142381Z ovnk_database_store/ovnkube-node-rk54c_nbdb
2026-07-27T21:40:31.853408877Z ovnk_database_store/ovnkube-node-jnlqv_nbdb
2026-07-27T21:40:31.862499697Z ovnk_database_store/ovnkube-node-jnlqv_sbdb
2026-07-27T21:40:31.882207939Z ovnk_database_store/ovnkube-node-qx29w_sbdb
2026-07-27T21:40:31.885036130Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:31.907388573Z INFO: Network log collection complete.
2026-07-27T21:40:32.098969644Z Gathering data for ns/openshift-console-operator...
2026-07-27T21:40:32.101142837Z Wrote inspect data to must-gather.
2026-07-27T21:40:32.129919135Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:32.132749271Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-27T21:40:32.233409074Z INFO: Getting status from alertmanager-main-0
2026-07-27T21:40:32.363245008Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:32.405940875Z Wrote inspect data to must-gather.
2026-07-27T21:40:32.569743086Z Gathering data for ns/openshift-console...
2026-07-27T21:40:32.582063231Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:32.584442166Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-27T21:40:32.712611892Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-27T21:40:32.715395199Z INFO: Fallback to identify the container image from release info
2026-07-27T21:40:32.798231497Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.020787420Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.025005499Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-27T21:40:33.113112628Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-27T21:40:33.267959826Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.270719481Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-27T21:40:33.419410225Z daemonset.apps/perf-node-gather-daemonset created
2026-07-27T21:40:33.500516010Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.503049684Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-27T21:40:33.579100044Z Waiting for performance profile collector pods to become ready: 1
2026-07-27T21:40:33.691607552Z Gathering data for ns/openshift-dns-operator...
2026-07-27T21:40:33.711133008Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.886464537Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:33.928982431Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:33.932374987Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-27T21:40:34.058374095Z Gathering data for ns/openshift-dns...
2026-07-27T21:40:34.117313049Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:34.291930622Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:34.294323665Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-27T21:40:34.473462401Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:34.694839831Z Wrote inspect data to /must-gather/istio.
2026-07-27T21:40:34.715892183Z Gathering data for ns/openshift-image-registry...
2026-07-27T21:40:34.733857242Z Waiting for performance profile collector pods to become ready: 2
2026-07-27T21:40:35.213021075Z Gathering data for ns/openshift-ingress-operator...
2026-07-27T21:40:35.404712809Z Done executing Istio gather script
2026-07-27T21:40:35.535663622Z Gathering data for ns/openshift-ingress...
2026-07-27T21:40:35.870965496Z Waiting for performance profile collector pods to become ready: 3
2026-07-27T21:40:35.891105897Z Gathering data for ns/openshift-ingress-canary...
2026-07-27T21:40:36.267203013Z Gathering data for ns/openshift-insights...
2026-07-27T21:40:36.992743397Z Waiting for performance profile collector pods to become ready: 4
2026-07-27T21:40:38.113905670Z Waiting for performance profile collector pods to become ready: 5
2026-07-27T21:40:38.605733036Z Gathering data for ns/kserve...
2026-07-27T21:40:38.891574733Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:39.237573684Z Waiting for performance profile collector pods to become ready: 6
2026-07-27T21:40:40.253988908Z Gathering data for ns/openshift-config...
2026-07-27T21:40:40.366642343Z Waiting for performance profile collector pods to become ready: 7
2026-07-27T21:40:40.547521469Z Gathering data for ns/openshift-config-managed...
2026-07-27T21:40:40.948785356Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-27T21:40:41.232942893Z Gathering data for ns/openshift-kube-apiserver...
2026-07-27T21:40:41.491726019Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-27T21:40:41.507039743Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-27T21:40:41.785891473Z Collecting performance related data for node ip-10-0-134-38.ec2.internal
2026-07-27T21:40:41.795674291Z Collecting performance related data for node ip-10-0-139-153.ec2.internal
2026-07-27T21:40:41.802942194Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-27T21:40:41.804305324Z Collecting performance related data for node ip-10-0-131-253.ec2.internal
2026-07-27T21:40:42.147925504Z Gathering data for ns/openshift-kube-scheduler...
2026-07-27T21:40:42.475263761Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-27T21:40:42.841278656Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-27T21:40:43.217907228Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-27T21:40:43.572815112Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-27T21:40:43.898624186Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:44.099637427Z Gathering data for ns/openshift-multus...
2026-07-27T21:40:45.556508197Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-27T21:40:46.817819608Z Gathering data for ns/openshift-host-network...
2026-07-27T21:40:47.210854768Z Gathering data for ns/openshift-network-diagnostics...
2026-07-27T21:40:47.810114029Z Gathering data for ns/openshift-network-node-identity...
2026-07-27T21:40:48.219578699Z Gathering data for ns/openshift-network-operator...
2026-07-27T21:40:48.588061766Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-27T21:40:48.860650280Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-27T21:40:48.903577468Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-27T21:40:49.376354450Z Gathering data for ns/openshift-apiserver-operator...
2026-07-27T21:40:49.697351083Z Gathering data for ns/openshift-apiserver...
2026-07-27T21:40:50.140564756Z Collecting kubelet logs for node ip-10-0-134-38.ec2.internal
2026-07-27T21:40:50.142727116Z Collecting kubelet logs for node ip-10-0-139-153.ec2.internal
2026-07-27T21:40:50.144526317Z Collecting kubelet logs for node ip-10-0-131-253.ec2.internal
2026-07-27T21:40:50.145886586Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-27T21:40:50.464890300Z Gathering data for ns/openshift-controller-manager...
2026-07-27T21:40:50.558883954Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-cbjts namespace
2026-07-27T21:40:50.564778938Z INFO: Node performance data collection complete.
2026-07-27T21:40:50.788406494Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-27T21:40:51.406699326Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-27T21:40:51.675472542Z Gathering data for ns/openshift-service-ca-operator...
2026-07-27T21:40:51.986518674Z Gathering data for ns/openshift-service-ca...
2026-07-27T21:40:52.325190134Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-27T21:40:53.212769005Z Wrote inspect data to must-gather.
2026-07-27T21:40:53.212812137Z error: inspection completed with the errors occurred while gathering data:
2026-07-27T21:40:53.212812137Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-27T21:40:53.343937485Z error: the server doesn't have a resource type "clusters"
2026-07-27T21:40:53.486460803Z Caches written to disk