2026-07-01T19:48:04.321165467Z [disk usage checker] Started
2026-07-01T19:48:04.324534670Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:04.530508041Z Gathering data for ns/openshift-cluster-version...
2026-07-01T19:48:04.679261298Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-01T19:48:04.790130294Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T19:48:04.924542460Z Gathering data for ns/default...
2026-07-01T19:48:05.276652186Z Gathering data for ns/openshift...
2026-07-01T19:48:05.722616034Z Gathering data for ns/kube-system...
2026-07-01T19:48:06.497091425Z Gathering data for ns/openshift-etcd...
2026-07-01T19:48:06.844351517Z Wrote inspect data to must-gather.
2026-07-01T19:48:06.844384789Z error: inspection completed with the errors occurred while gathering data:
2026-07-01T19:48:06.844384789Z     namespaces "assisted-installer" not found
2026-07-01T19:48:07.138109058Z Waiting on subprocesses to finish execution.
2026-07-01T19:48:07.143938487Z Executing Istio gather script
2026-07-01T19:48:07.145287015Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-01T19:48:07.146103559Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-01T19:48:07.147887935Z INFO: Gathering HAProxy config files
2026-07-01T19:48:07.151000983Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-01T19:48:07.175422408Z INFO: Collecting host service logs for crio
2026-07-01T19:48:07.175796452Z INFO: Collecting host service logs for kubelet
2026-07-01T19:48:07.176151929Z INFO: Collecting host service logs for rpm-ostreed
2026-07-01T19:48:07.176479388Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-01T19:48:07.176902127Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-01T19:48:07.177255966Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-01T19:48:07.177601066Z INFO: Collecting host service logs for NetworkManager
2026-07-01T19:48:07.177943405Z INFO: Collecting host service logs for openvswitch
2026-07-01T19:48:07.178286238Z INFO: Collecting host service logs for ovs-configuration
2026-07-01T19:48:07.178618058Z INFO: Collecting host service logs for ovsdb-server
2026-07-01T19:48:07.178973360Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-01T19:48:07.179447069Z INFO: Waiting for worker host service log collection to complete ...
2026-07-01T19:48:07.581357086Z INFO: Waiting for node performance related collection to complete ...
2026-07-01T19:48:07.675268316Z No resources found in openshift-etcd namespace.
2026-07-01T19:48:07.710403868Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-01T19:48:07.710445776Z INFO: on-disk MachineConfig config collection complete.
2026-07-01T19:48:07.817283317Z Wrote inspect data to must-gather.
2026-07-01T19:48:08.122389841Z ERROR: No running kube-apiserver pods found
2026-07-01T19:48:08.133924582Z error: the server doesn't have a resource type "performanceprofile"
2026-07-01T19:48:08.139224769Z No resources found
2026-07-01T19:48:08.204999307Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-01T19:48:08.384949178Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-01T19:48:08.435562939Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-c4nzb
2026-07-01T19:48:08.435647606Z insights-runtime-extractor-2wzcv
2026-07-01T19:48:08.435672140Z insights-runtime-extractor-lrzs5
2026-07-01T19:48:08.435691435Z insights-runtime-extractor-srhss
2026-07-01T19:48:08.542105224Z INFO: "metallb-operator" not detected. Skipping.
2026-07-01T19:48:08.600336488Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-01T19:48:08.850656903Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-01T19:48:09.122864878Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.156478363Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-01T19:48:09.156478363Z See 'oc rsync -h' for help and examples
2026-07-01T19:48:09.306394080Z INFO: Worker host service log collection to complete.
2026-07-01T19:48:09.379743650Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-01T19:48:09.391664576Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.394664436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T19:48:09.409766601Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-01T19:48:09.441079456Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.442353712Z No resources found
2026-07-01T19:48:09.450371503Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:09.566067442Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.610562083Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-01T19:48:09.636022389Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.721562552Z Wrote inspect data to must-gather.
2026-07-01T19:48:09.844023637Z tar: Removing leading `/' from member names
2026-07-01T19:48:09.849901074Z INFO: HAProxy config collection complete.
2026-07-01T19:48:09.852376503Z INFO: 'previous-logs' folder not found on ip-10-0-131-138.ec2.internal, skipping...
2026-07-01T19:48:09.864696482Z error: the server doesn't have a resource type "machineconfigs"
2026-07-01T19:48:09.958318042Z Wrote inspect data to must-gather.
2026-07-01T19:48:10.063084329Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-01T19:48:10.106697267Z Gathering data for ns/openshift-monitoring...
2026-07-01T19:48:10.174867415Z INFO: 'previous-logs' folder not found on ip-10-0-133-159.ec2.internal, skipping...
2026-07-01T19:48:10.220520168Z INFO: Getting rules from prometheus-k8s-0
2026-07-01T19:48:10.308877944Z Wrote inspect data to must-gather.
2026-07-01T19:48:10.333349516Z Wrote inspect data to must-gather.
2026-07-01T19:48:10.442733066Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-01T19:48:10.507960607Z INFO: 'previous-logs' folder not found on ip-10-0-134-25.ec2.internal, skipping...
2026-07-01T19:48:10.507994632Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-01T19:48:10.508041118Z INFO: Machine Config Daemon termination log collection complete.
2026-07-01T19:48:10.522686448Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-01T19:48:10.592312983Z Wrote inspect data to must-gather.
2026-07-01T19:48:10.697899581Z INFO: Getting status/config from prometheus-k8s-0
2026-07-01T19:48:10.744177385Z Wrote inspect data to must-gather.
2026-07-01T19:48:10.908646128Z INFO: INTERCONNECT MODE
2026-07-01T19:48:10.908678193Z INFO: Gathering ovn-kubernetes DBs
2026-07-01T19:48:10.975521172Z Wrote inspect data to must-gather.
2026-07-01T19:48:11.084431123Z INFO: Gathering OVN_Northbound from ovnkube-node-b2krn...
2026-07-01T19:48:11.084655235Z INFO: Gathering OVN_Northbound from ovnkube-node-dd4gr...
2026-07-01T19:48:11.084881898Z INFO: Gathering OVN_Northbound from ovnkube-node-p568j...
2026-07-01T19:48:11.085171479Z INFO: Gathering OVN_Southbound from ovnkube-node-b2krn...
2026-07-01T19:48:11.085456636Z INFO: Gathering OVN_Southbound from ovnkube-node-dd4gr...
2026-07-01T19:48:11.085767264Z INFO: Gathering OVN_Southbound from ovnkube-node-p568j...
2026-07-01T19:48:11.263322561Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-01T19:48:11.303351656Z Wrote inspect data to must-gather.
2026-07-01T19:48:11.437178258Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.449245027Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.453089394Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.456304169Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.465405942Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.468434140Z tar: Removing leading `/' from member names
2026-07-01T19:48:11.571528556Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-01T19:48:11.591648316Z Wrote inspect data to must-gather.
2026-07-01T19:48:11.840436003Z Wrote inspect data to must-gather.
2026-07-01T19:48:11.854782381Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-01T19:48:11.908483074Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-01T19:48:11.982658293Z INFO: Waiting for network log collection to complete ...
2026-07-01T19:48:11.982691831Z INFO: Waiting for ovnk database copies to complete ...
2026-07-01T19:48:11.983080987Z INFO: Copying ovnk databases complete.
2026-07-01T19:48:11.984668700Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-01T19:48:11.987220138Z ovnk_database_store/
2026-07-01T19:48:11.987267760Z ovnk_database_store/ovnkube-node-b2krn_nbdb
2026-07-01T19:48:12.002283472Z ovnk_database_store/ovnkube-node-p568j_nbdb
2026-07-01T19:48:12.011668215Z ovnk_database_store/ovnkube-node-p568j_sbdb
2026-07-01T19:48:12.033965698Z ovnk_database_store/ovnkube-node-dd4gr_sbdb
2026-07-01T19:48:12.057509453Z ovnk_database_store/ovnkube-node-dd4gr_nbdb
2026-07-01T19:48:12.057586835Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-01T19:48:12.065463862Z ovnk_database_store/ovnkube-node-b2krn_sbdb
2026-07-01T19:48:12.088285442Z INFO: Network log collection complete.
2026-07-01T19:48:12.098226330Z Wrote inspect data to must-gather.
2026-07-01T19:48:12.125249462Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:12.151381654Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-01T19:48:12.330091914Z Wrote inspect data to must-gather.
2026-07-01T19:48:12.368272994Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:12.372137834Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-01T19:48:12.597086046Z INFO: Getting status from alertmanager-main-0
2026-07-01T19:48:12.599062254Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:12.622172980Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-01T19:48:12.628533414Z INFO: Fallback to identify the container image from release info
2026-07-01T19:48:12.687607519Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T19:48:12.822295989Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:12.824785241Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-01T19:48:12.865209294Z Gathering data for ns/openshift-network-console...
2026-07-01T19:48:13.011296716Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:13.218217134Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:13.222914372Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-01T19:48:13.283295857Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-01T19:48:13.347809521Z Gathering data for ns/openshift-console-operator...
2026-07-01T19:48:13.436567186Z daemonset.apps/perf-node-gather-daemonset created
2026-07-01T19:48:13.488534949Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:13.579662393Z Waiting for performance profile collector pods to become ready: 1
2026-07-01T19:48:13.690370125Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:13.693417297Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-01T19:48:13.831485455Z Gathering data for ns/openshift-console...
2026-07-01T19:48:13.880079433Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.094138463Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.096537552Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-01T19:48:14.272895959Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.445645658Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-01T19:48:14.475828448Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.478040741Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-01T19:48:14.505697638Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:14.660034374Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.718488757Z Waiting for performance profile collector pods to become ready: 2
2026-07-01T19:48:14.874375534Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:14.878341544Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-01T19:48:15.010561625Z Gathering data for ns/openshift-dns-operator...
2026-07-01T19:48:15.076016567Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:15.267091954Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:15.270855058Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-01T19:48:15.351175024Z Gathering data for ns/openshift-dns...
2026-07-01T19:48:15.448002910Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:15.623960769Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:15.626391888Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-01T19:48:15.804115458Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:15.849394871Z Waiting for performance profile collector pods to become ready: 3
2026-07-01T19:48:16.023916704Z Wrote inspect data to /must-gather/istio.
2026-07-01T19:48:16.082248506Z Gathering data for ns/openshift-image-registry...
2026-07-01T19:48:16.625242902Z Gathering data for ns/openshift-ingress-operator...
2026-07-01T19:48:16.734005444Z Done executing Istio gather script
2026-07-01T19:48:16.991530559Z Waiting for performance profile collector pods to become ready: 4
2026-07-01T19:48:17.000964880Z Gathering data for ns/openshift-ingress...
2026-07-01T19:48:17.442194203Z Gathering data for ns/openshift-ingress-canary...
2026-07-01T19:48:17.878952759Z Gathering data for ns/openshift-insights...
2026-07-01T19:48:18.114178852Z Waiting for performance profile collector pods to become ready: 5
2026-07-01T19:48:19.226556260Z Waiting for performance profile collector pods to become ready: 6
2026-07-01T19:48:19.510683540Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:20.268205985Z Gathering data for ns/kserve...
2026-07-01T19:48:20.349787873Z Waiting for performance profile collector pods to become ready: 7
2026-07-01T19:48:21.465158840Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-01T19:48:21.746586290Z Collecting performance related data for node ip-10-0-131-138.ec2.internal
2026-07-01T19:48:21.753341519Z Collecting performance related data for node ip-10-0-134-25.ec2.internal
2026-07-01T19:48:21.762292787Z Collecting performance related data for node ip-10-0-133-159.ec2.internal
2026-07-01T19:48:22.189459469Z Gathering data for ns/openshift-config...
2026-07-01T19:48:22.532339814Z Gathering data for ns/openshift-config-managed...
2026-07-01T19:48:23.115725077Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-01T19:48:23.472864887Z Gathering data for ns/openshift-kube-apiserver...
2026-07-01T19:48:23.797573680Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-01T19:48:24.168830918Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-01T19:48:24.516934142Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:24.551781198Z Gathering data for ns/openshift-kube-scheduler...
2026-07-01T19:48:24.962158455Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-01T19:48:25.375276908Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-01T19:48:25.763556129Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-01T19:48:26.117815742Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-01T19:48:26.742025236Z Gathering data for ns/openshift-multus...
2026-07-01T19:48:28.338201778Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-01T19:48:29.521874122Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:29.987050276Z Gathering data for ns/openshift-host-network...
2026-07-01T19:48:30.151305205Z Collecting kubelet logs for node ip-10-0-131-138.ec2.internal
2026-07-01T19:48:30.153015975Z Collecting kubelet logs for node ip-10-0-134-25.ec2.internal
2026-07-01T19:48:30.154716345Z Collecting kubelet logs for node ip-10-0-133-159.ec2.internal
2026-07-01T19:48:30.455097591Z Gathering data for ns/openshift-network-diagnostics...
2026-07-01T19:48:30.526843032Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-26hz8 namespace
2026-07-01T19:48:30.532877527Z INFO: Node performance data collection complete.
2026-07-01T19:48:31.082778645Z Gathering data for ns/openshift-network-node-identity...
2026-07-01T19:48:31.576880893Z Gathering data for ns/openshift-network-operator...
2026-07-01T19:48:32.018898182Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-01T19:48:32.367607877Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-01T19:48:32.892687365Z Gathering data for ns/openshift-apiserver-operator...
2026-07-01T19:48:33.255692286Z Gathering data for ns/openshift-apiserver...
2026-07-01T19:48:33.693763128Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-01T19:48:34.082786599Z Gathering data for ns/openshift-controller-manager...
2026-07-01T19:48:34.429605560Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-01T19:48:34.528569922Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-01T19:48:35.111061108Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-01T19:48:35.423545387Z Gathering data for ns/openshift-service-ca-operator...
2026-07-01T19:48:35.768883822Z Gathering data for ns/openshift-service-ca...
2026-07-01T19:48:36.163037031Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-01T19:48:37.180243464Z Wrote inspect data to must-gather.
2026-07-01T19:48:37.180287348Z error: inspection completed with the errors occurred while gathering data:
2026-07-01T19:48:37.180287348Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-01T19:48:37.315288567Z error: the server doesn't have a resource type "clusters"
2026-07-01T19:48:37.454830538Z Caches written to disk