2026-07-07T03:06:51.144554736Z [disk usage checker] Started
2026-07-07T03:06:51.148001833Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:06:51.342555103Z Gathering data for ns/openshift-cluster-version...
2026-07-07T03:06:51.477121432Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T03:06:51.568244200Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T03:06:51.653643032Z Gathering data for ns/default...
2026-07-07T03:06:51.965900021Z Gathering data for ns/openshift...
2026-07-07T03:06:52.389257907Z Gathering data for ns/kube-system...
2026-07-07T03:06:53.143135471Z Gathering data for ns/openshift-etcd...
2026-07-07T03:06:53.452354156Z Wrote inspect data to must-gather.
2026-07-07T03:06:53.452398836Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T03:06:53.452398836Z     namespaces "assisted-installer" not found
2026-07-07T03:06:53.972525219Z Waiting on subprocesses to finish execution.
2026-07-07T03:06:53.976090610Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T03:06:53.982058704Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T03:06:53.982510701Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T03:06:53.988321198Z Executing Istio gather script
2026-07-07T03:06:54.006602646Z INFO: Gathering HAProxy config files
2026-07-07T03:06:54.011300368Z INFO: Collecting host service logs for crio
2026-07-07T03:06:54.011684047Z INFO: Collecting host service logs for kubelet
2026-07-07T03:06:54.012036351Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T03:06:54.012393704Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T03:06:54.012763401Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T03:06:54.013111140Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T03:06:54.013517418Z INFO: Collecting host service logs for NetworkManager
2026-07-07T03:06:54.013877635Z INFO: Collecting host service logs for openvswitch
2026-07-07T03:06:54.014229672Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T03:06:54.014612330Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T03:06:54.014967728Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T03:06:54.015363533Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T03:06:54.153860386Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T03:06:54.776248519Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T03:06:54.799028374Z No resources found in openshift-etcd namespace.
2026-07-07T03:06:54.923595897Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T03:06:54.923707689Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T03:06:54.999766423Z No resources found
2026-07-07T03:06:55.011986876Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T03:06:55.030117521Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T03:06:55.033040297Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T03:06:55.185173784Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-94vfd
2026-07-07T03:06:55.185241296Z insights-runtime-extractor-dlvjt
2026-07-07T03:06:55.185272784Z insights-runtime-extractor-ffpwx
2026-07-07T03:06:55.185293166Z insights-runtime-extractor-sh5gt
2026-07-07T03:06:55.220858645Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T03:06:55.487949238Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T03:06:55.657161623Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T03:06:55.657161623Z See 'oc rsync -h' for help and examples
2026-07-07T03:06:55.879164886Z ERROR: No running kube-apiserver pods found
2026-07-07T03:06:55.895982575Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T03:06:55.915973541Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T03:06:56.049493299Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.154920076Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.230586284Z No resources found
2026-07-07T03:06:56.309216947Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:06:56.314193494Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T03:06:56.341711990Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.351960075Z INFO: Worker host service log collection to complete.
2026-07-07T03:06:56.420553335Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.485824760Z INFO: 'previous-logs' folder not found on ip-10-0-135-57.ec2.internal, skipping...
2026-07-07T03:06:56.516641080Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T03:06:56.528291658Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.604928223Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.631607382Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.655343761Z Gathering data for ns/openshift-monitoring...
2026-07-07T03:06:56.770174329Z tar: Removing leading `/' from member names
2026-07-07T03:06:56.776077018Z INFO: HAProxy config collection complete.
2026-07-07T03:06:56.866051932Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T03:06:56.901182649Z Wrote inspect data to must-gather.
2026-07-07T03:06:56.932519280Z INFO: 'previous-logs' folder not found on ip-10-0-138-74.ec2.internal, skipping...
2026-07-07T03:06:56.979663060Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T03:06:57.073572719Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T03:06:57.203882130Z Wrote inspect data to must-gather.
2026-07-07T03:06:57.255652902Z INFO: 'previous-logs' folder not found on ip-10-0-142-230.ec2.internal, skipping...
2026-07-07T03:06:57.255690253Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T03:06:57.255743310Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T03:06:57.349588000Z Wrote inspect data to must-gather.
2026-07-07T03:06:57.476780549Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T03:06:57.504768219Z Wrote inspect data to must-gather.
2026-07-07T03:06:57.547862863Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T03:06:57.588924152Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T03:06:57.770253764Z Wrote inspect data to must-gather.
2026-07-07T03:06:57.813419369Z INFO: INTERCONNECT MODE
2026-07-07T03:06:57.813501834Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T03:06:57.949068157Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T03:06:57.987133760Z INFO: Gathering OVN_Northbound from ovnkube-node-9q4km...
2026-07-07T03:06:57.987353666Z INFO: Gathering OVN_Northbound from ovnkube-node-s4lcd...
2026-07-07T03:06:57.987581608Z INFO: Gathering OVN_Northbound from ovnkube-node-x5ll6...
2026-07-07T03:06:57.987855078Z INFO: Gathering OVN_Southbound from ovnkube-node-9q4km...
2026-07-07T03:06:57.988104732Z INFO: Gathering OVN_Southbound from ovnkube-node-s4lcd...
2026-07-07T03:06:57.988422129Z INFO: Gathering OVN_Southbound from ovnkube-node-x5ll6...
2026-07-07T03:06:58.031136053Z Wrote inspect data to must-gather.
2026-07-07T03:06:58.313332705Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T03:06:58.322488593Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.327985342Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.342148051Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.360951530Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.378331325Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.391644654Z tar: Removing leading `/' from member names
2026-07-07T03:06:58.440123621Z Wrote inspect data to must-gather.
2026-07-07T03:06:58.475027463Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T03:06:58.637918936Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T03:06:58.695059565Z Wrote inspect data to must-gather.
2026-07-07T03:06:58.782042576Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T03:06:58.866809597Z INFO: Waiting for network log collection to complete ...
2026-07-07T03:06:58.866911312Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T03:06:58.867441537Z INFO: Copying ovnk databases complete.
2026-07-07T03:06:58.869305553Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-07T03:06:58.871401713Z ovnk_database_store/
2026-07-07T03:06:58.871401713Z ovnk_database_store/ovnkube-node-s4lcd_nbdb
2026-07-07T03:06:58.881277608Z ovnk_database_store/ovnkube-node-9q4km_sbdb
2026-07-07T03:06:58.909038103Z ovnk_database_store/ovnkube-node-x5ll6_sbdb
2026-07-07T03:06:58.936699064Z ovnk_database_store/ovnkube-node-9q4km_nbdb
2026-07-07T03:06:58.944118195Z Wrote inspect data to must-gather.
2026-07-07T03:06:58.947471240Z ovnk_database_store/ovnkube-node-x5ll6_nbdb
2026-07-07T03:06:58.959611132Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T03:06:58.961126958Z ovnk_database_store/ovnkube-node-s4lcd_sbdb
2026-07-07T03:06:58.989758580Z INFO: Network log collection complete.
2026-07-07T03:06:59.011669105Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:06:59.131932231Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T03:06:59.258752332Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:06:59.262173747Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T03:06:59.283224069Z Wrote inspect data to must-gather.
2026-07-07T03:06:59.364622015Z Gathering data for ns/openshift-network-console...
2026-07-07T03:06:59.387818298Z INFO: Getting status from alertmanager-main-0
2026-07-07T03:06:59.506229617Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:06:59.606170491Z Wrote inspect data to must-gather.
2026-07-07T03:06:59.720358517Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:06:59.724158184Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T03:06:59.876855313Z Gathering data for ns/openshift-console-operator...
2026-07-07T03:06:59.928660798Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T03:06:59.932304732Z INFO: Fallback to identify the container image from release info
2026-07-07T03:06:59.949865546Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:00.159880703Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:00.163397549Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T03:07:00.287284668Z Gathering data for ns/openshift-console...
2026-07-07T03:07:00.408188137Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:00.624811832Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:00.628257465Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T03:07:00.667329375Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T03:07:00.820346664Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T03:07:00.821629060Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:00.871668221Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T03:07:00.990960854Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T03:07:01.053063805Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:01.055557206Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T03:07:01.252877313Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:01.329534216Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:07:01.493972028Z Gathering data for ns/openshift-dns-operator...
2026-07-07T03:07:01.498101844Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:01.500847230Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T03:07:01.684594097Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:01.806811548Z Gathering data for ns/openshift-dns...
2026-07-07T03:07:01.880342656Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:01.882882919Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T03:07:02.046366665Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:02.147596591Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T03:07:02.261403132Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:02.264269523Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T03:07:02.458786075Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:02.459793776Z Gathering data for ns/openshift-image-registry...
2026-07-07T03:07:02.683717647Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:02.687233388Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T03:07:02.853624363Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:03.012801536Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T03:07:03.059147856Z Wrote inspect data to /must-gather/istio.
2026-07-07T03:07:03.296618338Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T03:07:03.387494437Z Gathering data for ns/openshift-ingress...
2026-07-07T03:07:03.751483923Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T03:07:03.792890099Z Done executing Istio gather script
2026-07-07T03:07:04.140208655Z Gathering data for ns/openshift-insights...
2026-07-07T03:07:04.413457076Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T03:07:05.536599011Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T03:07:06.334311687Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:07:06.405800965Z Gathering data for ns/kserve...
2026-07-07T03:07:06.662348783Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T03:07:07.799855092Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T03:07:08.013116182Z Gathering data for ns/openshift-config...
2026-07-07T03:07:08.074718284Z Collecting performance related data for node ip-10-0-135-57.ec2.internal
2026-07-07T03:07:08.083478902Z Collecting performance related data for node ip-10-0-142-230.ec2.internal
2026-07-07T03:07:08.091095229Z Collecting performance related data for node ip-10-0-138-74.ec2.internal
2026-07-07T03:07:08.330040983Z Gathering data for ns/openshift-config-managed...
2026-07-07T03:07:08.848093552Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T03:07:09.217631267Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T03:07:09.550699668Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T03:07:09.909862988Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T03:07:10.225933354Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T03:07:10.524045603Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T03:07:10.879705334Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T03:07:11.256073477Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T03:07:11.341129960Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:07:11.574522685Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T03:07:12.078074711Z Gathering data for ns/openshift-multus...
2026-07-07T03:07:13.576701924Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T03:07:14.869468274Z Gathering data for ns/openshift-host-network...
2026-07-07T03:07:15.201694606Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T03:07:15.769995735Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T03:07:16.178306478Z Gathering data for ns/openshift-network-operator...
2026-07-07T03:07:16.346201662Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:07:16.563680231Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T03:07:16.610462710Z Collecting kubelet logs for node ip-10-0-135-57.ec2.internal
2026-07-07T03:07:16.612728702Z Collecting kubelet logs for node ip-10-0-142-230.ec2.internal
2026-07-07T03:07:16.614856691Z Collecting kubelet logs for node ip-10-0-138-74.ec2.internal
2026-07-07T03:07:16.881529076Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T03:07:17.008199974Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-h4pjz namespace
2026-07-07T03:07:17.016699861Z INFO: Node performance data collection complete.
2026-07-07T03:07:17.414838687Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T03:07:17.711861332Z Gathering data for ns/openshift-apiserver...
2026-07-07T03:07:18.113635682Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T03:07:18.415963156Z Gathering data for ns/openshift-controller-manager...
2026-07-07T03:07:18.690611065Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T03:07:19.335362584Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T03:07:19.624621401Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T03:07:19.938345297Z Gathering data for ns/openshift-service-ca...
2026-07-07T03:07:20.304669690Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T03:07:21.237824233Z Wrote inspect data to must-gather.
2026-07-07T03:07:21.237866287Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T03:07:21.237866287Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T03:07:21.350704418Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-07T03:07:21.367824042Z error: the server doesn't have a resource type "clusters"
2026-07-07T03:07:21.525568553Z Caches written to disk