2026-07-28T17:06:21.991237958Z [disk usage checker] Started
2026-07-28T17:06:21.994659613Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:22.372231087Z Gathering data for ns/openshift-cluster-version...
2026-07-28T17:06:22.514042849Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T17:06:22.611588317Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:06:22.696992682Z Gathering data for ns/default...
2026-07-28T17:06:23.023665914Z Gathering data for ns/openshift...
2026-07-28T17:06:23.459146353Z Gathering data for ns/kube-system...
2026-07-28T17:06:24.331203553Z Gathering data for ns/openshift-etcd...
2026-07-28T17:06:24.663827704Z Wrote inspect data to must-gather.
2026-07-28T17:06:24.663992312Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T17:06:24.663992312Z     namespaces "assisted-installer" not found
2026-07-28T17:06:25.064442217Z Waiting on subprocesses to finish execution.
2026-07-28T17:06:25.075302083Z Executing Istio gather script
2026-07-28T17:06:25.075897283Z INFO: Collecting host service logs for crio
2026-07-28T17:06:25.076213329Z INFO: Collecting host service logs for kubelet
2026-07-28T17:06:25.076229766Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T17:06:25.076572047Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T17:06:25.076904083Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T17:06:25.077107096Z INFO: Gathering HAProxy config files
2026-07-28T17:06:25.077636192Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T17:06:25.077973359Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T17:06:25.078305794Z INFO: Collecting host service logs for NetworkManager
2026-07-28T17:06:25.078695540Z INFO: Collecting host service logs for openvswitch
2026-07-28T17:06:25.079051090Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T17:06:25.079506730Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T17:06:25.079860279Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T17:06:25.080229430Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T17:06:25.086374333Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T17:06:25.089296563Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T17:06:25.191799567Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T17:06:25.984861394Z Wrote inspect data to must-gather.
2026-07-28T17:06:26.055320868Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T17:06:26.063554006Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T17:06:26.234961104Z No resources found
2026-07-28T17:06:26.247086575Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T17:06:26.364506504Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T17:06:26.364603402Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T17:06:26.377345051Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-6vwmk
2026-07-28T17:06:26.377408196Z insights-runtime-extractor-9wc8w
2026-07-28T17:06:26.377433454Z insights-runtime-extractor-h6jhq
2026-07-28T17:06:26.377452092Z insights-runtime-extractor-zk6l7
2026-07-28T17:06:26.446953947Z INFO: Worker host service log collection to complete.
2026-07-28T17:06:26.450131701Z No resources found in openshift-etcd namespace.
2026-07-28T17:06:26.521770082Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T17:06:26.561130684Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T17:06:26.705373716Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T17:06:26.723381403Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T17:06:27.190816801Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:27.202869645Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T17:06:27.264949326Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.269016702Z ERROR: No running kube-apiserver pods found
2026-07-28T17:06:27.329048879Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.362840618Z Gathering data for ns/openshift-monitoring...
2026-07-28T17:06:27.373356265Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T17:06:27.373356265Z See 'oc rsync -h' for help and examples
2026-07-28T17:06:27.383201986Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:06:27.557755336Z No resources found
2026-07-28T17:06:27.560288962Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.594296992Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.662064762Z INFO: 'previous-logs' folder not found on ip-10-0-136-8.ec2.internal, skipping...
2026-07-28T17:06:27.668475082Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T17:06:27.687803655Z tar: Removing leading `/' from member names
2026-07-28T17:06:27.697537530Z INFO: HAProxy config collection complete.
2026-07-28T17:06:27.699687250Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.784850831Z Wrote inspect data to must-gather.
2026-07-28T17:06:27.927593548Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T17:06:27.990412357Z INFO: 'previous-logs' folder not found on ip-10-0-140-33.ec2.internal, skipping...
2026-07-28T17:06:28.087606547Z Wrote inspect data to must-gather.
2026-07-28T17:06:28.124912185Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T17:06:28.217293284Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T17:06:28.314035727Z INFO: 'previous-logs' folder not found on ip-10-0-140-97.ec2.internal, skipping...
2026-07-28T17:06:28.314079360Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T17:06:28.314090488Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T17:06:28.401296391Z Wrote inspect data to must-gather.
2026-07-28T17:06:28.416131624Z Wrote inspect data to must-gather.
2026-07-28T17:06:28.601665610Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T17:06:28.695097237Z Wrote inspect data to must-gather.
2026-07-28T17:06:28.741442679Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T17:06:28.810789901Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T17:06:28.862591190Z Wrote inspect data to must-gather.
2026-07-28T17:06:29.035305033Z INFO: INTERCONNECT MODE
2026-07-28T17:06:29.035337092Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T17:06:29.095698126Z Wrote inspect data to must-gather.
2026-07-28T17:06:29.214119941Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T17:06:29.229012651Z INFO: Gathering OVN_Northbound from ovnkube-node-2dc94...
2026-07-28T17:06:29.229514761Z INFO: Gathering OVN_Northbound from ovnkube-node-h47gx...
2026-07-28T17:06:29.229803885Z INFO: Gathering OVN_Northbound from ovnkube-node-kgsxf...
2026-07-28T17:06:29.230172298Z INFO: Gathering OVN_Southbound from ovnkube-node-2dc94...
2026-07-28T17:06:29.230612416Z INFO: Gathering OVN_Southbound from ovnkube-node-h47gx...
2026-07-28T17:06:29.231031076Z INFO: Gathering OVN_Southbound from ovnkube-node-kgsxf...
2026-07-28T17:06:29.442370602Z Wrote inspect data to must-gather.
2026-07-28T17:06:29.523441100Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T17:06:29.610185649Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.624492471Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.679227307Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.682596140Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.692912558Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.696191360Z tar: Removing leading `/' from member names
2026-07-28T17:06:29.702651527Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T17:06:29.805779934Z Wrote inspect data to must-gather.
2026-07-28T17:06:29.876676415Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T17:06:30.005718605Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T17:06:30.078479258Z W0728 17:06:30.078419    1409 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T17:06:30.085669625Z Wrote inspect data to must-gather.
2026-07-28T17:06:30.099507004Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:30.212329455Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T17:06:30.214525475Z INFO: Waiting for network log collection to complete ...
2026-07-28T17:06:30.214570800Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T17:06:30.214958647Z INFO: Copying ovnk databases complete.
2026-07-28T17:06:30.216592421Z 4.6M	must-gather/network_logs/ovnk_database_store
2026-07-28T17:06:30.218697575Z ovnk_database_store/
2026-07-28T17:06:30.218697575Z ovnk_database_store/ovnkube-node-kgsxf_nbdb
2026-07-28T17:06:30.234526767Z ovnk_database_store/ovnkube-node-kgsxf_sbdb
2026-07-28T17:06:30.265046556Z ovnk_database_store/ovnkube-node-2dc94_sbdb
2026-07-28T17:06:30.300024397Z ovnk_database_store/ovnkube-node-2dc94_nbdb
2026-07-28T17:06:30.311782453Z ovnk_database_store/ovnkube-node-h47gx_sbdb
2026-07-28T17:06:30.314238688Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T17:06:30.338134219Z ovnk_database_store/ovnkube-node-h47gx_nbdb
2026-07-28T17:06:30.352493574Z INFO: Network log collection complete.
2026-07-28T17:06:30.364931801Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:30.368633688Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T17:06:30.431801149Z Wrote inspect data to must-gather.
2026-07-28T17:06:30.440810458Z Gathering data for ns/openshift-network-console...
2026-07-28T17:06:30.595870923Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:30.744930174Z Wrote inspect data to must-gather.
2026-07-28T17:06:30.758630723Z INFO: Getting status from alertmanager-main-0
2026-07-28T17:06:30.846995130Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:30.851095728Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T17:06:30.985375323Z Gathering data for ns/openshift-console-operator...
2026-07-28T17:06:31.084665504Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T17:06:31.089606426Z INFO: Fallback to identify the container image from release info
2026-07-28T17:06:31.100757759Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:31.321289779Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:31.325959240Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T17:06:31.441312288Z Gathering data for ns/openshift-console...
2026-07-28T17:06:31.546345313Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:31.768699626Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:31.772001236Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T17:06:31.873235081Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T17:06:31.967582261Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:32.029579045Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T17:06:32.079263657Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T17:06:32.195655600Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T17:06:32.196850410Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:32.201654219Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T17:06:32.309765565Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:32.378729789Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:32.633383291Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:32.637240163Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T17:06:32.722497332Z Gathering data for ns/openshift-dns-operator...
2026-07-28T17:06:32.829622073Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:33.033867393Z Wrote inspect data to /must-gather/istio.
2026-07-28T17:06:33.078259586Z Gathering data for ns/openshift-dns...
2026-07-28T17:06:33.329292759Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T17:06:33.750880865Z Done executing Istio gather script
2026-07-28T17:06:33.832226244Z Gathering data for ns/openshift-image-registry...
2026-07-28T17:06:34.430895530Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T17:06:34.478734522Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T17:06:34.880268768Z Gathering data for ns/openshift-ingress...
2026-07-28T17:06:35.284383696Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T17:06:35.609242401Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T17:06:35.729086827Z Gathering data for ns/openshift-insights...
2026-07-28T17:06:36.724549786Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T17:06:37.314549009Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:37.854224757Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T17:06:38.322587634Z Gathering data for ns/kserve...
2026-07-28T17:06:39.009767826Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T17:06:40.138477633Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T17:06:40.408678100Z Collecting performance related data for node ip-10-0-140-33.ec2.internal
2026-07-28T17:06:40.415364425Z Collecting performance related data for node ip-10-0-136-8.ec2.internal
2026-07-28T17:06:40.423142372Z Collecting performance related data for node ip-10-0-140-97.ec2.internal
2026-07-28T17:06:40.853769385Z Gathering data for ns/openshift-config...
2026-07-28T17:06:41.227187776Z Gathering data for ns/openshift-config-managed...
2026-07-28T17:06:41.788150112Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T17:06:42.149349690Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T17:06:42.318965310Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:42.470874160Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T17:06:42.819683675Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T17:06:43.206145377Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T17:06:43.548052264Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T17:06:43.913696738Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T17:06:44.290268057Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T17:06:44.635215588Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T17:06:45.272202845Z Gathering data for ns/openshift-multus...
2026-07-28T17:06:46.948080969Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T17:06:47.323466830Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:48.603246824Z Gathering data for ns/openshift-host-network...
2026-07-28T17:06:48.946520891Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T17:06:49.001886113Z Collecting kubelet logs for node ip-10-0-140-33.ec2.internal
2026-07-28T17:06:49.003636603Z Collecting kubelet logs for node ip-10-0-136-8.ec2.internal
2026-07-28T17:06:49.005320139Z Collecting kubelet logs for node ip-10-0-140-97.ec2.internal
2026-07-28T17:06:49.456411796Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qccvn namespace
2026-07-28T17:06:49.463924738Z INFO: Node performance data collection complete.
2026-07-28T17:06:49.587537866Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T17:06:50.034627294Z Gathering data for ns/openshift-network-operator...
2026-07-28T17:06:50.455282777Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T17:06:50.744996183Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T17:06:51.284909779Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T17:06:51.636017611Z Gathering data for ns/openshift-apiserver...
2026-07-28T17:06:52.086254841Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T17:06:52.328980008Z [disk usage checker] Volume usage percentage: current = 28 ; allowed = 70
2026-07-28T17:06:52.443489885Z Gathering data for ns/openshift-controller-manager...
2026-07-28T17:06:52.749880699Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T17:06:53.450863344Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T17:06:53.805222092Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T17:06:54.139709349Z Gathering data for ns/openshift-service-ca...
2026-07-28T17:06:54.549367712Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T17:06:55.496603632Z Wrote inspect data to must-gather.
2026-07-28T17:06:55.496653029Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T17:06:55.496653029Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T17:06:55.638933904Z error: the server doesn't have a resource type "clusters"
2026-07-28T17:06:55.792708834Z Caches written to disk