2026-07-30T17:37:29.972751240Z [disk usage checker] Started 2026-07-30T17:37:29.976214145Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:30.168673979Z Gathering data for ns/openshift-cluster-version... 2026-07-30T17:37:30.310454816Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-30T17:37:30.400641920Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-30T17:37:30.489298315Z Gathering data for ns/default... 2026-07-30T17:37:30.795704656Z Gathering data for ns/openshift... 2026-07-30T17:37:31.213195966Z Gathering data for ns/kube-system... 2026-07-30T17:37:31.983251608Z Gathering data for ns/openshift-etcd... 2026-07-30T17:37:32.322756587Z Wrote inspect data to must-gather. 2026-07-30T17:37:32.322790633Z error: inspection completed with the errors occurred while gathering data: 2026-07-30T17:37:32.322790633Z namespaces "assisted-installer" not found 2026-07-30T17:37:32.745714974Z Waiting on subprocesses to finish execution. 2026-07-30T17:37:32.756273741Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-30T17:37:32.758544238Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-30T17:37:32.759150395Z Executing Istio gather script 2026-07-30T17:37:32.761257594Z INFO: Gathering HAProxy config files 2026-07-30T17:37:32.764960845Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-30T17:37:32.800587250Z INFO: Waiting for node performance related collection to complete ... 2026-07-30T17:37:32.801061337Z INFO: Collecting host service logs for crio 2026-07-30T17:37:32.801428781Z INFO: Collecting host service logs for kubelet 2026-07-30T17:37:32.802723395Z INFO: Collecting host service logs for rpm-ostreed 2026-07-30T17:37:32.803006099Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-30T17:37:32.803312870Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-30T17:37:32.803629260Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-30T17:37:32.803923648Z INFO: Collecting host service logs for NetworkManager 2026-07-30T17:37:32.804213754Z INFO: Collecting host service logs for openvswitch 2026-07-30T17:37:32.806130020Z INFO: Collecting host service logs for ovs-configuration 2026-07-30T17:37:32.806528336Z INFO: Collecting host service logs for ovsdb-server 2026-07-30T17:37:32.806916083Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-30T17:37:32.807335998Z INFO: Waiting for worker host service log collection to complete ... 2026-07-30T17:37:33.129247040Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-30T17:37:33.129285719Z INFO: on-disk MachineConfig config collection complete. 2026-07-30T17:37:33.548400023Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-30T17:37:33.901801598Z INFO: "metallb-operator" not detected. Skipping. 2026-07-30T17:37:33.996474256Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-c795x 2026-07-30T17:37:33.996535560Z insights-runtime-extractor-2ppfb 2026-07-30T17:37:33.996557143Z insights-runtime-extractor-5f52v 2026-07-30T17:37:33.996592078Z insights-runtime-extractor-bjfhv 2026-07-30T17:37:34.105093297Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-30T17:37:34.125185276Z Wrote inspect data to must-gather. 2026-07-30T17:37:34.171054903Z error: the server doesn't have a resource type "performanceprofile" 2026-07-30T17:37:34.173734874Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-30T17:37:34.181809285Z No resources found in openshift-etcd namespace. 2026-07-30T17:37:34.308080275Z No resources found 2026-07-30T17:37:34.310597162Z INFO: Worker host service log collection to complete. 2026-07-30T17:37:34.716031618Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-30T17:37:34.752219335Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-30T17:37:34.842904365Z No resources found 2026-07-30T17:37:34.886016232Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-30T17:37:34.981624058Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.134179309Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:35.256871568Z ERROR: No running kube-apiserver pods found 2026-07-30T17:37:35.275997447Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.285566946Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.292658134Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-30T17:37:35.333749918Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-30T17:37:35.333749918Z See 'oc rsync -h' for help and examples 2026-07-30T17:37:35.363942659Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-30T17:37:35.371656224Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.380269344Z INFO: 'previous-logs' folder not found on ip-10-0-129-24.ec2.internal, skipping... 2026-07-30T17:37:35.402848696Z Gathering data for ns/openshift-monitoring... 2026-07-30T17:37:35.501401948Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.512351106Z tar: Removing leading `/' from member names 2026-07-30T17:37:35.512586525Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.519656201Z INFO: HAProxy config collection complete. 2026-07-30T17:37:35.697302975Z error: the server doesn't have a resource type "machineconfigs" 2026-07-30T17:37:35.727395065Z INFO: 'previous-logs' folder not found on ip-10-0-130-254.ec2.internal, skipping... 2026-07-30T17:37:35.765101082Z Wrote inspect data to must-gather. 2026-07-30T17:37:35.897207304Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-30T17:37:35.947127680Z INFO: Getting rules from prometheus-k8s-0 2026-07-30T17:37:36.020037340Z INFO: 'previous-logs' folder not found on ip-10-0-136-68.ec2.internal, skipping... 2026-07-30T17:37:36.020147474Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-30T17:37:36.020200544Z INFO: Machine Config Daemon termination log collection complete. 2026-07-30T17:37:36.088306057Z Wrote inspect data to must-gather. 2026-07-30T17:37:36.150846242Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-30T17:37:36.162647558Z Wrote inspect data to must-gather. 2026-07-30T17:37:36.324127965Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-30T17:37:36.360000442Z Wrote inspect data to must-gather. 2026-07-30T17:37:36.415110997Z INFO: Getting status/config from prometheus-k8s-0 2026-07-30T17:37:36.585749813Z Wrote inspect data to must-gather. 2026-07-30T17:37:36.675252454Z INFO: INTERCONNECT MODE 2026-07-30T17:37:36.675285279Z INFO: Gathering ovn-kubernetes DBs 2026-07-30T17:37:36.817593687Z Wrote inspect data to must-gather. 2026-07-30T17:37:36.852231117Z INFO: Gathering OVN_Northbound from ovnkube-node-hv7vb... 2026-07-30T17:37:36.852474194Z INFO: Gathering OVN_Northbound from ovnkube-node-p8znm... 2026-07-30T17:37:36.852724347Z INFO: Gathering OVN_Northbound from ovnkube-node-tblxl... 2026-07-30T17:37:36.853041719Z INFO: Gathering OVN_Southbound from ovnkube-node-hv7vb... 2026-07-30T17:37:36.853325908Z INFO: Gathering OVN_Southbound from ovnkube-node-p8znm... 2026-07-30T17:37:36.853629270Z INFO: Gathering OVN_Southbound from ovnkube-node-tblxl... 2026-07-30T17:37:36.996979638Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-30T17:37:37.182832316Z Wrote inspect data to must-gather. 2026-07-30T17:37:37.254119993Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.259521909Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.267479451Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.334038280Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.361677495Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.378640261Z tar: Removing leading `/' from member names 2026-07-30T17:37:37.420751909Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-30T17:37:37.441085987Z Wrote inspect data to must-gather. 2026-07-30T17:37:37.657238709Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-30T17:37:37.691069649Z Wrote inspect data to must-gather. 2026-07-30T17:37:37.699031714Z INFO: Waiting for network log collection to complete ... 2026-07-30T17:37:37.699070178Z INFO: Waiting for ovnk database copies to complete ... 2026-07-30T17:37:37.699601553Z INFO: Copying ovnk databases complete. 2026-07-30T17:37:37.707081807Z 3.4M must-gather/network_logs/ovnk_database_store 2026-07-30T17:37:37.710238460Z ovnk_database_store/ 2026-07-30T17:37:37.710272142Z ovnk_database_store/ovnkube-node-p8znm_sbdb 2026-07-30T17:37:37.740023852Z ovnk_database_store/ovnkube-node-hv7vb_nbdb 2026-07-30T17:37:37.743932526Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-30T17:37:37.748111631Z ovnk_database_store/ovnkube-node-hv7vb_sbdb 2026-07-30T17:37:37.771434637Z ovnk_database_store/ovnkube-node-tblxl_sbdb 2026-07-30T17:37:37.794006038Z ovnk_database_store/ovnkube-node-p8znm_nbdb 2026-07-30T17:37:37.803659255Z ovnk_database_store/ovnkube-node-tblxl_nbdb 2026-07-30T17:37:37.815994597Z INFO: Network log collection complete. 2026-07-30T17:37:37.869780204Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:37.871143756Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-30T17:37:37.930938421Z Wrote inspect data to must-gather. 2026-07-30T17:37:38.040232030Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-30T17:37:38.117311845Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:38.119936280Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-30T17:37:38.169614601Z Wrote inspect data to must-gather. 2026-07-30T17:37:38.340634113Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:38.449618422Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-30T17:37:38.451647489Z INFO: Fallback to identify the container image from release info 2026-07-30T17:37:38.512962154Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-30T17:37:38.516782000Z INFO: Getting status from alertmanager-main-0 2026-07-30T17:37:38.576003714Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:38.580660109Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-30T17:37:38.703263262Z Gathering data for ns/openshift-network-console... 2026-07-30T17:37:38.799697717Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:38.993254260Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:38.998276840Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-30T17:37:39.124786393Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-30T17:37:39.215785670Z Gathering data for ns/openshift-console-operator... 2026-07-30T17:37:39.245493195Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:39.274238424Z daemonset.apps/perf-node-gather-daemonset created 2026-07-30T17:37:39.433317672Z Waiting for performance profile collector pods to become ready: 1 2026-07-30T17:37:39.474520236Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:39.477464769Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-30T17:37:39.660484789Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:39.687227826Z Gathering data for ns/openshift-console... 2026-07-30T17:37:39.898267097Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:39.900812833Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-30T17:37:40.062245489Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:40.243249434Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:40.269309160Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:40.272655966Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-30T17:37:40.321653679Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-30T17:37:40.453999622Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:40.568131386Z Waiting for performance profile collector pods to become ready: 2 2026-07-30T17:37:40.660042615Z Wrote inspect data to /must-gather/istio. 2026-07-30T17:37:40.884638570Z Gathering data for ns/openshift-dns-operator... 2026-07-30T17:37:41.212493344Z Gathering data for ns/openshift-dns... 2026-07-30T17:37:41.374575113Z Done executing Istio gather script 2026-07-30T17:37:41.688419081Z Waiting for performance profile collector pods to become ready: 3 2026-07-30T17:37:41.853649986Z Gathering data for ns/openshift-image-registry... 2026-07-30T17:37:42.337233211Z Gathering data for ns/openshift-ingress-operator... 2026-07-30T17:37:42.668476308Z Gathering data for ns/openshift-ingress... 2026-07-30T17:37:42.826939460Z Waiting for performance profile collector pods to become ready: 4 2026-07-30T17:37:43.029553578Z Gathering data for ns/openshift-ingress-canary... 2026-07-30T17:37:43.507199176Z Gathering data for ns/openshift-insights... 2026-07-30T17:37:43.966694686Z Waiting for performance profile collector pods to become ready: 5 2026-07-30T17:37:45.085486455Z Waiting for performance profile collector pods to become ready: 6 2026-07-30T17:37:45.247789052Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:46.220992314Z Waiting for performance profile collector pods to become ready: 7 2026-07-30T17:37:46.393868382Z Gathering data for ns/kserve... 2026-07-30T17:37:47.344290385Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-30T17:37:47.635828469Z Collecting performance related data for node ip-10-0-130-254.ec2.internal 2026-07-30T17:37:47.642966916Z Collecting performance related data for node ip-10-0-136-68.ec2.internal 2026-07-30T17:37:47.649456543Z Collecting performance related data for node ip-10-0-129-24.ec2.internal 2026-07-30T17:37:48.270794418Z Gathering data for ns/openshift-config... 2026-07-30T17:37:48.628001482Z Gathering data for ns/openshift-config-managed... 2026-07-30T17:37:49.175733111Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-30T17:37:49.522021156Z Gathering data for ns/openshift-kube-apiserver... 2026-07-30T17:37:49.820274878Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-30T17:37:50.151841365Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-30T17:37:50.253048685Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:50.506053828Z Gathering data for ns/openshift-kube-scheduler... 2026-07-30T17:37:50.817401631Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-30T17:37:51.172898572Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-30T17:37:51.521206591Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-30T17:37:51.847404356Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-30T17:37:52.441124725Z Gathering data for ns/openshift-multus... 2026-07-30T17:37:53.988022198Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-30T17:37:55.259687922Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:37:55.443689638Z Gathering data for ns/openshift-host-network... 2026-07-30T17:37:55.778652427Z Gathering data for ns/openshift-network-diagnostics... 2026-07-30T17:37:56.251699863Z Collecting kubelet logs for node ip-10-0-130-254.ec2.internal 2026-07-30T17:37:56.253936433Z Collecting kubelet logs for node ip-10-0-136-68.ec2.internal 2026-07-30T17:37:56.255527142Z Collecting kubelet logs for node ip-10-0-129-24.ec2.internal 2026-07-30T17:37:56.362205553Z Gathering data for ns/openshift-network-node-identity... 2026-07-30T17:37:56.655026092Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5wwbm namespace 2026-07-30T17:37:56.660577023Z INFO: Node performance data collection complete. 2026-07-30T17:37:56.836683540Z Gathering data for ns/openshift-network-operator... 2026-07-30T17:37:57.228688371Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-30T17:37:57.538448695Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-30T17:37:58.018876643Z Gathering data for ns/openshift-apiserver-operator... 2026-07-30T17:37:58.358652398Z Gathering data for ns/openshift-apiserver... 2026-07-30T17:37:58.778000366Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-30T17:37:59.114583688Z Gathering data for ns/openshift-controller-manager... 2026-07-30T17:37:59.420672193Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-30T17:38:00.075763231Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-30T17:38:00.264312049Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-07-30T17:38:00.368361626Z Gathering data for ns/openshift-service-ca-operator... 2026-07-30T17:38:00.682243858Z Gathering data for ns/openshift-service-ca... 2026-07-30T17:38:01.044832833Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-30T17:38:01.951922757Z Wrote inspect data to must-gather. 2026-07-30T17:38:01.951974291Z error: inspection completed with the errors occurred while gathering data: 2026-07-30T17:38:01.951974291Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-30T17:38:02.082940336Z error: the server doesn't have a resource type "clusters" 2026-07-30T17:38:02.243303400Z Caches written to disk