2026-07-13T23:47:48.096049933Z [disk usage checker] Started 2026-07-13T23:47:48.099279667Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:47:48.331113857Z Gathering data for ns/openshift-cluster-version... 2026-07-13T23:47:48.471852114Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-13T23:47:48.577900986Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T23:47:48.679140431Z Gathering data for ns/default... 2026-07-13T23:47:48.969698677Z Gathering data for ns/openshift... 2026-07-13T23:47:49.352583189Z Gathering data for ns/kube-system... 2026-07-13T23:47:50.062438347Z Gathering data for ns/openshift-etcd... 2026-07-13T23:47:50.331337776Z Wrote inspect data to must-gather. 2026-07-13T23:47:50.331373697Z error: inspection completed with the errors occurred while gathering data: 2026-07-13T23:47:50.331373697Z namespaces "assisted-installer" not found 2026-07-13T23:47:50.580492485Z Waiting on subprocesses to finish execution. 2026-07-13T23:47:50.589135864Z Executing Istio gather script 2026-07-13T23:47:50.589405711Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-13T23:47:50.593052332Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-13T23:47:50.593660645Z INFO: Gathering HAProxy config files 2026-07-13T23:47:50.605068910Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-13T23:47:50.609620719Z INFO: Collecting host service logs for crio 2026-07-13T23:47:50.609943870Z INFO: Collecting host service logs for kubelet 2026-07-13T23:47:50.610316369Z INFO: Collecting host service logs for rpm-ostreed 2026-07-13T23:47:50.610648383Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-13T23:47:50.610974349Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-13T23:47:50.611319766Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-13T23:47:50.611653640Z INFO: Collecting host service logs for NetworkManager 2026-07-13T23:47:50.611987749Z INFO: Collecting host service logs for openvswitch 2026-07-13T23:47:50.612332083Z INFO: Collecting host service logs for ovs-configuration 2026-07-13T23:47:50.613751845Z INFO: Collecting host service logs for ovsdb-server 2026-07-13T23:47:50.614870839Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-13T23:47:50.615405735Z INFO: Waiting for worker host service log collection to complete ... 2026-07-13T23:47:50.616396015Z INFO: Waiting for node performance related collection to complete ... 2026-07-13T23:47:51.063190239Z No resources found in openshift-etcd namespace. 2026-07-13T23:47:51.316641863Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-13T23:47:51.445533524Z Wrote inspect data to must-gather. 2026-07-13T23:47:51.516901295Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-992z8 2026-07-13T23:47:51.516901295Z insights-runtime-extractor-hldl6 2026-07-13T23:47:51.516901295Z insights-runtime-extractor-vt7g6 2026-07-13T23:47:51.516901295Z insights-runtime-extractor-z5fl2 2026-07-13T23:47:51.684244813Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-13T23:47:51.842638719Z error: the server doesn't have a resource type "performanceprofile" 2026-07-13T23:47:52.090522550Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-13T23:47:52.090609010Z INFO: on-disk MachineConfig config collection complete. 2026-07-13T23:47:52.116901243Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-13T23:47:52.118744641Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-13T23:47:52.187337446Z INFO: Worker host service log collection to complete. 2026-07-13T23:47:52.203143416Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-13T23:47:52.264283673Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T23:47:52.377091186Z ERROR: No running kube-apiserver pods found 2026-07-13T23:47:52.399903388Z INFO: "metallb-operator" not detected. Skipping. 2026-07-13T23:47:52.459409012Z No resources found 2026-07-13T23:47:52.489684747Z Wrote inspect data to must-gather. 2026-07-13T23:47:52.622331562Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-13T23:47:52.622331562Z See 'oc rsync -h' for help and examples 2026-07-13T23:47:52.725726997Z No resources found 2026-07-13T23:47:52.831246281Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-13T23:47:52.865689495Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-13T23:47:52.872786208Z Wrote inspect data to must-gather. 2026-07-13T23:47:52.888780592Z Wrote inspect data to must-gather. 2026-07-13T23:47:52.923496301Z Wrote inspect data to must-gather. 2026-07-13T23:47:52.987652239Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.030264784Z Gathering data for ns/openshift-monitoring... 2026-07-13T23:47:53.036923455Z tar: Removing leading `/' from member names 2026-07-13T23:47:53.042796019Z INFO: HAProxy config collection complete. 2026-07-13T23:47:53.062717631Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-13T23:47:53.114687012Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:47:53.134519696Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.190748860Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.216198939Z error: the server doesn't have a resource type "machineconfigs" 2026-07-13T23:47:53.223464020Z INFO: 'previous-logs' folder not found on ip-10-0-131-116.ec2.internal, skipping... 2026-07-13T23:47:53.390947105Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-13T23:47:53.399400847Z INFO: Getting rules from prometheus-k8s-0 2026-07-13T23:47:53.456873347Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.487330657Z INFO: 'previous-logs' folder not found on ip-10-0-131-13.ec2.internal, skipping... 2026-07-13T23:47:53.612488723Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.720515004Z Wrote inspect data to must-gather. 2026-07-13T23:47:53.759306160Z INFO: 'previous-logs' folder not found on ip-10-0-140-52.ec2.internal, skipping... 2026-07-13T23:47:53.759340766Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-13T23:47:53.759373560Z INFO: Machine Config Daemon termination log collection complete. 2026-07-13T23:47:53.798447267Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-13T23:47:53.808083170Z INFO: Getting status/config from prometheus-k8s-0 2026-07-13T23:47:54.004114324Z INFO: INTERCONNECT MODE 2026-07-13T23:47:54.004114324Z INFO: Gathering ovn-kubernetes DBs 2026-07-13T23:47:54.009890590Z Wrote inspect data to must-gather. 2026-07-13T23:47:54.154213568Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-13T23:47:54.163822102Z INFO: Gathering OVN_Northbound from ovnkube-node-fzjjm... 2026-07-13T23:47:54.164091124Z INFO: Gathering OVN_Northbound from ovnkube-node-nffdh... 2026-07-13T23:47:54.164445240Z INFO: Gathering OVN_Northbound from ovnkube-node-rhgkn... 2026-07-13T23:47:54.164761703Z INFO: Gathering OVN_Southbound from ovnkube-node-fzjjm... 2026-07-13T23:47:54.165042249Z INFO: Gathering OVN_Southbound from ovnkube-node-nffdh... 2026-07-13T23:47:54.165337409Z INFO: Gathering OVN_Southbound from ovnkube-node-rhgkn... 2026-07-13T23:47:54.220236565Z Wrote inspect data to must-gather. 2026-07-13T23:47:54.288344425Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-13T23:47:54.493932988Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:54.496532841Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.496627899Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.527270051Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.551099244Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.558830746Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.568482747Z Wrote inspect data to must-gather. 2026-07-13T23:47:54.595052726Z tar: Removing leading `/' from member names 2026-07-13T23:47:54.609179808Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-13T23:47:54.676348820Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-13T23:47:54.737743840Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:54.741839212Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-13T23:47:54.807962465Z Wrote inspect data to must-gather. 2026-07-13T23:47:54.952449519Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-13T23:47:54.969947014Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:55.025923667Z INFO: Waiting for network log collection to complete ... 2026-07-13T23:47:55.025970970Z INFO: Waiting for ovnk database copies to complete ... 2026-07-13T23:47:55.026466786Z INFO: Copying ovnk databases complete. 2026-07-13T23:47:55.028544142Z 2.6M must-gather/network_logs/ovnk_database_store 2026-07-13T23:47:55.030887100Z ovnk_database_store/ 2026-07-13T23:47:55.030940677Z ovnk_database_store/ovnkube-node-fzjjm_nbdb 2026-07-13T23:47:55.037918806Z ovnk_database_store/ovnkube-node-nffdh_sbdb 2026-07-13T23:47:55.048220979Z Wrote inspect data to must-gather. 2026-07-13T23:47:55.055338436Z ovnk_database_store/ovnkube-node-fzjjm_sbdb 2026-07-13T23:47:55.072200822Z ovnk_database_store/ovnkube-node-rhgkn_nbdb 2026-07-13T23:47:55.079455455Z ovnk_database_store/ovnkube-node-rhgkn_sbdb 2026-07-13T23:47:55.096142255Z ovnk_database_store/ovnkube-node-nffdh_nbdb 2026-07-13T23:47:55.109177455Z INFO: Network log collection complete. 2026-07-13T23:47:55.191245855Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:55.193746952Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-13T23:47:55.206474683Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-13T23:47:55.256334283Z W0713 23:47:55.256287 1497 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-13T23:47:55.261765558Z Wrote inspect data to must-gather. 2026-07-13T23:47:55.293224533Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T23:47:55.390344953Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:55.495170354Z Gathering data for ns/openshift-network-console... 2026-07-13T23:47:55.560700788Z Wrote inspect data to must-gather. 2026-07-13T23:47:55.596584764Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:55.599378394Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-13T23:47:55.627341084Z INFO: Getting status from alertmanager-main-0 2026-07-13T23:47:55.848343796Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:55.861624466Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-13T23:47:55.865868479Z INFO: Fallback to identify the container image from release info 2026-07-13T23:47:55.948737846Z Gathering data for ns/openshift-console-operator... 2026-07-13T23:47:56.055830670Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:56.058386037Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-13T23:47:56.206699885Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:56.304898721Z Gathering data for ns/openshift-console... 2026-07-13T23:47:56.393693248Z Wrote inspect data to /must-gather/istio. 2026-07-13T23:47:56.617513680Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d 2026-07-13T23:47:56.758711670Z daemonset.apps/perf-node-gather-daemonset created 2026-07-13T23:47:56.826222343Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-13T23:47:56.901599162Z Waiting for performance profile collector pods to become ready: 1 2026-07-13T23:47:57.064042753Z Done executing Istio gather script 2026-07-13T23:47:57.320180979Z Gathering data for ns/openshift-dns-operator... 2026-07-13T23:47:57.618645439Z Gathering data for ns/openshift-dns... 2026-07-13T23:47:58.009048527Z Waiting for performance profile collector pods to become ready: 2 2026-07-13T23:47:58.122385724Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:47:58.164910106Z Gathering data for ns/openshift-image-registry... 2026-07-13T23:47:58.587480514Z Gathering data for ns/openshift-ingress-operator... 2026-07-13T23:47:58.873287442Z Gathering data for ns/openshift-ingress... 2026-07-13T23:47:59.131087996Z Waiting for performance profile collector pods to become ready: 3 2026-07-13T23:47:59.171279290Z Gathering data for ns/openshift-ingress-canary... 2026-07-13T23:47:59.519176232Z Gathering data for ns/openshift-insights... 2026-07-13T23:48:00.238729742Z Waiting for performance profile collector pods to become ready: 4 2026-07-13T23:48:01.246627402Z Gathering data for ns/openshift-config... 2026-07-13T23:48:01.361379610Z Waiting for performance profile collector pods to become ready: 5 2026-07-13T23:48:01.503062227Z Gathering data for ns/openshift-config-managed... 2026-07-13T23:48:01.876368185Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-13T23:48:02.102574516Z Gathering data for ns/openshift-kube-apiserver... 2026-07-13T23:48:02.324939025Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-13T23:48:02.469438903Z Waiting for performance profile collector pods to become ready: 6 2026-07-13T23:48:02.586721563Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-13T23:48:02.885308555Z Gathering data for ns/openshift-kube-scheduler... 2026-07-13T23:48:03.111831509Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-13T23:48:03.127493081Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:48:03.361263726Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-13T23:48:03.574588918Z Waiting for performance profile collector pods to become ready: 7 2026-07-13T23:48:03.651680997Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-13T23:48:03.948471800Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-13T23:48:04.377251472Z Gathering data for ns/openshift-multus... 2026-07-13T23:48:04.671345068Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-13T23:48:04.880576312Z Collecting performance related data for node ip-10-0-131-116.ec2.internal 2026-07-13T23:48:04.887627220Z Collecting performance related data for node ip-10-0-131-13.ec2.internal 2026-07-13T23:48:04.895369524Z Collecting performance related data for node ip-10-0-140-52.ec2.internal 2026-07-13T23:48:05.807895308Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-13T23:48:07.038736599Z Gathering data for ns/openshift-host-network... 2026-07-13T23:48:07.363935461Z Gathering data for ns/openshift-network-diagnostics... 2026-07-13T23:48:07.873422107Z Gathering data for ns/openshift-network-node-identity... 2026-07-13T23:48:08.132610180Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:48:08.264381985Z Gathering data for ns/openshift-network-operator... 2026-07-13T23:48:08.603046119Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-13T23:48:08.896519224Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-13T23:48:09.346480720Z Gathering data for ns/openshift-apiserver-operator... 2026-07-13T23:48:09.616421594Z Gathering data for ns/openshift-apiserver... 2026-07-13T23:48:09.980922571Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-13T23:48:10.221196878Z Gathering data for ns/openshift-controller-manager... 2026-07-13T23:48:10.473404169Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-13T23:48:11.076308410Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-13T23:48:11.343998376Z Gathering data for ns/openshift-service-ca-operator... 2026-07-13T23:48:11.657884243Z Gathering data for ns/openshift-service-ca... 2026-07-13T23:48:12.110301007Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-13T23:48:12.837986740Z Collecting kubelet logs for node ip-10-0-131-116.ec2.internal 2026-07-13T23:48:12.839652678Z Collecting kubelet logs for node ip-10-0-131-13.ec2.internal 2026-07-13T23:48:12.841256975Z Collecting kubelet logs for node ip-10-0-140-52.ec2.internal 2026-07-13T23:48:13.040435054Z Wrote inspect data to must-gather. 2026-07-13T23:48:13.040471017Z error: inspection completed with the errors occurred while gathering data: 2026-07-13T23:48:13.040471017Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-13T23:48:13.137402816Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-13T23:48:13.146697374Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-4t7lz namespace 2026-07-13T23:48:13.151677542Z INFO: Node performance data collection complete. 2026-07-13T23:48:13.269547451Z error: the server doesn't have a resource type "clusters" 2026-07-13T23:48:13.459366115Z Caches written to disk