2026-07-14T19:04:29.016703046Z [disk usage checker] Started
2026-07-14T19:04:29.020518934Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:29.218375405Z Gathering data for ns/openshift-cluster-version...
2026-07-14T19:04:29.380337042Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T19:04:29.490582565Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T19:04:29.575212082Z Gathering data for ns/default...
2026-07-14T19:04:29.896171383Z Gathering data for ns/openshift...
2026-07-14T19:04:30.421374703Z Gathering data for ns/kube-system...
2026-07-14T19:04:31.159869068Z Gathering data for ns/openshift-etcd...
2026-07-14T19:04:31.509463407Z Wrote inspect data to must-gather.
2026-07-14T19:04:31.509552063Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T19:04:31.509552063Z     namespaces "assisted-installer" not found
2026-07-14T19:04:31.644044839Z Waiting on subprocesses to finish execution.
2026-07-14T19:04:31.656429254Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-14T19:04:31.658587908Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-14T19:04:31.661536722Z Executing Istio gather script
2026-07-14T19:04:31.664008189Z INFO: Gathering HAProxy config files
2026-07-14T19:04:31.664512463Z INFO: Collecting host service logs for crio
2026-07-14T19:04:31.664874665Z INFO: Collecting host service logs for kubelet
2026-07-14T19:04:31.665238254Z INFO: Collecting host service logs for rpm-ostreed
2026-07-14T19:04:31.665610320Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-14T19:04:31.666012434Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-14T19:04:31.666361622Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-14T19:04:31.666879409Z INFO: Collecting host service logs for NetworkManager
2026-07-14T19:04:31.667232830Z INFO: Collecting host service logs for openvswitch
2026-07-14T19:04:31.667598568Z INFO: Collecting host service logs for ovs-configuration
2026-07-14T19:04:31.667981683Z INFO: Collecting host service logs for ovsdb-server
2026-07-14T19:04:31.668345460Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-14T19:04:31.668837217Z INFO: Waiting for worker host service log collection to complete ...
2026-07-14T19:04:31.676821729Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-14T19:04:31.789704691Z INFO: Waiting for node performance related collection to complete ...
2026-07-14T19:04:32.282230409Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-14T19:04:32.282322189Z INFO: on-disk MachineConfig config collection complete.
2026-07-14T19:04:32.488682591Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-14T19:04:32.507706749Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-14T19:04:32.667714599Z INFO: "metallb-operator" not detected. Skipping.
2026-07-14T19:04:32.721125207Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-14T19:04:32.741448421Z No resources found in openshift-etcd namespace.
2026-07-14T19:04:32.816257490Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-q5dbq
2026-07-14T19:04:32.816257490Z insights-runtime-extractor-5ll5x
2026-07-14T19:04:32.816257490Z insights-runtime-extractor-bflmb
2026-07-14T19:04:32.816257490Z insights-runtime-extractor-jprdd
2026-07-14T19:04:32.841100389Z ERROR: No running kube-apiserver pods found
2026-07-14T19:04:32.906394415Z error: the server doesn't have a resource type "performanceprofile"
2026-07-14T19:04:33.005705271Z INFO: Worker host service log collection to complete.
2026-07-14T19:04:33.030570700Z No resources found
2026-07-14T19:04:33.063980050Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-14T19:04:33.350386780Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-14T19:04:33.355298965Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.483951472Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-14T19:04:33.483951472Z See 'oc rsync -h' for help and examples
2026-07-14T19:04:33.544609903Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.713075584Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.833674378Z No resources found
2026-07-14T19:04:33.880425048Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.933114428Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.981692178Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T19:04:33.982214414Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-14T19:04:33.987377505Z Wrote inspect data to must-gather.
2026-07-14T19:04:33.992775445Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-14T19:04:33.994619723Z Wrote inspect data to must-gather.
2026-07-14T19:04:34.048087630Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:34.116531539Z tar: Removing leading `/' from member names
2026-07-14T19:04:34.124483100Z INFO: HAProxy config collection complete.
2026-07-14T19:04:34.157438649Z error: the server doesn't have a resource type "machineconfigs"
2026-07-14T19:04:34.159293527Z INFO: 'previous-logs' folder not found on ip-10-0-135-121.ec2.internal, skipping...
2026-07-14T19:04:34.244198878Z Gathering data for ns/openshift-monitoring...
2026-07-14T19:04:34.349616864Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-14T19:04:34.361784846Z Wrote inspect data to must-gather.
2026-07-14T19:04:34.366799415Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-14T19:04:34.477909970Z INFO: 'previous-logs' folder not found on ip-10-0-136-248.ec2.internal, skipping...
2026-07-14T19:04:34.482965301Z INFO: Getting rules from prometheus-k8s-0
2026-07-14T19:04:34.595730836Z Wrote inspect data to must-gather.
2026-07-14T19:04:34.625683972Z Wrote inspect data to must-gather.
2026-07-14T19:04:34.753049713Z INFO: 'previous-logs' folder not found on ip-10-0-143-80.ec2.internal, skipping...
2026-07-14T19:04:34.753081599Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-14T19:04:34.753093857Z INFO: Machine Config Daemon termination log collection complete.
2026-07-14T19:04:34.775133182Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-14T19:04:34.874166684Z Wrote inspect data to must-gather.
2026-07-14T19:04:34.908255788Z INFO: Getting status/config from prometheus-k8s-0
2026-07-14T19:04:34.977625034Z Wrote inspect data to must-gather.
2026-07-14T19:04:35.149549255Z INFO: INTERCONNECT MODE
2026-07-14T19:04:35.149578524Z INFO: Gathering ovn-kubernetes DBs
2026-07-14T19:04:35.158017570Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-14T19:04:35.189640037Z Wrote inspect data to must-gather.
2026-07-14T19:04:35.316521468Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-14T19:04:35.322876355Z INFO: Gathering OVN_Northbound from ovnkube-node-cgjz6...
2026-07-14T19:04:35.323140249Z INFO: Gathering OVN_Northbound from ovnkube-node-s88lr...
2026-07-14T19:04:35.323369771Z INFO: Gathering OVN_Northbound from ovnkube-node-tsq2g...
2026-07-14T19:04:35.323820057Z INFO: Gathering OVN_Southbound from ovnkube-node-cgjz6...
2026-07-14T19:04:35.324154269Z INFO: Gathering OVN_Southbound from ovnkube-node-s88lr...
2026-07-14T19:04:35.324461106Z INFO: Gathering OVN_Southbound from ovnkube-node-tsq2g...
2026-07-14T19:04:35.358420638Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:35.434887504Z Wrote inspect data to must-gather.
2026-07-14T19:04:35.621343471Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.623706609Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.659268068Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-14T19:04:35.666110554Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.694285275Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:35.696055079Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.697494339Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-14T19:04:35.697697213Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.712157036Z tar: Removing leading `/' from member names
2026-07-14T19:04:35.808219453Z Wrote inspect data to must-gather.
2026-07-14T19:04:35.871542739Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T19:04:35.929097190Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:35.945957975Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-14T19:04:36.052882098Z Wrote inspect data to must-gather.
2026-07-14T19:04:36.161443129Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:36.163966924Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-14T19:04:36.200330864Z INFO: Waiting for network log collection to complete ...
2026-07-14T19:04:36.200368860Z INFO: Waiting for ovnk database copies to complete ...
2026-07-14T19:04:36.200753072Z INFO: Copying ovnk databases complete.
2026-07-14T19:04:36.202645939Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-14T19:04:36.205183338Z ovnk_database_store/
2026-07-14T19:04:36.205183338Z ovnk_database_store/ovnkube-node-cgjz6_nbdb
2026-07-14T19:04:36.213979999Z ovnk_database_store/ovnkube-node-cgjz6_sbdb
2026-07-14T19:04:36.226703260Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-14T19:04:36.229086129Z ovnk_database_store/ovnkube-node-tsq2g_nbdb
2026-07-14T19:04:36.236196216Z ovnk_database_store/ovnkube-node-s88lr_nbdb
2026-07-14T19:04:36.244399460Z ovnk_database_store/ovnkube-node-s88lr_sbdb
2026-07-14T19:04:36.262401102Z ovnk_database_store/ovnkube-node-tsq2g_sbdb
2026-07-14T19:04:36.281593725Z INFO: Network log collection complete.
2026-07-14T19:04:36.291501137Z Wrote inspect data to must-gather.
2026-07-14T19:04:36.384424818Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:36.489791210Z W0714 19:04:36.489743    1559 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-14T19:04:36.497422739Z Wrote inspect data to must-gather.
2026-07-14T19:04:36.558952984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T19:04:36.595893513Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:36.598542766Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-14T19:04:36.624527112Z INFO: Getting status from alertmanager-main-0
2026-07-14T19:04:36.770170074Z Gathering data for ns/openshift-network-console...
2026-07-14T19:04:36.793164089Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-14T19:04:36.795780078Z INFO: Fallback to identify the container image from release info
2026-07-14T19:04:36.840900624Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:37.034539562Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:37.037335520Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-14T19:04:37.205001573Z Gathering data for ns/openshift-console-operator...
2026-07-14T19:04:37.208835197Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:37.399886893Z Wrote inspect data to /must-gather/istio.
2026-07-14T19:04:37.411775357Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-14T19:04:37.537539370Z daemonset.apps/perf-node-gather-daemonset created
2026-07-14T19:04:37.634271691Z Gathering data for ns/openshift-console...
2026-07-14T19:04:37.667707840Z Waiting for performance profile collector pods to become ready: 1
2026-07-14T19:04:38.096178158Z Done executing Istio gather script
2026-07-14T19:04:38.164956113Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-14T19:04:38.687477505Z Gathering data for ns/openshift-dns-operator...
2026-07-14T19:04:38.785768733Z Waiting for performance profile collector pods to become ready: 2
2026-07-14T19:04:38.969590409Z Gathering data for ns/openshift-dns...
2026-07-14T19:04:39.055737336Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:39.741531672Z Gathering data for ns/openshift-image-registry...
2026-07-14T19:04:39.884996156Z Waiting for performance profile collector pods to become ready: 3
2026-07-14T19:04:40.261602464Z Gathering data for ns/openshift-ingress-operator...
2026-07-14T19:04:40.575886917Z Gathering data for ns/openshift-ingress...
2026-07-14T19:04:40.971911521Z Gathering data for ns/openshift-ingress-canary...
2026-07-14T19:04:41.001994340Z Waiting for performance profile collector pods to become ready: 4
2026-07-14T19:04:41.351952287Z Gathering data for ns/openshift-insights...
2026-07-14T19:04:42.105373823Z Waiting for performance profile collector pods to become ready: 5
2026-07-14T19:04:43.189919838Z Gathering data for ns/openshift-config...
2026-07-14T19:04:43.207575079Z Waiting for performance profile collector pods to become ready: 6
2026-07-14T19:04:43.460622790Z Gathering data for ns/openshift-config-managed...
2026-07-14T19:04:43.886176155Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-14T19:04:44.062006826Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:44.160451092Z Gathering data for ns/openshift-kube-apiserver...
2026-07-14T19:04:44.310733694Z Waiting for performance profile collector pods to become ready: 7
2026-07-14T19:04:44.421197160Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-14T19:04:44.750296499Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-14T19:04:45.094464404Z Gathering data for ns/openshift-kube-scheduler...
2026-07-14T19:04:45.386000704Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-14T19:04:45.446741976Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-14T19:04:45.692739083Z Collecting performance related data for node ip-10-0-136-248.ec2.internal
2026-07-14T19:04:45.700821172Z Collecting performance related data for node ip-10-0-135-121.ec2.internal
2026-07-14T19:04:45.709690365Z Collecting performance related data for node ip-10-0-143-80.ec2.internal
2026-07-14T19:04:45.777490176Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-14T19:04:46.232150056Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-14T19:04:46.762456677Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-14T19:04:47.543940270Z Gathering data for ns/openshift-multus...
2026-07-14T19:04:49.066824108Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:49.248512516Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-14T19:04:50.665673357Z Gathering data for ns/openshift-host-network...
2026-07-14T19:04:51.052265909Z Gathering data for ns/openshift-network-diagnostics...
2026-07-14T19:04:51.592244874Z Gathering data for ns/openshift-network-node-identity...
2026-07-14T19:04:52.011537304Z Gathering data for ns/openshift-network-operator...
2026-07-14T19:04:52.427164142Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-14T19:04:52.715762551Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-14T19:04:53.290318372Z Gathering data for ns/openshift-apiserver-operator...
2026-07-14T19:04:53.640068192Z Gathering data for ns/openshift-apiserver...
2026-07-14T19:04:54.053334459Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-14T19:04:54.074984720Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-14T19:04:54.333215870Z Gathering data for ns/openshift-controller-manager...
2026-07-14T19:04:54.379184073Z Collecting kubelet logs for node ip-10-0-136-248.ec2.internal
2026-07-14T19:04:54.381672434Z Collecting kubelet logs for node ip-10-0-135-121.ec2.internal
2026-07-14T19:04:54.384083951Z Collecting kubelet logs for node ip-10-0-143-80.ec2.internal
2026-07-14T19:04:54.655533830Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-14T19:04:54.714679457Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-h4nks namespace
2026-07-14T19:04:54.719417540Z INFO: Node performance data collection complete.
2026-07-14T19:04:55.360985495Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-14T19:04:55.672626221Z Gathering data for ns/openshift-service-ca-operator...
2026-07-14T19:04:56.057511141Z Gathering data for ns/openshift-service-ca...
2026-07-14T19:04:56.488478446Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-14T19:04:57.449911838Z Wrote inspect data to must-gather.
2026-07-14T19:04:57.449953455Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T19:04:57.449953455Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-14T19:04:57.575610023Z error: the server doesn't have a resource type "clusters"
2026-07-14T19:04:57.711436935Z Caches written to disk