2026-07-12T23:54:48.255524428Z [disk usage checker] Started
2026-07-12T23:54:48.258766690Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-12T23:54:48.438300622Z Gathering data for ns/openshift-cluster-version...
2026-07-12T23:54:48.569848738Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-12T23:54:48.638722493Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-12T23:54:48.709315898Z Gathering data for ns/default...
2026-07-12T23:54:48.960318408Z Gathering data for ns/openshift...
2026-07-12T23:54:49.357074106Z Gathering data for ns/kube-system...
2026-07-12T23:54:49.976206465Z Gathering data for ns/openshift-etcd...
2026-07-12T23:54:50.230243628Z Wrote inspect data to must-gather.
2026-07-12T23:54:50.230271688Z error: inspection completed with the errors occurred while gathering data:
2026-07-12T23:54:50.230271688Z     namespaces "assisted-installer" not found
2026-07-12T23:54:50.674305279Z Waiting on subprocesses to finish execution.
2026-07-12T23:54:50.678818696Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-12T23:54:50.688592752Z INFO: Collecting host service logs for crio
2026-07-12T23:54:50.688626180Z Executing Istio gather script
2026-07-12T23:54:50.689028515Z INFO: Collecting host service logs for kubelet
2026-07-12T23:54:50.689381782Z INFO: Collecting host service logs for rpm-ostreed
2026-07-12T23:54:50.689721726Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-12T23:54:50.690065491Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-12T23:54:50.690417447Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-12T23:54:50.690740977Z INFO: Collecting host service logs for NetworkManager
2026-07-12T23:54:50.691083510Z INFO: Collecting host service logs for openvswitch
2026-07-12T23:54:50.691426517Z INFO: Collecting host service logs for ovs-configuration
2026-07-12T23:54:50.691765783Z INFO: Collecting host service logs for ovsdb-server
2026-07-12T23:54:50.692478418Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-12T23:54:50.692933720Z INFO: Waiting for worker host service log collection to complete ...
2026-07-12T23:54:50.702516060Z INFO: Gathering HAProxy config files
2026-07-12T23:54:50.723762005Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-12T23:54:50.731891149Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-12T23:54:50.755632982Z INFO: Waiting for node performance related collection to complete ...
2026-07-12T23:54:51.236449126Z No resources found in openshift-etcd namespace.
2026-07-12T23:54:51.489078403Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-12T23:54:51.490406390Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-12T23:54:51.588154195Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-12T23:54:51.669392180Z INFO: "metallb-operator" not detected. Skipping.
2026-07-12T23:54:51.701028786Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-hpbwb
2026-07-12T23:54:51.701078277Z insights-runtime-extractor-fcm8q
2026-07-12T23:54:51.701098932Z insights-runtime-extractor-kp46t
2026-07-12T23:54:51.701118590Z insights-runtime-extractor-lhm2x
2026-07-12T23:54:51.729337148Z Wrote inspect data to must-gather.
2026-07-12T23:54:51.853239185Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-12T23:54:51.941394593Z No resources found
2026-07-12T23:54:51.964514035Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-12T23:54:51.964542994Z INFO: on-disk MachineConfig config collection complete.
2026-07-12T23:54:52.166237348Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-12T23:54:52.166237348Z See 'oc rsync -h' for help and examples
2026-07-12T23:54:52.221318874Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-12T23:54:52.317220854Z INFO: Worker host service log collection to complete.
2026-07-12T23:54:52.408720534Z error: the server doesn't have a resource type "performanceprofile"
2026-07-12T23:54:52.481477416Z INFO: HAProxy config collection complete.
2026-07-12T23:54:52.522137671Z Wrote inspect data to must-gather.
2026-07-12T23:54:52.639763417Z ERROR: No running kube-apiserver pods found
2026-07-12T23:54:52.640681796Z INFO: 'previous-logs' folder not found on ip-10-0-130-128.ec2.internal, skipping...
2026-07-12T23:54:52.805246678Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-12T23:54:52.832741737Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-12T23:54:52.911296587Z Gathering data for ns/openshift-monitoring...
2026-07-12T23:54:52.941865055Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-12T23:54:52.976340312Z Wrote inspect data to must-gather.
2026-07-12T23:54:52.978036454Z No resources found
2026-07-12T23:54:53.091168110Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-12T23:54:53.141903593Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.143350848Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.185573147Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.236483828Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.269454386Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-12T23:54:53.324808746Z error: the server doesn't have a resource type "machineconfigs"
2026-07-12T23:54:53.352724049Z INFO: 'previous-logs' folder not found on ip-10-0-130-51.ec2.internal, skipping...
2026-07-12T23:54:53.466956468Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.476101968Z INFO: Getting rules from prometheus-k8s-0
2026-07-12T23:54:53.505104783Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-12T23:54:53.625362984Z INFO: 'previous-logs' folder not found on ip-10-0-137-234.ec2.internal, skipping...
2026-07-12T23:54:53.625473973Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-12T23:54:53.625532529Z INFO: Machine Config Daemon termination log collection complete.
2026-07-12T23:54:53.718284962Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.729568348Z Wrote inspect data to must-gather.
2026-07-12T23:54:53.860935806Z INFO: Getting status/config from prometheus-k8s-0
2026-07-12T23:54:53.875565615Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-12T23:54:53.951250982Z Wrote inspect data to must-gather.
2026-07-12T23:54:54.068887145Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-12T23:54:54.078048080Z Wrote inspect data to must-gather.
2026-07-12T23:54:54.232202465Z INFO: INTERCONNECT MODE
2026-07-12T23:54:54.232236520Z INFO: Gathering ovn-kubernetes DBs
2026-07-12T23:54:54.257297618Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-12T23:54:54.258755615Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:54.295183395Z Wrote inspect data to must-gather.
2026-07-12T23:54:54.379140842Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-12T23:54:54.415170219Z INFO: Gathering OVN_Northbound from ovnkube-node-4gcn5...
2026-07-12T23:54:54.415522453Z INFO: Gathering OVN_Northbound from ovnkube-node-d5nb9...
2026-07-12T23:54:54.415801822Z INFO: Gathering OVN_Northbound from ovnkube-node-l9hjg...
2026-07-12T23:54:54.416134262Z INFO: Gathering OVN_Southbound from ovnkube-node-4gcn5...
2026-07-12T23:54:54.416432061Z INFO: Gathering OVN_Southbound from ovnkube-node-d5nb9...
2026-07-12T23:54:54.418590933Z INFO: Gathering OVN_Southbound from ovnkube-node-l9hjg...
2026-07-12T23:54:54.515215393Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-12T23:54:54.519965947Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:54.528814883Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-12T23:54:54.551149361Z Wrote inspect data to must-gather.
2026-07-12T23:54:54.698007351Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.732056772Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.755635391Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.772502488Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.775004764Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.775540211Z tar: Removing leading `/' from member names
2026-07-12T23:54:54.864725163Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:54.875724720Z Wrote inspect data to must-gather.
2026-07-12T23:54:54.891265806Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-12T23:54:55.097700189Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:55.101363331Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-12T23:54:55.123678616Z W0712 23:54:55.123632    1411 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-12T23:54:55.129992224Z Wrote inspect data to must-gather.
2026-07-12T23:54:55.139558844Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-12T23:54:55.149003188Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-12T23:54:55.237538976Z INFO: Waiting for network log collection to complete ...
2026-07-12T23:54:55.237570008Z INFO: Waiting for ovnk database copies to complete ...
2026-07-12T23:54:55.237981454Z INFO: Copying ovnk databases complete.
2026-07-12T23:54:55.239930172Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-12T23:54:55.242757727Z ovnk_database_store/
2026-07-12T23:54:55.242757727Z ovnk_database_store/ovnkube-node-d5nb9_nbdb
2026-07-12T23:54:55.251326610Z ovnk_database_store/ovnkube-node-4gcn5_sbdb
2026-07-12T23:54:55.267250478Z ovnk_database_store/ovnkube-node-d5nb9_sbdb
2026-07-12T23:54:55.288121632Z ovnk_database_store/ovnkube-node-l9hjg_sbdb
2026-07-12T23:54:55.305310716Z ovnk_database_store/ovnkube-node-4gcn5_nbdb
2026-07-12T23:54:55.312594257Z ovnk_database_store/ovnkube-node-l9hjg_nbdb
2026-07-12T23:54:55.323005011Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:55.323300460Z INFO: Network log collection complete.
2026-07-12T23:54:55.346343425Z Gathering data for ns/openshift-network-console...
2026-07-12T23:54:55.440957716Z Wrote inspect data to must-gather.
2026-07-12T23:54:55.527684222Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:55.530910229Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-12T23:54:55.534986328Z INFO: Getting status from alertmanager-main-0
2026-07-12T23:54:55.722687016Z Wrote inspect data to must-gather.
2026-07-12T23:54:55.770813316Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:55.783622819Z Gathering data for ns/openshift-console-operator...
2026-07-12T23:54:55.960762797Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:55.963249082Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-12T23:54:55.973203695Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-12T23:54:55.975252297Z INFO: Fallback to identify the container image from release info
2026-07-12T23:54:56.127978354Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:56.131642412Z Gathering data for ns/openshift-console...
2026-07-12T23:54:56.297991469Z Wrote inspect data to /must-gather/istio.
2026-07-12T23:54:56.520862664Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-12T23:54:56.599355733Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-12T23:54:56.641452551Z daemonset.apps/perf-node-gather-daemonset created
2026-07-12T23:54:56.778225599Z Waiting for performance profile collector pods to become ready: 1
2026-07-12T23:54:56.954417150Z Done executing Istio gather script
2026-07-12T23:54:57.049985674Z Gathering data for ns/openshift-dns-operator...
2026-07-12T23:54:57.314270893Z Gathering data for ns/openshift-dns...
2026-07-12T23:54:57.784891107Z Gathering data for ns/openshift-image-registry...
2026-07-12T23:54:57.882897389Z Waiting for performance profile collector pods to become ready: 2
2026-07-12T23:54:58.181885164Z Gathering data for ns/openshift-ingress-operator...
2026-07-12T23:54:58.275979130Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-12T23:54:58.442933529Z Gathering data for ns/openshift-ingress...
2026-07-12T23:54:58.700482057Z Gathering data for ns/openshift-ingress-canary...
2026-07-12T23:54:59.001034455Z Waiting for performance profile collector pods to become ready: 3
2026-07-12T23:54:59.018247637Z Gathering data for ns/openshift-insights...
2026-07-12T23:55:00.109062446Z Waiting for performance profile collector pods to become ready: 4
2026-07-12T23:55:00.690122484Z Gathering data for ns/openshift-config...
2026-07-12T23:55:00.897510481Z Gathering data for ns/openshift-config-managed...
2026-07-12T23:55:01.232256949Z Waiting for performance profile collector pods to become ready: 5
2026-07-12T23:55:01.276550348Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-12T23:55:01.485105206Z Gathering data for ns/openshift-kube-apiserver...
2026-07-12T23:55:01.687029201Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-12T23:55:01.921630075Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-12T23:55:02.152201259Z Gathering data for ns/openshift-kube-scheduler...
2026-07-12T23:55:02.353663749Z Waiting for performance profile collector pods to become ready: 6
2026-07-12T23:55:02.373039129Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-12T23:55:02.602106629Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-12T23:55:02.851443470Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-12T23:55:03.110972036Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-12T23:55:03.281418259Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-12T23:55:03.461491619Z Waiting for performance profile collector pods to become ready: 7
2026-07-12T23:55:03.516662350Z Gathering data for ns/openshift-multus...
2026-07-12T23:55:04.580061248Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-12T23:55:04.664943353Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-12T23:55:04.805169725Z Collecting performance related data for node ip-10-0-137-234.ec2.internal
2026-07-12T23:55:04.812056037Z Collecting performance related data for node ip-10-0-130-128.ec2.internal
2026-07-12T23:55:04.819506193Z Collecting performance related data for node ip-10-0-130-51.ec2.internal
2026-07-12T23:55:05.855789423Z Gathering data for ns/openshift-host-network...
2026-07-12T23:55:06.167501528Z Gathering data for ns/openshift-network-diagnostics...
2026-07-12T23:55:06.613305319Z Gathering data for ns/openshift-network-node-identity...
2026-07-12T23:55:06.957427009Z Gathering data for ns/openshift-network-operator...
2026-07-12T23:55:07.309247882Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-12T23:55:07.560792179Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-12T23:55:07.984475708Z Gathering data for ns/openshift-apiserver-operator...
2026-07-12T23:55:08.223300771Z Gathering data for ns/openshift-apiserver...
2026-07-12T23:55:08.286309313Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-12T23:55:08.546573505Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-12T23:55:08.759600174Z Gathering data for ns/openshift-controller-manager...
2026-07-12T23:55:08.983769732Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-12T23:55:09.568395329Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-12T23:55:09.817513370Z Gathering data for ns/openshift-service-ca-operator...
2026-07-12T23:55:10.076859885Z Gathering data for ns/openshift-service-ca...
2026-07-12T23:55:10.379275859Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-12T23:55:11.166278842Z Wrote inspect data to must-gather.
2026-07-12T23:55:11.166317794Z error: inspection completed with the errors occurred while gathering data:
2026-07-12T23:55:11.166317794Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-12T23:55:12.643572201Z Collecting kubelet logs for node ip-10-0-137-234.ec2.internal
2026-07-12T23:55:12.645206945Z Collecting kubelet logs for node ip-10-0-130-128.ec2.internal
2026-07-12T23:55:12.646871469Z Collecting kubelet logs for node ip-10-0-130-51.ec2.internal
2026-07-12T23:55:12.926009245Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kw7kr namespace
2026-07-12T23:55:12.930558969Z INFO: Node performance data collection complete.
2026-07-12T23:55:13.050305814Z error: the server doesn't have a resource type "clusters"
2026-07-12T23:55:13.209205174Z Caches written to disk