2026-06-11T05:49:47.290801712Z [disk usage checker] Started
2026-06-11T05:49:47.294365936Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:49:47.486068683Z Gathering data for ns/openshift-cluster-version...
2026-06-11T05:49:47.611936371Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T05:49:47.696864885Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T05:49:47.782058891Z Gathering data for ns/default...
2026-06-11T05:49:48.044137790Z Gathering data for ns/openshift...
2026-06-11T05:49:48.430919104Z Gathering data for ns/kube-system...
2026-06-11T05:49:49.103820085Z Gathering data for ns/openshift-etcd...
2026-06-11T05:49:49.388070369Z Wrote inspect data to must-gather.
2026-06-11T05:49:49.388158587Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T05:49:49.388158587Z     namespaces "assisted-installer" not found
2026-06-11T05:49:49.761374823Z Waiting on subprocesses to finish execution.
2026-06-11T05:49:49.767611948Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-11T05:49:49.769722078Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-11T05:49:49.780802059Z INFO: Collecting host service logs for crio
2026-06-11T05:49:49.781189476Z INFO: Collecting host service logs for kubelet
2026-06-11T05:49:49.781537962Z INFO: Collecting host service logs for rpm-ostreed
2026-06-11T05:49:49.781907669Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-11T05:49:49.782234400Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-11T05:49:49.782537150Z Executing Istio gather script
2026-06-11T05:49:49.782554712Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-11T05:49:49.782862613Z INFO: Collecting host service logs for NetworkManager
2026-06-11T05:49:49.783215866Z INFO: Collecting host service logs for openvswitch
2026-06-11T05:49:49.783534070Z INFO: Collecting host service logs for ovs-configuration
2026-06-11T05:49:49.783959155Z INFO: Collecting host service logs for ovsdb-server
2026-06-11T05:49:49.784302796Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-11T05:49:49.784656894Z INFO: Gathering HAProxy config files
2026-06-11T05:49:49.784740220Z INFO: Waiting for worker host service log collection to complete ...
2026-06-11T05:49:49.788742220Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-11T05:49:49.824540876Z INFO: Waiting for node performance related collection to complete ...
2026-06-11T05:49:50.579308756Z No resources found in openshift-etcd namespace.
2026-06-11T05:49:50.594934917Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-11T05:49:50.598501085Z INFO: "metallb-operator" not detected. Skipping.
2026-06-11T05:49:50.659628858Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-11T05:49:50.659673935Z INFO: on-disk MachineConfig config collection complete.
2026-06-11T05:49:50.718190395Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-11T05:49:50.751740035Z INFO: Collecting Insights Archives from insights-operator-896549976-6dd22
2026-06-11T05:49:50.751780775Z insights-runtime-extractor-755nx
2026-06-11T05:49:50.751798975Z insights-runtime-extractor-9l8qp
2026-06-11T05:49:50.751850809Z insights-runtime-extractor-jhp9p
2026-06-11T05:49:50.819399471Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-11T05:49:50.864133515Z No resources found
2026-06-11T05:49:50.898748409Z error: the server doesn't have a resource type "performanceprofile"
2026-06-11T05:49:51.183306671Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-11T05:49:51.287419547Z INFO: Worker host service log collection to complete.
2026-06-11T05:49:51.408905185Z Wrote inspect data to must-gather.
2026-06-11T05:49:51.517562038Z ERROR: No running kube-apiserver pods found
2026-06-11T05:49:51.518349492Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-11T05:49:51.631349655Z No resources found
2026-06-11T05:49:51.687269948Z Gathering data for ns/openshift-monitoring...
2026-06-11T05:49:51.731371993Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-11T05:49:51.731371993Z See 'oc rsync -h' for help and examples
2026-06-11T05:49:51.759018805Z Wrote inspect data to must-gather.
2026-06-11T05:49:51.819147761Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-11T05:49:51.820413192Z Wrote inspect data to must-gather.
2026-06-11T05:49:51.843067995Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.061974157Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.111276214Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-11T05:49:52.169221695Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.173410268Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.173941711Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T05:49:52.202159751Z INFO: HAProxy config collection complete.
2026-06-11T05:49:52.296255219Z INFO: 'previous-logs' folder not found on ip-10-0-132-144.ec2.internal, skipping...
2026-06-11T05:49:52.300344141Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:49:52.352786455Z error: the server doesn't have a resource type "machineconfigs"
2026-06-11T05:49:52.487481525Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-11T05:49:52.504010244Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.536361331Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-11T05:49:52.571933000Z INFO: Getting rules from prometheus-k8s-0
2026-06-11T05:49:52.583799192Z INFO: 'previous-logs' folder not found on ip-10-0-134-8.ec2.internal, skipping...
2026-06-11T05:49:52.783746166Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.792798047Z Wrote inspect data to must-gather.
2026-06-11T05:49:52.843902243Z INFO: 'previous-logs' folder not found on ip-10-0-136-180.ec2.internal, skipping...
2026-06-11T05:49:52.843943567Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-11T05:49:52.843957433Z INFO: Machine Config Daemon termination log collection complete.
2026-06-11T05:49:52.955324908Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-11T05:49:52.965482203Z INFO: Getting status/config from prometheus-k8s-0
2026-06-11T05:49:53.027178288Z Wrote inspect data to must-gather.
2026-06-11T05:49:53.179717129Z Wrote inspect data to must-gather.
2026-06-11T05:49:53.312602452Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-11T05:49:53.320960436Z INFO: INTERCONNECT MODE
2026-06-11T05:49:53.320996476Z INFO: Gathering ovn-kubernetes DBs
2026-06-11T05:49:53.341068494Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T05:49:53.379844790Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-11T05:49:53.411778921Z Wrote inspect data to must-gather.
2026-06-11T05:49:53.498234493Z INFO: Gathering OVN_Northbound from ovnkube-node-pjww7...
2026-06-11T05:49:53.498597880Z INFO: Gathering OVN_Northbound from ovnkube-node-qxlpb...
2026-06-11T05:49:53.498911595Z INFO: Gathering OVN_Northbound from ovnkube-node-s4h5h...
2026-06-11T05:49:53.499217865Z INFO: Gathering OVN_Southbound from ovnkube-node-pjww7...
2026-06-11T05:49:53.499492476Z INFO: Gathering OVN_Southbound from ovnkube-node-qxlpb...
2026-06-11T05:49:53.499763823Z INFO: Gathering OVN_Southbound from ovnkube-node-s4h5h...
2026-06-11T05:49:53.504034964Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:53.726307881Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-11T05:49:53.782657223Z Wrote inspect data to must-gather.
2026-06-11T05:49:53.829814006Z tar: Removing leading `/' from member names
2026-06-11T05:49:53.853222092Z tar: Removing leading `/' from member names
2026-06-11T05:49:53.864183949Z tar: Removing leading `/' from member names
2026-06-11T05:49:53.876796273Z tar: Removing leading `/' from member names
2026-06-11T05:49:53.894001043Z tar: Removing leading `/' from member names
2026-06-11T05:49:53.920164391Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:53.923114081Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-11T05:49:53.934521683Z tar: Removing leading `/' from member names
2026-06-11T05:49:54.040938836Z W0611 05:49:54.040777    1403 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-11T05:49:54.048763073Z Wrote inspect data to must-gather.
2026-06-11T05:49:54.055685048Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-11T05:49:54.161127560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T05:49:54.166515207Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:54.325421455Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-11T05:49:54.366858797Z INFO: Waiting for network log collection to complete ...
2026-06-11T05:49:54.366896230Z INFO: Waiting for ovnk database copies to complete ...
2026-06-11T05:49:54.367279007Z INFO: Copying ovnk databases complete.
2026-06-11T05:49:54.368801322Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-06-11T05:49:54.371264553Z ovnk_database_store/
2026-06-11T05:49:54.371314828Z ovnk_database_store/ovnkube-node-qxlpb_nbdb
2026-06-11T05:49:54.377344978Z Gathering data for ns/openshift-network-console...
2026-06-11T05:49:54.379419296Z ovnk_database_store/ovnkube-node-pjww7_nbdb
2026-06-11T05:49:54.386990326Z ovnk_database_store/ovnkube-node-qxlpb_sbdb
2026-06-11T05:49:54.400475811Z Wrote inspect data to must-gather.
2026-06-11T05:49:54.403754064Z ovnk_database_store/ovnkube-node-pjww7_sbdb
2026-06-11T05:49:54.412392759Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:54.417865525Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-11T05:49:54.420364784Z ovnk_database_store/ovnkube-node-s4h5h_nbdb
2026-06-11T05:49:54.427756123Z ovnk_database_store/ovnkube-node-s4h5h_sbdb
2026-06-11T05:49:54.449595003Z INFO: Network log collection complete.
2026-06-11T05:49:54.621512526Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:54.701003038Z Wrote inspect data to must-gather.
2026-06-11T05:49:54.734530460Z INFO: Getting status from alertmanager-main-0
2026-06-11T05:49:54.840683590Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:54.842963303Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-11T05:49:54.891812630Z Gathering data for ns/openshift-console-operator...
2026-06-11T05:49:55.028009292Z Wrote inspect data to must-gather.
2026-06-11T05:49:55.077192010Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:55.282515825Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-11T05:49:55.282582085Z Gathering data for ns/openshift-console...
2026-06-11T05:49:55.284443160Z INFO: Fallback to identify the container image from release info
2026-06-11T05:49:55.287753869Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:55.289943004Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-11T05:49:55.456787464Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:55.634265126Z Wrote inspect data to /must-gather/istio.
2026-06-11T05:49:55.760437314Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-11T05:49:55.936274099Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-11T05:49:56.068791663Z daemonset.apps/perf-node-gather-daemonset created
2026-06-11T05:49:56.203534965Z Waiting for performance profile collector pods to become ready: 1
2026-06-11T05:49:56.318609449Z Gathering data for ns/openshift-dns-operator...
2026-06-11T05:49:56.320687645Z Done executing Istio gather script
2026-06-11T05:49:56.605214058Z Gathering data for ns/openshift-dns...
2026-06-11T05:49:57.155508715Z Gathering data for ns/openshift-image-registry...
2026-06-11T05:49:57.310489496Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:49:57.322058340Z Waiting for performance profile collector pods to become ready: 2
2026-06-11T05:49:57.583011234Z Gathering data for ns/openshift-ingress-operator...
2026-06-11T05:49:57.875034409Z Gathering data for ns/openshift-ingress...
2026-06-11T05:49:58.195286356Z Gathering data for ns/openshift-ingress-canary...
2026-06-11T05:49:58.424985637Z Waiting for performance profile collector pods to become ready: 3
2026-06-11T05:49:58.545147538Z Gathering data for ns/openshift-insights...
2026-06-11T05:49:59.535004600Z Waiting for performance profile collector pods to become ready: 4
2026-06-11T05:50:00.289560385Z Gathering data for ns/openshift-config...
2026-06-11T05:50:00.536371650Z Gathering data for ns/openshift-config-managed...
2026-06-11T05:50:00.656374069Z Waiting for performance profile collector pods to become ready: 5
2026-06-11T05:50:00.918357565Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-11T05:50:01.199324094Z Gathering data for ns/openshift-kube-apiserver...
2026-06-11T05:50:01.446608461Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-11T05:50:01.713403633Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-11T05:50:01.788087105Z Waiting for performance profile collector pods to become ready: 6
2026-06-11T05:50:02.018277405Z Gathering data for ns/openshift-kube-scheduler...
2026-06-11T05:50:02.257429758Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-11T05:50:02.315954347Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:50:02.516099064Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-11T05:50:02.864760535Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-11T05:50:02.909654554Z Waiting for performance profile collector pods to become ready: 7
2026-06-11T05:50:03.160045855Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-11T05:50:03.612348264Z Gathering data for ns/openshift-multus...
2026-06-11T05:50:04.012884942Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-11T05:50:04.232295926Z Collecting performance related data for node ip-10-0-134-8.ec2.internal
2026-06-11T05:50:04.239549718Z Collecting performance related data for node ip-10-0-132-144.ec2.internal
2026-06-11T05:50:04.246462567Z Collecting performance related data for node ip-10-0-136-180.ec2.internal
2026-06-11T05:50:05.056027152Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-11T05:50:06.215273052Z Gathering data for ns/openshift-host-network...
2026-06-11T05:50:06.532590447Z Gathering data for ns/openshift-network-diagnostics...
2026-06-11T05:50:07.062914966Z Gathering data for ns/openshift-network-node-identity...
2026-06-11T05:50:07.321309041Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:50:07.461927341Z Gathering data for ns/openshift-network-operator...
2026-06-11T05:50:07.798980160Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-11T05:50:08.041362376Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-11T05:50:08.522569129Z Gathering data for ns/openshift-apiserver-operator...
2026-06-11T05:50:08.803951345Z Gathering data for ns/openshift-apiserver...
2026-06-11T05:50:09.169239291Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-11T05:50:09.458647226Z Gathering data for ns/openshift-controller-manager...
2026-06-11T05:50:09.727908805Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-11T05:50:10.354354290Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-11T05:50:10.651435869Z Gathering data for ns/openshift-service-ca-operator...
2026-06-11T05:50:10.934066796Z Gathering data for ns/openshift-service-ca...
2026-06-11T05:50:11.281389020Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-11T05:50:12.083490896Z Collecting kubelet logs for node ip-10-0-134-8.ec2.internal
2026-06-11T05:50:12.085207708Z Collecting kubelet logs for node ip-10-0-132-144.ec2.internal
2026-06-11T05:50:12.086811582Z Collecting kubelet logs for node ip-10-0-136-180.ec2.internal
2026-06-11T05:50:12.157231165Z Wrote inspect data to must-gather.
2026-06-11T05:50:12.157279147Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T05:50:12.157279147Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-11T05:50:12.325635620Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-11T05:50:12.374567689Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wtqp6 namespace
2026-06-11T05:50:12.380488467Z INFO: Node performance data collection complete.
2026-06-11T05:50:12.513360232Z error: the server doesn't have a resource type "clusters"
2026-06-11T05:50:12.662175422Z Caches written to disk