2026-07-14T14:48:21.658954884Z [disk usage checker] Started
2026-07-14T14:48:21.662545108Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-14T14:48:21.843562629Z Gathering data for ns/openshift-cluster-version...
2026-07-14T14:48:21.960910123Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T14:48:22.054121182Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:48:22.135729209Z Gathering data for ns/default...
2026-07-14T14:48:22.452102765Z Gathering data for ns/openshift...
2026-07-14T14:48:22.905208396Z Gathering data for ns/kube-system...
2026-07-14T14:48:23.635423413Z Gathering data for ns/openshift-etcd...
2026-07-14T14:48:23.956497311Z Wrote inspect data to must-gather.
2026-07-14T14:48:23.956537284Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T14:48:23.956537284Z     namespaces "assisted-installer" not found
2026-07-14T14:48:24.418715106Z Waiting on subprocesses to finish execution.
2026-07-14T14:48:24.428676622Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-14T14:48:24.432754722Z Executing Istio gather script
2026-07-14T14:48:24.434438524Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-14T14:48:24.439516007Z INFO: Waiting for node performance related collection to complete ...
2026-07-14T14:48:24.444147407Z INFO: Collecting host service logs for crio
2026-07-14T14:48:24.444492953Z INFO: Collecting host service logs for kubelet
2026-07-14T14:48:24.444843924Z INFO: Collecting host service logs for rpm-ostreed
2026-07-14T14:48:24.445198027Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-14T14:48:24.445545889Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-14T14:48:24.445877436Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-14T14:48:24.446241780Z INFO: Collecting host service logs for NetworkManager
2026-07-14T14:48:24.446585802Z INFO: Collecting host service logs for openvswitch
2026-07-14T14:48:24.446924971Z INFO: Collecting host service logs for ovs-configuration
2026-07-14T14:48:24.447304861Z INFO: Collecting host service logs for ovsdb-server
2026-07-14T14:48:24.447649526Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-14T14:48:24.448057118Z INFO: Waiting for worker host service log collection to complete ...
2026-07-14T14:48:24.449740508Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-14T14:48:24.450749349Z INFO: Gathering HAProxy config files
2026-07-14T14:48:24.898972189Z No resources found
2026-07-14T14:48:25.046897551Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-xgl4l
2026-07-14T14:48:25.046897551Z insights-runtime-extractor-j2kcz
2026-07-14T14:48:25.046897551Z insights-runtime-extractor-vw2zq
2026-07-14T14:48:25.046897551Z insights-runtime-extractor-z27b2
2026-07-14T14:48:25.142998754Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-14T14:48:25.217915710Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-14T14:48:25.381257198Z No resources found in openshift-etcd namespace.
2026-07-14T14:48:25.437597746Z Wrote inspect data to must-gather.
2026-07-14T14:48:25.592964497Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-14T14:48:25.593056569Z INFO: on-disk MachineConfig config collection complete.
2026-07-14T14:48:25.838570698Z INFO: "metallb-operator" not detected. Skipping.
2026-07-14T14:48:25.879188772Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-14T14:48:25.907604425Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-14T14:48:25.907604425Z See 'oc rsync -h' for help and examples
2026-07-14T14:48:25.928338768Z INFO: Worker host service log collection to complete.
2026-07-14T14:48:25.951045009Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-14T14:48:26.036487738Z error: the server doesn't have a resource type "performanceprofile"
2026-07-14T14:48:26.042220167Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-14T14:48:26.545042588Z ERROR: No running kube-apiserver pods found
2026-07-14T14:48:26.550983970Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-14T14:48:26.718370046Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:48:26.728290980Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-14T14:48:26.763422228Z INFO: 'previous-logs' folder not found on ip-10-0-131-121.ec2.internal, skipping...
2026-07-14T14:48:26.771536700Z Wrote inspect data to must-gather.
2026-07-14T14:48:26.834148771Z Wrote inspect data to must-gather.
2026-07-14T14:48:26.851881727Z Wrote inspect data to must-gather.
2026-07-14T14:48:26.879602106Z Wrote inspect data to must-gather.
2026-07-14T14:48:26.927885231Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-14T14:48:26.960526704Z No resources found
2026-07-14T14:48:27.135998850Z Wrote inspect data to must-gather.
2026-07-14T14:48:27.139062876Z INFO: Getting rules from prometheus-k8s-0
2026-07-14T14:48:27.156686182Z Wrote inspect data to must-gather.
2026-07-14T14:48:27.198317257Z Gathering data for ns/openshift-monitoring...
2026-07-14T14:48:27.269960271Z INFO: 'previous-logs' folder not found on ip-10-0-135-0.ec2.internal, skipping...
2026-07-14T14:48:27.311352384Z tar: Removing leading `/' from member names
2026-07-14T14:48:27.315435512Z INFO: HAProxy config collection complete.
2026-07-14T14:48:27.390015186Z error: the server doesn't have a resource type "machineconfigs"
2026-07-14T14:48:27.472565351Z Wrote inspect data to must-gather.
2026-07-14T14:48:27.587994333Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-14T14:48:27.606421584Z INFO: 'previous-logs' folder not found on ip-10-0-137-133.ec2.internal, skipping...
2026-07-14T14:48:27.606453404Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-14T14:48:27.606484998Z INFO: Machine Config Daemon termination log collection complete.
2026-07-14T14:48:27.694483081Z INFO: Getting status/config from prometheus-k8s-0
2026-07-14T14:48:27.767439099Z Wrote inspect data to must-gather.
2026-07-14T14:48:27.858474653Z Wrote inspect data to must-gather.
2026-07-14T14:48:28.032180700Z Wrote inspect data to must-gather.
2026-07-14T14:48:28.049544192Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-14T14:48:28.193604393Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-14T14:48:28.299638675Z Wrote inspect data to must-gather.
2026-07-14T14:48:28.377534367Z INFO: INTERCONNECT MODE
2026-07-14T14:48:28.377571082Z INFO: Gathering ovn-kubernetes DBs
2026-07-14T14:48:28.397800336Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-14T14:48:28.466290508Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-14T14:48:28.541770475Z Wrote inspect data to must-gather.
2026-07-14T14:48:28.542081785Z INFO: Gathering OVN_Northbound from ovnkube-node-8k7vq...
2026-07-14T14:48:28.542332676Z INFO: Gathering OVN_Northbound from ovnkube-node-sqg8s...
2026-07-14T14:48:28.542542772Z INFO: Gathering OVN_Northbound from ovnkube-node-w6c55...
2026-07-14T14:48:28.542832150Z INFO: Gathering OVN_Southbound from ovnkube-node-8k7vq...
2026-07-14T14:48:28.543153757Z INFO: Gathering OVN_Southbound from ovnkube-node-sqg8s...
2026-07-14T14:48:28.543459915Z INFO: Gathering OVN_Southbound from ovnkube-node-w6c55...
2026-07-14T14:48:28.853537005Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-14T14:48:28.886608840Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.891918114Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.914918540Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.927649290Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.938220190Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.959259516Z tar: Removing leading `/' from member names
2026-07-14T14:48:28.993094136Z Wrote inspect data to must-gather.
2026-07-14T14:48:29.073909705Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-14T14:48:29.162509797Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-14T14:48:29.179313151Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-14T14:48:29.257045244Z Wrote inspect data to must-gather.
2026-07-14T14:48:29.419032639Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:29.478177746Z INFO: Waiting for network log collection to complete ...
2026-07-14T14:48:29.478292225Z INFO: Waiting for ovnk database copies to complete ...
2026-07-14T14:48:29.478795877Z INFO: Copying ovnk databases complete.
2026-07-14T14:48:29.481360379Z 3.0M	must-gather/network_logs/ovnk_database_store
2026-07-14T14:48:29.483859762Z ovnk_database_store/
2026-07-14T14:48:29.483907917Z ovnk_database_store/ovnkube-node-8k7vq_nbdb
2026-07-14T14:48:29.493541409Z ovnk_database_store/ovnkube-node-8k7vq_sbdb
2026-07-14T14:48:29.510132532Z ovnk_database_store/ovnkube-node-w6c55_nbdb
2026-07-14T14:48:29.515461844Z Wrote inspect data to must-gather.
2026-07-14T14:48:29.517454838Z ovnk_database_store/ovnkube-node-w6c55_sbdb
2026-07-14T14:48:29.536585173Z ovnk_database_store/ovnkube-node-sqg8s_nbdb
2026-07-14T14:48:29.546047825Z ovnk_database_store/ovnkube-node-sqg8s_sbdb
2026-07-14T14:48:29.567340116Z INFO: Network log collection complete.
2026-07-14T14:48:29.647731973Z INFO: Getting status from alertmanager-main-0
2026-07-14T14:48:29.676470354Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:29.679751197Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-14T14:48:29.749135575Z W0714 14:48:29.749088    1502 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-14T14:48:29.757200097Z Wrote inspect data to must-gather.
2026-07-14T14:48:29.791627314Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-14T14:48:29.905174629Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:29.981728474Z Gathering data for ns/openshift-network-console...
2026-07-14T14:48:30.100473523Z Wrote inspect data to must-gather.
2026-07-14T14:48:30.128202349Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:30.130815295Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-14T14:48:30.352833283Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:30.412252595Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-14T14:48:30.414164382Z INFO: Fallback to identify the container image from release info
2026-07-14T14:48:30.433964013Z Gathering data for ns/openshift-console-operator...
2026-07-14T14:48:30.583596807Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:30.586037913Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-14T14:48:30.808676922Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:30.844469624Z Gathering data for ns/openshift-console...
2026-07-14T14:48:31.016695270Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:31.018886257Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-14T14:48:31.059384512Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-14T14:48:31.190620120Z daemonset.apps/perf-node-gather-daemonset created
2026-07-14T14:48:31.202211269Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:31.341941742Z Waiting for performance profile collector pods to become ready: 1
2026-07-14T14:48:31.434145597Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:31.436914243Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-14T14:48:31.462940582Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-14T14:48:31.622236524Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:31.734752835Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-14T14:48:31.839612950Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:31.843688540Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-14T14:48:32.026061251Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.033604758Z Gathering data for ns/openshift-dns-operator...
2026-07-14T14:48:32.225136047Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.228193230Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-14T14:48:32.396464598Z Gathering data for ns/openshift-dns...
2026-07-14T14:48:32.418591422Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.476689315Z Waiting for performance profile collector pods to become ready: 2
2026-07-14T14:48:32.596264066Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.598652764Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-14T14:48:32.773994067Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.968351249Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:32.971022115Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-14T14:48:33.087409761Z Gathering data for ns/openshift-image-registry...
2026-07-14T14:48:33.147486189Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:33.338554367Z Wrote inspect data to /must-gather/istio.
2026-07-14T14:48:33.616072342Z Waiting for performance profile collector pods to become ready: 3
2026-07-14T14:48:33.633975387Z Gathering data for ns/openshift-ingress-operator...
2026-07-14T14:48:34.020268525Z Gathering data for ns/openshift-ingress...
2026-07-14T14:48:34.041819881Z Done executing Istio gather script
2026-07-14T14:48:34.351176798Z Gathering data for ns/openshift-ingress-canary...
2026-07-14T14:48:34.752868571Z Waiting for performance profile collector pods to become ready: 4
2026-07-14T14:48:34.768096639Z Gathering data for ns/openshift-insights...
2026-07-14T14:48:35.868685418Z Waiting for performance profile collector pods to become ready: 5
2026-07-14T14:48:36.739453935Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:48:36.991792523Z Waiting for performance profile collector pods to become ready: 6
2026-07-14T14:48:37.074577390Z Gathering data for ns/kserve...
2026-07-14T14:48:38.108304032Z Waiting for performance profile collector pods to become ready: 7
2026-07-14T14:48:38.339226373Z Gathering data for ns/openshift-config...
2026-07-14T14:48:38.593233573Z Gathering data for ns/openshift-config-managed...
2026-07-14T14:48:39.015823489Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-14T14:48:39.236209973Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-14T14:48:39.302620373Z Gathering data for ns/openshift-kube-apiserver...
2026-07-14T14:48:39.513699539Z Collecting performance related data for node ip-10-0-131-121.ec2.internal
2026-07-14T14:48:39.521445131Z Collecting performance related data for node ip-10-0-137-133.ec2.internal
2026-07-14T14:48:39.533552675Z Collecting performance related data for node ip-10-0-135-0.ec2.internal
2026-07-14T14:48:39.621393255Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-14T14:48:39.994271193Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-14T14:48:40.337809720Z Gathering data for ns/openshift-kube-scheduler...
2026-07-14T14:48:40.704740907Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-14T14:48:41.087004854Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-14T14:48:41.459002763Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-14T14:48:41.744666056Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:48:41.777609755Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-14T14:48:42.355602051Z Gathering data for ns/openshift-multus...
2026-07-14T14:48:43.951092281Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-14T14:48:45.344464153Z Gathering data for ns/openshift-host-network...
2026-07-14T14:48:45.643729325Z Gathering data for ns/openshift-network-diagnostics...
2026-07-14T14:48:46.169369929Z Gathering data for ns/openshift-network-node-identity...
2026-07-14T14:48:46.575613424Z Gathering data for ns/openshift-network-operator...
2026-07-14T14:48:46.749195404Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:48:46.971527978Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-14T14:48:47.287089844Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-14T14:48:47.823665591Z Gathering data for ns/openshift-apiserver-operator...
2026-07-14T14:48:48.158004395Z Gathering data for ns/openshift-apiserver...
2026-07-14T14:48:48.236787213Z Collecting kubelet logs for node ip-10-0-131-121.ec2.internal
2026-07-14T14:48:48.238531681Z Collecting kubelet logs for node ip-10-0-137-133.ec2.internal
2026-07-14T14:48:48.240180147Z Collecting kubelet logs for node ip-10-0-135-0.ec2.internal
2026-07-14T14:48:48.601495195Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sn8bc namespace
2026-07-14T14:48:48.606441835Z INFO: Node performance data collection complete.
2026-07-14T14:48:48.638429252Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-14T14:48:48.983221464Z Gathering data for ns/openshift-controller-manager...
2026-07-14T14:48:49.283352105Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-14T14:48:49.969549131Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-14T14:48:50.321510028Z Gathering data for ns/openshift-service-ca-operator...
2026-07-14T14:48:50.653382414Z Gathering data for ns/openshift-service-ca...
2026-07-14T14:48:51.048050663Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-14T14:48:51.754672103Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-14T14:48:51.932882148Z Wrote inspect data to must-gather.
2026-07-14T14:48:51.932943211Z error: inspection completed with the errors occurred while gathering data:
2026-07-14T14:48:51.932943211Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-14T14:48:52.075808341Z error: the server doesn't have a resource type "clusters"
2026-07-14T14:48:52.205461036Z Caches written to disk