2026-06-09T07:25:28.374380217Z [disk usage checker] Started
2026-06-09T07:25:28.378368562Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:28.696451228Z Gathering data for ns/openshift-cluster-version...
2026-06-09T07:25:28.837331786Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-09T07:25:28.921756217Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T07:25:29.000955973Z Gathering data for ns/default...
2026-06-09T07:25:29.285784379Z Gathering data for ns/openshift...
2026-06-09T07:25:29.685950471Z Gathering data for ns/kube-system...
2026-06-09T07:25:30.325720181Z Gathering data for ns/openshift-etcd...
2026-06-09T07:25:30.633652734Z Wrote inspect data to must-gather.
2026-06-09T07:25:30.633694672Z error: inspection completed with the errors occurred while gathering data:
2026-06-09T07:25:30.633694672Z     namespaces "assisted-installer" not found
2026-06-09T07:25:31.108416644Z Waiting on subprocesses to finish execution.
2026-06-09T07:25:31.113283583Z INFO: Gathering HAProxy config files
2026-06-09T07:25:31.121557311Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-09T07:25:31.126678326Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-09T07:25:31.129570238Z Executing Istio gather script
2026-06-09T07:25:31.136784265Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-09T07:25:31.150967045Z INFO: Collecting host service logs for crio
2026-06-09T07:25:31.151306023Z INFO: Collecting host service logs for kubelet
2026-06-09T07:25:31.151627652Z INFO: Collecting host service logs for rpm-ostreed
2026-06-09T07:25:31.151922690Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-09T07:25:31.152247694Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-09T07:25:31.152548807Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-09T07:25:31.152833200Z INFO: Collecting host service logs for NetworkManager
2026-06-09T07:25:31.153132124Z INFO: Collecting host service logs for openvswitch
2026-06-09T07:25:31.153437302Z INFO: Collecting host service logs for ovs-configuration
2026-06-09T07:25:31.153746887Z INFO: Collecting host service logs for ovsdb-server
2026-06-09T07:25:31.154038154Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-09T07:25:31.154415690Z INFO: Waiting for worker host service log collection to complete ...
2026-06-09T07:25:31.228962769Z INFO: Waiting for node performance related collection to complete ...
2026-06-09T07:25:31.375034439Z No resources found in openshift-etcd namespace.
2026-06-09T07:25:31.567965754Z No resources found
2026-06-09T07:25:31.610042431Z INFO: "metallb-operator" not detected. Skipping.
2026-06-09T07:25:31.888529295Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-09T07:25:32.128865778Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-09T07:25:32.178263711Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-09T07:25:32.448782461Z error: the server doesn't have a resource type "performanceprofile"
2026-06-09T07:25:32.459303154Z INFO: Collecting Insights Archives from insights-operator-896549976-bzm47
2026-06-09T07:25:32.459303154Z insights-runtime-extractor-d2bsh
2026-06-09T07:25:32.459303154Z insights-runtime-extractor-m9sz6
2026-06-09T07:25:32.459303154Z insights-runtime-extractor-x8n6w
2026-06-09T07:25:32.571007815Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-09T07:25:32.659804047Z INFO: Worker host service log collection to complete.
2026-06-09T07:25:32.719941602Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-09T07:25:32.720105792Z INFO: on-disk MachineConfig config collection complete.
2026-06-09T07:25:32.763340785Z Wrote inspect data to must-gather.
2026-06-09T07:25:32.914919930Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-09T07:25:32.914919930Z See 'oc rsync -h' for help and examples
2026-06-09T07:25:32.936685627Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-09T07:25:33.098353575Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-09T07:25:33.157058573Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.168928175Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-09T07:25:33.208591398Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.470688056Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:33.474120299Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.617956822Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-09T07:25:33.654520449Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T07:25:33.681402669Z ERROR: No running kube-apiserver pods found
2026-06-09T07:25:33.689675726Z No resources found
2026-06-09T07:25:33.761983305Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.804012256Z INFO: 'previous-logs' folder not found on ip-10-0-133-250.ec2.internal, skipping...
2026-06-09T07:25:33.825745622Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.854842682Z error: the server doesn't have a resource type "machineconfigs"
2026-06-09T07:25:33.904911413Z Wrote inspect data to must-gather.
2026-06-09T07:25:33.934279502Z tar: Removing leading `/' from member names
2026-06-09T07:25:33.940776850Z INFO: HAProxy config collection complete.
2026-06-09T07:25:34.028863530Z INFO: Getting rules from prometheus-k8s-0
2026-06-09T07:25:34.072385066Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-09T07:25:34.139030126Z INFO: 'previous-logs' folder not found on ip-10-0-139-225.ec2.internal, skipping...
2026-06-09T07:25:34.150378615Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.155407727Z Gathering data for ns/openshift-monitoring...
2026-06-09T07:25:34.329683670Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.424818331Z INFO: 'previous-logs' folder not found on ip-10-0-142-75.ec2.internal, skipping...
2026-06-09T07:25:34.424882430Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-09T07:25:34.424882430Z INFO: Machine Config Daemon termination log collection complete.
2026-06-09T07:25:34.432522941Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.484899994Z INFO: Getting status/config from prometheus-k8s-0
2026-06-09T07:25:34.498831929Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-09T07:25:34.698001928Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.724218245Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.927257296Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-09T07:25:34.948922142Z Wrote inspect data to must-gather.
2026-06-09T07:25:34.998810574Z INFO: INTERCONNECT MODE
2026-06-09T07:25:34.998859630Z INFO: Gathering ovn-kubernetes DBs
2026-06-09T07:25:35.158475998Z Wrote inspect data to must-gather.
2026-06-09T07:25:35.161425163Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-09T07:25:35.169512138Z INFO: Gathering OVN_Northbound from ovnkube-node-8glcp...
2026-06-09T07:25:35.169768150Z INFO: Gathering OVN_Northbound from ovnkube-node-fbvwh...
2026-06-09T07:25:35.170012402Z INFO: Gathering OVN_Northbound from ovnkube-node-s8jw8...
2026-06-09T07:25:35.170361523Z INFO: Gathering OVN_Southbound from ovnkube-node-8glcp...
2026-06-09T07:25:35.170673597Z INFO: Gathering OVN_Southbound from ovnkube-node-fbvwh...
2026-06-09T07:25:35.170969010Z INFO: Gathering OVN_Southbound from ovnkube-node-s8jw8...
2026-06-09T07:25:35.293990244Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-09T07:25:35.472491721Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.494350691Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.537263945Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.542781883Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.548670910Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.566562340Z Wrote inspect data to must-gather.
2026-06-09T07:25:35.569643693Z tar: Removing leading `/' from member names
2026-06-09T07:25:35.628556466Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-09T07:25:35.666525210Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:35.764093807Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-09T07:25:35.868539235Z Wrote inspect data to must-gather.
2026-06-09T07:25:35.920007223Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-09T07:25:35.929668507Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:35.932391866Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-09T07:25:36.120426097Z INFO: Waiting for network log collection to complete ...
2026-06-09T07:25:36.120463234Z INFO: Waiting for ovnk database copies to complete ...
2026-06-09T07:25:36.120888183Z INFO: Copying ovnk databases complete.
2026-06-09T07:25:36.122462425Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-06-09T07:25:36.123700547Z Wrote inspect data to must-gather.
2026-06-09T07:25:36.124744332Z ovnk_database_store/
2026-06-09T07:25:36.124744332Z ovnk_database_store/ovnkube-node-fbvwh_sbdb
2026-06-09T07:25:36.144454970Z ovnk_database_store/ovnkube-node-8glcp_sbdb
2026-06-09T07:25:36.159994631Z ovnk_database_store/ovnkube-node-8glcp_nbdb
2026-06-09T07:25:36.168501621Z ovnk_database_store/ovnkube-node-fbvwh_nbdb
2026-06-09T07:25:36.172495441Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:36.178636017Z ovnk_database_store/ovnkube-node-s8jw8_sbdb
2026-06-09T07:25:36.196010348Z ovnk_database_store/ovnkube-node-s8jw8_nbdb
2026-06-09T07:25:36.207043810Z INFO: Network log collection complete.
2026-06-09T07:25:36.346928624Z INFO: Getting status from alertmanager-main-0
2026-06-09T07:25:36.352965921Z W0609 07:25:36.352910    1527 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-09T07:25:36.359000934Z Wrote inspect data to must-gather.
2026-06-09T07:25:36.402621201Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:36.405942587Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-09T07:25:36.458445588Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-09T07:25:36.621501346Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:36.635722686Z Gathering data for ns/openshift-network-console...
2026-06-09T07:25:36.638991130Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-09T07:25:36.641694185Z INFO: Fallback to identify the container image from release info
2026-06-09T07:25:36.823293221Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:36.825950672Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-09T07:25:37.047331429Z Gathering data for ns/openshift-console-operator...
2026-06-09T07:25:37.049838268Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:37.248067815Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:37.250732540Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-09T07:25:37.415433398Z Gathering data for ns/openshift-console...
2026-06-09T07:25:37.418989282Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:37.579173090Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-09T07:25:37.626880234Z Wrote inspect data to /must-gather/istio.
2026-06-09T07:25:37.703280656Z daemonset.apps/perf-node-gather-daemonset created
2026-06-09T07:25:37.847304270Z Waiting for performance profile collector pods to become ready: 1
2026-06-09T07:25:38.012783707Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-09T07:25:38.372857528Z Done executing Istio gather script
2026-06-09T07:25:38.504018481Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:38.541946436Z Gathering data for ns/openshift-dns-operator...
2026-06-09T07:25:38.826816387Z Gathering data for ns/openshift-dns...
2026-06-09T07:25:38.955027265Z Waiting for performance profile collector pods to become ready: 2
2026-06-09T07:25:39.491306201Z Gathering data for ns/openshift-image-registry...
2026-06-09T07:25:40.087589531Z Waiting for performance profile collector pods to become ready: 3
2026-06-09T07:25:40.143662275Z Gathering data for ns/openshift-ingress-operator...
2026-06-09T07:25:40.470550790Z Gathering data for ns/openshift-ingress...
2026-06-09T07:25:40.804561303Z Gathering data for ns/openshift-ingress-canary...
2026-06-09T07:25:41.213979077Z Waiting for performance profile collector pods to become ready: 4
2026-06-09T07:25:41.234397531Z Gathering data for ns/openshift-insights...
2026-06-09T07:25:42.326188156Z Waiting for performance profile collector pods to become ready: 5
2026-06-09T07:25:43.289226102Z Gathering data for ns/openshift-jobset-operator...
2026-06-09T07:25:43.451223724Z Waiting for performance profile collector pods to become ready: 6
2026-06-09T07:25:43.509366400Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:44.042331901Z Gathering data for ns/openshift-config...
2026-06-09T07:25:44.290217731Z Gathering data for ns/openshift-config-managed...
2026-06-09T07:25:44.565812798Z Waiting for performance profile collector pods to become ready: 7
2026-06-09T07:25:44.772602062Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-09T07:25:45.018180480Z Gathering data for ns/openshift-kube-apiserver...
2026-06-09T07:25:45.272778642Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-09T07:25:45.542229004Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-09T07:25:45.685388780Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-09T07:25:45.829283889Z Gathering data for ns/openshift-kube-scheduler...
2026-06-09T07:25:45.965191387Z Collecting performance related data for node ip-10-0-133-250.ec2.internal
2026-06-09T07:25:45.972785090Z Collecting performance related data for node ip-10-0-139-225.ec2.internal
2026-06-09T07:25:45.984656121Z Collecting performance related data for node ip-10-0-142-75.ec2.internal
2026-06-09T07:25:46.141970329Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-09T07:25:46.487425230Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-09T07:25:46.878258274Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-09T07:25:47.279996743Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-09T07:25:47.820932213Z Gathering data for ns/openshift-multus...
2026-06-09T07:25:48.514397360Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:49.167909472Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-09T07:25:50.346243713Z Gathering data for ns/openshift-host-network...
2026-06-09T07:25:50.626614249Z Gathering data for ns/openshift-network-diagnostics...
2026-06-09T07:25:51.126201672Z Gathering data for ns/openshift-network-node-identity...
2026-06-09T07:25:51.606023509Z Gathering data for ns/openshift-network-operator...
2026-06-09T07:25:51.943820534Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-09T07:25:52.174702880Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-09T07:25:52.603043723Z Gathering data for ns/openshift-apiserver-operator...
2026-06-09T07:25:52.912880589Z Gathering data for ns/openshift-apiserver...
2026-06-09T07:25:53.321849462Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-09T07:25:53.520475863Z [disk usage checker] Volume usage percentage: current = 45 ; allowed = 70
2026-06-09T07:25:53.579269405Z Gathering data for ns/openshift-controller-manager...
2026-06-09T07:25:53.812862551Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-09T07:25:54.546948894Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-09T07:25:54.801054694Z Gathering data for ns/openshift-service-ca-operator...
2026-06-09T07:25:54.830428351Z Collecting kubelet logs for node ip-10-0-133-250.ec2.internal
2026-06-09T07:25:54.832723192Z Collecting kubelet logs for node ip-10-0-139-225.ec2.internal
2026-06-09T07:25:54.835080647Z Collecting kubelet logs for node ip-10-0-142-75.ec2.internal
2026-06-09T07:25:55.138336805Z Gathering data for ns/openshift-service-ca...
2026-06-09T07:25:55.202112460Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kmqfn namespace
2026-06-09T07:25:55.208189380Z INFO: Node performance data collection complete.
2026-06-09T07:25:55.498022004Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-09T07:25:56.329964227Z Wrote inspect data to must-gather.
2026-06-09T07:25:56.330014130Z error: inspection completed with the errors occurred while gathering data:
2026-06-09T07:25:56.330014130Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-09T07:25:56.453783421Z error: the server doesn't have a resource type "clusters"
2026-06-09T07:25:56.546606771Z Caches written to disk