2026-07-07T16:19:41.518256367Z [disk usage checker] Started
2026-07-07T16:19:41.521447445Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:19:41.714080551Z Gathering data for ns/openshift-cluster-version...
2026-07-07T16:19:41.840097321Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T16:19:41.921946765Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:19:41.997280185Z Gathering data for ns/default...
2026-07-07T16:19:42.307241899Z Gathering data for ns/openshift...
2026-07-07T16:19:42.704316714Z Gathering data for ns/kube-system...
2026-07-07T16:19:43.382491748Z Gathering data for ns/openshift-etcd...
2026-07-07T16:19:43.678483507Z Wrote inspect data to must-gather.
2026-07-07T16:19:43.678516437Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T16:19:43.678516437Z     namespaces "assisted-installer" not found
2026-07-07T16:19:44.036125043Z Waiting on subprocesses to finish execution.
2026-07-07T16:19:44.040389350Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T16:19:44.041906909Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T16:19:44.052181924Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T16:19:44.054533163Z INFO: Gathering HAProxy config files
2026-07-07T16:19:44.063551400Z Executing Istio gather script
2026-07-07T16:19:44.072291782Z INFO: Collecting host service logs for crio
2026-07-07T16:19:44.072649856Z INFO: Collecting host service logs for kubelet
2026-07-07T16:19:44.073003522Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T16:19:44.073354395Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T16:19:44.073676922Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T16:19:44.074014685Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T16:19:44.074333556Z INFO: Collecting host service logs for NetworkManager
2026-07-07T16:19:44.074666520Z INFO: Collecting host service logs for openvswitch
2026-07-07T16:19:44.075011566Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T16:19:44.075360527Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T16:19:44.075704281Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T16:19:44.076178035Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T16:19:44.129391155Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T16:19:44.510723884Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T16:19:44.510816641Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T16:19:44.623983569Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-lvl7b
2026-07-07T16:19:44.623983569Z insights-runtime-extractor-5llcr
2026-07-07T16:19:44.623983569Z insights-runtime-extractor-glnt7
2026-07-07T16:19:44.623983569Z insights-runtime-extractor-jlcj5
2026-07-07T16:19:44.658014035Z No resources found
2026-07-07T16:19:44.885469922Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T16:19:44.911928935Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T16:19:45.069021511Z No resources found in openshift-etcd namespace.
2026-07-07T16:19:45.093077075Z Wrote inspect data to must-gather.
2026-07-07T16:19:45.101158428Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T16:19:45.366232641Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T16:19:45.591073369Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T16:19:45.595963341Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T16:19:45.595963341Z See 'oc rsync -h' for help and examples
2026-07-07T16:19:45.799165297Z INFO: Worker host service log collection to complete.
2026-07-07T16:19:45.886782343Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T16:19:45.907578625Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T16:19:45.936341171Z Wrote inspect data to must-gather.
2026-07-07T16:19:45.957789510Z INFO: 'previous-logs' folder not found on ip-10-0-134-97.ec2.internal, skipping...
2026-07-07T16:19:46.023992889Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T16:19:46.065645023Z No resources found
2026-07-07T16:19:46.070601942Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.077981609Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.256314260Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:19:46.381309484Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T16:19:46.405058670Z ERROR: No running kube-apiserver pods found
2026-07-07T16:19:46.467839599Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.485632134Z tar: Removing leading `/' from member names
2026-07-07T16:19:46.497930681Z INFO: HAProxy config collection complete.
2026-07-07T16:19:46.528441963Z Gathering data for ns/openshift-monitoring...
2026-07-07T16:19:46.533970639Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:19:46.536252297Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.562834119Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.680549261Z INFO: 'previous-logs' folder not found on ip-10-0-135-144.ec2.internal, skipping...
2026-07-07T16:19:46.726834641Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T16:19:46.746094069Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T16:19:46.809929157Z Wrote inspect data to must-gather.
2026-07-07T16:19:46.830033598Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T16:19:46.936283224Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T16:19:46.960801340Z INFO: 'previous-logs' folder not found on ip-10-0-139-180.ec2.internal, skipping...
2026-07-07T16:19:46.960838723Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T16:19:46.960871374Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T16:19:47.054845524Z Wrote inspect data to must-gather.
2026-07-07T16:19:47.159574833Z Wrote inspect data to must-gather.
2026-07-07T16:19:47.210840973Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T16:19:47.301881705Z Wrote inspect data to must-gather.
2026-07-07T16:19:47.320627644Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T16:19:47.519506712Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T16:19:47.528843726Z Wrote inspect data to must-gather.
2026-07-07T16:19:47.563696946Z INFO: INTERCONNECT MODE
2026-07-07T16:19:47.563748719Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T16:19:47.623063049Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T16:19:47.710560529Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:47.742989701Z INFO: Gathering OVN_Northbound from ovnkube-node-5bb8h...
2026-07-07T16:19:47.743346101Z INFO: Gathering OVN_Northbound from ovnkube-node-hcfqv...
2026-07-07T16:19:47.743635110Z INFO: Gathering OVN_Northbound from ovnkube-node-xtmhn...
2026-07-07T16:19:47.743998833Z INFO: Gathering OVN_Southbound from ovnkube-node-5bb8h...
2026-07-07T16:19:47.744295639Z INFO: Gathering OVN_Southbound from ovnkube-node-hcfqv...
2026-07-07T16:19:47.744575806Z INFO: Gathering OVN_Southbound from ovnkube-node-xtmhn...
2026-07-07T16:19:47.746040056Z Wrote inspect data to must-gather.
2026-07-07T16:19:47.909711095Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T16:19:48.035887280Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.081553003Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.094095502Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.112055201Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:48.115549509Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T16:19:48.126541235Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.129414374Z Wrote inspect data to must-gather.
2026-07-07T16:19:48.131638400Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.132890737Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T16:19:48.164744532Z tar: Removing leading `/' from member names
2026-07-07T16:19:48.277993372Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T16:19:48.356347797Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:48.399075895Z Wrote inspect data to must-gather.
2026-07-07T16:19:48.547460892Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T16:19:48.586045082Z INFO: Waiting for network log collection to complete ...
2026-07-07T16:19:48.586098968Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T16:19:48.586609633Z INFO: Copying ovnk databases complete.
2026-07-07T16:19:48.588973978Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:48.590309919Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-07T16:19:48.591512511Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T16:19:48.593125503Z ovnk_database_store/
2026-07-07T16:19:48.593125503Z ovnk_database_store/ovnkube-node-5bb8h_nbdb
2026-07-07T16:19:48.605562855Z ovnk_database_store/ovnkube-node-xtmhn_nbdb
2026-07-07T16:19:48.613632822Z ovnk_database_store/ovnkube-node-5bb8h_sbdb
2026-07-07T16:19:48.632683495Z ovnk_database_store/ovnkube-node-hcfqv_sbdb
2026-07-07T16:19:48.637811240Z Wrote inspect data to must-gather.
2026-07-07T16:19:48.649315310Z ovnk_database_store/ovnkube-node-hcfqv_nbdb
2026-07-07T16:19:48.657851704Z ovnk_database_store/ovnkube-node-xtmhn_sbdb
2026-07-07T16:19:48.677795345Z INFO: Network log collection complete.
2026-07-07T16:19:48.786524865Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:19:48.795209155Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:48.932281214Z INFO: Getting status from alertmanager-main-0
2026-07-07T16:19:48.944770122Z Wrote inspect data to must-gather.
2026-07-07T16:19:48.979375213Z Gathering data for ns/openshift-network-console...
2026-07-07T16:19:49.024261464Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:49.028441854Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T16:19:49.245014356Z Wrote inspect data to must-gather.
2026-07-07T16:19:49.258758402Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:49.427325939Z Gathering data for ns/openshift-console-operator...
2026-07-07T16:19:49.462823313Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:49.466117315Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T16:19:49.521295406Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T16:19:49.523909761Z INFO: Fallback to identify the container image from release info
2026-07-07T16:19:49.637633977Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:49.792592025Z Gathering data for ns/openshift-console...
2026-07-07T16:19:49.823593556Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:19:50.246974427Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T16:19:50.310487809Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T16:19:50.386605311Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T16:19:50.500973052Z Done executing Istio gather script
2026-07-07T16:19:50.521357223Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T16:19:50.842777701Z Gathering data for ns/openshift-dns-operator...
2026-07-07T16:19:51.136831127Z Gathering data for ns/openshift-dns...
2026-07-07T16:19:51.539517590Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:19:51.643938970Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T16:19:51.691938148Z Gathering data for ns/openshift-image-registry...
2026-07-07T16:19:52.106200641Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T16:19:52.389391415Z Gathering data for ns/openshift-ingress...
2026-07-07T16:19:52.688202783Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T16:19:52.751808495Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T16:19:53.047340386Z Gathering data for ns/openshift-insights...
2026-07-07T16:19:53.853606426Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T16:19:54.802481782Z Gathering data for ns/openshift-config...
2026-07-07T16:19:54.950496457Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T16:19:55.083992832Z Gathering data for ns/openshift-config-managed...
2026-07-07T16:19:55.495752607Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T16:19:55.728477049Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T16:19:55.973287379Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T16:19:56.073338898Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T16:19:56.257185035Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T16:19:56.522820805Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T16:19:56.546309673Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:19:56.752264655Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T16:19:57.017250241Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T16:19:57.192102718Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T16:19:57.315361487Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T16:19:57.648030670Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T16:19:58.094311854Z Gathering data for ns/openshift-multus...
2026-07-07T16:19:58.295586660Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T16:19:58.510673045Z Collecting performance related data for node ip-10-0-134-97.ec2.internal
2026-07-07T16:19:58.517447614Z Collecting performance related data for node ip-10-0-135-144.ec2.internal
2026-07-07T16:19:58.524383713Z Collecting performance related data for node ip-10-0-139-180.ec2.internal
2026-07-07T16:19:59.566700059Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T16:20:00.778007218Z Gathering data for ns/openshift-host-network...
2026-07-07T16:20:01.142794784Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T16:20:01.554396317Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:20:01.660322749Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T16:20:02.061635189Z Gathering data for ns/openshift-network-operator...
2026-07-07T16:20:02.421641113Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T16:20:02.680750509Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T16:20:03.137659069Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T16:20:03.421492736Z Gathering data for ns/openshift-apiserver...
2026-07-07T16:20:03.805258737Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T16:20:04.058838757Z Gathering data for ns/openshift-controller-manager...
2026-07-07T16:20:04.317972815Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T16:20:04.931866809Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T16:20:05.190500402Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T16:20:05.492324862Z Gathering data for ns/openshift-service-ca...
2026-07-07T16:20:05.849811246Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T16:20:06.545452810Z Collecting kubelet logs for node ip-10-0-134-97.ec2.internal
2026-07-07T16:20:06.547618448Z Collecting kubelet logs for node ip-10-0-135-144.ec2.internal
2026-07-07T16:20:06.549613136Z Collecting kubelet logs for node ip-10-0-139-180.ec2.internal
2026-07-07T16:20:06.560878721Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:20:06.761654998Z Wrote inspect data to must-gather.
2026-07-07T16:20:06.761690995Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T16:20:06.761690995Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T16:20:06.849396467Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mrdb2 namespace
2026-07-07T16:20:06.854429948Z INFO: Node performance data collection complete.
2026-07-07T16:20:06.985032437Z error: the server doesn't have a resource type "clusters"
2026-07-07T16:20:07.117213542Z Caches written to disk