2026-07-28T00:13:47.198073451Z [disk usage checker] Started
2026-07-28T00:13:47.201239567Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:13:47.400207402Z Gathering data for ns/openshift-cluster-version...
2026-07-28T00:13:47.538367927Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T00:13:47.620650084Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T00:13:47.699148947Z Gathering data for ns/default...
2026-07-28T00:13:47.986723343Z Gathering data for ns/openshift...
2026-07-28T00:13:48.375487233Z Gathering data for ns/kube-system...
2026-07-28T00:13:49.027028619Z Gathering data for ns/openshift-etcd...
2026-07-28T00:13:49.302153926Z Wrote inspect data to must-gather.
2026-07-28T00:13:49.302185443Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T00:13:49.302185443Z     namespaces "assisted-installer" not found
2026-07-28T00:13:49.669573267Z Waiting on subprocesses to finish execution.
2026-07-28T00:13:49.675146030Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T00:13:49.681383168Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T00:13:49.684557500Z Executing Istio gather script
2026-07-28T00:13:49.686303252Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T00:13:49.700498063Z INFO: Gathering HAProxy config files
2026-07-28T00:13:49.730393088Z INFO: Collecting host service logs for crio
2026-07-28T00:13:49.730848378Z INFO: Collecting host service logs for kubelet
2026-07-28T00:13:49.731157517Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T00:13:49.731444404Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T00:13:49.731741582Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T00:13:49.732015840Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T00:13:49.732289572Z INFO: Collecting host service logs for NetworkManager
2026-07-28T00:13:49.732611153Z INFO: Collecting host service logs for openvswitch
2026-07-28T00:13:49.732923394Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T00:13:49.733218603Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T00:13:49.733505040Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T00:13:49.733871161Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T00:13:49.783288702Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T00:13:50.049348009Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T00:13:50.049433195Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T00:13:50.103653908Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-kv888
2026-07-28T00:13:50.103825553Z insights-runtime-extractor-22q8s
2026-07-28T00:13:50.103851263Z insights-runtime-extractor-bh6wr
2026-07-28T00:13:50.103873337Z insights-runtime-extractor-cktm2
2026-07-28T00:13:50.144260599Z No resources found in openshift-etcd namespace.
2026-07-28T00:13:50.516998633Z Wrote inspect data to must-gather.
2026-07-28T00:13:50.525804374Z No resources found
2026-07-28T00:13:50.726310437Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T00:13:50.942872290Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T00:13:51.014629526Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T00:13:51.112278982Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T00:13:51.317795293Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T00:13:51.326141724Z INFO: Worker host service log collection to complete.
2026-07-28T00:13:51.443938068Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T00:13:51.451765107Z ERROR: No running kube-apiserver pods found
2026-07-28T00:13:51.471960995Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T00:13:51.471960995Z See 'oc rsync -h' for help and examples
2026-07-28T00:13:51.551803000Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T00:13:51.640340611Z Wrote inspect data to must-gather.
2026-07-28T00:13:51.901228635Z Wrote inspect data to must-gather.
2026-07-28T00:13:51.958330538Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T00:13:52.034511776Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T00:13:52.051098296Z INFO: 'previous-logs' folder not found on ip-10-0-132-108.ec2.internal, skipping...
2026-07-28T00:13:52.084193672Z tar: Removing leading `/' from member names
2026-07-28T00:13:52.092039063Z INFO: HAProxy config collection complete.
2026-07-28T00:13:52.148316025Z No resources found
2026-07-28T00:13:52.202756992Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T00:13:52.220941827Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:13:52.319142845Z Wrote inspect data to must-gather.
2026-07-28T00:13:52.343290169Z Wrote inspect data to must-gather.
2026-07-28T00:13:52.353895182Z Wrote inspect data to must-gather.
2026-07-28T00:13:52.410303949Z Gathering data for ns/openshift-monitoring...
2026-07-28T00:13:52.424156938Z Wrote inspect data to must-gather.
2026-07-28T00:13:52.432538095Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T00:13:52.565439701Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T00:13:52.587593405Z INFO: 'previous-logs' folder not found on ip-10-0-138-229.ec2.internal, skipping...
2026-07-28T00:13:52.686056914Z Wrote inspect data to must-gather.
2026-07-28T00:13:52.752806994Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T00:13:52.817872711Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T00:13:52.899640074Z INFO: 'previous-logs' folder not found on ip-10-0-140-17.ec2.internal, skipping...
2026-07-28T00:13:52.899709556Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T00:13:52.899722336Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T00:13:52.978973361Z Wrote inspect data to must-gather.
2026-07-28T00:13:53.006771673Z Wrote inspect data to must-gather.
2026-07-28T00:13:53.018766680Z Wrote inspect data to must-gather.
2026-07-28T00:13:53.177164869Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T00:13:53.224237018Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T00:13:53.245608620Z INFO: INTERCONNECT MODE
2026-07-28T00:13:53.245644994Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T00:13:53.389498019Z Wrote inspect data to must-gather.
2026-07-28T00:13:53.403032102Z INFO: Gathering OVN_Northbound from ovnkube-node-t4vmm...
2026-07-28T00:13:53.403232623Z INFO: Gathering OVN_Northbound from ovnkube-node-tvrbh...
2026-07-28T00:13:53.403402263Z INFO: Gathering OVN_Northbound from ovnkube-node-zkbp7...
2026-07-28T00:13:53.403713457Z INFO: Gathering OVN_Southbound from ovnkube-node-t4vmm...
2026-07-28T00:13:53.404056052Z INFO: Gathering OVN_Southbound from ovnkube-node-tvrbh...
2026-07-28T00:13:53.404294423Z INFO: Gathering OVN_Southbound from ovnkube-node-zkbp7...
2026-07-28T00:13:53.593518102Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T00:13:53.674949150Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.709324649Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.716588184Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.727583595Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.748385525Z Wrote inspect data to must-gather.
2026-07-28T00:13:53.757944742Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.762832159Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T00:13:53.767852202Z tar: Removing leading `/' from member names
2026-07-28T00:13:53.823137424Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:53.998340753Z Wrote inspect data to must-gather.
2026-07-28T00:13:54.009134046Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T00:13:54.035728002Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T00:13:54.056802576Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:54.059606222Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T00:13:54.222210194Z INFO: Waiting for network log collection to complete ...
2026-07-28T00:13:54.222250204Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T00:13:54.222593772Z INFO: Copying ovnk databases complete.
2026-07-28T00:13:54.224056305Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-28T00:13:54.226880779Z ovnk_database_store/
2026-07-28T00:13:54.226932365Z ovnk_database_store/ovnkube-node-tvrbh_nbdb
2026-07-28T00:13:54.228543284Z Wrote inspect data to must-gather.
2026-07-28T00:13:54.232607505Z ovnk_database_store/ovnkube-node-tvrbh_sbdb
2026-07-28T00:13:54.248459512Z ovnk_database_store/ovnkube-node-t4vmm_sbdb
2026-07-28T00:13:54.251907943Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T00:13:54.264454545Z ovnk_database_store/ovnkube-node-zkbp7_nbdb
2026-07-28T00:13:54.268840290Z ovnk_database_store/ovnkube-node-t4vmm_nbdb
2026-07-28T00:13:54.269941221Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:54.276815370Z ovnk_database_store/ovnkube-node-zkbp7_sbdb
2026-07-28T00:13:54.298744811Z INFO: Network log collection complete.
2026-07-28T00:13:54.447783483Z Wrote inspect data to must-gather.
2026-07-28T00:13:54.478457109Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:54.478765551Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T00:13:54.480635287Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T00:13:54.626441635Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T00:13:54.674307378Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:54.732193491Z Wrote inspect data to must-gather.
2026-07-28T00:13:54.798734405Z Gathering data for ns/openshift-network-console...
2026-07-28T00:13:54.839426745Z INFO: Getting status from alertmanager-main-0
2026-07-28T00:13:54.889027942Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:54.892084697Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T00:13:55.037487430Z Wrote inspect data to must-gather.
2026-07-28T00:13:55.116406739Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:55.240803528Z Gathering data for ns/openshift-console-operator...
2026-07-28T00:13:55.304136822Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T00:13:55.307943249Z INFO: Fallback to identify the container image from release info
2026-07-28T00:13:55.329114623Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:55.332803050Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T00:13:55.481775438Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:55.606933912Z Gathering data for ns/openshift-console...
2026-07-28T00:13:55.653437868Z Wrote inspect data to /must-gather/istio.
2026-07-28T00:13:55.950942092Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T00:13:56.078454379Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T00:13:56.153537172Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T00:13:56.213969606Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T00:13:56.319458749Z Done executing Istio gather script
2026-07-28T00:13:56.642404633Z Gathering data for ns/openshift-dns-operator...
2026-07-28T00:13:56.956573523Z Gathering data for ns/openshift-dns...
2026-07-28T00:13:57.226322522Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:13:57.317318255Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T00:13:57.518489677Z Gathering data for ns/openshift-image-registry...
2026-07-28T00:13:57.971081084Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T00:13:58.259934629Z Gathering data for ns/openshift-ingress...
2026-07-28T00:13:58.419210577Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T00:13:58.556549817Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T00:13:58.955055957Z Gathering data for ns/openshift-insights...
2026-07-28T00:13:59.540285526Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T00:14:00.638795085Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T00:14:00.860187250Z Gathering data for ns/openshift-config...
2026-07-28T00:14:01.152783783Z Gathering data for ns/openshift-config-managed...
2026-07-28T00:14:01.528770129Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T00:14:01.746255623Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T00:14:01.816851678Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T00:14:02.045760746Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T00:14:02.232493206Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:14:02.343993688Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T00:14:02.653479976Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T00:14:02.843403571Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T00:14:02.943295343Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T00:14:03.197345680Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T00:14:03.487802185Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T00:14:03.758548930Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T00:14:03.944655868Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T00:14:04.191424959Z Collecting performance related data for node ip-10-0-138-229.ec2.internal
2026-07-28T00:14:04.198950141Z Collecting performance related data for node ip-10-0-140-17.ec2.internal
2026-07-28T00:14:04.207639851Z Collecting performance related data for node ip-10-0-132-108.ec2.internal
2026-07-28T00:14:04.326777639Z Gathering data for ns/openshift-multus...
2026-07-28T00:14:05.823846448Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T00:14:07.051018954Z Gathering data for ns/openshift-host-network...
2026-07-28T00:14:07.237941322Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:14:07.344292197Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T00:14:07.838437488Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T00:14:08.216832444Z Gathering data for ns/openshift-network-operator...
2026-07-28T00:14:08.562896433Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T00:14:08.806905437Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T00:14:09.310310422Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T00:14:09.577708339Z Gathering data for ns/openshift-apiserver...
2026-07-28T00:14:09.943801097Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T00:14:10.180646957Z Gathering data for ns/openshift-controller-manager...
2026-07-28T00:14:10.429831866Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T00:14:11.079075952Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T00:14:11.339370918Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T00:14:11.647727278Z Gathering data for ns/openshift-service-ca...
2026-07-28T00:14:11.882141843Z Collecting kubelet logs for node ip-10-0-138-229.ec2.internal
2026-07-28T00:14:11.883951283Z Collecting kubelet logs for node ip-10-0-140-17.ec2.internal
2026-07-28T00:14:11.885709785Z Collecting kubelet logs for node ip-10-0-132-108.ec2.internal
2026-07-28T00:14:11.994498654Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T00:14:12.225407979Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-6gfpx namespace
2026-07-28T00:14:12.231757857Z INFO: Node performance data collection complete.
2026-07-28T00:14:12.243377543Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T00:14:12.899518303Z Wrote inspect data to must-gather.
2026-07-28T00:14:12.899559321Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T00:14:12.899559321Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T00:14:13.017158160Z error: the server doesn't have a resource type "clusters"
2026-07-28T00:14:13.146709715Z Caches written to disk