2026-05-28T07:47:14.023891310Z [disk usage checker] Started
2026-05-28T07:47:14.027780117Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-28T07:47:14.217136540Z Gathering data for ns/openshift-cluster-version...
2026-05-28T07:47:14.353657182Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T07:47:14.429114588Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T07:47:14.502768596Z Gathering data for ns/default...
2026-05-28T07:47:14.762682416Z Gathering data for ns/openshift...
2026-05-28T07:47:15.140280059Z Gathering data for ns/kube-system...
2026-05-28T07:47:15.808946462Z Gathering data for ns/openshift-etcd...
2026-05-28T07:47:16.094890477Z Wrote inspect data to must-gather.
2026-05-28T07:47:16.094920307Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T07:47:16.094920307Z     namespaces "assisted-installer" not found
2026-05-28T07:47:16.193498303Z Waiting on subprocesses to finish execution.
2026-05-28T07:47:16.201999458Z INFO: Gathering HAProxy config files
2026-05-28T07:47:16.204370361Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T07:47:16.205305668Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T07:47:16.205345496Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T07:47:16.215108089Z INFO: Collecting host service logs for crio
2026-05-28T07:47:16.215417405Z INFO: Collecting host service logs for kubelet
2026-05-28T07:47:16.215775199Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T07:47:16.216060103Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T07:47:16.216447629Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T07:47:16.216675349Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T07:47:16.216958374Z INFO: Collecting host service logs for NetworkManager
2026-05-28T07:47:16.217253933Z INFO: Collecting host service logs for openvswitch
2026-05-28T07:47:16.217787880Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T07:47:16.218087095Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T07:47:16.218419544Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T07:47:16.218865275Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T07:47:16.255030042Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T07:47:16.721770111Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T07:47:16.841995482Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T07:47:16.993430250Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T07:47:17.040449900Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T07:47:17.156804788Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T07:47:17.181519079Z Wrote inspect data to must-gather.
2026-05-28T07:47:17.198665336Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T07:47:17.198691758Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T07:47:17.254624184Z No resources found in openshift-etcd namespace.
2026-05-28T07:47:17.334975761Z No resources found
2026-05-28T07:47:17.373024788Z INFO: Worker host service log collection to complete.
2026-05-28T07:47:17.373756823Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-qlwjx
2026-05-28T07:47:17.373797274Z insights-runtime-extractor-25xlb
2026-05-28T07:47:17.373818430Z insights-runtime-extractor-c489f
2026-05-28T07:47:17.373837866Z insights-runtime-extractor-fs5hf
2026-05-28T07:47:17.448548940Z ERROR: No running kube-apiserver pods found
2026-05-28T07:47:17.525759218Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T07:47:17.593499910Z Wrote inspect data to must-gather.
2026-05-28T07:47:17.874660544Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T07:47:17.874660544Z See 'oc rsync -h' for help and examples
2026-05-28T07:47:17.884213585Z Wrote inspect data to must-gather.
2026-05-28T07:47:17.929581218Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T07:47:17.948875158Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T07:47:18.095503352Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T07:47:18.115646613Z Gathering data for ns/openshift-monitoring...
2026-05-28T07:47:18.116171413Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.198141382Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.229933389Z INFO: 'previous-logs' folder not found on ip-10-0-129-23.ec2.internal, skipping...
2026-05-28T07:47:18.270241153Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T07:47:18.288791435Z No resources found
2026-05-28T07:47:18.357948759Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T07:47:18.373163661Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.401811914Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.441048842Z tar: Removing leading `/' from member names
2026-05-28T07:47:18.447286229Z INFO: HAProxy config collection complete.
2026-05-28T07:47:18.569756712Z INFO: 'previous-logs' folder not found on ip-10-0-130-201.ec2.internal, skipping...
2026-05-28T07:47:18.587315699Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.611223093Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.677743592Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T07:47:18.814120115Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.821479982Z INFO: 'previous-logs' folder not found on ip-10-0-130-4.ec2.internal, skipping...
2026-05-28T07:47:18.821664239Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T07:47:18.821664239Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T07:47:18.859956876Z Wrote inspect data to must-gather.
2026-05-28T07:47:18.954790793Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T07:47:19.035853305Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-28T07:47:19.060244232Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T07:47:19.093727540Z Wrote inspect data to must-gather.
2026-05-28T07:47:19.153432888Z Wrote inspect data to must-gather.
2026-05-28T07:47:19.337569164Z Wrote inspect data to must-gather.
2026-05-28T07:47:19.359283832Z INFO: INTERCONNECT MODE
2026-05-28T07:47:19.359320314Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T07:47:19.443299656Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T07:47:19.548734087Z Wrote inspect data to must-gather.
2026-05-28T07:47:19.555530287Z INFO: Gathering OVN_Northbound from ovnkube-node-2xgnb...
2026-05-28T07:47:19.555817250Z INFO: Gathering OVN_Northbound from ovnkube-node-llfn6...
2026-05-28T07:47:19.556160621Z INFO: Gathering OVN_Northbound from ovnkube-node-sgxm2...
2026-05-28T07:47:19.556452167Z INFO: Gathering OVN_Southbound from ovnkube-node-2xgnb...
2026-05-28T07:47:19.556881679Z INFO: Gathering OVN_Southbound from ovnkube-node-llfn6...
2026-05-28T07:47:19.557198654Z INFO: Gathering OVN_Southbound from ovnkube-node-sgxm2...
2026-05-28T07:47:19.630540903Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T07:47:19.713934680Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T07:47:19.831175297Z tar: Removing leading `/' from member names
2026-05-28T07:47:19.847947607Z tar: Removing leading `/' from member names
2026-05-28T07:47:19.878534634Z Wrote inspect data to must-gather.
2026-05-28T07:47:19.914414851Z tar: Removing leading `/' from member names
2026-05-28T07:47:19.914492034Z tar: Removing leading `/' from member names
2026-05-28T07:47:19.927064449Z tar: tar: Removing leading `/' from member names
2026-05-28T07:47:19.927064449Z Removing leading `/' from member names
2026-05-28T07:47:20.010576476Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T07:47:20.096088463Z Wrote inspect data to must-gather.
2026-05-28T07:47:20.258772911Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T07:47:20.294285462Z W0528 07:47:20.294237    1324 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T07:47:20.304292029Z Wrote inspect data to must-gather.
2026-05-28T07:47:20.315150789Z INFO: Waiting for network log collection to complete ...
2026-05-28T07:47:20.315183814Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T07:47:20.315614437Z INFO: Copying ovnk databases complete.
2026-05-28T07:47:20.317738384Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-05-28T07:47:20.320593998Z ovnk_database_store/
2026-05-28T07:47:20.320593998Z ovnk_database_store/ovnkube-node-llfn6_nbdb
2026-05-28T07:47:20.324198382Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T07:47:20.358614272Z ovnk_database_store/ovnkube-node-llfn6_sbdb
2026-05-28T07:47:20.380416568Z ovnk_database_store/ovnkube-node-sgxm2_nbdb
2026-05-28T07:47:20.390146711Z ovnk_database_store/ovnkube-node-2xgnb_nbdb
2026-05-28T07:47:20.402699813Z ovnk_database_store/ovnkube-node-2xgnb_sbdb
2026-05-28T07:47:20.427674169Z ovnk_database_store/ovnkube-node-sgxm2_sbdb
2026-05-28T07:47:20.450273481Z INFO: Network log collection complete.
2026-05-28T07:47:20.487635815Z Gathering data for ns/openshift-network-console...
2026-05-28T07:47:20.598070809Z Wrote inspect data to must-gather.
2026-05-28T07:47:20.621701398Z INFO: Getting status from alertmanager-main-0
2026-05-28T07:47:20.832295302Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T07:47:20.834889423Z INFO: Fallback to identify the container image from release info
2026-05-28T07:47:20.891692215Z Gathering data for ns/openshift-console-operator...
2026-05-28T07:47:21.209414441Z Gathering data for ns/openshift-console...
2026-05-28T07:47:21.372696138Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T07:47:21.474069139Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T07:47:21.599488609Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T07:47:21.660898795Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T07:47:21.998197269Z Gathering data for ns/openshift-dns-operator...
2026-05-28T07:47:22.303162649Z Gathering data for ns/openshift-dns...
2026-05-28T07:47:22.713724308Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T07:47:22.818543990Z Gathering data for ns/openshift-image-registry...
2026-05-28T07:47:23.219375617Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T07:47:23.551885376Z Gathering data for ns/openshift-ingress...
2026-05-28T07:47:23.835986860Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T07:47:23.843569179Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T07:47:24.040763201Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-28T07:47:24.193363739Z Gathering data for ns/openshift-insights...
2026-05-28T07:47:24.941133549Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T07:47:25.992006351Z Gathering data for ns/openshift-config...
2026-05-28T07:47:26.061747790Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T07:47:26.225402033Z Gathering data for ns/openshift-config-managed...
2026-05-28T07:47:26.592256132Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T07:47:26.870173920Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T07:47:27.093020849Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T07:47:27.179910560Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T07:47:27.339567252Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T07:47:27.574227647Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T07:47:27.790492426Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T07:47:28.032946949Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T07:47:28.295859133Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T07:47:28.323236135Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T07:47:28.576440257Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T07:47:28.996865444Z Gathering data for ns/openshift-multus...
2026-05-28T07:47:29.045409310Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-28T07:47:29.392753342Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T07:47:29.613916374Z Collecting performance related data for node ip-10-0-129-23.ec2.internal
2026-05-28T07:47:29.621058734Z Collecting performance related data for node ip-10-0-130-4.ec2.internal
2026-05-28T07:47:29.629118491Z Collecting performance related data for node ip-10-0-130-201.ec2.internal
2026-05-28T07:47:30.469431611Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T07:47:32.118872861Z Gathering data for ns/openshift-host-network...
2026-05-28T07:47:32.438928165Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T07:47:32.901510667Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T07:47:33.256989122Z Gathering data for ns/openshift-network-operator...
2026-05-28T07:47:33.619398823Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T07:47:33.907602184Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T07:47:34.050382423Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-05-28T07:47:34.344226010Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T07:47:34.610076707Z Gathering data for ns/openshift-apiserver...
2026-05-28T07:47:35.002426399Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T07:47:35.278371609Z Gathering data for ns/openshift-controller-manager...
2026-05-28T07:47:35.506594439Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T07:47:36.102637733Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T07:47:36.358596325Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T07:47:36.631956846Z Gathering data for ns/openshift-service-ca...
2026-05-28T07:47:36.956053692Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T07:47:37.211034196Z Collecting kubelet logs for node ip-10-0-129-23.ec2.internal
2026-05-28T07:47:37.213391661Z Collecting kubelet logs for node ip-10-0-130-4.ec2.internal
2026-05-28T07:47:37.215756334Z Collecting kubelet logs for node ip-10-0-130-201.ec2.internal
2026-05-28T07:47:37.556704311Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T07:47:37.562430961Z INFO: Node performance data collection complete.
2026-05-28T07:47:37.841356600Z Wrote inspect data to must-gather.
2026-05-28T07:47:37.841394078Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T07:47:37.841394078Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T07:47:37.965264375Z error: the server doesn't have a resource type "clusters"
2026-05-28T07:47:38.087736404Z Caches written to disk