2026-04-16T22:42:50.350724733Z [disk usage checker] Started
2026-04-16T22:42:50.354121290Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:42:50.577198475Z Gathering data for ns/openshift-cluster-version...
2026-04-16T22:42:50.717624589Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:42:50.808871534Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:42:50.902214803Z Gathering data for ns/default...
2026-04-16T22:42:51.225681745Z Gathering data for ns/openshift...
2026-04-16T22:42:51.675406135Z Gathering data for ns/kube-system...
2026-04-16T22:42:52.326155494Z Gathering data for ns/openshift-etcd...
2026-04-16T22:42:52.664217688Z Wrote inspect data to must-gather.
2026-04-16T22:42:52.664252629Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:42:52.664252629Z     namespaces "assisted-installer" not found
2026-04-16T22:42:52.882919659Z Waiting on subprocesses to finish execution.
2026-04-16T22:42:52.892414570Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T22:42:52.895361909Z INFO: Gathering HAProxy config files
2026-04-16T22:42:52.896824914Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T22:42:52.904689171Z INFO: Collecting host service logs for crio
2026-04-16T22:42:52.905039428Z INFO: Collecting host service logs for kubelet
2026-04-16T22:42:52.905372898Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T22:42:52.905700547Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T22:42:52.906065769Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T22:42:52.906394569Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T22:42:52.906713098Z INFO: Collecting host service logs for NetworkManager
2026-04-16T22:42:52.907062580Z INFO: Collecting host service logs for openvswitch
2026-04-16T22:42:52.907396205Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T22:42:52.907753454Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T22:42:52.908103309Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T22:42:52.908510591Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T22:42:52.916028324Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T22:42:53.032875441Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T22:42:53.491592220Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T22:42:53.496096134Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T22:42:53.760755522Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T22:42:54.041461844Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-2qjf4
2026-04-16T22:42:54.041461844Z insights-runtime-extractor-6wx9g
2026-04-16T22:42:54.041461844Z insights-runtime-extractor-th4nh
2026-04-16T22:42:54.041461844Z insights-runtime-extractor-vdsxv
2026-04-16T22:42:54.111508898Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T22:42:54.180342845Z No resources found in openshift-etcd namespace.
2026-04-16T22:42:54.278321761Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T22:42:54.299165719Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T22:42:54.299935317Z No resources found
2026-04-16T22:42:54.320895956Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T22:42:54.323163244Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T22:42:54.468445647Z Wrote inspect data to must-gather.
2026-04-16T22:42:54.652235926Z ERROR: No running kube-apiserver pods found
2026-04-16T22:42:54.687255213Z INFO: Worker host service log collection to complete.
2026-04-16T22:42:54.716850781Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T22:42:54.716850781Z See 'oc rsync -h' for help and examples
2026-04-16T22:42:55.235834584Z INFO: 'previous-logs' folder not found on ip-10-0-133-16.ec2.internal, skipping...
2026-04-16T22:42:55.270212433Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T22:42:55.323588361Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T22:42:55.357339157Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T22:42:55.385799701Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:42:55.428214765Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:42:55.476120826Z Gathering data for ns/kuadrant-system...
2026-04-16T22:42:55.704932003Z Wrote inspect data to must-gather.
2026-04-16T22:42:55.754385829Z No resources found
2026-04-16T22:42:55.774600251Z Wrote inspect data to must-gather.
2026-04-16T22:42:55.793865779Z Wrote inspect data to must-gather.
2026-04-16T22:42:55.854191603Z Wrote inspect data to must-gather.
2026-04-16T22:42:55.864388250Z Wrote inspect data to must-gather.
2026-04-16T22:42:55.980643562Z INFO: HAProxy config collection complete.
2026-04-16T22:42:56.064045684Z INFO: 'previous-logs' folder not found on ip-10-0-133-72.ec2.internal, skipping...
2026-04-16T22:42:56.085206368Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T22:42:56.172483631Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T22:42:56.226976080Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:42:56.246545195Z Wrote inspect data to must-gather.
2026-04-16T22:42:56.434148236Z INFO: 'previous-logs' folder not found on ip-10-0-135-106.ec2.internal, skipping...
2026-04-16T22:42:56.434247798Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T22:42:56.434305717Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T22:42:56.485189885Z W0416 22:42:56.485147    1078 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T22:42:56.510407489Z Wrote inspect data to must-gather.
2026-04-16T22:42:56.565881504Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:42:56.657648880Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T22:42:56.716563690Z Wrote inspect data to must-gather.
2026-04-16T22:42:56.812403742Z Gathering data for ns/openshift-monitoring...
2026-04-16T22:42:56.865482665Z Wrote inspect data to must-gather.
2026-04-16T22:42:57.089915634Z Wrote inspect data to must-gather.
2026-04-16T22:42:57.097109509Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T22:42:57.175772209Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T22:42:57.436654330Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T22:42:57.438261554Z Wrote inspect data to must-gather.
2026-04-16T22:42:57.458829392Z INFO: INTERCONNECT MODE
2026-04-16T22:42:57.458859722Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T22:42:57.665628643Z INFO: Gathering OVN_Northbound from ovnkube-node-6bqmn...
2026-04-16T22:42:57.665910864Z INFO: Gathering OVN_Northbound from ovnkube-node-fwm7d...
2026-04-16T22:42:57.666165842Z INFO: Gathering OVN_Northbound from ovnkube-node-hm4t5...
2026-04-16T22:42:57.666416494Z INFO: Gathering OVN_Southbound from ovnkube-node-6bqmn...
2026-04-16T22:42:57.666648563Z INFO: Gathering OVN_Southbound from ovnkube-node-fwm7d...
2026-04-16T22:42:57.666996066Z INFO: Gathering OVN_Southbound from ovnkube-node-hm4t5...
2026-04-16T22:42:57.797911524Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T22:42:57.881176518Z Wrote inspect data to must-gather.
2026-04-16T22:42:58.032034679Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.036870956Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.050511116Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.108952996Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.120421317Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.148950386Z tar: Removing leading `/' from member names
2026-04-16T22:42:58.306936016Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T22:42:58.374485649Z Wrote inspect data to must-gather.
2026-04-16T22:42:58.745234384Z Wrote inspect data to must-gather.
2026-04-16T22:42:58.803684792Z INFO: Waiting for network log collection to complete ...
2026-04-16T22:42:58.803733518Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T22:42:58.804259994Z INFO: Copying ovnk databases complete.
2026-04-16T22:42:58.806475948Z 4.9M	must-gather/network_logs/ovnk_database_store
2026-04-16T22:42:58.809446796Z ovnk_database_store/
2026-04-16T22:42:58.809492958Z ovnk_database_store/ovnkube-node-6bqmn_nbdb
2026-04-16T22:42:58.824497203Z ovnk_database_store/ovnkube-node-fwm7d_nbdb
2026-04-16T22:42:58.833126115Z INFO: Getting status from alertmanager-main-0
2026-04-16T22:42:58.837921661Z ovnk_database_store/ovnkube-node-6bqmn_sbdb
2026-04-16T22:42:58.872958876Z ovnk_database_store/ovnkube-node-hm4t5_sbdb
2026-04-16T22:42:58.904071862Z ovnk_database_store/ovnkube-node-hm4t5_nbdb
2026-04-16T22:42:58.915004730Z ovnk_database_store/ovnkube-node-fwm7d_sbdb
2026-04-16T22:42:58.952149273Z INFO: Network log collection complete.
2026-04-16T22:42:59.053460119Z W0416 22:42:59.053408    1436 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T22:42:59.065187501Z Wrote inspect data to must-gather.
2026-04-16T22:42:59.314418841Z Gathering data for ns/openshift-network-console...
2026-04-16T22:42:59.439218640Z Wrote inspect data to must-gather.
2026-04-16T22:42:59.805009670Z Wrote inspect data to must-gather.
2026-04-16T22:42:59.866711647Z Gathering data for ns/openshift-console-operator...
2026-04-16T22:43:00.102042577Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T22:43:00.104238820Z INFO: Fallback to identify the container image from release info
2026-04-16T22:43:00.348140297Z Gathering data for ns/openshift-console...
2026-04-16T22:43:00.436929050Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:43:00.539441634Z Wrote inspect data to must-gather.
2026-04-16T22:43:00.650970007Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T22:43:00.791806632Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T22:43:00.872690733Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T22:43:00.932699281Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T22:43:01.251278966Z Gathering data for ns/openshift-dns-operator...
2026-04-16T22:43:01.577979404Z Gathering data for ns/openshift-dns...
2026-04-16T22:43:02.073796767Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T22:43:02.209775516Z Gathering data for ns/openshift-image-registry...
2026-04-16T22:43:02.686753716Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T22:43:03.053704459Z Gathering data for ns/openshift-ingress...
2026-04-16T22:43:03.214515042Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T22:43:03.628917324Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T22:43:04.197135942Z Gathering data for ns/openshift-insights...
2026-04-16T22:43:04.331084433Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T22:43:05.444028243Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:43:05.454613657Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T22:43:06.592139409Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T22:43:06.914898139Z Gathering data for ns/openshift-lws-operator...
2026-04-16T22:43:07.462836197Z Gathering data for ns/kserve...
2026-04-16T22:43:07.733022556Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T22:43:08.864012533Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T22:43:09.136799945Z Collecting performance related data for node ip-10-0-133-72.ec2.internal
2026-04-16T22:43:09.155296545Z Collecting performance related data for node ip-10-0-133-16.ec2.internal
2026-04-16T22:43:09.155408167Z Collecting performance related data for node ip-10-0-135-106.ec2.internal
2026-04-16T22:43:09.225733049Z Gathering data for ns/openshift-config...
2026-04-16T22:43:09.619961296Z Gathering data for ns/openshift-config-managed...
2026-04-16T22:43:10.193160260Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T22:43:10.451527348Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:43:10.621722851Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T22:43:10.958620450Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T22:43:11.268275876Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T22:43:11.657399735Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T22:43:12.005093620Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T22:43:12.394608816Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T22:43:12.755223859Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T22:43:13.090564042Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T22:43:13.655105293Z Gathering data for ns/openshift-multus...
2026-04-16T22:43:15.080040260Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T22:43:15.458468803Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:43:16.596273868Z Gathering data for ns/openshift-host-network...
2026-04-16T22:43:16.936453781Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T22:43:17.587462468Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T22:43:17.723403583Z Collecting kubelet logs for node ip-10-0-133-72.ec2.internal
2026-04-16T22:43:17.725155305Z Collecting kubelet logs for node ip-10-0-135-106.ec2.internal
2026-04-16T22:43:17.726857036Z Collecting kubelet logs for node ip-10-0-133-16.ec2.internal
2026-04-16T22:43:18.159641879Z Gathering data for ns/openshift-network-operator...
2026-04-16T22:43:18.161049570Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T22:43:18.165312800Z INFO: Node performance data collection complete.
2026-04-16T22:43:18.565267855Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T22:43:18.839815068Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T22:43:19.299970088Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T22:43:19.646684506Z Gathering data for ns/openshift-apiserver...
2026-04-16T22:43:20.062076073Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T22:43:20.364628795Z Gathering data for ns/openshift-controller-manager...
2026-04-16T22:43:20.463057490Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-16T22:43:20.681382803Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T22:43:21.317193947Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T22:43:21.639964795Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T22:43:21.956071959Z Gathering data for ns/openshift-service-ca...
2026-04-16T22:43:22.342693991Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T22:43:23.293736844Z Wrote inspect data to must-gather.
2026-04-16T22:43:23.293797440Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:43:23.293797440Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T22:43:23.431254366Z error: the server doesn't have a resource type "clusters"
2026-04-16T22:43:23.601388151Z Caches written to disk