Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It only takes a minute to sign up.

Clustering - so do you mean an actual cluster - MariaDB Galera Cluster, or just normal master-slave replication, or something else? Or maybe you don't really care, maybe it's more the high availability you're after?
– dbdemonJan 21 at 14:11

we need a cluster for true HA, we've got a bunch of replicas atm but managing failures is high maintenance when you least need it.
– Paddy CarrollJan 21 at 14:44

High availability for 2 sites with ~30 ms latency between each: You can have a Galera cluster on each site, and have asynchronous master-slave or master-master replication between them. The latency depends on the link between the sites.

1-2 local copies at each site for Backups/reporting etc: Galera cluster has a full copy of the databases in each node.

Sub 30 minute site failover time: Failover can be automated with a database proxy such as MariaDB MaxScale or ProxySQL, or a more general proxy like haproxy.

Minimal application re-engineering: MariDB Galera cluster works much like a single MariaDB server instance from the application's point of view, but note that every table must have a primary key (a single or compound key), and that only the InnoDB storage engine is supported. Also note that unless you're only ever writing to a single node at a time (e.g. read-write splitting which can be configured e.g. with MaxScale), your application needs to take into account that transactions can fail both on individual statements within the transaction and after the commit - see Galera’s big gotcha for MySQL users for details.

Assume AWS or Vault KMS: I don't know.

You may want to look into a solution such as SeveralNines' ClusterControl which I believe can automate a lot of this.