While this works fine for different ARM arch levels, the only case
where this breaks down is for VFP and iWMMXt and such, since they are
optional and not a property of the arch level, so another solution
needs to be found for those.

You can handle this, to an extent, with the target triplet-
arm-linux-gnu is pretty generic, so it's better to have names like
armv5l-linux-gnu (little endian) armv5b-linux-gnu (big endian),
armv5l-softvfp-linux-gnu (little endian soft floating point). It can
get pretty long. That said, I don't see anywhere in the near future
that Fedora is going to support a multitude of arm variants. Big and
Little endian with an implicit baseline (v5?) is probably enough
distinction for a while to come.