The ORM isn't intended to be a complete abstraction to SQL; it's just a shortcut for the most common queries. Dropping to raw SQL (or .extra if you really need to) isn't a violation of the ORM concept; it's part of the ORM concept — at least, as understood in Django.

The docs talk about ​aggregating annotations, but not about re-annotating annotations. I'm not sure it's possible to support it while keeping the code complexity under control.