Our image handling and resizing functions should magically detect whether IM is available and usable (much like Gallery does) and use them where appropriate, as it's a lot more efficient than PHP's built-in GD functions.

This is probably the oldest to-do on my list too. Postponed it 2 years ago as ImageMagick was relatively rare. Perhaps it's time to look into this again, would also need to reevaluate which serves us better IM or GD (that are available in PHP 5.2.4+).

I would really like to see Imagemagick support added to 3.5, to complement our focus on the media, upload, and gallery user experience.

This ticket needs someone (or a few someones) to grab it by the horns and run with it. If your strong suit is API and backend development but want to contribute to the media/upload project in 3.5, this is definitely a great way to help.

I'm curious in how we should implement this. The implementation would be something like how it is done with the HTTP API.
I'm also curious if this will effect server load like in case of WordPress.com.

See also #18543 - wrapping our GD functions in methods that can allow for using stream wrappers (CDN integration, etc.).

Yes, I do picture an API that looks a lot like the HTTP API. (This could be done as an interface, though an abstract base class would probably make sense, in case there is any shared functionality.) We would support both GD and IM in core, with a preference toward IM, and replace all of our direct GD function calls with this new API.

So, what's the best way to inventory all the current usages of GD library functions in core? Just iterating through all the functions available on ​http://www.php.net/manual/en/ref.image.php seems like a pretty brute force way to go.

Woot! Awesome work. I'll get started on the NetPM and IM mappings to try to see whether we can design a versatile interface. Then it will just be a matter of creating some adaptors and then replacing all the function calls with method calls.