What determines which MediaTypeFormatter is used?

I've setup a default MediaTypeFormatter to be JSON. I'm wondering why the Accept header in an Http request doesn't have priority.
For example I am making the same request (Get or Post) to a Uri that looks like: http://localhost:1234/Email/test@example.net

Here are the different headers I'm trying:
1. (No Headers)

User-Agent: Fiddler

JSON is returned as expected.

2. Accept: text/xml

User-Agent: Fiddler
Accept: text/xml

XML is returned as expected

3. Providing an Accept header and Content-Type for the post (or even a get)

User-Agent: Fiddler
Accept: text/xml
Content-Type: text/json

JSON is returned even though I'm asking for text/xml back.

I might be making too many assumptions, or my logic is off. But shouldn't the Accept header determine the MediaTypeFormatter that is used. I'm sure that users won't be posting Json and wanting xml back, but shouldn't this work regardless?