Use EXPLAIN in front of your query to see what the database is actually doing.

How large in MB and rows are the two tables?

What is your sort_buffer_size and tmp_table_size server variables set to?
Due to the combination of left join and order by mysql will need to create a temporary table and sort it.

Why do you have "DISTINCT ih.* "?
My guess is that you probably have a primary key on that table which means that the distinct is useless.
Using DISTINCT out of habit when there is no need for it is sloppy programming and should be avoided.

The size of invoiceheader is around 3.6MB and invoiceresponsibleuser is just under 300KB.

My sort_buffer_size is set to 2 097 144 and the tmp_table_size is 33 554 432.

Why distinct is used i dont really know, i was hired to clean up the Java code in a project and realized that what was really needed was to clean up some of the database queries. Database not being my field of expertise i looked to this forum for help.