I know that the yellow paper does not specify how transactions are to be ordered in a block and this is up to the miner to decide. But I am interested in how this is practically handled (I assume not calling some rand function to get ordering). This answer to a related question suggests that gas price plays a role. How does, e.g. geth order transactions during the mining process?

// This method first sorts the separates the list of transactions into individual
// sender accounts and sorts them by nonce. After the account nonce ordering is
// satisfied, the results are merged back together by price, always comparing only
// the head transaction from each account. This is done via a heap to keep it fast

Once sorted, the transactions are further screened in commitTransactions() to remove any with "low gas".

Parity

For Parity things are a little easier: there's a CLI option. This will, at least, allow ordering to be changed in some basic ways.

I think it's a bit more involved that just "x before y" - it's basically sorting by both at the same time. "SortByPriceAndNonce sorts the transactions by price in such a way that the nonce orderings within a single account are maintained."
– Richard HorrocksMar 28 '17 at 14:27

The "nonce height" is checked. I assume this guarantees nonces are in the right order for a given user.

Then the selected strategy is applied.

If all equal, they are ordered in the order they came in.

TL;DR

In the end, keep in mind this is open source software, and there is no rules on how transactions must be ordered. Each miner is free to send transactions in the order he wishes, so there is no way to guarantee the transaction order, but it looks like a high gas price and a reasonable gas limit should secure you a good spot in most cases.

I think Horrock's answer is a little hard to understand, and the "sorting" is a weird type of sorting that needs a bit more explanation.

Say you're the miner, and you have 100 transactions to sort through. Say 80 unique people sent 80 unique transactions. Then say 10 unique people each sent 2 transactions. That's 10 "first" transactions, and 10 "second" transactions, for those 10 people. 100 total.

The idea is, group the 80 unique transactions and the 10 "first" transactions, into a group G.

while gasLimitNotReached:
T = highestGasPrice(G)
G.remove(T)
if T was some Account."first":
G.append(Account."second")
Commit(T)

I think you can see how this extends into some accounts having three transactions or four transactions. The idea is that only let G have transactions from unique accounts. If the same account has many txs, then only consider the smallest nonce, regardless of the gas price of those larger nonces. Then, loop over G, poping in order of gas price. Everytime you get rid of a transaction, check if that account has any more transactions of higher nonce, and then add that tx back into G if it exists.