A Beginners’ Guide to x86-64 Instruction Encoding

The encoding of x86 and x86-64 instructions is well documented in Intel or AMD’s manuals. However, they are not quite easy for beginners to start with to learn encoding of the x86-64 instructions. In this post, I will give a list of useful manuals for understanding and studying the x86-64 instruction encoding, a brief introduction and an example to help you get started with the formats and encodings of the x86-64 instructions. For further more details, you may go on to read the reference documents listed using the technique shown in this post.

Before you move on to following parts, let’s make clear of one especially confusing part for beginners about the assembly syntax when you read various documents: there are AT&T syntax and Intel syntax. In Intel documents, it is usually in Intel syntax. With GNU tool chains on Linux, the default syntax used is usually the AT&T one.

The most significant different between these 2 syntaxes is that AT&T and Intel syntax use the opposite order for source and destination operands. Intel syntax uses "dest, source" while the AT&T syntax uses "source, dest". Note that instructions with more than one source operand, such as the enter instruction, do not have reversed order. For the representation of operands with the SIB or displacement, the formats are different. For example, the Intel syntax is [rdi+0xa] for 0xa(%rdi) in AT&T syntax. For a list of notable differences, please check AT&T Syntax versus Intel Syntax.

Reference documents for x86-64 instruction’s encoding

Here is a list of references and useful documents I will refer to in this post and you can further check later too to encode more instructions.

Tools and tips for finding out an x86-64 instruction’s encoding

To quickly find out the encoding of an instruction, you can use the GNU assembler as and the objdump tool together. For example, to find out the encoding of the instruction addq 10(%rdi), %r8, you can do it as follows.

There will be 2 operand parts for the RM encoding. The first part will be ModRM:reg(r,w) and the second part will be ModRM:r/m(r). "Figure 2-4. Memory Addressing Without an SIB Byte; REX.X Not Used" from Volume 2 shows the encoding for this case.

The REX.R and REX.B bits and the ModeRM byte will be decided accordingly. There are 3 parts in the ModRM byte: ‘mod’, ‘reg’ and ‘r/m’.

There is a table "Table 2-2. 32-Bit Addressing Forms with the ModR/M Byte" (it is for 32-bit operands. But from 2.2.1.1, "In 64-bit mode, these formats do not change. Bits needed to define fields in the 64-bit context are provided by the addition of REX prefixes" and hence the same value can be used) in Volume 2 which shows mapping of the operands combinations to the bits values of ‘mod’.

Although the table applies to 64-bit modes too, it does not show the additional registers like r8. Hence, we only use it to find out bits for ‘Mod’ only for the addq instruction we are encoding it. As 0xa can be encoded in a byte, we can use disp8 to keep the instruction encoding short. From the row of [EDI]+disp8 (actually, all disp8 ones share the same ‘Mod’ bits),

Mod = 01 (in bits)

For the encoding of the registers, I compiled a table for the general purpose 64-bit registers for your reference:

The ‘‘ in the ‘.Reg’ are usually a bit in the REX prefix, such as REX.B and REX.R, depending on specific instructions and operand combinations.

For the addq instruction in this case, r8 is 1.000 and rdi is 0.111. Hence, in bits, we get

reg = 000
r/m = 111
REX.B = 0 (from `rdi`)
REX.R = 1 (from `r8`)

Now, let’s put them together.

By putting the ‘WRXB’ bits ([BITS: 0100WRXB]) together, we get the REX prefix for this instruction is

0100 1100

Together with the 03 in REX.W+03/r from the reference for the ADD instruction, the opcode part, in hexadecimal, is

4c 03

By putting the mod, reg and r/m together, we get the ModRM byte (in bits)

01 000 111

which is, in hexadecimal,

47

Following the ModRM byte is the displacement is 0xa(10‘s hexadecimal representation) in one byte (disp8).

Putting all these together, we finally get the encoding of add r8,[rdi+0xa]:

4c 03 47 0a

In this example, to show the process, I have shown how to manually do an instruction’s encoding which is usually done by the assembler. You may use the same method to encode all other instruction by checking the reference documents for details of every instruction/operand combinations’ cases. Enjoy low level system programming!

Please share if you like this post:

Eric Z Ma

Eric is a father and systems guy. Eric is interested in building high-performance and scalable distributed systems and related technologies. The views or opinions expressed here are solely Eric's own and do not necessarily represent those of any third parties.