Linux x86-64 system calls

Applications can communicate with the Linux kernel through system calls to perform tasks such as reading and writing to files.
You're going to learn how to perform system calls on behalf of any program using the command line interface of Proctal on the x86-64 architecture.

How system calls work

On x86-64 Linux, system calls are performed by running the syscall instruction. The instruction passes control to the kernel which figures out what to do by looking up the system call number in the rax register.
The kernel looks up arguments in the following registers, in the given order:

  • rdi

  • rsi

  • rdx

  • r10

  • r8

  • r9

When the kernel returns control back to the program, the rax register contains the return value.
The registers that are used as arguments, and the registers rcx and r11 are not guaranteed to hold the same values after the system call returns control back to the program.
You can find tables of system call numbers and their corresponding number of arguments on the internet.

Calling write

Here's how you need to set up the registers.

  • rax system call number

  • rdi file descriptor

  • rsi buffer address

  • rdx size of buffer

The system call number is 1 and the file descriptor for standard output is usually 1. All that is left is a buffer.
The allocate command lets you allocate some space in memory. You can use it to create a buffer.

$ proctal allocate --pid=12345 -rw 14

The buffer starts at address 89AF8C08. Using the write command you can fill the buffer with data.

$ proctal write --pid=12345 --address=89AF8C08 --type=text 'Hello, world!' $'\n'

Now you can make the call. Using the execute command run the following assembly code. It sets up the registers and then uses the syscall instruction to make the call.

proctal execute --pid=12345 <<EOD
	mov	rax, 1
	mov	rdi, 1
	mov	rsi, 0x89AF8C08
	mov	rdx, 14

You should now see the contents of the buffer in the output of the program.
All that is left to do is to deallocate the memory block that was used as a buffer.

$ proctal deallocate --pid=12345 89AF8C08