Custom Shellcode Encoder – X+1 XOR 0xAA (Linux/x86)
Encoders are quite useful in cases where there are restricted characters in an application being exploited. Popular encoders can be found in Metasploit like shikata_ga_nai and many more. To demonstrate how encoders work, I've created a very basic encoder which adds 1 byte to each shellcode characters and the result gets XOR'd with 0xAA. The formula goes something like this:
(X + 1) xor 0xAA = Y, where X is a byte of the shellcode and Y is the encoded byte
Y in this case can be transformed back to X using the formula:
(Y xor 0xAA) - 1 = X, where Y is the encoded byte and X is the original shellcode byte
To do this, suppose we have an execve NASM program that runs /bin/sh:
global _start
global _start
section .text
_start:
xor eax, eax
push eax
push eax
push eax
push ...