The x86 Countdown Encoder, Explained

In our last installment of the Shellcode Signature Series for security practitioners, we walked through the Metasploit shellcode encoder, Shikata Ga Nai. In this article, we’ll explore the x86 Countdown Encoder, which is not as complicated as Shikata Ga Nai, but incorporates several techniques that will be useful when analyzing the additional signatures in this series. There is no polymorphism or randomization when this shellcode is constructed, so it should result in a simple signature. 

The “core” logic of the x86 Countdown Encoder is hardcoded within a single function—decoder_stub.  At the beginning of the code, ECX is cleared and then loaded with the length of the shellcode minus 1.

A common technique used to obtain EIP is Call $ + 4:

The generated shellcode includes call instruction, E8 FF FF FF FF, which can be broken down into the opcode, ‘E8’. This call instruction and the operand, ‘FF FF FF FF (-1),’ calls the function that is located 1 byte before the next instruction.

The next instruction is 0x0000000B. That instruction minus 1 is 0x0000000A (shown below in bold). The call also pushes the address of the next instruction onto the stack for the return. Ultimately, the shellcode needs that return address so it can address the encoded buffer relative to it.

0x00000006 e8ffffffff             call 0xa
0x0000000b c15e304c            rcr  dword [esi + 0x30], 0x4c

When the call instruction is executed, our code will now look like this:

0x00000006 e8ffffffff             call 0xa
0x0000000a ffc1                   inc  ecx //Call lands here
0x0000000c 5e                     pop  esi // Get rtn addr from stack
0x0000000d 304c0e07               xor  byte [esi + ecx + 7], cl

The assembler keeps the full length of the first instruction, and 0xA is now recognized as a valid instruction. Our buffer length, contained in ECX, is incremented and EIP is loaded into ESI. Now we can start referencing locations to the XOR. The XOR operates on the entire buffer going backwards from end to start. The key starts as the length of shellcode and then decrements each iteration until it reaches 0.

0x00000011 e2fa                   loop 0xd

The loop completes when ECX is 0, then execution continues to the shellcode. A review of the encoder’s source code reveals that the following bytes of this core logic are directly hardcoded into the decoder:

"\xe8\xff\xff\xff" +  # call $+4       
"\xff\xc1" +          # inc ecx
"\x5e" +              # pop esi
"\x30\x4c\x0e\x07" +  # xor_loop: xor [esi + ecx + 0x07], cl
 "\xe2\xfa"            # loop xor_loop

Since there is no content dynamically generated by the encoder other than the length that is calculated and preloaded into ECX, this will result in a signature that closely resembles the source material.

rule x86_countdown
{
    meta:
        author = "Nick Hoffman / Jeremy Humble"
        description = "X86 Countdown Encoder"
    strings:
        $op_decode = {e8 ff ff ff
                      ff c1
                      5e
                      30 4c 0e 07
                      e2 fa}
        $ascii_decode = /e8ffffffffc15e304c0e07e2fa/ nocase
    condition:
        any of them
}

Rather than using a static single-byte XOR, this encoder adds a small twist by including a decrement operation; however, because these bytes are hardcoded into the shellcode generator, they are not going to change unless the author manually patches the shellcode, which is trivial, but rarely done.  

Dependence on the LOOP operation and preloading ECX with the shellcode length allows the author to do three things in an efficient way:

1. XOR value is changed at each step
2. Index is tracked
3. Loop can exit with no checking or compares

To make writing signatures more difficult, junk instructions of varying size and length could be included between the steps. This method would force the detection tool to employ something like emulation or use large variable jumps in the Yara pattern between instructions, which will increase the signature’s false positive rate. Using registers other than ECX and performing a more traditional loop would make the payload bigger and affect other registers; however, it would make it slightly more difficult to write detection signatures.

When looking at the Yara signature provided above, a keen observer will notice that peppering in a single NOP instruction after the POP would make the signature fail. To account for this, defenders would need to insert padding in between instructions, allowing for intermediate instructions. A large consequence of wildcarding is the potential of triggering on false positives.

Check back for our next article in the Shellcode Signature Series for additional insights on the evolving threat landscape.