Custom Encoding For Shellcode

Tired of your payloads constantly getting detected? Tried MSFVenom and still have had no luck with making it past EDR? 

Then read on for to how to make your own encoder.


Before we get into the weeds, I have to credit the project that this was heavily based on by It can be found by following the hyperlink. I took it one step further by combining it into one Windows based script and showing applications as well as analyzing the detection's.

Now onto the fun:

The code for the project can be found here. One should have familiarity with Powershell and hex while working through this. Knowledge of assembly isn't required, but things will make more sense with it. should be credited for heavily for all of this, but especially for commenting the assembly file, it helps a lot with understand it.

The first step is to encode our hex payload. For this we will use Powershell. We run through the hex encoded string and XOR it, add to it, and then XOR it again, before returning the new encoded string.

Running this returns our new encoded string and the size of it. 

The size is needed for our assembly to properly know how many bytes to process.

With the size and newly encoded shellcode, we can then move on to our assembly file. 

This is a quick assembly file that loops through our encoding scheme in reverse. The math operations happen in reverse of what we use in our encoding scheme because we want to generate instructions for undoing it. This way, when we compile our code, we get back the encoded shellcode as well as the assembly operations for decoding it. We can run it through nasm nasm -f win64 .\decode.asm -o Decode and examine the output in CFF Explorer. This looks like below.

The highlighted section starts at EB 1E and, you might notice, ends at the last character of our payload, 3F 20. This blocks includes our decode scheme and our encoded payload. So when we load the payload into an application, it knows how to decode it as shown below. The encoded version is on the left and the decoded is on the right.

Applying what we've done:

With this shellcode, we can insert it into an application using something like Frampton

Frampton is a tool to insert shellcode into code caves within applications. Code caves are a whole different topic to get into but the long and short of it is that after compiling programs there's empty space left over. This space can be filled with our code that's either pointing to an external resource or by finding a cave large enough to fit our whole payload. Once Frampton finds and inserts into this code cave, it changes the execution point to start where our cave is filled, and works from there. 

It's not super elegant because we're not controlling where the code cave is, it makes file sections that are normally not executable now executable, and the application flow doesn't work as you might normally expect afterwards. Because the PE has a new entry point outside of the .text section, which is normally the only executable section of a file, AV has a higher chance of flagging it as malicious. 

Analyzing detection:

We've inserted our encoded shellcode into an application and made it executable. So what does the Virustotal detection look like? 

Just about all the vendors are detecting the invalid signature in PuTTY now since we've changed the file and the remaining vendors are detecting the unusual entry point. Compare this with the detection of the same payload encoded using Shikata Ga Nai.

We can see a big jump in detections as well as some vendors identifying the specific encoding scheme used.