Advanced Malware Tradecraft and EDR Evasion: Reflective DLL Injection, AMSI Bypass and Direct Syscalls

Advanced Malware Tradecraft and EDR Evasion: Reflective Loaders, Syscall Unhooking and In-Memory Payload Engineering

Modern offensive security operations increasingly rely on stealth rather than raw exploitation alone. After initial compromise and privilege escalation, attackers must survive inside heavily monitored environments protected by:

• EDR platforms
• Behavioral analytics
• ETW monitoring
• AMSI scanning
• Kernel callbacks
• Userland API hooks
• Memory scanners

Traditional malware techniques now trigger immediate detection because modern security products inspect:

• API call patterns
• Memory permissions
• Thread creation
• RWX pages
• Import tables
• Syscall behavior

Key Takeaway

Modern malware tradecraft is fundamentally about minimizing observable indicators while maintaining execution reliability inside monitored systems.

Table of Contents

• In-Memory Malware
• Reflective DLL Injection
• Userland Hooking
• Direct Syscalls
• Syscall Unhooking
• AMSI Internals
• ETW Evasion
• APC Injection
• Process Hollowing
• Detection Mathematics
• Related Articles

Understanding In-Memory Malware

Modern offensive tooling increasingly avoids writing payloads to disk.

Disk artifacts are dangerous because:

• AV engines scan files
• EDR products hash executables
• Filesystem monitoring detects anomalies
• Forensic analysts recover deleted binaries

In-memory execution attempts to avoid these detections entirely.

Traditional Malware Flow

\[ Disk \rightarrow Execute \rightarrow Detect \]

Modern Fileless Flow

\[ Memory \rightarrow Execute \]

Why Fileless Malware Matters

Eliminating filesystem artifacts dramatically reduces forensic visibility and static signature detection opportunities.

Reflective DLL Injection

Reflective DLL Injection allows a DLL to load itself directly from memory without using:

• LoadLibrary()
• Standard Windows loaders

This bypasses many traditional monitoring points.

Reflective Loading Stages

1. Allocate memory
2. Copy PE sections
3. Resolve imports
4. Apply relocations
5. Call entry point

Memory Mapping Mathematics

\[ VirtualAddress = BaseAddress + SectionOffset \]

Reflective Loader Example

VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_EXECUTE_READWRITE)

CLI Output Example

Expand Reflective Loader Debug Output

[+] PE Header Parsed
[+] Import Table Resolved
[+] Relocations Applied
[+] Entry Point Executed

Userland API Hooking

EDR products frequently hook:

• ntdll.dll
• kernel32.dll
• kernelbase.dll

to monitor suspicious behavior.

Hooking Logic

\[ API \rightarrow Hook \rightarrow EDR \]

Hooked functions often include:

• VirtualAlloc()
• CreateRemoteThread()
• WriteProcessMemory()
• NtProtectVirtualMemory()

Typical Hook

jmp edr_monitor

This redirects execution into monitoring engines.

Direct Syscalls

Direct syscalls bypass userland API hooks by invoking kernel syscalls directly.

Traditional Execution

\[ Application \rightarrow API \rightarrow Hook \rightarrow Kernel \]

Direct Syscall Flow

\[ Application \rightarrow Syscall \rightarrow Kernel \]

64-bit Syscall Instruction

mov r10, rcx
mov eax, syscall_id
syscall
ret

Why Direct Syscalls Work

Most EDR products hook high-level APIs rather than raw syscall stubs.

Important Insight

Direct syscalls reduce visibility by bypassing userland monitoring layers entirely.

Syscall Unhooking

Advanced malware frequently restores clean syscall stubs from disk-backed ntdll.dll mappings.

Unhooking Strategy

• Map clean ntdll.dll
• Locate syscall stubs
• Restore original bytes
• Remove JMP hooks

Hook Detection Logic

\[ ExpectedBytes \neq CurrentBytes \]

indicates API tampering.

Typical Clean Syscall Stub

4c 8b d1
b8 18 00 00 00
0f 05
c3

AMSI Internals

Antimalware Scan Interface (AMSI) allows applications to submit scripts and memory buffers to security products.

PowerShell heavily integrates AMSI.

AMSI Flow

\[ PowerShell \rightarrow AMSI \rightarrow AV Engine \]

Why AMSI Became Important

Fileless PowerShell malware became extremely popular in offensive operations.

Common AMSI Targets

• AmsiScanBuffer()
• AmsiOpenSession()
• AmsiInitialize()

Patch Concept

\[ ScanResult = Clean \]

regardless of actual content.

Detection Reality

Modern EDR platforms increasingly monitor AMSI tampering itself rather than only the malicious payload.

ETW Evasion

Event Tracing for Windows (ETW) provides detailed telemetry for:

• Process creation
• Thread execution
• Image loads
• CLR execution
• PowerShell logging

ETW Visibility

\[ Application \rightarrow ETW \rightarrow EDR \]

ETW Providers

Provider	Purpose
Microsoft-Windows-Kernel-Process	Process telemetry
Microsoft-Windows-PowerShell	Script execution
CLR ETW	.NET execution

ETW Tampering Goal

\[ Telemetry \rightarrow Null \]

APC Injection

Asynchronous Procedure Calls allow code execution inside existing threads.

APC Injection Workflow

1. Open target process
2. Allocate memory
3. Write payload
4. Queue APC
5. Resume thread

APC Execution Logic

\[ Thread \rightarrow APCQueue \rightarrow Payload \]

Example API Chain

QueueUserAPC(shellcode, thread, NULL)

Process Hollowing

Process hollowing creates legitimate processes before replacing their memory with malicious payloads.

Classic Hollowing Flow

1. Create suspended process
2. Unmap legitimate image
3. Write malicious image
4. Patch thread context
5. Resume execution

Process Hollowing Equation

\[ LegitimateProcess + MaliciousMemory = StealthExecution \]

CLI Example

CreateProcess(..., CREATE_SUSPENDED)

Thread Stack Spoofing

Modern EDR products inspect thread call stacks for suspicious execution flow.

Attackers increasingly spoof:

• Return addresses
• Call stacks
• Thread origins

Call Stack Logic

\[ ReturnAddress \rightarrow TrustedModule \]

instead of attacker memory.

Why Stack Spoofing Matters

Behavioral detection engines heavily analyze execution provenance and suspicious call stack transitions.

Indirect Syscalls

Direct syscalls themselves increasingly trigger detections.

Attackers responded with:

• Indirect syscalls
• Heaven's Gate
• Syscall trampolines

Indirect Syscall Logic

\[ Jump \rightarrow CleanSyscallStub \]

instead of calling syscall instructions directly from attacker memory.

Sleep Obfuscation

EDR products scan sleeping malware memory regions.

Attackers increasingly:

• Encrypt payloads during sleep
• Unmap executable pages
• Use timer callbacks
• Perform memory re-encryption

Encrypted Sleep Concept

\[ Payload \rightarrow Encrypt \rightarrow Sleep \]

then:

\[ Wake \rightarrow Decrypt \rightarrow Execute \]

Detection and Evasion Mathematics

Entropy-Based Detection

\[ Entropy = -\sum p(x)\log_2 p(x) \]

High entropy regions frequently indicate:

• Encrypted payloads
• Packed malware
• Compressed shellcode

Behavioral Correlation

\[ SuspicionScore = APIAnomalies + MemoryAnomalies + ExecutionAnomalies \]

Memory Permission Logic

\[ RWX = HighRisk \]

because simultaneous writable and executable pages are suspicious.

Machine Learning and Modern EDR

Modern EDR systems increasingly rely on:

• Behavioral AI
• Execution graph analysis
• Memory entropy analysis
• Thread anomaly detection
• Graph neural networks

Important machine learning concepts:

• Entropy in Machine Learning
• Information Gain
• Precision vs Recall
• Bias Variance Tradeoff

Previous Parts of This Series

• Part 1 — Weaponizing Connect-Back Shellcode
• Part 2 — Advanced Shellcode and ROP Exploitation
• Part 3 — Modern Binary Exploitation
• Part 4 — Kernel Exploitation and Privilege Escalation
• Part 6 - Advanced Active Directory Exploitation: Kerberos Abuse, Golden Tickets, NTLM Relays and Enterprise Lateral Movement

Related Security Articles

• Evolution of IDS and IPS
• Deep Packet Inspection Internals
• HTTP Tunneling Detection
• Modern Security Architectures
• Advanced Packet Inspection

Final Thoughts

Modern offensive security increasingly focuses on stealth engineering rather than only exploitation itself.

Advanced operators must understand:

• Memory telemetry
• Behavioral analytics
• Syscall internals
• Windows kernel architecture
• Userland monitoring
• Execution graph analysis

As EDR platforms continue evolving, malware tradecraft increasingly depends on:

• Indirect syscalls
• Memory-only execution
• Reflective loaders
• Encrypted sleep cycles
• Behavioral evasion
• Stack spoofing

Understanding these techniques provides critical insight into how modern offensive tooling attempts to survive inside heavily monitored enterprise environments.