Modern Binary Exploitation: Format String Attacks, Heap Sprays, EMET Bypass & Clang Security Internals

Modern Binary Exploitation: Format Strings, Heap Sprays, Clang Hardening and EMET Bypassing

Modern exploit development is no longer limited to traditional stack overflows and basic shellcode injection. As operating systems evolved, exploit mitigation technologies dramatically changed the offensive security landscape.

Modern exploit developers must understand:

• Advanced format string exploitation
• Heap spraying
• Compiler-generated mitigations
• ASLR bypassing
• SafeStack internals
• EMET bypass techniques
• GRSEC and PAX hardening
• Memory corruption engineering
• Runtime exploit mitigations

Key Takeaway

Modern exploitation is fundamentally an arms race between offensive payload engineering and defensive memory hardening.

Table of Contents

• Format String Exploitation
• Heap Spray Attacks
• Clang vs GCC Security
• AddressSanitizer and SafeStack
• Compiler Hardening
• EMET Internals
• EMET Bypass Strategies
• GRSEC and PAX
• SELinux Hardening
• Exploit Mathematics
• Related Articles

Advanced Format String Exploitation

Format string vulnerabilities occur when attacker-controlled input is passed directly into formatting functions like:

• printf()
• fprintf()
• sprintf()
• snprintf()

without specifying a fixed format string.

Vulnerable Example

printf(user_input);

Instead of:

printf("%s", user_input);

Why This Becomes Dangerous

Format specifiers allow attackers to:

• Read stack memory
• Leak pointers
• Leak libc addresses
• Write arbitrary values
• Corrupt execution flow

Understanding Stack Parameters

Format strings can reference arbitrary stack parameters using:

\[ \%6\$s \]

This accesses:

\[ Parameter_6 \]

inside the stack frame.

Exploit Discovery Payload

AAAAAAAA %p %p %p %p %p %p %p

CLI Output Example

Expand Format String Leak Output

AAAAAAAA 0xffffd1a0 0x80484f0 0xf7e12420 0x41414141

When:

\[ 0x41414141 \]

appears, attackers know they reached their controlled input.

Exploit Insight

Format string vulnerabilities often become powerful ASLR bypass primitives because they leak randomized memory addresses.

Heap Spray Attacks

Heap spraying attempts to fill memory with attacker-controlled payloads.

The goal is to increase the probability that:

\[ EIP \rightarrow Payload \]

after memory corruption occurs.

Classic Spray Addresses

Address	Purpose
0x0a0a0a0a	Predictable spray marker
0x0c0c0c0c	Heap spray filler
0x20202020	Whitespace spray
0x14141414	Controlled payload zone

Heap sprays became extremely common in:

• Browser exploitation
• Flash exploitation
• Java exploitation
• PDF exploitation

Heap Spray Mathematics

If:

\[ N = Number\ of\ sprayed\ blocks \]

then probability increases:

\[ P(success) \propto N \]

because attackers increase memory coverage.

EMET Heap Spray Protection

EMET mitigated heap spraying by pre-allocating common spray addresses.

This blocks attacker-controlled allocations at:

• 0x0a0a0a0a
• 0x0c0c0c0c
• 0x20202020

Attackers responded with:

• Randomized spraying
• Partial sprays
• Address discovery
• Heap feng shui

Clang vs GCC Security

Modern exploitation increasingly depends on compiler behavior.

Clang and GCC generate different stack layouts, instruction ordering and mitigations.

Why This Matters

Exploit reliability depends heavily on:

• Stack alignment
• Saved register placement
• Function prologues
• Stack cleanup logic

Typical Function Prologue

push ebp
mov ebp, esp
sub esp, 0x8

Compilers generate different instruction patterns even for identical source code.

Compiler Security Flags

Flag	Purpose
-fPIE	Position Independent Executable
-fPIC	Position Independent Code
-fstack-protector	Stack canaries
-fsanitize=safe-stack	SafeStack protection
-fsanitize=cfi	Control Flow Integrity

AddressSanitizer and SafeStack

AddressSanitizer (ASAN) instruments binaries to detect:

• Use-after-free
• Out-of-bounds access
• Stack corruption
• Heap corruption

ASAN Compilation

clang -fsanitize=address vulnerable.c

SafeStack Concept

SafeStack separates:

• Safe variables
• Unsafe variables

into different memory regions.

This complicates traditional stack smashing attacks.

Interesting Exploitation Observation

SafeStack does not always terminate execution cleanly.

Example crash:

\[ EBP = 0x41414141 \]

This indicates attacker-controlled frame corruption.

GDB Register Example

Expand GDB Register State

eax 0x1
ebp 0x41414141
eip 0xb7db6ebf
esp 0xbfffe830

Offensive Insight

Not every mitigation fails safely. Some protections still expose attacker-controlled states useful for exploitation research.

Compiler Hardening Internals

Modern compilers increasingly inject defensive logic automatically.

Stack Canary Logic

\[ Canary_{expected} = Canary_{runtime} \]

If:

\[ Canary_{expected} \neq Canary_{runtime} \]

then:

\[ __stack\_chk\_fail() \]

terminates the process.

Typical Canary Validation

xor rax, QWORD PTR fs:0x28
je safe_return
call __stack_chk_fail

Forcing Compiler Protections

Organizations often enforce mitigations globally.

Example Secure Flags

CFLAGS="-fstack-protector-all -fPIE -Wl,-z,relro,-z,now"

Checking Compiler Defaults

gcc -dumpspecs

The Problem with Binary Distributions

Most Linux distributions ship precompiled binaries.

Security depends heavily on how those binaries were compiled.

Questions offensive researchers ask:

• Was PIE enabled?
• Was RELRO enabled?
• Was NX enforced?
• Was ASLR compiled properly?
• Were performance flags prioritized over security?

EMET Internals

Microsoft EMET introduced multiple advanced exploit mitigations.

Important EMET Features

Feature	Purpose
SEHOP	Protect exception handlers
DEP	Prevent code execution
ASLR	Randomize memory
EAF	Protect export tables
ROPGuard	Detect ROP chains
ASR	Attack surface reduction

Export Address Table Filtering

Traditional Windows shellcode walks:

• kernel32.dll
• ntdll.dll

to dynamically resolve APIs.

EMET monitors these operations.

Attackers therefore evolved:

• Indirect resolution
• ROP-based API lookup
• Hash-based imports
• Kernel-assisted lookups

EMET Bypass Strategies

No mitigation is perfect.

Attackers discovered:

• WOW64 gadget abuse
• Deep Hook bypasses
• ROP chain obfuscation
• Memory pivot tricks

WOW64 Advantage

WOW64 permits:

\[ 32bit \leftrightarrow 64bit \]

transitions.

This dramatically increases gadget availability.

Why This Matters

More gadgets means more flexible ROP chains and better mitigation bypass opportunities.

GRSEC and PAX

GRSEC and PAX introduced some of the strongest Linux hardening features ever developed.

Important Protections

• Kernel hardening
• Enhanced ASLR
• RAP protection
• Thread stack protection
• mmap restrictions
• mprotect restrictions

PAX Executable Space Protection

PAX prevents:

\[ Writable \cap Executable \]

memory mappings.

This breaks many classic shellcode techniques.

mprotect Restriction

Attackers frequently rely on:

\[ mprotect() \]

to create executable memory pages.

PAX heavily restricts this behavior.

KERNHEAP

KERNHEAP hardens Linux kernel heap allocators.

This reduces reliability of:

• Kernel UAF exploitation
• Heap corruption
• Kernel heap spraying

SELinux

SELinux introduces mandatory access controls into Linux.

It controls:

• Process execution
• File access
• Socket permissions
• Capability usage

SELinux Access Logic

\[ Subject \rightarrow Object \rightarrow Policy \]

Even successful exploitation may fail if SELinux blocks post-exploitation actions.

Exploit Development Mathematics

ASLR Entropy

\[ Entropy = \log_2(Randomized\ States) \]

Heap Spray Coverage

\[ Coverage = BlockSize \times SprayCount \]

Canary Validation

\[ Canary_{stack} \oplus Canary_{fs} \]

ROP Probability

\[ P(success) \propto GadgetDensity \]

Machine Learning and Modern Exploit Detection

Modern EDR systems increasingly use machine learning to detect:

• ROP execution patterns
• Stack pivots
• Heap anomalies
• Execution entropy
• Behavioral outliers

Important concepts:

• Entropy in Machine Learning
• Information Gain
• Precision vs Recall
• Bias Variance Tradeoff

Previous Parts of This Series

• Part 1 — Weaponizing Connect-Back Shellcode
• Part 2 — Advanced Shellcode and ROP Exploitation
• Part 4 - Kernel Exploitation and Privilege Escalation: SMEP, SMAP, KASLR and Advanced Linux Kernel Attacks

Related Security Articles

• Evolution of IDS and IPS
• Deep Packet Inspection Internals
• HTTP Tunneling Detection
• Modern Security Architectures
• Advanced Packet Inspection

Final Thoughts

Modern exploit development requires far more than simple memory corruption. Today's offensive security researchers must understand:

• Compiler internals
• Mitigation architectures
• Heap allocators
• Kernel hardening
• Runtime protections
• ROP chain engineering
• Detection systems

As defensive technologies improve, offensive techniques continue evolving through:

• Dynamic payloads
• ROP obfuscation
• Function hashing
• Heap manipulation
• Behavioral evasion

Understanding both attack methodology and defensive architecture remains essential for modern exploit development research.