Dynamic Analysis with x64dbg - Complete Guide

x64dbg is a powerful, open-source debugger for Windows that has become the go-to tool for malware analysts and reverse engineers. This comprehensive guide covers everything from basic setup to advanced analysis techniques used in real-world malware investigations.

Table of Contents

• Introduction to Dynamic Analysis
• Setting Up Your Analysis Environment
• Understanding the x64dbg Interface
• Basic Analysis Techniques
• Advanced Analysis Techniques
• Practical Malware Analysis Examples
• Detecting and Bypassing Anti-Analysis
• Best Practices and Tips

Introduction to Dynamic Analysis

Dynamic analysis involves executing malware in a controlled environment to observe its behavior. Unlike static analysis, which examines code without execution, dynamic analysis reveals runtime behavior, API calls, network communications, and system modifications.

Why x64dbg?

• Free and Open Source - No licensing costs, actively maintained community
• Both 32-bit and 64-bit Support - x32dbg and x64dbg variants
• Powerful Plugin Ecosystem - Extensive plugin support for specialized tasks
• Modern Interface - Intuitive GUI with customizable layouts
• Advanced Features - Memory search, pattern scanning, scripting support

Dynamic vs. Static Analysis

Dynamic Analysis	Static Analysis
Observes actual behavior	Examines code structure
Reveals runtime secrets	Faster initial assessment
Detects packed/obfuscated code	No execution required
Requires safe environment	Safe to perform

Setting Up Your Analysis Environment

Virtual Machine Requirements

Critical: Always perform dynamic analysis in an isolated virtual machine. Never run malware on your host system.

Recommended VM Configuration:

• OS: Windows 10 or Windows 11 (latest updates)
• RAM: 4GB minimum, 8GB recommended
• Storage: 50GB minimum for tools and samples
• Network: Isolated or monitored network segment
• Snapshots: Clean baseline snapshot before analysis

Installing x64dbg

1. Download from official repository: https://x64dbg.com/
2. Extract to a dedicated tools directory (e.g., C:\Tools\x64dbg)
3. Run as Administrator for full system access
4. Configure plugins and themes as needed

Essential Plugins

• ScyllaHide - Anti-debugging protection bypass
• xAnalyzer - Automatic analysis and API documentation
• APIInfo - Enhanced API call information
• Multiline Ultimate Assembler - Advanced assembly editing
• SwissArmyKnife - Collection of useful analysis tools

Additional Tools Setup

# Process Monitor (ProcMon) - File/Registry monitoring
# Wireshark - Network traffic analysis  
# Process Explorer - Advanced process monitoring
# RegShot - Registry comparison
# Fakenet-NG - Network simulation
# API Monitor - API call monitoring

Understanding the x64dbg Interface

Main Windows Overview

1. CPU Window (Main Disassembly)

The primary interface showing disassembled code, registers, memory dump, and stack.

Disassembly Pane:

Address   | Bytes        | Disassembly           | Comments
00401000  | 55           | push ebp              | Function prologue
00401001  | 8BEC         | mov ebp, esp          | Set up stack frame
00401003  | 83EC 0C      | sub esp, 0Ch          | Allocate local space
00401006  | 6A 00        | push 0                | Push parameter

Registers Pane:

EAX = 00000000    ESI = 7FFE0030    GS = 0000
EBX = 00000000    EDI = 00000000    SS = 0023
ECX = 00000000    EBP = 0019FFA0    DS = 0023
EDX = 7FFE0304    ESP = 0019FF94    ES = 0023

2. Memory Window

Displays raw memory contents in hex and ASCII format:

Address   | 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F | ASCII
00401000  | 55 8B EC 83 EC 0C 6A 00 68 B0 21 40 00 E8 C4 00 | U.....j.h.!@....
00401010  | 00 00 83 C4 08 6A 00 6A 00 6A 00 68 C0 21 40 00 | .....j.j.j.h.!@.

3. Stack Window

Shows the current stack contents and function parameters:

Address   | Value        | Comments
0019FF94  | 00401018     | Return address
0019FF98  | 00000000     | Parameter 1
0019FF9C  | 7FFE0030     | Parameter 2

Navigation and Control

Essential Keyboard Shortcuts:

F2           - Set/Remove breakpoint
F7           - Step into
F8           - Step over  
F9           - Run/Continue
Ctrl+F9      - Run until return
Ctrl+G       - Go to address
Ctrl+F       - Search in disassembly
Ctrl+B       - Binary search
Ctrl+L       - Go to line
Space        - Assemble instruction

Basic Analysis Techniques

1. Loading and Initial Reconnaissance

Loading a Sample:

1. File → Open → Select your malware sample
2. Choose the appropriate debugger (x32dbg for 32-bit, x64dbg for 64-bit)
3. The debugger pauses at the entry point
4. Examine the initial state before execution

Initial Assessment Commands:

# Check if sample is packed
- Look for unusual section names (UPX0, UPX1, .packed, etc.)
- Check entropy of sections
- Examine import table
# Useful x64dbg commands:
!info                    # Basic file information
!pe                     # PE header analysis  
!entropy                # Section entropy analysis
!imports                # Import table
!exports                # Export table

2. Setting Strategic Breakpoints

Common API Breakpoints for Malware Analysis:

# File Operations
bp CreateFileA
bp CreateFileW
bp WriteFile
bp ReadFile
bp DeleteFileA
bp MoveFileA
# Registry Operations  
bp RegOpenKeyA
bp RegOpenKeyW
bp RegSetValueA
bp RegCreateKeyA
bp RegDeleteKeyA
# Network Operations
bp WSAStartup
bp socket
bp connect
bp send
bp recv
bp InternetOpenA
bp InternetConnectA
bp HttpOpenRequestA
# Process/Thread Operations
bp CreateProcessA
bp CreateProcessW
bp CreateThread
bp OpenProcess
bp VirtualAlloc
bp VirtualProtect
# Crypto Operations
bp CryptAcquireContextA
bp CryptCreateHash
bp CryptEncrypt
bp CryptDecrypt

Setting Conditional Breakpoints:

# Break only when specific conditions are met
bp CreateFileA, "arg(1) == \"important.txt\""
bp RegSetValueA, "arg(2) == \"Software\\\\Microsoft\\\\Windows\\\\CurrentVersion\\\\Run\""
bp VirtualAlloc, "arg(2) > 0x10000"  # Large allocations

3. API Call Monitoring

Using API Logger Plugin:

The API Logger plugin automatically logs all API calls with parameters and return values:

CreateFileA("C:\temp\malware.exe", GENERIC_WRITE, FILE_SHARE_READ, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL) = 0x74
WriteFile(0x74, "MZ\x90\x00\x03...", 1024, &bytesWritten, NULL) = TRUE
RegOpenKeyA(HKEY_LOCAL_MACHINE, "SOFTWARE\Microsoft\Windows\CurrentVersion\Run", &hKey) = ERROR_SUCCESS
RegSetValueA(hKey, "WindowsUpdate", "C:\temp\malware.exe", 21) = ERROR_SUCCESS

4. Memory Analysis Techniques

Monitoring Memory Allocations:

# Track VirtualAlloc calls for shellcode injection
bp VirtualAlloc
# When hit, examine:
- Size of allocation (arg 2)
- Protection flags (arg 4) - look for PAGE_EXECUTE_READWRITE
- Return address for allocated memory

Memory Search Operations:

# Search for strings in memory
Ctrl+B → Search for "cmd.exe", "powershell", IP addresses
# Search for patterns
Ctrl+B → Search for MZ header: 4D 5A
         Search for PE signature: 50 45 00 00
         Search for shellcode patterns: 90 90 90 90 (NOP sleds)

Dumping Memory Regions:

# Right-click in memory window → Dump to file
# Useful for extracting:
- Unpacked malware
- Injected shellcode  
- Decrypted configurations
- Dropped payloads

Advanced Analysis Techniques

1. Unpacking Packed Malware

Generic Unpacking Procedure:

1. Identify Packer: Use tools like PEiD or Detect It Easy
2. Find OEP (Original Entry Point): Look for characteristic patterns
3. Set Breakpoints on Unpacking APIs:

# Common unpacking breakpoints
bp VirtualAlloc           # Memory allocation for unpacked code
bp VirtualProtect         # Changing memory permissions
bp WriteProcessMemory     # Writing unpacked code
bp SetThreadContext       # Redirecting execution

ESP Trick for Finding OEP:

1. Run until first VirtualAlloc or VirtualProtect
2. Note ESP value
3. Set hardware breakpoint on ESP (memory access)
4. Continue execution
5. When hit, check if you're at the OEP
6. Dump memory region

2. Analyzing Shellcode

Identifying Shellcode Characteristics:

# Common shellcode patterns:
90 90 90 90              # NOP sled
31 C0                    # XOR EAX, EAX (common instruction)
64 8B 30                 # MOV ESI, DWORD PTR FS:[EAX] (PEB access)
8B 76 0C                 # MOV ESI, DWORD PTR DS:[ESI+C] (LDR access)

Shellcode Analysis Workflow:

1. Locate shellcode in memory
2. Set breakpoint at shellcode start
3. Step through execution
4. Identify API resolution techniques:
   • PEB walking
   • Hash-based API resolution
   • Direct API addresses
5. Document payload functionality

3. Process Injection Analysis

Common Injection Techniques to Monitor:

DLL Injection:

bp OpenProcess           # Target process opening
bp VirtualAllocEx        # Memory allocation in target
bp WriteProcessMemory    # Writing DLL path
bp CreateRemoteThread    # Executing LoadLibrary

Process Hollowing:

bp CreateProcessA        # Creating suspended process
bp ZwUnmapViewOfSection  # Unmapping original image
bp VirtualAllocEx        # Allocating new memory
bp WriteProcessMemory    # Writing malicious image
bp SetThreadContext      # Redirecting execution
bp ResumeThread          # Starting malicious process

APC Injection:

bp QueueUserAPC          # Queuing APC
bp VirtualAllocEx        # Memory allocation
bp WriteProcessMemory    # Writing shellcode

4. Anti-Analysis Detection and Bypass

Common Anti-Debugging Checks:

IsDebuggerPresent API:

bp IsDebuggerPresent
# When hit, modify return value:
- Set EAX to 0 (false)
- Or patch the function to always return 0

PEB BeingDebugged Flag:

# Manual check and patch:
1. Go to FS:[30] (PEB address)
2. Offset +0x02 is BeingDebugged flag
3. Set to 0 to bypass check

NtGlobalFlag Check:

# PEB+0x68 contains NtGlobalFlag
# Normal value: 0x0
# Debugger value: 0x70
# Patch to 0x0 to bypass

Using ScyllaHide Plugin:

ScyllaHide automatically bypasses many common anti-debugging techniques:

• PEB flags modification
• API hooks for debugging detection
• Exception-based detection bypass
• Timing attack mitigation

Practical Malware Analysis Examples

Example 1: Analyzing a Simple Dropper

Scenario:

A suspicious executable that appears to drop additional payloads.

Analysis Steps:

1. Load sample in x64dbg
2. Set file operation breakpoints:

   bp CreateFileA
   bp CreateFileW  
   bp WriteFile

3. Run and observe file creation
4. When CreateFile hits:
   • Check ARG1 for filename
   • Note the file path and name
   • Continue execution
5. When WriteFile hits:
   • Examine ARG2 (buffer) to see what's being written
   • Dump the buffer contents
   • Check if it's PE file (MZ header)
6. Extract dropped payloads for further analysis

Example 2: Analyzing Persistence Mechanisms

Scenario:

Malware that establishes persistence on the system.

Analysis Steps:

1. Set registry operation breakpoints:

   bp RegOpenKeyA
   bp RegOpenKeyW
   bp RegSetValueA
   bp RegSetValueW

2. Monitor common persistence locations:
   • HKLM\Software\Microsoft\Windows\CurrentVersion\Run
   • HKCU\Software\Microsoft\Windows\CurrentVersion\Run
   • HKLM\System\CurrentControlSet\Services
3. Document registry modifications
4. Check for scheduled task creation:

   bp CreateProcessA  # Look for schtasks.exe execution

Example 3: Network Communication Analysis

Scenario:

Malware that communicates with C2 servers.

Analysis Steps:

1. Set network API breakpoints:

   bp WSAStartup
   bp socket
   bp connect  
   bp send
   bp recv
   bp InternetOpenA
   bp HttpOpenRequestA

2. When connect() hits:
   • Examine ARG2 (sockaddr structure) for IP and port
   • Convert binary IP to dotted decimal
3. When send() hits:
   • Examine ARG2 (buffer) for transmitted data
   • Look for HTTP headers, commands, or encrypted data
4. Document network IOCs

Detecting and Bypassing Anti-Analysis

VM Detection Techniques

Registry-based Detection:

# Monitor registry queries for VM artifacts:
bp RegQueryValueA
bp RegQueryValueW
# Common VM registry keys:
HKLM\SYSTEM\ControlSet001\Services\VBoxService
HKLM\SOFTWARE\VMware, Inc.\VMware Tools
HKLM\HARDWARE\DESCRIPTION\System\SystemBiosVersion

File-based Detection:

# Monitor file existence checks:
bp GetFileAttributesA
bp CreateFileA
# Common VM files:
C:\windows\system32\drivers\vboxmouse.sys
C:\windows\system32\drivers\vmhgfs.sys
C:\windows\system32\vboxdisp.dll

Hardware-based Detection:

# Monitor CPUID instructions
# Set breakpoint on CPUID (opcode: 0F A2)
# Check for VMware signatures in registers

Timing-based Anti-Analysis

RDTSC Timing Checks:

# Set breakpoint on RDTSC instruction (opcode: 0F 31)
# When hit, manipulate EDX:EAX to control timing
# Or use ScyllaHide to automatically handle timing checks

Sleep/Delay Techniques:

bp Sleep
bp WaitForSingleObject
bp GetTickCount
# When hit, either:
# 1. Patch the call to return immediately
# 2. Modify time values
# 3. Skip the instruction entirely

Control Flow Obfuscation

Junk Code and Dead Code:

• Look for nonsensical instructions
• Use static analysis to identify unreachable code
• Focus on actual execution flow

Control Flow Flattening:

• Identify dispatcher loops
• Track state variables
• Map the actual execution flow

Best Practices and Tips

Environment Setup

• Always use isolated VMs - Never analyze on host systems
• Take snapshots - Create clean baselines before analysis
• Monitor network traffic - Use tools like Wireshark or Fakenet
• Document everything - Keep detailed analysis logs
• Use multiple tools - Combine x64dbg with other analysis tools

Analysis Methodology

1. Static reconnaissance first - Basic file analysis before execution
2. Set strategic breakpoints - Don't just run blindly
3. Monitor key APIs - Focus on behavior-revealing functions
4. Extract IOCs - Document IPs, domains, file paths, registry keys
5. Validate findings - Cross-reference with other tools

Performance Tips

• Use conditional breakpoints - Reduce false positives
• Disable unnecessary plugins - Improve debugging performance
• Use hardware breakpoints - More reliable than software BP
• Clear breakpoints regularly - Remove unused breakpoints

Common Pitfalls to Avoid

• Don't trust the malware - Verify all observed behavior
• Watch for time bombs - Some malware delays execution
• Check for network dependencies - Some malware needs internet access
• Be aware of anti-analysis - Use evasion detection techniques
• Don't overlook persistence - Check for system modifications

Advanced Scripting

x64dbg Script Example:

// Automatic API logging script
bp CreateFileA
$breakpointcondition = 1
$breakpointcommand = log "CreateFile called with: {s:arg(1)}"
bp RegSetValueA  
$breakpointcondition = 1
$breakpointcommand = log "Registry value set: {s:arg(2)} = {s:arg(5)}"

Conclusion

Dynamic analysis with x64dbg is a powerful technique for understanding malware behavior and extracting actionable intelligence. By combining systematic methodology with the right tools and techniques, analysts can efficiently reverse engineer even sophisticated threats.

Remember that dynamic analysis is most effective when combined with static analysis, network monitoring, and system-level observation. The key to successful malware analysis is understanding both the individual techniques and how to apply them as part of a comprehensive analysis workflow.

Additional Resources

• Official x64dbg Documentation
• x64dbg GitHub Repository
• x64dbg Help Documentation
• Plugin repositories and community resources
• Malware analysis training courses and certifications

Stay safe, analyze responsibly, and always use proper isolation when handling malware samples.