Windows API Internals: From User Mode to Kernel Land

Date: 2025-11-24 Tags: Windows, Kernel, Reverse Engineering, C++ Author: Wissam Ztaoui

Introduction

When you call CreateFile in C++, what actually happens? Most developers live in the comfortable abstraction of kernel32.dll. But for malware analysts, game cheat developers, and systems programmers, the real magic happens much deeper. This guide peels back the layers of the Windows OS, from the Win32 subsystem down to the kernel syscall dispatcher.

1. The Layered Architecture

Windows is not a monolith. It is a layered cake of abstractions.

The Subsystems

• Win32 Subsystem (csrss.exe): The traditional Windows API.
• NT Subsystem: The native API layer.
• Win32k.sys: The kernel-mode component of the GUI subsystem.

The DLL Hierarchy

1. Application: Calls CreateFileW in kernel32.dll.
2. KernelBase.dll: kernel32.dll is mostly a forwarder. The logic often lives here.
3. Ntdll.dll: The last frontier in User Mode. This exports the “Native API” (functions starting with Nt or Zw, e.g., NtCreateFile).
4. Syscall: ntdll.dll executes the syscall (x64) or sysenter (x86) instruction to transition to Ring 0.

2. The Native API (ntdll.dll)

ntdll.dll is the bridge. It contains no significant logic; it is a wrapper around syscalls.

; Example of NtCreateFile in ntdll.dll (x64)
mov r10, rcx      ; Save first argument
mov eax, 0x55     ; Syscall number (changes between Windows versions!)
syscall           ; Transition to Kernel Mode
ret

Key Concept: Syscall numbers (SSDT Indices) are not stable. They change between Windows 7, 10, 11, and even service packs. Tools like “Hell’s Gate” dynamically resolve these numbers to bypass EDR hooks.

3. The Kernel Transition (Ring 3 $\to$ Ring 0)

When syscall executes:

1. CPU switches to Ring 0.
2. Instruction Pointer (RIP) jumps to the address stored in the MSR_LSTAR register.
3. This address points to KiSystemCall64 in ntoskrnl.exe.

SSDT (System Service Descriptor Table)

The kernel uses the syscall number from EAX as an index into the SSDT to find the address of the actual kernel function (e.g., NtCreateFile inside ntoskrnl.exe).

4. The PE File Format (Portable Executable)

Every .exe and .dll follows the PE format. Understanding this is crucial for manual mapping and injection.

Structure

1. DOS Header: Starts with MZ. Contains a pointer to the PE Header.
2. PE Header: Starts with PE\0\0. Contains architecture (x86/x64), timestamp, and entry point address.
3. Optional Header: Contains “ImageBase” (preferred load address) and “AddressOfEntryPoint”.
4. Section Table: Defines segments like .text (code), .data (globals), .rdata (imports/constants).

IAT (Import Address Table)

When an executable loads, the Windows Loader fills the IAT with the actual memory addresses of imported functions (like MessageBoxA).

• API Hooking: By overwriting a pointer in the IAT, you can redirect calls from the target application to your own function.

5. API Hooking Techniques

IAT Hooking

• Pros: Easy to implement.
• Cons: Only affects imports. Doesn’t work if the app uses GetProcAddress.

Inline Hooking (Trampoline)

Overwriting the first few bytes of the target function with a JMP to your code.

1. Read the first 5 bytes of MessageBoxA.
2. Save them (to execute later).
3. Overwrite with JMP MyHook.
4. MyHook: Do malicious stuff $\to$ Execute saved bytes $\to$ Jump back.

Conclusion

The Windows API is a rabbit hole. Understanding ntdll, syscalls, and the PE format gives you superpowers: you can hide processes, inject code, and understand exactly how the OS manages your software.