Syscall Hook Detector

 

Introduction

This program checks if functions in ntdll.dll are hooked or modified. It looks at the first few bytes of each function to see if they match the expected code. If the bytes are different, it indicates the function may be hooked. The program uses memory reading techniques (Portable Executable file structure manipulation, image below) to inspect the functions and report their status. It helps detect tampering or alterations in system functions.

In this program, we are going to be specifically playing with the Export Directory located within the NT Headers of a PE (Portable Executable file), more specifically in the DataDirectory field of the IMAGE_OPTIONAL_HEADER.

Introduction to Syscalls

• Syscalls (System Calls) are the mechanism by which programs interact with the operating system kernel to perform tasks that require higher privileges or direct access to system resources.
• When a program needs to perform an operation like reading a file, allocating memory, or creating a process, it issues a syscall to request that the kernel perform the action on its behalf.
• This is necessary because user-mode applications do not have direct access to kernel-level resources for security and stability reasons.
• In Windows, syscalls are exposed through functions like NtCreateFile, NtQuerySystemInformation, ZwReadFile, etc., which are implemented in the ntdll.dll library.
• These functions serve as an abstraction layer between user-mode applications and the underlying operating system kernel, allowing programs to request services in a controlled, secure manner.

Introduction to Hooks

• Syscall hooks refer to a method of modifying the normal behavior of system calls, usually by intercepting them and redirecting their execution to custom code. This modification is often used to monitor, modify, or even bypass the intended behavior of syscalls.
• In the context of malicious software or debugging tools, syscall hooks can be used to hide activity (e.g., hiding processes or files) or alter the function of critical system calls to avoid detection or modify system behavior.

Workflow and Walkthrough

Below shows the stub for for NtReadVirtualMemory on a system with no EDR present, meaning the syscall NtReadVirtualMemory is not hooked:

We can see the NtReadVirtualMemory syscall stub starts with instructions:

00007ffc`d6dcc780 4c8bd1          mov     r10,rcx
00007ffc`d6dcc783 b83f000000      mov     eax,3Fh
...

... which roughly translates to the following opcodes :

4c 8b d1 b8

Below shows an example of how NtReadVirtualMemory syscall stub looks like when it's hooked by an EDR:

We can see that the hooked NtReadVirtualMemory syscall stub starts with different instructions :

jmp 0000000047980084

... which roughly translates to the following opcodes :

e9 0f 64 f8 c7

All x64 Windows SYSCALLs must follow this general calling convention :

MOV R10, RCX
MOV EAX, <SYSCALL_NUMBER>h
SYSCALL
RETN

...as a result of this standard pattern, hook detection becomes trivial.

Workflow

Based on these discoveries, we can map out and brainstorm :

• Hooks are typically implemented by replacing the first few bytes of a function’s code with a JMP or CALL instruction. This redirection changes the flow of execution, causing the program to jump to another address where custom behavior (e.g., logging or blocking actions) is executed instead of the original function.
• The program detects syscall hooks by examining the first few bytes of the functions in ntdll.dll, particularly those starting with Nt or Zw (common prefixes for syscalls).
• Prologue inspection is used to check if the first bytes of these functions match a known unhooked pattern (e.g., 0x4c 0x8b 0xd1 0xb8), which is characteristic of unmodified syscalls.
• The program uses ReadProcessMemory to retrieve the first few bytes of each syscall function and then compares them to a predefined pattern.
• If the bytes don't match, it indicates that the function might be hooked. The program specifically looks for JMP or CALL opcodes (0xE9 or 0xFF), which are commonly used to redirect the execution flow in a hook.

That`s it! Nothing more, nothing less. For those who understand concepts better visually - below is a graph of our plan

Key Features

• Dynamic Function Scanning: Loads ntdll.dll into memory using LoadLibraryA and retrieves the process handle with GetCurrentProcess() to access its functions.

• PE Header Parsing: Extracts the DOS Header and NT Headers to locate the Export Directory, where the addresses and names of exported functions are stored.

• Export Directory Iteration: Uses AddressOfFunctions, AddressOfNames, and AddressOfNameOrdinals to loop through all exported functions in ntdll.dll.

• Syscall Detection: The isSyscall function checks if a function starts with Nt or Zw, signaling it's a syscall and requiring special hook checks.

• Memory Integrity Check: ReadProcessMemory reads function memory and compares the first bytes against expected patterns, detecting hooks or modifications.

• Hook Detection: Identifies functions with JMP or CALL instructions (common hooks) and reports their status, distinguishing between unhooked and modified functions.