Threat Research

Captain Hook—Pirating AVs to Bypass Exploit Mitigations

By Udi Yavo | July 19, 2016

FortiGuard Labs Threat Analysis Report: This blog originally appeared on the enSilo website on July 19, 2016, and is republished here for threat research purposes. enSilo was acquired by Fortinet in October 2019.

The enSilo threat research team (now part of FortiGuard Labs) found six different common security issues that stem from the incorrect implementation of code hooking and injection techniques. These issues were found in more than 15 different products. The most impactful discovery was that three different hooking engines also suffer from these kind problems, including the most popular commercial hooking engine in the world – Microsoft Detours (scheduled patch, August 2016). Practically, it means that thousands of products are affected.

Usere Mode Hooks

User-mode hooks are used by most of the endpoint security vendors today, specifically Anti-Virus (AV) products and Anti-Exploitation products such as EMET. Beyond their usage in security, hooks are used in other invasive applications such as Application Performance Management (APM) technologies to track performance bottlenecks.

Hooking itself is a very intrusive coding operation where function calls (mainly operating system functions) are intercepted in order to alter or augment their behavior.

Given the sensitivity of hooking implementations, we sought to find their robustness. For our research, we investigated more than a dozen popular security products. Our findings were depressing—we revealed six different security problems and vulnerabilities stemming from this practice.

Hooking in a Nutshell

The use of hooks allows intrusive software to intercept and monitor sensitive API calls. In particular, security products use hooking to detect  malicious activity. For example, most Anti-Exploitation solutions monitor memory allocation functions, such as VirtualAlloc and VirtualProtect, in an attempt to detect vulnerability exploitation.

On the other side of the security spectrum, hooks are also used extensively by malware for various nefarious purposes, the most popular being Man-In-The-Browser (MITB) attacks.

The most common form of hooking in real-life products, especially security products, is inline hooking. Inline hooking is performed by overwriting the first few instructions in the hooked function and redirecting it to the hooking function. Although there are other forms of hooking, such as Import Address Table (IAT)-hooking, this research focuses only on inline hooks.

Hooking in user-mode is usually implemented within a DLL that is loaded into a process address space. We refer to this DLL as the “Hooking Engine”.

Our research dives into inline user-mode hooking. We also take a deep look into injection techniques, specifically kernel-to-user injections since these are usually used to load the hooking engine into the process address space. Kernel-to-user injections are not trivial to implement, and accordingly, some of the most severe issues that we found were not in the hooking engine itself but rather in the implementation of the kernel-to-user injection.

Under-the-Hood of Inline User-Mode Hooking

Although hooking is quite common and there are several common hooking libraries out there, such as Microsoft Detours, it seems that most security vendors develop their own hooking engines. That said, apart from a few exceptions, most of these in-house inline hooking implementations are pretty much similar.

Inline Hooking on 32-Bit Processes

Hooking 32-bit functions is straight forward most of the time. The hooking engine disassembles the first few instructions of the target function in order to replace it with a five-byte jmp instruction. After at least five bytes of disassembled instructions are found, the hooking engine copies the instructions to a dynamically allocated code stub and follows with a jmp that returns the code to the original function. At that stage, the hooking engine overwrites the instructions with a jmp to the actual hooking function.

For example, let’s see how at how a hook on InternetConnectW looks in a windbg: 

Figure 1: InternetConnectW before the hook is set (Marked in red are the instructions that will be replaced)
Figure 2: After the hook is set

We can see that the jmp instruction leads to 0x178940, which is the hooking function itself. Disassembling the code at 0x178940 provides more insight:

Figure 3: Disassembled code at 0x178940

This code calls the original InternetConnectW function, leading to:

Figure 4: Original instructions of the function followed by a jmp

As shown, the original instructions of the function are followed by a jmp to the original function.

In other 32-bit hooking scenarios, hooking is not that straightforward. For example, if one of the first instructions is a relative call it must be fixed before being copied.

A nice read on possible hooking issues can be found in Binary Hooking Problems, by Gil Dabah.

Inline Hooking on 64-Bit Processes

Hooking on 64-bit processes is a bit more difficult than on 32-bit processes because the address space is much larger. This means that a five-bytes jmp instruction might not be enough in order to install a x64 hook since it is limited to a 2GB range from its location.

There are several solutions to this problem, some of them are described in Trampolines in X64, also by Gil Dabah.

The most common solution to this issue is to allocate a code stub within a 2GB range from the hooked function and use the following code template:

MOV RAX, <Hooking Function>


For example, let’s take a look at a hook on the 64-bit version of InternetConnectA.

Figure 5: The original InternetConnectA function
Figure 6: The function after the hook is set.

As shown, the function jumps to 0x7fefe1ff000.

Figure 7: Disassembling the code in address0x7fefe1ff000

If we follow the hooking function like we did in the 32-bit version, we get to the following code stub that redirects the execution back to the original function: 

Figure 8: 64-bit code stub

Injecting the Hook Engine

Regardless of the way the hooking engine is implemented, a prerequisite for it to do its job is to inject it into the target process. Most vendors use kernel-to-user DLL injections to perform this. In this section, we cover the most common methods used by security vendors.

Import Injection

Import injection is quite common and is relatively clean as it doesn’t require any code modifications. And as far as we know, this injection technique was never used by malware.

It works by adding an import to the main image. These are the steps for import injection:

  1. Register the load image callback using PsSetLoadImageNotifyRoutine and then wait for the main module to load.
  2. After the main module is loaded, the import table is copied to a different location and a new row that imports the hook engine is added to the beginning of the table. The RVA of the import table is modified to point to the new table. This is how it looks in Internet Explorer:
Figure 9: Internet Explorer patched import table

This is the new import table:

Figure 10: The new import table

    3.  When the module completes loading, the RVA of the original import table is restored.

Entrypoint Patching

To the best of our knowledge, this kind of injection method was first used by the infamous Duqu malware and is well documented. It is also used by security vendors.

These are the steps for entrypoint patching:

  1. Register the load image callback using PsSetLoadImageNotifyRoutine and then wait for main module to load.
  2. Read the instructions from the entrypoint and allocate a payload to load the hook engine.
  3. Patch the entry point with a jmp to the payload. This is what entry point patching looks like in Internet Explorer:
Figure 11: Internet Explorer patched entrypoint

    4. When the payload executes, it first loads the hooking engine and then restores the bytes that were copied from the original image.

Figure 12: Restoring the bytes from the original image

User-APC Kernel-to-User DLL Injection

Kernel-to-user DLL injection using User Mode APC (Asynchronous Procedure Call) is probably the most documented and common method. This method was also extensively used by malware, TDL, and Zero-Access, for example.

For detailed information on this injection method we refer the reader to:


This method is also quite common in security software. As far as we know, the NTDLL.DLL/User32.DLL patching was never used by malware.

This is how it works:

  1. Register the load image callback using PsSetLoadImageNotifyRoutine and then wait for the target module to load.
  2. Once the module is loaded, a payload for loading the hook engine is injected into the process and a function that will be called during the startup of the process is patched with a jmp or push/ret to the payload. On user32.dll, the patched function used is usually UserClientDllInitialize. On ntdll.dll, the patched function is usually LdrLoadDLL.  In this case, the push/ret sequence is used to divert execution to the injected payload.
Figure 13: LdrLoadDLL is used for injection

    3. Once the payload executes it loads the hook engine and restores the original code in the patched function.


Our Blackhat talk—Captain Hook: Pirating AVs to Bypass Exploit Mitigations,” held Wednesday, August 3, 2016 from 4:20PM–5:00PM—focused precisely on these hooking and injection implementations. We reviewed the security issues we found in various security and non-security products, as well as with hooking engines. 

Learn more about FortiGuard Labs threat research and the FortiGuard Security Subscriptions and Services portfolioSign up for the weekly Threat Brief from FortiGuard Labs. 

Learn more about Fortinet’s free cybersecurity training initiative or about the Fortinet Network Security Expert programNetwork Security Academy program, and FortiVet program.