So I wanted to hook Direct3D9 and override some of the calls with my own so I could either track the call itself or even change the incoming parameters or the returned results. There are a number of uses for this type of thing ranging from profiling, cheating in 3D games to full screen monitor spanning.
This sort of thing has been done before in C++ many times and there are a number of sources of information and samples on how to do it such as over at
GameDev.net. However, I wanted to be able to write my code in C# both because I felt it was time to learn the language a little bit and I have worked with C++ enough to know that I don't really want to deal with all the headaches associated with such a low level language.
First off, I needed to be able to hook a Direct3D9 application with C#. Luckily, there is a free (and extremely well documented) library available that does just that called
EasyHook. Getting it up and running was quite easy since it came with a sample application that did almost exactly what I wanted, it just hooked CreateFile in kernel32.dll instead of Direct3DCreate9 in d3d9.dll which was a simple change.
So now anytime Direct3DCreate9 was called by the hooked application instead of calling the real Direct3DCreate9 it would instead call my C# function, MyDirect3DCreate9. Initially, MyDirect3DCreate9 would call the real Direct3DCreate9 (in d3d9.dll) and return whatever result the real one returned (as an IntPtr).
Of course , Direct3DCreate9 returns an IDirect3D9 interface which then does all the work. At this point things get tricky because I wanted to be able to override the member funtions of IDirect3D9, not just the exported functions in d3d9.dll (of which there are only a few). My initial thought was to use the COM interop services provided in the .NET framework and implement IDirect3D9 on my own. My IDirect3D9 object could then instantiate a real IDirect3D9 object (from d3d9.dll) and call those functions as necessary.
Unfortunately, after much trial and error I was unable to get such a system working. I think the main problem stems from the fact that Direct3DCreate9 needed to return an IDirect3D9 object to the calling native code (the hooked game) and it wasn't too happy getting a managed IDirect3D9 implementation instead.
Then I came across
this site which, although a total hack, allowed me to do exactly what I wanted. Basically, we know what the IDirect3D9 virtual function table has in it because we have the C++ header file d3d9.h, which defines exactly what the functions look like. In order to fool the calling application into thinking it has a real IDirect3D9 object all we have to do is provide it with an object containing a virtual function table containing 17 members (as seen in d3d9.h) all of which have signitures that it is familiar with.
First things first we create a real IDirect3D9 object and take a look at it's virtual function table. We can't modify the original table because it's protected by the OS but we can point the IDirect3D9 at a different virtual function table. So what we do is create our own virtual function table in unprotected memory and copy all the function pointers from the original into it. Then we tell IDirect3D9 to point at our custom table instead of the real one.
Now that we have a writable virtual function table we can then change some of those functions to point at our own C# functions instead. Those C# functions can then call into the original function table when we want to call the real IDirect3D9 functions!
Here are the DllImport statements for calling into kernel32 (for memory management) and d3d9. You will notice that I have also created an IDirect3D9 structure in C# that has a signature that matches the native IDirect3D9 signature (which only has an array of function pointers). This will allow me to receive an IDirect3D9 object from native code and get to the virtual functions from managed code.
Here is my custom IDirect3D9 class. It's not really an IDirect3D9 implementation but instead it's a container where I keep all my IDirect3D function overrides, store the native IDirect3D9 object and manage the virtual function table.