I & D - part 2

One thing I've learned through having a blog is that I really suck at having a blog. Writing an article takes more time than I thought it would take, and there's a lot of interesting stuff to do besides writing a blog entry.

Anyway, I'll dive straight into part 2 and I'll talk a bit about bindings. This is a surprisingly relevant topic if you want to work with D.

What do I mean when I say "bindings"?

A programming language is basically useless without some functionality. The most beautiful looping construct in the world is just an intellectual exercise unless you have a set of records or files or vertices to iterate over. Some functionality is provided by the standard library (in some cases, like with .Net for example it's a huge set of functionality, and in other cases, like with C or D it's much smaller). But, no matter how much pre-built code your language of choice comes with, sooner or later you will need to interface with some third-party library and this is what I'm talking about here.

I'll be presenting a few cases of third-party bindings I had to implement with boxen in increasing order of effort necessary.

1. The Win32 API

Of course, since boxen is currently exclusively a Windows application (yeah, my Linux/xlib port is seriously behind schedule) I'm of course using the almighty Win32 API. For such a big and complicated API using it in boxen was surprisingly easy - just head over to dsource.org and you'll find the relevant bindings there. boxen has a file called "platform.d" which contains most of the platform specific code for both Windows and Linux, and it publicly pulls in the win32 files I need - like this:

As you can probably imagine it wasn't exactly 100% straightforward, but it was a relatively painless experience. Download the code, adjust some compiler flags and you're good to go.

2. COM

COM is an essential part of Windows programming. Sooner or later you will probably have to interface with COM if you develop non-trivial Windows applications. Below is a sample of boxen's WASAPI interface, which is based around COM, but COM is also essential in the ASIO interface (which I will talk about a bit further down) and the drag/drop system.

Thankfully D makes it somewhat easy to deal with COM. I had some trouble initially with my COM interfaces crashing, until I figured out that (for WASAPI interfaces) I need to declare them in an "extern (Windows)" block, which makes the D compiler emit the correct machine code when calling members of the interface.

A quick note here if you don't know what I'm talking about: there are different ways of how you can implement function calls on the machine code level. This is a bit how power plugs works: there are different standards and if you have, for example, a device with a power plug that's used in Denmark, you can't plug it in here in Austria, because the plugs are different (although the voltage and frequency are the same AFAIR). None of those standards is inherently "better" than another one, they're just different because people have defined them differently and with slightly different priorities. D, for example, uses its own way to define function calls, and Windows uses another one. So if you want your D code to call a Windows interface you need to tell the D compiler that this interface uses the Windows calling convention instead of the D calling convention - by surrounding the definition of the interface with "extern (Windows)". There are other conventions, like "extern (C)" or "extern (Pascal)". To find out which convention your third-party API uses simply consult its documention.

As you can see in the screenshot it's very easy to define a COM interface in your code - just derive it from IUnknown. I remember it was a bit of a hassle to get this working with Tango (which is the standard library I'm using), but I can't actually remember what it was, sorry. YMMV, of course. Another point worth noting here is that you will still need the actual GUIDs of the interfaces - in the case of WASAPI DMD already ships with uuid.lib which can simply be linked against your project. In other cases you might need to define them as literals in your code or obtain them at runtime from somewhere.

D is no Visual Basic or .Net when it comes to COM - it's not as simple as checking a checkbox in the IDE. But it's reasonably simple to include such a complex beast as COM in your D project.

3. Cairo

Cairo is probably one of the best examples of how you will create bindings (and you will do that!) for D. It's a C library, it's open source and available as both statically and dynamically linked version. Since boxen is closed source I of course link dynamically against Cairo.

C is sometimes called the lingua franca of programming languages, and the C ABI is also the most commonly used format for libraries you will find out there. It's virtually guaranteed that your D project will at one point or another link against C code (even more so outside the Windows world), so the line "extern (C)" will be a frequent resident in your source files.

There's not much to be said about linking against C binaries. It just works. DMD treats C binaries like first-class citizens, and since D allows you to write structured code and supports pointers it feels completely natural to use C functions. The only hassle that remains is dealing with null-terminated strings, and that's just something you have to accept.

I'm listing Cairo as number 3 because it was actually the first binding I ever ported from one language to another. I don't actually remember if there was a Cairo binding project on dsource at the time, I think there probably was, but I couldn't get it to work. Anyway, I ported almost the whole API over to D, which was an enormous amount of work.

In the process I learned how to do a port like that, and I learned a lot about how programming works at this level. I especially learned that you don't actually need the full API ported, usually it's only a handful of functions and structs that can get you started. Today I can start with a DLL and use it in a D project in 10 minutes. I've done this over and over and I've gotten very efficient by now.

This is both a good thing and a bad thing.

The good thing is that I've learned a lot about how it works and I can get going from the worst case (no bindings exist for D) in a very short time. The bad thing is that writing bindings for libraries is just wasted time.

It's not helped by the fact that existing bindings that you can find on dsource often simply don't work or they are for the wrong version of D. Sometimes they are monoliths of code that would probably be very amazing but you need to spend hours to get them working - like Derelict, for example. Derelict is a binding for OpenGL (among other things) which I simply couldn't get to work after 15 minutes, so I started writing my own OpenGL binding in the same amount of time. This is what boxen now uses for the experimental video playback.

It seems to me that linking C libraries is very easy in D, but you shouldn't have to do it in the first place. Still, since it's easy and painless you can't really complain about it too much, but it's something you need to consider before choosing the language for your project.

It should probably be mentioned that there is a tool called "htod" which can automatically port C headers to D. My success with it always has been a bit underwhelming, but try it for yourself before attempting to do a port manually.

4. ASIO

With the tools that I've mentioned so far - support for "extern (C)", "extern (Windows)" and IUnknown - it's possible to cover 99.9% of all bindings. Sure, it's often more work than necessary, but nothing that you can't handle. But what if there isn't a built-in way to link against an API or a certain COM interface?

ASIO works on top of COM, like WASAPI does. But while WASAPI's COM objects use "extern (Windows)" as described above, ASIO uses a special calling convention know as "thiscall" which is not supported by DMD (it's only supported by Microsoft's C++ compiler, AFAIK). But since boxen supports ASIO you may now rightfully ponder how this technological wizardry is possible. Alas, the answer to this riddle is the mysterious witchcraft of inline assembler:

As you can probably see I'm manually manipulating the stack and calling the interface function with a hard-coded offset. (Edit: when I wrote this code I still was pretty much thinking in C++ - I should have used the "auto" keyword instead of "long" and the ".ptr" property instead of the "&foo[0]" trick to obtain a pointer to the array.)

I would strongly recommend against doing this. In retrospect I think it would have been far easier to just write a small library adapter in Visual Studio and use that, but - as with so many other things - it was a tremendously important learning experience for me and I think I've grown a lot as a programmer because of this.

For a hobby project I think this is fine, but for everything else it's just stupid.

But it just goes to show that there's simply no limit to how far you can go with D, even if it means that the only thing that propels you forward is the recoil from shooting yourself in the foot.

In conclusion

That's about all I wanted to write about in part 2. I hope this gives you some insight into one important aspect of working with D and it shows you what you can expect if you need to interface with third-party libraries. D forces you to get your hands dirty more often than needed. Often you find that there are no bindings for a library you need to use or that you can't get bindings to work, so you'll have to do extra work. But D gives you a very elegant and comfortable shovel to get the extra work done quickly and reliably. And for the few cases where the shovel doesn't work it still has a toothbrush that will always work.

And I'll take this opportunity to wish you all a Merry Christmas and a Happy New Year! See you in 2012 :)