Archive

A great deal has happened in wxHaskell land over the past few weeks, so I thought a summary was worthwhile

wxHaskell 0.90 Released

A significant update to wxHaskell was released on April 14th. This brings in all of the work done by Dave Tapley, Eric and many others to provide support for wxWidgets 2.9.

Supporting wxWidgets 2.9 is important for quite a number of reasons – not least because at some point it will become wxWidgets 3.0 and will be the new stable version. However in the short term the main benefit is support for 64bit OS platforms – notably MacOS X Snow Leopard and Lion.

The slightly odd version numbering convention was chosen to allow wxHaskell 0.13 to evolve without being excessively constrained over version numbering. In any case, it would be nice to get to version 1.0 at some time soon – perhaps when wxWidgets 3.0 is released.

Most of the future wxHaskell development effort will go on the new branch.

wxHaskell 0.13 Branch Created

On many systems, particularly almost all Linux distributions, wxWidgets 2.8.x remains the standard ‘package’ for wxWidgets, so we continue to support this for those who would prefer to use the packages provided by their distro. It also allows Windows users without C++ development environments to use the wxPack binary installers for wxWidgets.

Experimental GitHub Repository

I have created an experimental GitHub repository. It is right up to date at the time of writing and contains two active branches: master is the wxWidgets 2.9 repo and WXWIDGETS_2_8 is (unsurprisingly) the wxWidgets 2.8 branch.

I’m especially interested in feedback on whether moving definitively to GitHub would be a good thing – the main criteria for judgement being whether it makes it easier to receive contributions from others.

Those who follow the wxHaskell developer or users lists will know that over the last few months there has been a flurry of activity on wxHaskell, spurred on by Dave Tapley. I wanted to summarise what has been happening, and what stands in the way of a release supporting wxWidgets 2.9

<blatent-plug>Dave is working on wxHaskell with the agreement of his employer, Mentics Inc. I encourage any readers who have the opportunity to place business in Mentics’ direction to do so in appreciation of this generous donation of time and talent to the Haskell community.</blatent-plug>

Why wxWidgets 2.9?

Most Linux distributions currently supply wxWidgets 2.8.x libraries from their packaging systems, and Windows has the pre-built and readily installable wxPack containing wxWidgets 2.8.12, so why are we moving to an unstable release of wxWidgets?

There are many reasons, but two stand out in particular:

It is the future. wxWidgets 3.0 will be released from the 2.9 line, and provides major improvements. These include:

Much improved Unicode support.

Lots of new controls: ribbon bars, wxPropertyGrid, wxWebView, wxTreeListCtrl, wxRichToolTip, wxRichMessageDialog. Together, these simplify the creation of GUIs with a ‘modern’ look and feel.

The stc library (used by StyledTextControl) is now part of the main library build (as is SVG, which I would like to wrap for wxHaskell in the near future).

Support for 64bit OS X builds, which we have been unable to support on wxHaskell due to underlying lack of support in wxWidgets.

We need to clean up some legacy ‘cruft’ in the code. Daan originally wrote wxHaskell for wxWidgets 2.4.x, and things have moved forward a long way since then. This is an opportunity to remove deprecated functions and offer cleaner APIs.

What is changing?

The changes listed exist today in Dave’s development repository at DarcsDen, and will be mainlined in our master repository at code.haskell.org starting the coming weekend.

Newly wrapped classes

PropertyGrid, related helper types and sample code (Dave Tapley)

ListView (Dave Tapley)

Reinstated features

StyledTextCtrl (Dave Tapley)

OpenGL (Me)

Ability to use wxHaskell in GHCi (Dave Tapley)

Build system improvements

Some old Eiffel legacy code in the build system has been removed. Everything is now either C++ or Haskell (Eric Kow).

The C wrapper for wxWidgets has been moved into a separate project, wxC and built as a shared library (Dave Tapley).

Work to support OS X Lion (Eric and Alessandro Vermulen).

A Haskell native implementation of wx-config for use on Windows platforms. We are not sure if we will use this as yet – it is experimental (Eric).

Bugfixes and ‘build experience’ contributions from all of the above and several others , including Shelarcy, Maciek Makowski, Henning Thielemann, Peter Simons.

What is left to do before there is a release to Hackage?

Quite a bit!

The code in Dave’s repo is reasonably well tested on Linux (Ubuntu), but has currently received insufficient love on Windows or Mac. We will need it to work reliably on all three platforms before it can be released.

I think that the main blocker right now is probably determining the correct configuration for a Windows build. We have a couple of options here. We could develop Eric’s Haskell wx-config replacement so that it has been tested for a good subset of all possible wxWidgets build configurations, or we could fix wx-config-win to the same end. This has raised some philosophical questions on ‘Open Source and community’ which I am thinking of blogging on separately.

Most of all, when we have candidate builds ready, I hope that wxHaskell users will help out by trying to install the new version on as many machines as possible, so that we can be sure that we have everything working.

Some Background

As some of you may have noticed, there has been a minor flurry of activity (if that’s not an oxymoron) from me recently. The wxHaskell repo went offline after the attack on code.haskell.org and I decided to look at support for wxWidgets 2.9 as well as 2.8 while I was waiting for the repo to be restored.

Something I thought I had fixed (would have known better if I had been reading my mail more carefully) was a bug found by Eric when using the native Mac wxWidgets. Basically Eric was seeing an error dialog indicating an assertion failure, which it seemed to be safe to ignore, when starting an application.

I’m afraid this causes the (surprise!) event handlers for some list boxes to stop working😦

That’ll teach me not to take the time to understand a problem before ‘fixing’ it. OK, so Laziness and Impatience failed. Time for Hubris to take over and I tweeted that I’d look into the problem properly.

In fact, the code in wxEventHandler_GetClosure() is rather dubious in a couple of respects. The first is that the static global ‘getCallback’, is clearly a very long-standing hack (the repo source code notes this – I tend to remove comments to keep blog entries to some sort of sane length). The second is that wxWidgets doesn’t document wxEvtHandler::SearchDynamicEventTable() at all. Since wxWidgets is generally very well documented, this is a warning…

How does Event handling normally work in wxWidgets?

There are a couple of ways to do event handing in wxWidgets. The simplest, which I’m not going to explore further, is to use event macros. This won’t help us in wxHaskell as we want to define our event handlers in Haskell. The solution is to use wxEventHandler::Connect() to associate a function call with an event. This means that our supplied function will get called when the event occurs. A little plumbing will be needed to allow the called function be be written in Haskell, which we’ll look at later.

If the object is a wxWindow, ProcessEvent is recursively called on the window’s wxValidator. If this returns true, the function exits.

SearchEventTable is called for this event handler. If this fails, the base class table is tried, and so on until no more tables exist or an appropriate function was found, in which case the function exits.

The search is applied down the entire chain of event handlers (usually the chain has a length of one). If this succeeds, the function exits.

If the object is a wxWindow and the event is set to set to propagate (in the library only wxCommandEvent based events are set to propagate), ProcessEvent is recursively applied to the parent window’s event handler. If this returns true, the function exits.

Finally, ProcessEvent is called on the wxApp object.

Connecting event handlers in wxHaskell

Now that the outline of event handling in wxWidgets is clear, let’s look at events in wxHaskell. I’m going to do this starting from the WXCore library as the WX library, while simpler to use, is just a Haskell wrapper around WXCore.

Widget wrappers need to provide at least two event-handler related functions: one to set the event handler and another to retrieve it. The setters and getters resemble the following (this example for Button):

The setter calls a generic event handler setter for everything deriving from Window with the control to which the handler applies, a list of the events for which the handler applies and an event handler function which will be called when the event occurs. One small point to note is that the event handler is passed twice: once as a piece of state and a second time wrapped in a lambda function.

The getter function uses an unsafe function to obtain the event handler for the requested event.

In other words, we eventually transform the event handler for any given control into a generic event handler for a Window a (remember, using phantom types, wxHaskell arranges that all controls eventually resolve to a Window a), and this contains a function (strictly an IO computation) which is run when the event is triggered, and a computation which is run when the event handler is deleted. This computation takes a True argument if the owner is deleted and a False argument if we are simply disconnecting the event handler.

We are finally getting close to where we connect up with the wxWidgets framework. The Closure a type is derived (using phantom types) from wxObject a which is itself an instance of Object a, which is basically a way of storing a C pointer.

In createClosure we wrap some state information, an event handler action and an action to perform when the event handler is disconnected. We obtain stable pointers to these (if they are not already stable pointers) and construct a closure which contains them. This construction can be safely passed to the C++ world without risk of being garbage collected in the Haskell world.

If we now go back to evtHandlerOnEventConnect, it should hopefully be clear that we are calling evtHandlerConnect with a window, a set of event IDs to handle and a closure containing the Haskell components which will be invoked when an event arrives.

Having looked at how event handlers get connected in wxHaskell, next time we’ll look at how they get called in some more detail.

This is the second in a series about refactoring parts of wxHaskell so that it will work as expected in GHCi. There’s nothing very specific to wxHaskell here – it’s much more about implementing a custom build system in Cabal, and how to use the Cabal API.

The work described is still somewhat in progress, so there are probably a few wrinkles in the cross-platform support, but it is basically working on my development machine, so it’s slightly more than half-baked…

What we are going to do is to add a build system capable of building a shared library to the Distribution.Simple. This is not a particularly clever build system – it doesn’t do any dependency tracking, for example, which makes it somewhat painful for developers. However, the main use case for Cabal is that it is an easy way for library users to install software on their machines, and in this case dependency tracking is not really an issue as the code is only built once.

If you want real dependency tracking, use Make (or better, something like Scons – if I had a pound for every Make-based build system I’ve worked with which doesn’t quite track dependencies correctly, I’d have enough to buy a paper copy of Learn You a Haskell for Great Good).

Extending the cabal build description

In the hope of creating something reasonably re-usable (if there is enough interest, I’m happy to work with the Cabal team to get something like this into Cabal by default), I’m going to add some custom stanzas. These use the “x-something” support already built into Cabal:

x-dll-sources: a whitespace separated list of all of the C or C++ source files which need to be compiled into a shared library.

x-dll-name: the basename of the shared library we are going to create. This will expand to basename.dll on Windows, libbasename.so on most Unix systems and basename.dylib on OS X.

x-dll-extra-libraries: a list of the libraries with which the shared library must be linked at runtime. This is present to stop the library user from having to worry about this. Restrictions on some platforms mean that any libraries used here really ought to be available in a shared form on all supported platforms.

Updating the list of libraries to link

When installing a Cabal package, Cabal informs GHC of all of the libraries which need to be linked for the package to run. In this implementation, this means the shared library we are building and any libraries in x-dll-extra-libraries. These need to be added to the list of libraries which Cabal passes to ghc-pkg when the package is installed, and a good place to extract this information is in the configuration hook.

The configuration hook runs when you execute runhaskell Setup.hs configure (or as the first stage of a cabal install). You need something like the following in your Setup.hs:

The configure hook is, rather unadventurously, named myConfHook and the build hook, which we will discuss at length later, and which is responsible for building the shared library, is myBuildHook.

Much of the effort in working with both build and configuration hooks is in finding where the information you need is stored and unwrapping and rewrapping the data structures (doubtless someone with greater Haskell-fu than me would do this in a shorter and infinitely more elegant way, but this works, so… whatever). The Cabal documentation is definitely your friend, and in this case, we start from UserHooks, which is the container for all of the hooks available in Cabal.

We are interested in creating a confHook function, and the documentation says that this has type confHook :: (GenericPackageDescription, HookedBuildInfo) -> ConfigFlags -> IOLocalBuildInfo. It’s also worth noting, to understand what follows, that simpleUserHooks simply gives us the default set of hooks envisaged by the Cabal designers.

The key lines are highlighted in red. The customFieldsBI libbi stanza returns an association list [(key :: String, value :: String)]containing all of the entries in the cabal file which start “x-“. The parseDLLsfunction extracts the contents of all of the stanzas whose key string matches an entry in the match list – “x-dll-name” and “x-dll-extra-libraries” in this case – and wraps them in a BuildInfo structure. We add the BuildInfo thus extracted to the extraLibsstanza.

The operation of parseDLLs is straightforward: We simply update the extraLibs field of an empty BuildInfo with a complete list of the libraries provided in the selected x_stanzas. It is worth noting that it is legal to have multiple libraries in a stanza, provided that they are separated by newlines (hence the use of lines when we are constructing the list of DLLs).

After executing myConfHook, we have all of the libraries we need to link with our Haskell package (note: not the libraries we link with our DLL!), and these will be added to the list of libraries included when the package is used.

Compiling C or C++ code from within Cabal

This requires a user hook to be executed during the Cabal build phase (i.e. when you type cabal build at the command line). As I mentioned earlier, the build hook we will be executing is myBuildHook. The example shown is from an early (but working) draft. I’ll discuss why this is not quite sufficient in a later posting, but it shows the principle.

As with the configuration hook, much of the code is simply extracting the required information from the various structures passed into the build hook. Most of the information is pretty standard for compiling any C or C++ shared library: a set of source files, the paths to search for include files, libraries to link and so on. Cabal already knows where to find compilers, linkers, archivers and the like (on Windows, GHC includes a copy of the MinGW development system, which contains a port of gcc and GNU binutils), and you can look up the full path to any of these tools using lookupProgram.

The most important lines in myBuildHook invoke the compiler and link the shared library, respectively, and are marked in red.

The CompileCxx function compiles a single C or C++ source file to object code. In the version shown there is no dependency management whatsoever, which means that every file is compiled every time cabal build is invoked. This is very inefficient for development, since there are quite a number of files to compile, and most are recompiled needlessly. A proper C++ build system would check whether the modification time of the source file or any of the files it includes is newer than the modification time of the corresponding object file. This is a lot of work to get right (most large scale C++ build systems I have encountered don’t quite get it correct!), and I have taken the judgement that since most Cabal users only compile the library the one time they install it, it is an unnecessary effort. That said, if you look in the code above, you will notice a commented out call to a function needsCompiling which checks only the modification time of the source file to compile. This is fragile (it will do the wrong thing if an include file has been changed, but not the source), but it speeds things up for development. It will not (and should not) go into released code.

I’d also like to mention the normalizePath function. This is present to fix some behaviour in System.FilePath which turns out to be, shall we say, infelicitous. The problem is an age-old one. Way back in the days of MS-DOS 1.0, which didn’t support directories, Microsoft chose to use the forward slash ‘/’ character for command line switches, which meant that it was unavailable for use as a path separator when, in about 1983, MS-DOS 2.0 came along. At that time, Microsoft chose the backslash ‘\’ character as a file separator, and software engineers across the world have been dealing with the fact that Unix chose the forward slash (some 10 years earlier) ever since.

The infelicitous behaviour comes about because of the way that MinGW (and Cygwin) try to get around this irritation. Because software originally written for Unix often hard-codes the forward slash as the directory separator, both MinGW and Cygwin treat the ‘/’ character as though it is a ‘\’ on Windows. This is OK as far as it goes, but it is fragile. In particular, many Unix derived tools generate paths with the ‘/’ separator even on Windows. When these are concatenated with paths produced using System.FilePath (or by Windows executables), you end up with paths containing both sets of separators (e.g. c:/path/to\some\windows/location.c). It turns out that some programs are not robust in the presence of both types of separators, so the only solution is to normalize all paths so that they contain only the correct path separator.

The code is shown above. Nothing to be especially proud of (in particular, I’m sure there must be something which does the same as replace in Data.List, but I couldn’t find it in about 30 seconds of looking!), but it works. Personally I believe that it would be better if System.FilePath did such normalization, but it’s probably a religious argument, and I’m not trying to flame anyone.

I think that’s about enough for this posting. I’ll be continuing shortly with a look at linking the shared library, and a digression into C++ static destructors. Apologies to those who value the beauty of Haskell and are offended by the ugliness of static destructors, but lifting of stones is sometimes necessary, even if what we find under them is not particularly palatable. Linking libraries isn’t much more attractive either😉

I’ve been spending a bit of time, on and off, over the last few months thinking about one of the most annoying bugs in wxHaskell, and how to fix it: that wxHaskell is now unusable from within GHCi.

There are actually two separate bugs: the first, which prevents things from working at all, is that on Windows, there is no DLL version of libstdc++; the second is that you cannot restart a wxWidgets session in GHCi after the application has terminated.

The first bug is a consequence of the fact that, on Windows, GHC uses the MinGW compiler suite to do C/C++ compilation and all linking (including with Haskell objects). The version of MinGW included in GHC does not support dynamic linking of libstdc++.

When wxHaskell was fully cabalized, one of the changes was to replace a complex and fragile build system for the C++ coe in wxHaskell with a much simple cabal-based build. The consequence of this is that the C++ code and wxcore Haskell 0bject code reside in the same library, which means that wxcore needs to be linked against libstdc++.

For compiled binaries, this is not so much of a problem: the static libstdc++ is linked with the other code and all works well. However, in the GHCi case, it is fatal: GHCi doesn’t know how to statically link non-Haskell object code, although it can load DLLs. When you try to load load wxcore, GHCi complains that it cannot load stdc++ (Loading package wxcore-0.12.1.6 … <interactive>: stdc++: The specified module could not be found. can’t load .so/.DLL for stdc++ (addDLL: could not load DLL))

The second problem: inability to restart a GUI, is a consequence of wxWidgets using static destructors. These are only executed when the application fully terminates, which is not the case in a GHCi.

It turns out that a good solution to both of these problems is to repackage the C++ part of wxHaskell as a DLL. The reasons are:

The C++ code in the DLL needs to be linked with libstdc++, but this can be done at DLL link time. If libstdc++ is statically linked with the DLL, wxcore no longer needs to depend on linking to libstdc++ (only wxc.dll).

If the DLL is dynamically unloaded, the C++ static destructors will be executed (as far as I can tell, this should be true for Linux .so shared libraries as well as Windows DLLs). This means that so long as the wxc shared library is loaded at application start-up and unloaded at application termination, we should be able to correctly restart a GUI.

GHCi can load DLLs, so wxHaskell should become usable on GHCi again (since the libstdc++ requirement would have gone away).

The problem, and it has occupied quite a bit of my rather limited spare time, is that this requires quite a bit of re-architecture.

Changes to compile wxc as a DLL

This is probably the simplest part, since wxc was compiled as a DLL in an older version of the build system. The downside is that I would like to use cabal to do the building, since asking anyone to have prerequisites much beyond Haskell Platform greatly reduces the attractiveness of a library.

Changes to use dynamically loaded libraries.

Most usage of shared libraries is pseudo-static, that’s to say that the addresses of functions exported from the shared library are fixed-up when the application is loaded by the linker-loader. In my case, I want to do things fully dynamically. The outline code is fairly straightforward (caveat: this is outline code – I haven’t run it!).

The problem is that I have about 2,000 functions which need to be exported in this way, and the only sane approach will be to automate things. I also need to find the correct locations to load and unload libraries, and to ensure that all function pointers are invalidated when I unload the DLL.

I now have most of the work done on changes to the build system, and my ideas thought through for the dynamically loaded libraries. In the next few entries (which will hopefully be more closely spaced than of late), we’ll look at these in some more detail.

One aside: I mentioned doing some work to enable Swig to be used as a wrapper generator for the wxc wrapper layer. I still want to do this, but it has been put aside in the “too much work right now” pile. Doing things as I am planning is less maintainable in the long run, but stands a fighting chance of being finished sometime this decade, and I want people to use wxHaskell more than I need to have an awesome system for automating the generation of wrappers over C++ code.

Edit: Please note that the issue described in this post has been fixed on newer versions of Haskell Platform, so you shouldn’t need to follow the instructions here.

Dan Haraj asked a question about installing wxHaskell on a Windows 7 machine. Since he was having problems, and since I have never verified wxHaskell installation on Windows 7, I agree to look into the problem.

Bottom line is that wxHaskell is fine on Windows 7, but there’s a problem with the Haskell Platform 2010.1.0.0 Windows installation. The issue is that the GHC version used doesn’t include C++ support, and wxHaskell needs this to build.

You have a couple of options for installing wxHaskell on Windows:

Just use an older Haskell Platform installer – these work fine.

Copy the required C++ support into Haskell Platform 2010.1.0.0 yourself from a MinGW installation. This is a horrible hack, as the files you will be copying to not correspond exactly to the MinGW version shipped in the Haskell Platform. They are close enough to work, however, if doing this doesn’t mortally wound your sense of software aesthetics.

You will need a copy of MinGW with gcc 3.4.5 installed, including C++ support. The automated installer at Sourceforge will install the same versions I used – just remember to tick the box marked ‘C++’ in the installer.

In looking at this issue, I’m not really addressing my TODO list in the correct order, but Konstantin Chugalinskiy asked a question on the subject on the wxhaskell-users mailing list. His specific question was about using wxWebConnect to provide access to a WebKit browser component from wxHaskell, but the problem is more general.

An aspect of wxWidgets which has been problematic when it comes to maintenance is that there is actually a very considerable number of valid build configurations, and this continues to increase as time passes. When we moved to the Cabalized build system six months or so ago, one change which was a consequence is that we only wrap the ‘default’ subset of wxWidgets. This has the unfortunate side-effect that some features which were previously supported are no longer available. The main among these are:

OpenGL

Styled Text Control

I would prefer for these optional components to be wrapped up as separate Cabalized libraries (so if you want Styled Text Control support you would need to do something like ‘cabal install wxstc’), so started looking into the problem.

wxDirect is not a general purpose wrapper generator

The most obvious approach is to use wxdirect to generate the Haskell bindings, and this immediately causes a problem, which is this: wxdirect was really designed as a single purpose wrapper generator. It is very closely coupled to the C coding style used by the C wrappers for wxWidgets in wxHaskell, which are hand-written in a specific style. In addition, its internal design and outputs are really only intended to generate WxCore.

In the past, shelarcy made changes to wxdirect to enable it to generate Haskell bindings for the Styled Text Control, but these changes were made in a way which really supports STC only, and which still requires hand-written bindings.

Generating bindings automatically

At the moment, WxCore relies on a set of C language bindings to wxWidgets which need to be written and maintained manually. These work well, and have the considerable advantage of generally being very thin wrappers over the C++ code. The problem, however, as shown up in a couple of earlierpostings to this blog, is that it is very easy for errors to creep in, and for new (or old) functions not to be wrapped.

About a year ago, on the wxhaskell-devel list there was some discussion on trying to generate the C bindings automatically. At the time, the discussions centred around using Doxygen to generate the information required to generate the C bindings. There was a proof of concept by Mads Lindstroem, using scripts written in Python to do much of the work. I have decided to see if I can make any progress in this direction (if it proves to be too much work, I’ll just make changes to wxDirect, but this will mean staying with hand-written C wrappers).

Looking at the Doxygen approach, one problem is that it seems to require a complete copy of the wxWidgets source repository (to get at the Doxygen sources) – plus I’d rather work in Haskell than Python (*) and I have never succeeded in getting *any* of the XML libraries for Haskell to work properly on my Windows development box.

(*) Nothing against Python – it is a fine language, but my spare time is very limited, and I prefer not to spend it re-learning a language I have mostly forgotten.

The most preferable solution would be to generate the wrapper code from the wxWidgets headers directly, since that would avoid transcription bugs and scale nicely to the problem of wrapping other wxWidgets components if needed. There are a couple of ways to approach this:

gccxml generates an XML representation of the parse tree of a piece of C++ code. It suffers from a couple of issues: I couldn’t get it to build on Windows, and I’d have to get an XML library for Haskell working on Windows. It also appears not to be very actively maintained.

SWIG is designed for exactly this sort of thing, but doesn’t support Haskell.

Now, SWIG support for Haskell would actually be pretty sweet, as it would make binding to many C++ libraries a great deal easier. I’m going to look into it – after all, how hard can it be? Probably the most significant issue to work through is that the WxCore API has been stable for a very long time now, so whatever approach to wrapping I take needs to come out with exactly what we have today (minus the bugs, obviously).