we run our Jubula test cases inside a virtual machine (Windows 7).
The test executor is triggerd by an Jenkins job.

The tests are successful as long as somebody is connected to the virtual machine, e.g. via MS Remote Desktop.
If there is nobody connected the tests run, however they are not successful. The screenshots Jubula takes if an error occurs are completely black.

Is it possible to run the tests inside the virtual machine without the requirement to be connected to it?

I tried different Windows systems inside the virtual machine (Windows 7 Enterprise 64bit, Windows Server 2008 64bit and Windows XP SP3 32bit), but always the same the tests fail and the screenshots are only black screens.

Sorry, no idea what's going on there. As I said above that is close to our own setup. I don't know if there is a configuration option in ESXi which would prevent disconnected VMs from GUI interaction.

There are a few things to consider:

The Hudson/Jenkins connector can't be run as a service. Starting it via ssh might work depending on your setup. The best solutions is to autologin a test account and start the Hudson/Jenkins connector during login.

Start the AUTagent(s) from inside the Hudson/Jenkins job (and don't forget to stop them when the job is done).

You may get better performance starting the test executor on your Hudson/Jenkins master.

I'm not sure that any of the above points are the cause for your particular problem. Just check them and let's see if anything gets better if you change the setup.

we encountered the same annoying behaviour.
Our solution currently is to connect to the vm on a "native" console.

When connected per "remote desktop" all tests do work.
When leaving the "remote desktop session" then all tests result in a failure, with all their screenshots being completely black.

When connecting to the "native" console i.e. through vmware vsphere client all tests do work, even when the vsphere client has been closed in the meanwhile. (Obviously this way the native session is kept open, so this supposedly will be the reason for it to work).

Like most ui testing tools, Jubula needs an active desktop session to work correctly. This means that your tests need to be run as a user with access to the desktop (i.e. not as a service) and that the desktop must not be locked.

Most Windows versions only allow one active desktop session at a time. This means that when you log in using RDP, the local desktop will be locked. Once you terminate the RDP session, it's desktop session is also terminated. At this point, there's no active desktop session anymore. Also, when the RDP window is minimized on the client, the desktop on the server does not redraw by default. There's a work-around for this using a registry setting, which unfortunately must be set on the client side.

We do the following:

Use autologin to log in the user running the tests.

Disallow RDP sessions.

Use VNC (TightVNC) or the remoting capability of our VM server (XenServer) to watch tests remotely.

We used "TweakUI" to set up automatic login with an dedicated user running the autagent service. This service needs the "interactive" flag to be allowed to use the currently open native session.
Autagent can be wrapped as a win service by using http://kenai.com/projects/winsw/pages/Home

Another trick for Win systems:
There are some "enhancements" that allow more than one active sessions on today's windows systems.
Google is your friend.

That way it is still possible to connect via rdp with another user.
Regards,
R.O.

We had exactly the same problem, windows is geared towards human user interaction, not auto/batch processing.

What I finally did is write and run a deamon on our linux box that brings up an rdp session to the windows box continuously. The rdp on linux runs in an X frame buffer in memory so it is completely human-less. You can then even make screen shots from the in-memory 'screen', either manually or at a certain interval. I did not manage to pull off video although it should technically be possible. Both machine run in AWS by the way.

The only downside is that when you connect to the windows machine by rdp manually you will be kicked after a while by the daemon. You can of course stop the daemon before a manual connect but that is more hassle.