To help newly registered users get more familiar with the wiki (and maybe older users too) there is now a {{Welcome to the wiki}} template. Have a look at it and feel free to add it to new users discussion pages (and perhaps your own).

FlightGear Headless

Status (08/2013)

I only need the graphical output for the real aircraft and want to disable the unnecessary graphical outputs (to reduce work load).[1]

— Rahlf, Jonas (ext.)

I'm interested in the capability of doing multiple builds with different versions, branches and options and in doing some kind of automated testing on the resulting builds[2]

— Pat

I'm working on a prototype of this - with no X11 / VLC required at all. It will load scenery tiles (for ground-intersections) but no views or rendering. But don't be in a rush for something to appear in Git, it's just an experiment.[3]

— James Turner

Make subsystems create-able and removable from commands. Only some subsystems are supported so far, since many have non-default constructors or other complexities.

With this, change, it's possible to dynamically add and remove the traffic-manager at runtime, for example:

allow running flightgear in a server/test mode with only some subsystems, and no rendering

Obviously supporting a standalone 'fgcanvas' would be quite a small extension from those. I'm not worrying about dynamic dependencies or automatic subsystem creation for the moment - I expect the user / defaults to have defined a set of subsystems that work without crashing You're correct of course that Nasal has many assumptions about subsystems, but I think that can be improved incrementally on the Nasal side.

For the test mode, I really want to start Nasal-the-langauge very early, without loading all the modules in Nasal/ immediately. (Or maybe load a 'safe' subset). That's going to take some thought and I didn't get that far yet![5]

— James Turner

could some thought be given to producing a benchmark suite for Flightgear. It would need to take in all of the, by now well known, variables - making it by no means a simple beast to manage.
If this could be automated in some way it would be much easier to capture, and then submit, consistent data. [6]

Regression Testing in FlightGear

While introducing unit tests and regression tests into the FlightGear project has been repeatedly brought up by several long-term contributors [1] and core developers [2], it isn't yet in wide or regular use in FlightGear, even though it is generally understood to be a worthwhile addition to FlightGear in order to do automated testing of individual features, for example when preparing releases [3].

And while there are indeed some minor build tests provided by both, the SimGear and FlightGear projects, such test cases aren't really commonly provided our updated by developers when introducing modified or new code.
Also, these are just low level tests for specific APIs - and do not lend themselves to be used for testing high level features.

Increasingly, FlightGear users are facing issues that are highly specific to their usage of FlightGear so that it isn't directly or easily possible to reproduce certain issues without exactly reproducing possibly an entire flight including identical startup and runtime settings, a fact that is also frequently acknowledged by FlightGear core developers [4].

This is however not only a tedious and long-winded process, but also a process that may require certain usage patterns and background information or a specific set of skills (such as for example landing a specific aircraft on an aircraft carrier).

In fact, the corresponding bug reports are often fairly long winded and complicated in that they try to provide all information necessary in order to allow developers to redo a certain flight segment that resulted in an error (see for example [5] or [6], [7]).

These obstacles in debugging such highly specific issues are also highlighted by core developers to severely limit the troubleshooting process
[8].

Background

FlightGear in its current form is an application that was primarily designed as an interactive graphical simulator, in other words, it is meant to be used by a user sitting in front of one or multiple screens, controlled by means such as a keyboard, mouse and other optional hardware such as joysticks/yokes and possibly also rudder/yaw pedals.

While confining FlightGear's design and use cases to this standard use scenario was of course very valid and feasible (as this is definitely the primary use) this restriction isn't necessarily ideal or even appropriate for the project to eventually be able to leverage itself for increasingly important purposes such as automated unit testing or automated benchmarking of individual FlightGear components in order to do regression testing.

This is an approach that is already used by the jsbsim project to some extent [9].

This RFC is meant to discuss the possible merits and approaches of allowing FlightGear to be used non-interactively, i.e. in an automated fashion such as for example by invoking it via shell scripts, so that FlightGear doesn't necessarily have to rely on user input or even a graphical output window in order to do a certain, well-defined and limited job, such as for example running certain subsystems for benchmarking purposes or by running scripted flights to fly standard patterns in order to generally help test aircraft that are considered for inclusion in upcoming FlightGear releases.

While there are certainly various thinkable scenarios for employing such facilities in other interesting contexts, this RFC will merely focus on the benefits for FlightGear itself.

Introducing Regression Tests to FlightGear

The task of introducing regression tests isn't that easily achieved in FlightGear's case:

FlightGear has largely become an independent system and platform, so while it would be fairly straightforward (but still very tedious) to introduce individual unit tests in order to validate the correct behavior of low level C++ components, such as the SimGear APIs, it wouldn't really be that easy to properly test the various abstract, high level features that are provided by FlightGear as a functionality provider and simulation framework/platform with all its various subsystems providing support for abstract features.

In fact, conventional regression tests would inevitably fail when it comes to supporting base package resources, simply because FlightGear is the sole target platform for these resources.

While base package resources do generally make use of well-understood and established technologies or standards (i.e. textures, XML, scripts, 3D models, text files etc), it is only the specific combination of these resources inside FlightGear, that define a real purpose and use.

So, doing proper regression testing for such high level features would be very difficult without writing lots of redundant test code, which would probably end up being a maintenance burden in the long term - probably resulting in a situation similar to the current one, where tests simply end up being neglected and ignored at some point.

Thus, this discussion of bringing regression testing to FlightGear favors an approach where FlightGear itself is used as the regression testing framework.

So this isn't about doing low-level unit testing for individual FlightGear C++ code, but much more abstractly do regression testing by making use of the FlightGear platform to test abstract FlightGear features by making use of FlightGear's native support for technologies such as XML, scripting and networking.

Goals

Leverage FlightGear as its own regression testing framework, for purposes such as for example:

Approach

Due to FlightGear's extensive support for flexible software interfaces (such as e.g. networking, scripting and XML), FlightGear can in many scenarios theoretically already be used for serving as its own test platform.

In fact, the major obstacle really limiting FlightGear to be used by automated/scripted tests is its reliance on having a graphical output window available and opened.

If FlightGear provided an option to be run in non-interactive/headless mode, so that it wouldn't necessarily create a visible output window but could just run silently in a shell environment, it could already be easily used by shell scripts to do simple things such as for example profiling the fgfs process while running a specific Nasal script non-interactively and automatically terminating afterwards.

It's worth pointing out that this is indeed already possible: Nasal scripts can terminate the simulator by invoking an fgcommand, so this really isn't that much off the table and would facilitate scenarios where Nasal scripts may run certain test suites and automatically report status back to the caller (shell script). So, this would be just one scenario for running fgfs non-interactively in order to profile the Nasal interpreter.
Another possible use might be scripted flights to have aircraft fly standard patterns or instrument approaches, while using a network interface such as the telnet facility to monitor the state of the flight during all phases of the flight.

Also, FlightGear's reliance on user input via means such as the mouse/keyboard and other hardware peripherals doesn't really pose a real problem, because all of these inputs are already internally handled by a combination of XML and scripting, so that emulating arbitrary user input by making use of scripts or by automatically writing to the property tree via network sockets is fairly straightforward and could also be accomplished by running shell scripts, that may for example invoke "netcat" specifically for this purpose.

The recent additions to the Autopilot and Route Manager systems in FlightGear also make it increasingly feasible to create completely scripted test flights for automatically doing certain portions of a flight without relying on user input.

Milestones

allow FlightGear to be run without creating a visible GUI window, i.e. in "headless" mode, using a --headless parameter Pending

allow FlightGear to be optionally compiled and run without any sound support/dependencies (OpenAL) Done

allow replay buffers to be saved to a file in order to be replayed for automated test/demo flights, so that users can share saved replay buffers when reporting a bug Done (supported via the flight recorder subsystem)