Wednesday, 28 November 2012

This post has a bit of a misleading title (it's more the wrapper API we're improving than the documentation) but whatever, we'll run with it.

So just to explain the previous screenshot (and probably my java-related tirades on Twitter and Google Plus for anyone who happens to follow me there) ...

That screenshot was "visual" confirmation of some MapGuide API wrapper enhancement work I've been doing to address some of the pain points of using the MapGuide API (namely Java, but this work has some cross-pollinating benefits for .net as well)

The Problem

For the 15 people in the room who actually use the MapGuide Java wrapper API in its current form (I kid, though it could possibly be true), you've probably had to deal with these annoying issues:

The Java proxy of MgException being a checked exception. Thus having to pollute your java code with "throws MgException" in calling methods or having to try/catch an exception that probably isn't thrown 99% of the time and if you do catch it, you're probably just gonna log or display that exception anyway.

Java class method naming being a verbatim transplant of their C++ counterparts (ie. They're in UpperCamelCase instead of lowerCamelCase)

Needing to have the web-based API reference on hand as doxygen commentary is lost in translation to the target language. Auto-complete in your Java IDE of choice gives you no documentation, because none is transferred across.

Needing to have the web-based API reference on hand to also find out if a given class or method is deprecated, because nothing in the proxy classes will tell you that to trigger any compiler warnings

Though the memory leaks were plugged for the 2.4 release, the other problems are still present. Despite my own reservations about Java (the language), Java (the platform/ecosystem) can't be ignored. Just because we're using SWIG to churn out bindings to 3 different languages doesn't mean we can't do some tweaking here and there for a given language. We already do this for .net (to support properties and serializable exceptions), so why not give the Java wrapper some love as well?

The Solution

For those who are wondering how we actually build bindings to the MapGuide API in 3 different languages, here's a high-level overview:

As already mentioned, SWIG is the tool used to generate the language bindings to the MapGuide API, but in order to enforce some level of encapsulation in the wrapper APIs (because there's C++ internals in some MapGuide API classes that shouldn't be exposed in the wrapper classes), we run a custom pre-processor (IMake.exe) through the MapGuide C++ headers to generate a "sanitized" input file for SWIG to do its thing.

If you ever look at a any MapGuide C++ header and wonder what those PUBLISHED_API, INTERNAL_API and EXTERNAL_API markers are for, they are for IMake to generate an encapsulated view of that particular class for SWIG. Similarly, the __get and __set markers you see at the end of some method declarations are hints to IMake to generate .net properties for that particular method.

The same IMake tool is also used to generate the constants.php for PHP and the necessary constants source files for .net and Java

The solution consists of 3 parts:

Modifying SWIG to support our java-specific requirements

(Ab)using some SWIG directives for purposes beyond their original scope

Modifying the IMake preparation tool to take advantage of these (ab)used SWIG directives

Modifying SWIG

This part was simpler than I thought. The SWIG source code was not as hostile as I originally thought and hacking the java module to support what we need was relatively straightforward. SWIG was basically modified to support 2 optional java-only flags:

To omit the "throws MgException" clause in any java method generated

To output java methods in lowerCamelCase. Since all MapGuide API C++ methods are guaranteed to be in UpperCamelCase, this is a simple case of lower-casing the first letter of each method that SWIG writes out.

These flags are optional, because if we are going to provide a new enhanced Java wrapper API, we'd still want to keep the existing one around for compatibility purposes, with the enhanced wrapper being an opt-in choice.

Normally, these 2 directives are used to alter class/method visibility of the respective generated proxy classes

For example, if we apply this SWIG directive:

%javaclassmodifiers MgMap::MgMap() "protected"

We get the following java proxy class method declaration:

public class MgMap
{
...
protected MgMap()
{
....
}
}

Similarly if we apply this typemap:

%typemap(javaclassmodifiers) MgMap "private"

We get the following java proxy class declaration:

private class MgMap
{
...
}

The trick with these directives is that that SWIG does no validation of the visibility string, meaning we can put any arbitrary content in there, say things like javadoc comments and annotations (or XML comments and attributes in .net). In fact, this technique is what the authors of SWIG recommend as a way of being able to document your proxy classes

Meaning with these directives, it is now possible for us to not only pass down (converted) doxygen comments, but also pass down deprecation intent as well, because sometimes an API reference is not enough to tell you something is deprecated. You ideally want csc or javac to throw a compiler warning at your face while compiling against the wrapper API library to truly drive the point home that you should not be using this class or method.

Modifying IMake

So we have the means to transfer documentation and deprecation from the C++ classes, but surely we aren't going to manually transcribe the documentation fragments of the hundreds of classes that are in the MapGuide API? Of course not!

The IMake tool already does a pass through all the MapGuide C++ headers. And if you look at the constants.php generated by IMake, you can see that it does pick up and collect doxygen commentary along the way (albeit dumped verbatim into the target language, making it un-usable as javadoc or .net documentation). So the IMake tool was modified to do the following:

Translate these collected doxygen fragments to their javadoc or .net counterparts

Write these fragments out as the aforementioned SWIG directives for the target language

Other assorted codegen magic

Sadly, Java being the dinosaur of a programming language that it is compared to C# (in my opinion), does not support something similar to partial classes, which would've made augmentation of generated java proxy classes so much simpler, so in order to add java.util.Collection and java.util.Iterable support, we have to once again look at SWIG directives for ways we can augment the generated java classes.

The MgException class (from which all MapGuide exceptions inherit from) extends a pre-defined java AppThrowable class, which itself extends from Exception. As a result, all proxy exceptions are checked as a result. Making these exception classes un-checked as a simple case of re-basing the AppThrowable class to extend from RuntimeException instead.

The End Result

So how well do these changes fare? We can verify by running javadoc against the generated proxy Java class source files. Have another look at that same screenshot of the javadoc from our "enhanced" Java wrapper API:

And also a simple reminder from IMake that we've still got lots of room for improvement in the API documentation content

If we dive down to the method level, here's what we currently have in doxygen:

And here's what gets transplanted to javadoc

As you can see, most of the key doxygen directives have matching/compatible javadoc equivalents allowing for easy conversion by the IMake tool.

All MapGuide exceptions are now un-checked, as they extend RuntimeException

For .net, we get the same result as going through the DoxyTransform tool. In fact, this API enhancement work will render the existing .net documentation approach obsolete, as this approach does the same thing but supports multiple target languages.

So the javadoc output is proof that we have working transplanted documentation, now comes the important bit: making this documentation easily consumable from within a Java IDE, which is incidentally where I'm currently scratching my head.

What is the official/recommended way to consume Javadoc in a Java IDE (Eclipse, Netbeans, etc)? Is there an official way? Or does each IDE have its own way? I'd appreciate some pointers from someone more knowledgeable in Java tooling than myself. Feel free to comment below.

At the moment, from what I can gather there's 2 ways to bundle up this documentation for consumption within a Java IDE:

I tried both approaches through IntelliJ IDEA and it seems to pick up both forms:

Here's the result of ctrl+Q with the -sources jar

And the result of ctrl+Q with the -javadoc jar

But I'd still prefer to only have to choose one method that will work on all the major Java IDEs. I'm just not sure which one it is. I'm currently leaning on the -source version, but I'd still like an "official" answer.

But what about PHP?

You might have noticed that PHP is a conspicuous absence from this MapGuide API enhancement work.

Well the truth is that PHP poses the following problems:

Besides constants.php there is no actual PHP source files to "compile" into. So there's no actual place to drop our translated doxygen comments into.

There is no officially defined standard for documenting PHP code. Even if there was, the real aim of this API enhancement work is to improve integrated API documentation (ie. From within an IDE). If all the PHP documentation solutions ultimately produce something similar to what we already have in our API reference (a web-based documentation source), then what is the real point? We already have that!

What's left?

Besides answering the Java IDE documentation question, this API enhancement will need a requisite test run on Linux (especially the SWIG/IMake stuff). All the viewers still work as before on Windows post-SWIG/IMake surgery, indicating the wrapper APIs haven't been negatively affected in any way.

Also I have to draft up an RFC for these enhancements, for eventual inclusion into the next major release of MapGuide Open Source (2.5). If everything pans out, this next release will have 4 Web API options:

PHP

.net (with XML documentation files for Visual Studio)

Java (the old crufty one, kept for compatibility purposes) with Java IDE documentation support (whatever it will be)

Java (the enhanced one with the changes described here) with Java IDE documentation support (whatever it will be)

Sadly, Mono did not want to cooperate with us despite our code (and the SharpDevelop version we're using) sticking to the Mono compatible subset of the Base Class Library + Windows Forms. So the zip distribution (which you would install for Mono) does not include the IronPython REPL functionality and uses the existing FDO Expression Editor implementation on Mono.

Local resource previews

As part of eating our own dogfood, previews for certain resources will now use our own built-in map viewer component. The main benefits of using a locally-based previewing mechanism that:

It is generally snappier as we don't have to spin up an external web browser to show the preview

It is smarter about the initial view if we're previewing layers where the scale range is not 0 - Infinity

Our preview UI has more niceties over the browser-based previewing mechanism

If this feature is not working for you or you prefer the old way of previewing things, you can turn this option off in the Application preferences.

Updated SDK

This is the first release in the Maestro 5.0 series to include a matching SDK. This includes some new samples

A basic sample Add-In demonstrating how to integrate with the Maestro application

All these samples have been migrated to .net Framework 4.0 / VS 2010 (You can use the VS 2012 express editions as well if you so choose). You cannot use the Maestro 5.0 API in a version of the .net Framework that is older than 4.0.

As I hinted to when I started on the road to Maestro 5.0, the new SDK also contains a build tool to "roll your own" LocalNative provider, which lets you use the MaestroAPI over the official MapGuide API. This is a much more convenient way than me having to build a LocalNative provider for n specific versions of MapGuide

Simply point to where your official MapGuide .net API assemblies are and hit Build to build your LocalNative provider. Then you can use this provider by registering the dll in the ConnectionProviders.xml of your MaestroAPI application.

Please Note: Because this is still a beta, the Maestro APIs are not set in stone. APIs may still change from now until the final release.

Other cool stuff

We've tweaked the WMS Configuration UI to better handle some of the more obscure WMS servers I've encountered in my journey thus far.

You can now edit the FDO logical class name/description

You can also swap the class name/description individually or in a batch. This is a convenience function for some WMS Servers where the layer names actually end as the FDO logical class description instead of class name when processed by the WMS FDO provider.

We've also made the FDO class name encoding/decoding in Maestro to behave more like its FDO counterpart so that problems like this should no longer happen anymore. We've backed this up with the same test suite and data used to exercise the name encoding/decoding in FDO proper. Basically, you shouldn't get strange looking FDO class names anymore. They should get properly encoded and decoded.

When deleting a resource with dependent resources, we're now nice enough to tell you what they are

For folders, we just give you a catch-all question which may or may not be true, simply because we don't want to waste CPU cycles computing affected references if we happen to be trashing a big folder.

There is also a new context menu option for selected resources that allows you to compile a full dependency list for a given resource

This produces a list of resources that the selected resource either directly or indirectly depends on, which can be saved to a text file

This feature combines nicely with an improvement to the Package builder to allow packaging an explicit list of resource ids instead of a particular folder. Because sometimes, you just want to package a specific resource and everything it depends on, instead of a particular folder which may then package a whole bunch of other junk which you don't particularly want.

On the theming front, ColorBrewer color ramps can now be reversed if you so desire.

We've improved your ability to rapidly translate Web Layout and Flexible Layout documents, but previewing such documents (or any browser-based preview actually) in the AJAX/Fusion viewer will still default to using the en locale. You can now choose what locale that browser-based previews will default to in the Application preferences.

This has the effect of basically tacking a locale=your_selected_locale to whatever preview URLs you launch from Maestro

Note: FDO Toolbox is still a semi-retired project. Don't expect any major new features. I've put this release out because I believe every new version of FDO should at least have a matching version of FDO Toolbox that uses that particular version.

Monday, 5 November 2012

Eating your own dog food, also called dogfooding, is a slang term used to reference a scenario in which a company (usually, a software company) uses its own product to demonstrate the quality and capabilities of the product

For the next beta of Maestro, we're doing some dogfooding of our own. The new Map Viewer component, will now be the default method for previewing the following types of resources across all connection types

Layer Definitions

Map Definitions

Watermark Definitions

Where previously, a web browser would be launched (if using a http connection) or a mg-desktop based viewer would be loaded (if using a local connection), for such resources we have our own preview UI built on top of the Maestro Map Viewer component (which also works in Mono)

The main benefit of this method of previewing becomes apparent when you try to preview layers with restricted scale ranges. The default method (launching the AJAX viewer) for such layers will usually result in seeing nothing as we don't have the ability to pass on the initial opening scale to the AJAX viewer nor do we have the required information to adjust the preview bounds. That's why as a workaround, we put in a "Zoom to Scale" command under the "Tools" menu of the preview Web Layout to let you easily zoom into the layer's visible scale range.

With the new preview method, the preview map viewer will automatically adjust its initial scale to fit into the layer's scale range, eliminating the need to manually zoom into the layer's visible scale range.

This feature is optional, and if you still prefer the old way of previewing resources, you can simply uncheck the option.

Next beta is almost here. There's some Mono-related bugs/annoyances to fix up