This week we had some fun at the Esri International User Conference but still also made time to do some rock solid coding for Python Add-in’s. One of the issues we were encountering was the repetitious and disconnected manner in which the Python Add-in Assistant works, well, not fair to blame the assistant, but more about how the XML remains static while the python addin module (your ***_addin.py file) goes through far more edits than the config.xml. Take this simple example of a toolbar with a button and a combo box, begin with:

and then add the toolbar, button and combo box:

Click save an you’ll end up with stubbed classes for the button and the combo box in your ***_addin.py (menus_addin.py in this case) file and the information specified in the UI all sitting on your config.xml. The content of which looks like:

This is definitely not cool from a maintainability standpoint and given how touchy Python Add-ins to any miss type or typographic error we set about turn this stubbing out process into one that is more aligned with a build out process.

We’ve done a fair amount of XML processing (reading and writing using Python for working with things like XML Workspace Documents and configuration files and more so we decided to take this on in much the same light. Build ourselves a set of Python classes that understand their nested nature and know how to behave to serialize to XML.

Some of this can be automated using modules like generateDS but even that seems to be overly verbose and not always perfect in picking up the details on serialization. You can also rely on your IDE for XML manipulation (we use PyCharm and it has some really nice tools included for XML handling). No shortage of ways to get from XML to Python classes.

Now on to the use of the classes. Let’s write the python code that will serialize to the same as that stored in the config.xml file. In this particular case it would look like:

Really? Just 32 lines of beautiful python, yup, that is all it takes (including imports!). Notice how native Python types are used for collections (i.e. lists) and the idea of passing objects around instead of strings for id and refID being embedded throughout the code is a very subtle but a very powerful addition.

In certain cases we have set some meaningful defaults on the classes so that we do not need to be as verbose. Then to generate the XML we just do something like:

This is of course a very simple example and we’ve taken this much much further by writing additional code that can inspect our classes and use that information to automatically populate the tool, button, menu, etc. Really awesome when you can use Python to manipulate Python to generate Python to run Python.

Now that we are auto-generating our binary toolboxes fully from code we have also been able to introduce a really nice build process that generates our “calling code” scripts, tool validator code, and then validates all of our tool code (classes and functions), tool parameters, stylesheets, referenced/supporting files used for defaults, and, gasp, documentation too at the parameter and tool level.

Some might think it overly involved to have a build process for script tools but having been building toolbox tools since the inception of this capability, back when ArcToolbox was a stand-alone application (shout out to anyone else who was building tools then!), this should just be an obvious evolution. It might not make a lot of sense for a throw-away tool or just building one or two tools but having built, well, probably more tools than come with the core of ArcGIS over the years this again just makes good sense.

While the ArcGIS product continues to evolve and improve there have been some portions that have degraded, in particular, the documentation tools for documenting tools. In 8.x this was all done in code so it was up to the developer – too bad developers generally write in API speak and in terms of nuts and bolts, not in terns of usage or usability. Issues in 9.x days where if you changed the name of a parameter your documentation got dropped, no warning, no message, it just got dropped. Fast forward to 10.x, what do we have? Documentation for tools is handled the same as editing metadata on a feature class. Not sure about that – what is the extent of a tool? and what makes sense as a thumbnail? It just seems forced and the new editor is functionally inferior to that from 9.x days.

We decided that we needed a better way, a way that allowed our developers and analysts and specialists to all work in the same manner and on the same platform and contribute to better documentation. In our case we adopted reStructuredText as our standard for documentation creation (mark down is fast to write and works really well with source control). We had actually adopted it several years ago for our Python package documentation standard for parameters and also as part of a movement towards separating content from presentation for other types of technical documentation and for generating conditional documentation. The same technology choice being used in three or four different ways means we get to become proficient and much more comfortable in that technology and learnings become universally applied.

In our build process for tools and toolboxes now we perform the necessary conversions from mark-down into mark-up compatible with binary toolboxes. Yes, it does mean needing to dive into dark areas of ArcGIS that probably were not intended to be used in this way but the rewards are great and the time saved phenomenal and the consistency across tools is outstanding – mostly due to the way in which documentation can be reused without copying – and leads to greatly enhanced usability.

The functionality for generating toolboxes, tools, and their documentation is included in our pygp package.

After working through the idiosyncrasies of python toolboxes over the last few months we’ve found ourselves saying that python toolboxes are good but binary toolboxes are better.

In our trials we did come up with some really nice patterns and processes for use with python toolboxes. Examples include: lazy imports with correct import context to avoid compilation of modules when ArcGIS loads, auto generated documentation (more on this in another post), ability to use .net executables with messaging in a python toolbox, and a half dozen or so other items that are now included in pygp.

One thing worth praising of python toolboxes is that the parameter class got some better smarts. We decided that we wild build around this and ended up developing a set of patterns for use to build script tools. So what? Well it is pretty significant because now from Python we can actually automatically build our binary toolboxes including the script tools and all associated documentation and this includes full handling for tool validators.

The whole process is pretty awesome and is saving us hours and hours of effort especially when working in multiple computing environments (dev/test/prod).

Python code building script tools in binary toolboxes. This is what python toolboxes should have been.

With the mapping functionality of ArcGIS 10 layers gained some additional functionality but at the same time they lost some too depending on how they are accessed and/or used. In 9.x days (or when using a 9.x geoprocessor object) you would get a string (ok ok, not a string but a Results object and then you would getOutput(0) and then have a string) from the MakeFeatureLayer tool and this was a FeatureLayer, more specifically, it was a GPFeatureLayer.

In 10.x days the MakeFeatureLayer still returns a Results object (yey!) but when you go after the getOutput(0) you get a mapping layer object (when done via arcpy) or an object of type geoprocessing Layer Object when done via arcgisscripting. Cool that we get an object but not so cool for legacy code since the new object returned does not always play nice with legacy coding approaches.

The good news is that you can still access the same functionality, you just need to be selective about how you go about it. The following snippet shows an approach for generating a GPFeatureLayer (dataType=’FeatureLayer’) to access the fidSet attribute (result is empty for this particular test) but also getting a map layer at the same time.

I’ve held a twitter account for some time now and have decided to use it for more than just reading newsfeeds. Follow me over on twitter @jlhumber where I will post insights into Python usually as it pertains to ArcGIS.

EDIT: I’ve confirmed that this is an actual email from esri and it did include username, password, and login location all in the same message. Groan, I sure hope online is more secure than my business account information.

Last night I received an email from Esri, an email stating that I had requested a password reset on my account. Of course this was a little disconcerting because I had not requested a password reset. Of course I thought this was spam but since my Esri global username is not the same as my email address it did seem somewhat legit. But still, seemed like spam and made me wonder, how did my details get leaked?

Then earlier today I received an email from GIS-centric newsletter. Nothing I had ever heard of before and certainly nothing I had signed up for. It seems as though Esri account information has been recently lifted and sold into the market.