Aggregated Live Feeds in ArcGIS

I recently spoke with ESRI’s Derrick Burke and Paul Dodd about a methodology they’ve developed to aggregate live data feeds in ArcGIS.

Derrick Burke (l) and Paul Dodd (r)

Derrick Burke is the Technology Team Lead in ESRI’s Technical Marketing department. He holds a BA in Geography from SUNY Geneseo, a Masters in Geography, Urban Planning, and Sustainable Development from UNC Charlotte, and an MBA in Finance from the University of Redlands. Derrick has worked at ESRI for more than eight years, first within professional services as a developer and then in technical marketing focusing on creating prototypes on new technology and presentations.

Paul Dodd is the GIS Systems Administration Team Lead in the Technical Marketing Department. A Computer Science Major at California State University, Paul has more than 25 years of experience in the computer industry working with Mainframe, Mini, and Micro computer systems and software. For more than 11 years at ESRI, Paul has working with ArcSDE in conjunction with various Oracle and Microsoft database products.

You’ve developed a technique you’re calling “Aggregated Live Feeds”…can you tell me more about it?

We have developed a methodology on how to aggregate Internet accessible data (eg. USGS/NOAA xml, GeoRSS, and so on) as services through ArcGIS Server. We load the aggregated data into layers within ArcSDE, expose them as Spatial Views, and serve through ArcGIS server in near real time. Any client that can consume ArcGIS Server services can leverage these services (ArcGIS Desktop, ArcGIS Server, ArcGIS Explorer, etc). This technique was developed in response to the desire to show near real-time data (such as weather data) and analysis within all of our ESRI products.

A running load script.

What are the benefits of using this methodology?

Depending on the feed requirements, data is processed and loaded on the server side every few minutes to every few hours, taking the load off clients. Continuously needing to poll for fresh data can carry a heavy penalty, especially in browser-based applications. Processing feed data on the server side allows the client to poll for data only when needed, via standard ArcGIS Server protocols. Managing these feeds centrally can ease the demand on network resources by using a few systems to download feed content from the Internet, rather than potentially a large numbers of users. Clients can then access these local services as operational layers or fuse them with other base maps and operational content.

How was this built?

Our methodology uses simple batch scripting with a handful of public domain command line utilities to download and pre-process the feed data. The scripts then use ArcSDE command line functions to push this data to the database. The scripts also incorporate logic to track the process, making sure they run as expected. If a load should fail, an alert email is sent to an administrator. Scripts run every 5 minutes, 30 minutes, once an hour, or once a day depending on the appropriate need. The current methodology can handle shapefiles, raw ASCII (like CSV and some custom formats), and XML (like RSS, GeoRSS, and CAP). There are even utilities that allow the scripts to handle de-compressing files. Once the data is pushed into ArcSDE, ArcGIS Server services are authored and served, monitoring/notification on the availability of these services are provide through the service monitor available on arcscripts.esri.com.

Weather data feed.

Can you give the readers of this blog some examples of how could this methodology be used for scientific applications?

Sure, these aggregated live feeds have been used in applications ranging from homeland security to environmental analysis. For example, in one of our latest demonstrations a light weight browser application will call an analysis ArcGIS Server geoprocessing service to perform plume modeling (eg. a contaminant leak) based upon an aggregated ArcGIS Server service containing the latest wind velocity and direction. The analysis produces a plume which can then be chained to other ArcGIS Server analysis such as identifying the demographics of the area.

Do you plan on sharing this with the ESRI user community?

We plan to post this methodology on ESRI’s ArcScripts, blogs, and the new resource center. We’ve had many clients interested in doing this and in fact some clients have created their own tools that perform similar functions.

Any possibility this will lead to new functionality in ArcGIS?

This methodology was designed to work with previous, current, and future releases of our software because it’s more of a methodology than customizing our core tools. Because this idea is becoming more popular, the development team is thinking about supplying aggregator tools as part of the core software, but that project is still in design.