GlobeXplorer

GlobeXplorer provides web services access to satellite and aerial imagery over the internet. GlobeXplorer manages relationships with dozens of source imagery providers and then provides simple web access to a multi-terrabyte unified collection of images.

GlobeXplorer clients range from individual end-users with Web Map Server (WMS) clients to consumer-oriented web portals serving millions of hits a day. This high-load, high-volume environment requires a very reliable technical solution that scales up efficiently to support millions of requests per day.

The GlobeXplorer (GX) architecture does not store imagery inside the database itself, but uses the database to store metadata about available imagery. The GX middleware handles incoming image requests by first querying the database to determine image availability, then retrieving the actual images from storage and delivers them to the client. So, each image request generates one or more database hits on the database metadata.

As a startup in 1999, GX first implemented their system on Oracle 8i, but found early on that Oracle could not handle the load generated by the GX system. Informix offered to migrate the system to their database, using the “Spatial Blade”, and by 2001 the system was running completely on Informix.

GlobeXplorer continued to grow year-over-year, adding clients for their services, and imagery to their archive. As a result, their infrastructure grew as well and by 2004 they were running 11 CPUs of Informix, each of which cost about $30,000 to license. Looking to the future, GlobeXplorer recognized that further linear scaling of the system would not be economically viable — another doubling of capacity could incur a half million dollar software bill.

After researching options, GlobeXplorer settled on PostGIS as a possible alternative to their Informix back-ends and began running load tests in 2004. The engineering group created code that allowed the Informix and PostGIS systems to run in parallel, and the DBA team tested tuning options. When performance was acceptable, the system was gradually migrated to a pure PostGIS basis, and by the end of 2004 all the production databases had been moved to PostGIS.

Since then, GlobeXplorer has continued to leverage PostGIS, adding massive stores of vector data to the information available to clients: all US roads, watersheds, flood plains, and 32 million parcel polygons. GlobeXplorer serves maps from those vector tables using the UMN Mapserver. GX also uses PostGIS to process spatial data from logs to drive the acquisition of new imagery.

The GlobeXplorer system currently serves over 1 million image requests a day on average (with occasional peaks of over 5 million requests). The data preparation and content management systems use PostGIS. The team is now working to migrate the billing system to PostgreSQL as well.

PostGIS has provided GlobeXplorer equivalent performance and stability to their old proprietary systems, while allowing them to scale up their infrastructure on an economical basis. As the load on their systems has grown, GlobeXplorer has been able to add new database nodes without acquiring extra expensive software.

“We probably could not afford a commercial solution for our runtime database needs”, says GlobeXplorer DBA Greg Williamson, “Even with fewer, faster CPUs we would still be paying out a huge chunk of money.”

As GlobeXplorer continues to grow, it adds data, clients and new services, all of which add demands on their database infrastructure. No matter what needs to be supported next, Williamson says, “PostGIS is likely to be an integral part of it.”