What's new and upcoming in PostgreSQL

PostgreSQL 9.5 Grouping Sets with PostGIS spatial aggregates

One of the features coming in PostgreSQL 9.5 is the triumvirate GROUPING SETS, CUBE, and ROLLUP nicely covered in Bruce's recent slide deck. The neatest thing about PostgreSQL development is that when improvements happen, they don't just affect the core, but can be taken advantage of by extensions, without even lifting a finger. Such is the case with these features.

One of the things I was curious about with these new set of predicates is Would they work with any aggregate function?. I assumed they would, so decided to put it to the test, by using it with PostGIS ST_Union function (using PostGIS 2.2.0 development). This feature was not something the PostGIS Development group planned on supporting, but by the magic of PostgreSQL, PostGIS accidentally supports it. The grouping sets feature is particularly useful if you want to aggregate data multiple times, perhaps for display using the same dataset. It allows you to do it with a single query that in other PostgreSQL versions would require a UNION query. This is a rather boring example but hopefully you get the idea.

Using PostgreSQL Contribs

PostgreSQL OGR FDW update and PostGIS 2.2 news

PostGIS 2.2 is planned to reach feature freeze June 30th 2015 so we can make the September PostgreSQL 9.5 curtain call with confidence. Great KNN enhancements for PostgreSQL 9.5 only users. I've been busy getting all my ducks lined up. A lot on tiger geocoder and address standardizer extension to be shipped with windows builds, story for later. One other feature we plan to ship with the windows PostGIS 2.2 builds is the ogr_fdw ogr_fdw Foreign data wrapper extension. I've been nagging Paul Ramsey a lot about issues with it, this in particular https://github.com/pramsey/pgsql-ogr-fdw/issues/25, and after some prodding, he finally put his nose in and fixed them and pinged Even Rouault for some help on a GDAL specific item.

Needless to say, I've been super happy with the progress and support I've gotten with ogr_fdw development and really enjoying my ogr_fdw use. The XLSX reading a file saved after the connection was open required a fix in GDAL 2.0 branch (which missed GDAL 2.0.0 release, so because of this, this new package contains a GDAL 2.0.1ish library. Hopeful GDAL 2.0.1 will be out before PostGIS 2.2.0 comes out so I can release without guilt with this fix.

What's packaged and where do I get it

One of the things we decided to take out of the new build is CURL support (it's still in for packaged www_fdw). We added in sqlite3 support (which opens up a couple of key spatial drivers needed by some of our clients). The main reason for the change is since we are shipping this with PostGIS 2.2, we didn't want the extra CURL dependency and none of our clients or users funding our work had a need for any of the ogr www based drivers (Google Fusion tables, CartoDb, WFS etc.). If you have issue with that, let us know.
We've also included the executables ogr2ogr.exe, ogrinfo.exe which gets built with GDAL. This we did mostly so you can see available formats and do basic troubleshooting: with command like:

ODBC Support

The most sort after driver support at least for windows users is the ODBC support (particularly for connecting to SQL Server databases). Someone wrote me about this trying to use the old odbc_fdw we talked about a while ago, and I told him to use ogr_fdw instead since its much more robust and support 9.3 and 9.4

I mentioned in Querying MS Access and other ODBC data sources with OGR_FDW it's use with MS Access, but still plan to follow up soon with a SQL Server example.
With SQL Server there are two drivers, which both rely on ODBC support. The ODBC connection form requires a system dsn. The MSSQLSpatial you can do with a DSNLess connection, but there is an extra environment variable you need to set. More on that in our followup article.

Using PostgreSQL Contribs

After installing PostgreSQL 9.4 and PostGIS following An Almost Idiot's guide to installing PostgreSQL, PostGIS, and pgRouting, on my CentOS 6.7 64-bit except replacing 9.3 references with equivalent 9.4 reference, I then proceeded to install ogr_fdw. To my disappointment, there are no binaries yet for that, which is not surprising, considering there aren't generally any binaries for any OS, except the windows ones I built which I will be packaging with PostGIS 2.2 windows bundle. Getting out of my windows comfort zone, I proceeded to build those on CentOS. Mainly because I have a client on CentOS where ogr_fdw I think is a perfect fit for his workflow and wanted to see how difficult of a feat this would be. I'll go over the steps I used for building and stumbling blocks I ran into in this article with hope it will be of benefit to those who find themselves in a similar situation.

UPDATE pgdg yum now has ogr_fdw as an offering. If you are on PostgreSQL 9.4, you can now install with : yum install ogr_fdw94

I should add I also experimented with building stuff from PGXN registry on my CentOS box, and one big peeve I have about that, is that for less than trivial extensions, you've got to go searching for the dependencies yourself. Not like some other extension systems like Node (NPM) and Python Pip that just install these things for you or warn you if they can't. At the very least it would be nice if it gave such a message rather than often cryptic ones. I know I'm spoiled, but I can dream. I'll save using PGXN for another article.

I'll start off with a couple of stumbling blocks you may have if you are new to compiling stuff on Linux. Most Linux/Unix folks can skip these sections, since they probably are already setup.
Everytime I spin up a Linux VM, it for some reason never has these things installed.

You need gcc-c++ and git

As with all platforms where you intend to compile stuff like PostGIS and PostgreSQL extensions, you need a functioning gcc-c++. You know you don't have one when you get messages like: No way to build binaries

sudo yum install gcc-c++

Should fix that

ogr_fdw not currently in pgxn, got to get from git repo

If you don't have git client already installed, just run this:

sudo yum install git

You need GDAL Develop and PostgreSQL develop

When you install PostGIS / PostgreSQL via Yum, it installs all needed to run PostGIS, which includes GDAL, but not necessarily GDAL development or PostgreSQL develop package. So if you do this:

The make installcheck part is very finicky because the postgres service account needs rights to read the data folder to reference the test dbf and on top of that you need to be a superuser
to build the foreign. So don't be too upset if your installcheck yields failure instead of:

My next test will be to experiment with UnixODBC to see if I can connect my windows SQL Server install to my CentOS PostgreSQL install.

For windows users, reading this. I've had a change of heart, and will be putting CURL support back in and with SSL support. I managed to get my curl binaries to use the packaged
EDB ssleasy32 and libeay32.dll by making sure I compile against the same version (and no higher).

Using PostgreSQL Contribs

Connecting to SQL Server from Linux using FDWs

There are two PostgreSQL FDWs (currently maintained) I know of for connecting to SQL Server from a Linux/Unix PostgreSQL box. There is the TDS Foreign Data wrapper (tds_fdw driver) which
relies on the Free TDS driver. This is a fairly light-weight FDW since it just relies on TDS which is commonly already available on Linux installs or an easy install away. Unfortunately when I tried to use it on windows (compiling my usual mingw64 way), while it compiled and installed, it crashed when I attempted to connect to my SQL Server 2008 R2 box table, so I gave up on it for the time being as a cross-platform solution. One thing I will say about it is that it accepts ad-hoc queries from what I can see, as a data source, which is pretty nice. So we may revisit it in the future to see if we can get it to work on windows.
I'm not sure if tds_fdw would support SQL Server spatial geometry columns though would be interesting to try.

The second option, which as you may have noticed, we spent much time talking about is the ogr_fdw foreign data driver. ogr_fdw utilizes UnixODBC on Linux, iODBC on MacOSX and Windows ODBC on windows for connecting to SQL Server. The ogr_fdw big downside is that it has a dependency on GDAL, which is a hefty FOSS swiss-army knife ETL tool that is a staple of all sorts of spatial folks doing both open source and proprietary development. The good thing about ogr_fdw, is that since it is a spatial driver, it knows how to translate SQL Server geometry to it's equivalent PostGIS form in addition to being able to handle most of the other not-so spatial columns.

Although GDAL is big, the good news is that, PostGIS relies on it as well since PostGIS 2.0 to support raster functionality. That was one of my ulterior motives for pushing raster into the PostGIS extension in 2.0:
There will come a day when PostgreSQL will need to reach out to vast different kinds of spatial and not-so spatial data and GDAL would be a convenient ring to do so, so lets start planting the roots..
This makes compiling and installing ogr_fdw on Linux pretty trivial if you already have PostGIS with raster support installed and even easier now that PGDG Yum packages it in the repo.

Until recently, I've only used ogr_fdw on Windows and have been very happy with it connecting to all sorts of datasources from open-street map extract, dbase files, excel spreadsheets, MS Access databases, and SQL Server, it's a real gem. That is not to say it couldn't stand for many improvements. For a good chunk of these like OSM and MySQL and SQLite, GDAL doesn't rely on ODBC and uses the native drivers directly. Recently people have been writing me about how they can use it on Linux to connect to SQL Server. Yes, my jaw dropped, Linux people want to connect their PostgreSQL to SQL Server, why the heck would they want to do that. I thought
maybe it's a good idea to try this out myself to experience first hand issues people are running into rather than simply relaying the information between people on what they tried that worked and didn't work. So here is my naive attempt to do so and distill the body of information that people have been sending me.

There are two UnixODBC drivers you can use for connecting to SQL Server. There is the TDS based one, and in theory, if you are on CentOS/Red Hat EL (5,6) or SUSE Linux EL 11, you can also use the Microsoft provided via: https://msdn.microsoft.com/en-us/library/hh568454%28v=sql.110%29.aspx which I have not attempted, but may in a future article.

We'll use the more commonly available TDS driver which I think works on pretty much all Linux/Unix and MacOSX systems. This I am doing on a CentOS 7 box.

If you installed ogr_fdw or PostGIS using, yum, then you probably have UnixODBC installed, to verify do this:

sudo yum list installed | grep unixODBC

We get this for output

unixODBC.x86_64 2.3.1-10.el7 @base

I should note, that if GDAL isn't compiled with UnixODBC support, you have a much harder hurdle to jump. Luckily as far as I can tell, I think GDAL is generally compiled with UnixODBC support on Linux
and possibly on Mac as well.

Next see what drivers you have installed already

odbcinst -d -q

On our CentOS 7 box, got returned:

[PostgreSQL]
[MySQL]

Setting up UnixODBC connection using TDS ODBC Driver

Here are the following steps. Much of this information is gleaned from unixODBC - MS SQL Server. You need to do this if FreeTDS didn't show up as an option when querying odbc manager.

Install FreeTDS Driversudo yum install freetds

This is a RedhatEL/CentOS/Fedora way, on Ubuntu you'd probably need to replace yum install with apt-get sudo yum install freetds

To do this we need to create a datasource template file and register that with odbcinst manager similar to what we did with the driver as follows:

Create a file call it tds-testmssql.datasource.template and has contents as follows. Note the [..] is the name you want to refer to your data source and should be different for each database you want to connect to.

Use the include ogr_fdw_info commandline tool to figure out the server and fdw for a table. Connecting to ODBC sources with GDAL/OGR is described in more detail here: http://www.gdal.org/drv_odbc.html I'll use my favorite example of information_schema.columns.
You should replace with what you want to connect to.

Now if you have a lot of tables, the CREATE SERVER process (and selecting from an FDW table) tends to take a while (like 30 seconds), you can winnow down the list a bit to only list tables you would ever want to connect to like so:

Someone mentioned to me that their list of tables that have compound keys didn't show up as options on Linux, but did when using ogr_fdw under windows. I haven't experimented with that to see
if I can replicate the issue.