Filling Holes in SRTM: Blackart, MicroDEM, or SRTM2BGL?

Recommended Posts

Hi all:let me start by pointing out that the following comments are in no way intended to criticize any of the authors of the tested freeware tools. As a user rather than a producer of FS utilities I am very grateful to anyone writing and sharing software. My interest is in finding a suitable combination of tools for generating the best possible mesh from the available source data.SRTM is the main source of free elevation data for areas outside of North America, and Matthew's SRTM2BGL probably the most often used tool for transforming the raw data into FS format, probably followed by MicroDEM. The big issue, of course, are the small and large void areas (holes) in the source data, and how to fill them. Both SRTM2BGL and MicroDEM offer interpolation functions as do a number of other freeware tools, such as 3DEM (www.visualizationsoftware.com/3dem.html) or SRTMfill (www.3dNature.com).Any mesh designer should be familiar with John Child's amazing DEM website (http://www.terrainmap.com/index.html#top) and it also contains a utility of his, called Blackart. Blackart has a wide range of functions, such as DEM construction from digitized contour maps, DEM editing of ASTER, SRTM, or other data, and also interpolation routines for filling holes in ASTER and SRTM tiles.I just completed a series of tests for a single SRTM tile of Patagonia, South America, and the comparison images are below. First, here's how to run an interpolation in Blackart (v 3.48), as the documentation is more geared toward Blackart's other features:1. Start Blackart and load the SRTM .hgt tile2. In the graphics window, select Image > Array Operations > Clip Negative Elevations to set all void areas (-32768) to zero.3. In the graphics window, select File > Input, then choose the number of iteration passes for one or both of the available algorithms (see John's description below)4. Switch to the control window and select Run > Run Blackart5. Once the interpolations are complete save the altered file; if you want to compile the SRTM data with SRTM2BGL, save it as .hgt.---Here's how John describes the two interpolation algorithms - in Greek ;-)"There are two choices: LSQR iterations and Laplacian iterations. The Laplacian iterations is the number of passes that the central difference approximation solver will make to compute an initial solution of the problem. This initial solution will be a low quality DEM that can be computed cheaply. This initial solution is used by the LSQR solver to produce the high quality DEM. The central difference solver is fast, so it is best to use a lot of iterations. 10,000 is not too many for most files. The second entry controls the number of iterations used by the slower but better LSQR solver. Again, it is not possible to specify too many iterations. However, the larger the number, the longer the program will take to compute. Thousands of iterations are usually required to produce a good result but this will depend on the size of your input file and the characteristics of the topography being interpolated." ( http://www.terrainmap.com/rm33.html#FAQ )---Here are the results of the comparison for the NE section of tile S48W075, with plenty of big and small holes in the raw data:1. SRTM2BGL with 20 interpolation passes http://forums.avsim.net/user_files/59154.jpg2. MicroDEM, with search radius of 2000http://forums.avsim.net/user_files/59155.jpg3. Blackart, with 0 LSQR and 1000 Laplacian iterationshttp://forums.avsim.net/user_files/59156.jpg4. Blackart, with 1000 LSQR and 0 Laplacian iterations http://forums.avsim.net/user_files/59157.jpgBig differences! Note also that LSQR interpolations reduce the overall detail of the raw data, at least in this example. I believe that choosing a higher number of iterations would reduce this effect but a test with 1000 LSQR and 10000 Laplacian iterations took more than an hour on my machine.It seems to me that Blackart is the winner here and that 1000 or 10000 Laplacian iterations, which take a few minutes per tile, should provide a solid base file for mesh making in SRTM2BGL or MicroDEM. If you have lots of time and/or only a few tiles, trying a high number of LSQR/Laplacian combinations will probably give even better results.Obviously, filling the bigger holes with other available data seems the logical improvement on simple interpolation. However, my tests with SRTM30/GTOPO30 or DTED0 data indicate that the generalization of the low-res meshes (mountains too low, valleys too high) leads to poor matches along the edges of the holes. Below is an image of a (different) SRTM tile filled with SRTM30 data. Unless fill data of similar resolution and quality are available, Blackart's approach seems to generate better looking and perhaps even more accurate representations.http://forums.avsim.net/user_files/59158.jpgI think I have to redo some of my Rocky Mountain meshes...Cheers, Holger

Share this post

Link to post

Share on other sites

Hi Holger.That is quite a tool!There is an alternative method for filling these holes. SRTM 1 and 3 arc second data can be merged ( or overlayed on ) to SRTM30 data.ftp://edcsgs9.cr.usgs.gov/pub/data/srtm/SRTM30/The end result is the holes are filled by SRTM30 data ( which is a newer replacement of the old GOTOPO30 data ). This new data can then be resampled at whatever LOD required. Some mountaintops and valleys will be using the cruder data, but the result may actually be closer to the true elevations.Dick

Share this post

Link to post

Share on other sites

Hi Dick:If I understand you correctly, i.e., filling bigger holes with the SRTM30 data, then your suggestion is countered by the last image of my post above: it just doesn't look good or is more accurate. Sorry about the long post - I guess it exceeds the average attention span, including mine ;-)Cheers, Holger

Share this post

Link to post

Share on other sites

Hi Holger.Sorry... I did just scan the post.However, any interpolation of large areas is going to be a fantasy. It may look better than the SRTM30 data, but it is not going to be accurate.I don't have a problem with that, as the terrain view is the main reason we would make mesh... as far as flight and landing, mesh is really of no consequence. We could fly from airport to airport over a world as smooth as a marble, and let the flattens and airfields jut out as required at their proper altitude and location.VFR flight is aided by mesh, as well as landclass and terrain polys and lines. It helps us orientate where we are.But there is no way any mathemetical interpolation is more accurate than the clunky SRTM30 1km data. But, I can see how you could argue it is not less accurate, as the span between data is so large with SRTM30. That would be true of voids filled that are less than 1km. For larger voids the SRTM30 data would be more accurate, although somewhat dull looking at LODs greater than 7.This may seem like hair-splitting to most readers, but I think you would understand my reasoning.It should be possible to fill large voids ( > 1km squared ) with SRTM30, and smaller voids interpolated by BlackArt. That would satisfy any objection I would have concerning accuracy. As far as looks, I'm sure the mesh massaged with BlackArt would be very nice.Dick

Share this post

Link to post

Share on other sites

Guest vlada stoje

Hi all,thanks for the perfect illustrations of different interpolators, Holger, the Blackart really has interesting results! Perhaps some next version will be able to insert also the different sources before the interpolation, so both sides of this coin would be solved? Ha, different sources, just searched the existing Aster dems and found the land of my childhoodhttp://www.volny.cz/stoje/scen/aster_srtm_01.gifthe Aster doesn

Share this post

Link to post

Share on other sites

Hi all:Dick, I agree with your assessment though, as a hair-splitter, I'd bet that the threshold for interpolation vs. filling is larger than 1km ;-) . What would help settling these questions is to make a formal comparison by using a complete mesh tile (e.g. a CDED 3-arcsec tile) as a basis, punching holes into it of various sizes, applying different procedures for interpolations and filling, and comparing the accuracy as the average (or local) discrepancy between the "true" and the modelled mesh. I won't have the time for that but I wouldn't be surprise if someone out there (outside the FS world) has done this already - I'll run a lit search on the university database.Vlada, Blackart already has some options for adding individual points of known elevation in the void areas, or merging with DTED0 data that have been interpolated to the same resolution as SRTM or ASTER. I didn't mention this above because my main point in this thread is that most mesh designers won't do anything "fancy" and simply use SRTM2BGL or MicroDEM for filling holes of all sizes. I suggest people use Blackart instead!Lucky that you've found a ASTER image without too big holes. It does look similar in quality to SRTM. However, I noticed that ASTER DEMs tend to have more inaccurate bumps and holes (you can spot a few in your image too, particularly in the flat areas), which can be more of a headache than missing data. Anyway, they could be useful for filling big holes in SRTM.Cheers, Holger

Share this post

Link to post

Share on other sites

Hi Holger.Yes, I did make a blanket statement without anything to back it, other than my intuition regarding the threshold size for filling missing data. If you could find something on this, it would be great.I think you're right about the flurry of new meshes. We are probably getting a lot of low quality BGLs... lots of errors, very bumpy oceans, etc.Blackart would help with this... and using DEM2BSQ would allow the use of a paint program to smoothen water heights for large areas. If a geographic projection of waterbodies was made into a bitmap of the same size as the DEM2BSQ output, then all heights for water polys could be made as -1000, or so, and then individually "filled" with the right height in the paint program. Just musing, here... but then our water polys could be heightless, like CFS2 water polys.Dick

Share this post

Link to post

Share on other sites

If you gentlemen will allow an amateur's comment, it seems to me that the important consideration is not so much accuracy (after all, who really knows the precise relief of his surrounding areas, much less that of foreign lands), so much as realism. What is really important is that jagged mountains look like that and not be interspersed with flat areas, that a plateau should be level and not contain random spikes.In this sense, Holger's comparison is extremely interesting in that it clearly shows which of those tools allows the creation of the most realistic altitude mesh. Blackart seems to be the clear winner.Just to add to Dick's comment about cleaning up extraneous water elevations, an important advantage to this is the dramatic reduction in file sizes - those data points in the water serve no useful prupose, and simply make for much bigger files. Some of us (me!) really dislike these multi-megabyte mesh files, and if they can easily be reduced, so much the better. Plus frame rates are enhanced withe smaller files.Best regards.Luis

Share this post

Link to post

Share on other sites

Hi all:Luis, thanks for those two very good points. I've been able to make simple water masks of large areas using free Landsat imagery but I guess not every SRTM mesh designer has the means to do the same. An alternative might be to use Autoasm's import function for the worldwide coastline data, generating a simple binary bitmap from it and merging the raw SRTM data with it. I've also tried the SRTM30 data because ocean areas are defined as 0m. Unfortunately, their coarse grid size "carves out" many areas along the coast as well, which would require some extensive prep work before merging.There's an exciting update to John Childs' Blackart on his website - http://www.terrainmap.com/index.html#topHe's been "lurking" in this forum and came across this very thread ;-)Turns out that the "blurring" of the LSQR procedure that is visible in the image above was unintentional and has been adressed in the new version. Nice to know that the FS mesh design community can give something back to the poeple out there in the "real world" of DEM/GIS.Cheers, Holger

AVSIM is a free service to the flight simulation community. AVSIM is staffed completely by volunteers and all funds donated to AVSIM go directly back to supporting the community. Your donation here helps to pay our bandwidth costs, emergency funding, and other general costs that crop up from time to time. Thank you for your support!

Donation Goals

AVSIM's 2018 Fundraising Goal

Donate to our annual general fund. This donation keeps our doors open and providing you service 24 x 7 x 365. Your donation here helps to pay our bandwidth costs, emergency funding, and other general costs that crop up from time to time. We reset this goal every new year for the following year's goal.