File size increase when projecting raster

Why when is it that when I project a raster from Alaska state plane zone 4 feet to Web Mercator does the file size increase so dramatically? It starts at ~29MB before it is projected and then increases to ~257MB after projection.

NAD_1983_2011_StatePlane_Alaska_4_FIPS_5004_US_Survey_Ft

WGS_1984_Web_Mercator_Auxiliary_Sphere

The only thing I can think of is that the distortion in web mercator at high latitudes is causing the raster to be "stretched" in some way and then leading to an increase in file size.

I thought I should summarize, although the post by Gabriel Upchurch summarized this pretty well – and your initial assumption [that this is caused by the inherent distortion of Web Mercator at high latitudes] was correct. I should have realized the details sooner.

In the Web Mercator projection, the scale increases as (1/(cosine of Latitude)) so at 60deg it is 2x larger (2x scale error), and your data is at about 70 degrees latitude, so the scale error is almost exactly a factor of 3. Cosine(70) = .3420. That means it is a proper implementation for a Web Mercator projection that a distance of 100 meters will be reported to appear to be 300 meters. I had not realized this distortion would be embedded into a reprojected file, but it is necessary for compatibility with the coordinate system. Multiplied in 2 dimensions, this (3x3) distortio is a factor of ~9, so your input file (29 MB) becomes almost 250 MB after reprojection.

This really emphasizes the disadvantage of using a projected coordinate system too far from its origin. If all of your work will be at high latitudes, it would be better to use a local coordinate system, assuming any measurements will be made on the map.

Note I did some testing, and verified that the ArcGIS Online web map compensates for the distortion, so it provides generally accurate measurements. (Exactly how accurate, I do not know – I would need to ask our experts…)

Also note that ArcGIS Desktop has options for how it reports measurements. It will default to "PLANAR" measurements when using the measure tool (which does NOT compensate for the scale distortion), but the tool has the option to choose "GEODESIC" which will provide accurate local measurements.

Options to consider:

The best option is to use the Mosaic Dataset to serve your imagery directly from its original format, and the application that consumes it can request any projection (AK State Plane, Web Mercator, other) and ArcGIS will project on the fly. For this, you will need our Image Extension, and I’m trying to confirm if your client already has it. The advantage is you won’t have to duplicate data on disk (either as export to Web Mercator *or* in raster tile cache format).

Sub-option here – you do not need the image extension to serve one single image e.g. an orthomosaic of a single project site – but I’m guessing you have more than one site/one image and you would want to use a Mosaic Dataset & thus the Image Extension…

If the web map needs to support measurements, consider building your web maps/apps in Alaska State Plane or another appropriate coordinate system. It is not mandatory for web maps/apps to use the Web Mercator projection, but that raises the question of what other data must appear in your maps…

If you want to proceed with services using Web Mercator, I think they can be fine for visualization at Latitude 70 degrees (but at even more extreme latitudes I think the distortion would be unacceptable). If your users expect to make any measurements, you will need to consider and test the Web Mercator scale issues carefully.

If you do proceed in Web Mercator, I’d suggest you cache directly from the Mosaic Dataset. The cache may increase in size due to the scaling, but you should use JPG format (assuming your images are continuous imagery or a hillshade of elevation data), and keep the scale changes due to the projection in mind – if you want the visual quality of a level 19 cache, you can actually start at level 18 (1/4 the data volume) and you won’t lose visual quality. Again, we should test a small area first to be sure!

Note I had forgotten that your data file was 32 bit float. If this is elevation data, then the raster tile cache format is not usable – it is limited to 8 bits, 3 bands. We can share elevation data as a compressed 3D surface (different cache format), so if you need to share elevation values, this is a separate discussion. But note that using a dynamic image service, shared from a Mosaic Dataset, solves this…

Also, looking at your image, raster_cut_small contains more columns and rows; the cells are smaller, and there's a higher level for pyramids. All of this results in more data contained within the file.

Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

For sure the extra size has to be coming from the increase in rows and columns. But why is this happing? I would think that it would have more or less the same number of rows and columns as the original because:

It appears you are confusing comments by two different people. You are exactly correct about the cell sizes since there was a conversion from feet to meters between the two systems. The cells are still the same size so this is not why the datasets are different sizes.

Did you symbolize the NoData as I (Gabriel Upchurch) suggested? My guess is that the dataset was rotated during the transformation which resulted in a large areas of NoData cells being added in the corners.

Thanks for checking the NoData. I did some testing this morning and was able to reproduce your issue and think I have figured out what is occurring. It appears you are encountering the common issue of projection distortion. In the case of Web Mercator the distortion is largely in distance and area and with your data being so far from the system's origin, the distortion is sizable. To put it simply, a 0.228 meter cell in Web Mercator does not cover an equivalent area on the ground as a 0.75 feet cell in Alaska State Plane. To represent the same area on the ground while complying with the specified output cell size, the software is adding additional cells. The behavior is further exacerbated because you are going from a local coordinate system that is highly accurate in terms of distance and area to a global coordinate system that is highly inaccurate in terms of distance and area.

The additional cells are not obvious because the software does a good job of maintaining the pattern in the data. I didn't have any AK data so I generated a random raster for testing purposes and I was unable to visually determine that cells were added. To confirm that cells were actually being added during the projection process I used the Raster to Point tool.

This first screenshot shows points representing the cell centers in the Alaska State Plane dataset.

This second screenshot shows points representing the cell centers after projecting to Web Mercator.

I wondered the same when Forest created the thread - he even mentioned that as a possibility. Alaska would definitely experience a high degree of distortion in web mercator - thanks for providing examples.

It does feel like this is the only possibility, other then the tool just having some error that is expressing it's self in this situation. But I would think that the tool would compensate for the distortion as one moved to different areas of the projection (kinda the point of projecting). I can manually set the cell size larger, like .4m and produce cells that are still smaller then .75ft when displayed in Arc, and not have loss in image quality.

I sent a sanitized raster to Coby, the esri person on this thread, for testing. So, we will see what they say...

what hat compression is being applied to the output image? Is it the same as the source. Under the Environments there is a raster storage setting called compression which specifies the compression used on the image. See the help topic

Depending in in the source compression and the resultant image you could see these size changes.

There is something fundamentally wrong, but so far I can't figure out what it is. Most of the dialogue above is correct - it's not a difference in compression or units - but the number of rows and columns of your output should not be dramatically different than the input. I can't figure out why that would happen. It almost seems like you have a custom projection, and something in that projection definition is confusing the software - e.g. it automatically figures out that 0.75 feet = 0.228 meters, but then when reprojecting, it "forgets" that the units are changing, so it's outputting 9 identical pixels for every single input pixel (oversampling from 0.75 units to 0.228 units of the *same* units).

A few questions (although I do not believe that these explain the problem)

1) why are your rasters in a database? I typically recommend against that - rather, store as *.tif on disk and access them via a mosaic dataset. Is there a reason you can't do this?

2) Is there a reason that you MUST reproject and duplicate your data? If you use a mosaic dataset, you can project on the fly and not consume disk space or processing time...

3) have you reproduced this with other rasters, or is this one doi\ng something unique?

If you could send me your input file - or a smaller one that is doing the same thing - I'd like to try to reproduce it.

1) I just put them into gdb for the projection processing as I have run into weird and unknown problems in the past when working with the tifs (this may still not be a problem. Last time I did this we still had a mixed environment 9.3 and 10.1. Now we have almost retired all of the 9.3 stuff)

2) The data must be projected for use in web maps. In the past we had a map service of a raster dataset that reprojected on the fly, but it was fuzzy when viewed at small scales. Now maybe a image service off of a mosaic dataset would not have this problem, but I understand that we would need an image server extension for ArcGIS server. The current plan is to have a cached map service on the hard drive of server in the same projection as the web mapping application.

3) This is a problem for all of our 2015 and 2012 ortho images.

Sadly, I can not share any part of the data as it is confidential; however, I will try and find something that I can share with you that has the same behavior.

[disclaimer - text here has been edited to correct a misstatement I made previously]

This may appear that Project Raster is in error but (as noted in my summary posted 10/12) it is actually honoring the definition of the Web Mercator projection which results in significant (but predictable) distortion at latitudes far from the equator.

I thought I should summarize, although the post by Gabriel Upchurch summarized this pretty well – and your initial assumption [that this is caused by the inherent distortion of Web Mercator at high latitudes] was correct. I should have realized the details sooner.

In the Web Mercator projection, the scale increases as (1/(cosine of Latitude)) so at 60deg it is 2x larger (2x scale error), and your data is at about 70 degrees latitude, so the scale error is almost exactly a factor of 3. Cosine(70) = .3420. That means it is a proper implementation for a Web Mercator projection that a distance of 100 meters will be reported to appear to be 300 meters. I had not realized this distortion would be embedded into a reprojected file, but it is necessary for compatibility with the coordinate system. Multiplied in 2 dimensions, this (3x3) distortio is a factor of ~9, so your input file (29 MB) becomes almost 250 MB after reprojection.

This really emphasizes the disadvantage of using a projected coordinate system too far from its origin. If all of your work will be at high latitudes, it would be better to use a local coordinate system, assuming any measurements will be made on the map.

Note I did some testing, and verified that the ArcGIS Online web map compensates for the distortion, so it provides generally accurate measurements. (Exactly how accurate, I do not know – I would need to ask our experts…)

Also note that ArcGIS Desktop has options for how it reports measurements. It will default to "PLANAR" measurements when using the measure tool (which does NOT compensate for the scale distortion), but the tool has the option to choose "GEODESIC" which will provide accurate local measurements.

Options to consider:

The best option is to use the Mosaic Dataset to serve your imagery directly from its original format, and the application that consumes it can request any projection (AK State Plane, Web Mercator, other) and ArcGIS will project on the fly. For this, you will need our Image Extension, and I’m trying to confirm if your client already has it. The advantage is you won’t have to duplicate data on disk (either as export to Web Mercator *or* in raster tile cache format).

Sub-option here – you do not need the image extension to serve one single image e.g. an orthomosaic of a single project site – but I’m guessing you have more than one site/one image and you would want to use a Mosaic Dataset & thus the Image Extension…

If the web map needs to support measurements, consider building your web maps/apps in Alaska State Plane or another appropriate coordinate system. It is not mandatory for web maps/apps to use the Web Mercator projection, but that raises the question of what other data must appear in your maps…

If you want to proceed with services using Web Mercator, I think they can be fine for visualization at Latitude 70 degrees (but at even more extreme latitudes I think the distortion would be unacceptable). If your users expect to make any measurements, you will need to consider and test the Web Mercator scale issues carefully.

If you do proceed in Web Mercator, I’d suggest you cache directly from the Mosaic Dataset. The cache may increase in size due to the scaling, but you should use JPG format (assuming your images are continuous imagery or a hillshade of elevation data), and keep the scale changes due to the projection in mind – if you want the visual quality of a level 19 cache, you can actually start at level 18 (1/4 the data volume) and you won’t lose visual quality. Again, we should test a small area first to be sure!

Note I had forgotten that your data file was 32 bit float. If this is elevation data, then the raster tile cache format is not usable – it is limited to 8 bits, 3 bands. We can share elevation data as a compressed 3D surface (different cache format), so if you need to share elevation values, this is a separate discussion. But note that using a dynamic image service, shared from a Mosaic Dataset, solves this…

Web AppBuilder does not publish, so I don't understand what you are trying to do. and no, it also does not reproject. However if you have an image service published (through image server) from a mosaic dataset, web appbuilder can request any (known) projection, and the server will reproject on the fly.

Can you start a new thread and explain more completely what you have, and what you're trying to do? Also let us know what version of ArcGIS Desktop (ArcMap? ArcGIS Pro?) you are using