I'm hoping to get some feedback on styling strategies in Geoserver WMS when dealing with dynamic (e.g. not stored with the layer) data and layers that have a large number of features. We've tried a number of different techniques, all with varying levels of performance (mostly poor) but can't help feeling like we're missing something (like we're going about this wrong from the out).

The Actual Question:
Is there a better way than a huge, brute force SLD to interpolate dynamically generated data against a GeoServer feature? We're basically looking for a dynamic color map that interpolates against values not stored with the layer.

The Details:
We're trying to create thematic maps that colorize each feature according to some external/dynamic value we've calculated using input from the front end. The resultant of which ends up being an interpolation of each feature in the layer matched with a color we've assigned via those calculations. These calculations change every time they're run, so there is no way to store this data with the layer.

Because the data is dynamic and not stored with the layer, the only means to translate this to Geoserver (we can see) is to interpolate by a known property of the layer...which means every feature in the layer needs to be represented in the Recode or Interpolate function. In situations where the layer has thousands of features, this interpolation can result in gigantic SLD files that cause performance issues when rendering on the client (and frankly, feels like a "brute force" method to get what we're after). Additionally, because of the nature of these calculations, there isn't any order which allows for using a "between" or similar operators.

SLD_BODY parameter with a dynamically generated SLD that uses a Recode or Interpolate function to apply colors by property to each feature. - This results in a 2-5Mb SLD file included with every request (and thus must be X-Form-Encoded/POST-ed to Geoserver rather than a simple inline GET) and renders very slowly (especially when panning zooming). Tiles themselves take upwards of 2-4 seconds to render.

SLD parameter with the same dynamically generated SLD from #1 but offloaded to the filesystem and included as a URL in the WMS request. - The size of the SLD is the same but because it's offloaded we can use a GET request and rendering is marginally faster.

Using WFS to pull GeoJSON and style on the client. - This requires a 15Mb+ GeoJSON file to be pulled from WFS, consumed and styled on the client. It does seem to work better on small areas (say for a single county) than #1/#2 but once you get into larger geometries the map becomes sluggish and/or strops responding entirely due to both the timing of the initial request and the load of adding such a large set of GeoJSON to the client.

Using the Vector Tiles plugin to pull GeoJSON and style on the client - This one seemed promising but due to the lack of viewparams support, it just doesn't work for our use case. Additionally, even prototyping this to work with Google Maps was extremely difficult and didn't always work (partial tiles, service lockups, etc).

Reformatting our SLD to make it as small as possible - We were able to get the size of the SLD down by using the in function (rather than using Recode or Interpolate) but even reducing the size to <1.5Mb (for a medium sized area) doesn't seem to reduce the rendering time by any noticeable amount.

E.g:

<Rule>
<Filter>
<PropertyIsEqualTo>
<ogc:Function name="in">
<ogc:PropertyName>CODE</ogc:PropertyName>
<ogc:Literal>060855053044</ogc:Literal>
<ogc:Literal>060374818001</ogc:Literal>
<ogc:Literal>060570001052</ogc:Literal>
...Create a rule over and over for each color grouping...ETC

So, our question is less about how to tune our Geoservers performance (it runs swimmingly for just about everything else we're asking it to do) and more about:

Is there a better way to interpolate dynamically generated data against a GeoServer feature? Is there an extension, service (WPS?) or approach generally used to do what we're after that doesn't require brute force?

The layer we're rendering in this particular case is from a ShapeFile store which really limits our options. We're going to see about changing that and taking a swipe at @JGH's proposed solution.

After spending a day or two working through the proposed solutions, we ended up taking JGH's suggestion and moved the layer(s) from ShapeFile stores into the database, created a cache table for the dynamic data, then used a SQL View to join the new database layer to the cache table. IMHO, Ian's solution is pretty darn clever and we're going to try and prototype that as well at some point...just between our low level of expertise with WPS and not being a Java shop, it's just a bit more than we're really able to manage at the moment. If/when we do use that solution, I'll post some details here on our experience.

In terms of performance here's a quick breakdown of what we were seeing for posterity. This is against a single non-production GeoServer instance over the wire (not local) so the actual timings are less relevant than the comparative between them.

2 Answers
2

You could store the computed categories (colors) on the server. You would basically have a table with sessionID (or requestID, or userID etc) / featureID / colorID. You then publish in geoserver a parameterized view between the layer and this table, the parameter beeing the sessionID (or requestID, or userID etc). The SLD then only has one entry per color (colorID1 -> red etc).

The table should be indexed on sessionID and on featureID, and you would need to do some housekeeping to remove old entries (ex, when a session expires)

Really interesting idea. I'm thinking through the implementation and where I'm stuck is how to get from the parameterized view to join that data against the feature we're rendering.
– mikeoNov 14 '17 at 2:15

Ahh I see. Ok so, I think were this gets stuck for us is that the layer we're rendering against is actually from a ShapeFile store (which means it can't be joined directly in the db unfortunately). However, this doesn't mean it HAS to be ShapeFile store, if we can move it into the database, I think this is a viable option.
– mikeoNov 14 '17 at 2:19

I'll edit the question to contain that info and I'll spend some time giving this a shot.
– mikeoNov 14 '17 at 2:20

Now, this is the kind of geoserver-vodoo-magic-sauce I was hoping to find but I unfortunately, I might need a bit of guidance on this one (we may not be magic-sauce-level-skilled yet). This WPS process would need to be compiled yes? Then installed into Geoserver. Then we'd send a WPS request (as outlined in the link) providing our offline/dynamic data set as a process-input (as JSON). Then the result of which we would save as a temporary layer and apply the necessary styles? Did I miss the mark anyplace here?
– mikeoNov 14 '17 at 18:11

10-4. OK I'll take a swing at this one too. Much appreciated Ian.
– mikeoNov 14 '17 at 18:13

Quick thought Ian, if it would be possible to post a pre-compiled .jar on your github (or maybe a "release"), it would really be huge for those of us without working JDK/maven environments. I realize you already do a lot for the community and there may be good reason for not doing so -- so no complaints, just something I noticed that would have made this a little less daunting for us. Thanks again for your help!
– mikeoNov 17 '17 at 0:06

You need to build it to match your version of geoserver and the geotools it is built with.
– Ian Turton♦Nov 17 '17 at 7:12