I think that recycling shmem segments would fix most of the issue. alternatively we could decide to spend time implementing the data pipe thing. I am fine with going in either direction. The data pipe approach will certainly be better in the long run but also require more work to get right.

This patch set gets read of the majority of shmem allocations. It's not great but it's simple. Eventually I'd like to replace that either by a proper data pipe as we have talked about a few times, or by a proper reference counted shmem type that would handle the cross process atomic refcount in a nicer way (storing the ref count along with the metadata in the front guard page instead of the visible part of the buffer among other things).
In the mean time this should do.

My plan in the very short term is to add an atomic counter in the shared memory header that indicates how many processes have ipc::Shmem instances referring to the shared memory segment. This will make it possible to remove the ugly/dangerous hack of storing that counter at the beginning of the buffer with the data for some shmems and not others.

This type will be used in later patches to implement recycling optimizations. The idea is to have a Shmem type separate from Shmem to avoid dangerous bugs, that stores an atomic reference count. This reference count is updated manually for now.
I learned (the unpleasant way) that while it is possible to store a Shmem in a C++ type and serialize it with with the ParamTraits mechanism, what you get on the other side is a Shmem that points to no shared memory at all. As a result there is an IPDL type RefCountedShmem and a set of static methods implemented in a fake class RefCountedShm (pretty confusing, suggestions to make this better within the constraints of IPDL very welcome).

Use the RefCountedShmem type instead of Shmem for small allocs, but don't use the ref counting at all. This should exactly the same thing as without the patch, the goal being to move the uninteresting boilerplate out of the next patch which deserves a more careful review.

This does the manual reference counting and keeps a single ref counted shmem in WebRenderBridgeChild to recycle it. We could build upon this to recycle any number of shmems, but recycling one gets rid of the vast majority of allocations with all the sites that I tested this on.

Comment on attachment 8941115[details][diff][review]
Part 4 - Take advantage of the ref counting to recycle a shmem.
Review of attachment 8941115[details][diff][review]:
-----------------------------------------------------------------
::: gfx/layers/wr/WebRenderBridgeChild.cpp
@@ +643,5 @@
> }
>
> +bool
> +WebRenderBridgeChild::AllocResourceShmem(size_t aSize, RefCountedShmem& aShm)
> +{
It's probably worth adding some more comments to this function. It took me some time to reason through what was going on.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #16)
> Comment on attachment 8941110[details][diff][review]
> Add a RefCountedShmem type
>
> Review of attachment 8941110[details][diff][review]:
> -----------------------------------------------------------------
>
> Can anything bad happen if the client is maliciously manipulating the
> refcount?
As far as I can tell it shouldn't be worse than what clients can already do with regular shmems (deallocate them too early or leak them). My understanding is that if the client deallocates the shmem too early and the host tries to read from it, an assert will blow up on the host side in ipdl code as we try to look a shmem up that isn't in the table anymore.