Hi,
Vignesh from the WebM team here. I added support for VP8/9 alpha channel in Chromium. Bottomline is, libvpx does NOT support encoding of alpha channel. It simply does not understand the notion of an alpha plane and ignores it.
The way we implemented it was more of a workaround in the container:
* Encode the YUV planes separately [1].
* Form a "fake" image with the A-plane as the Y-plane and with dummy U and V planes.
* Encode the "fake" image [2].
* Mux the outputs of [1] and [2] as a single frame in the container using WebM's BlockAdditional [3] element.
You could potentially do something similar in MediaRecorder, but this will mean running two instances of the encoder and sync'ing up on key frames. Sample encoder can be found in [4]. (A not very well written) Design spec is in [5].
[3] https://matroska.org/technical/specs/index.html#BlockAdditional
[4] FFmpeg can encode VPx streams with alpha channel when given an input with alpha channel: https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/libvpxenc.c#L995
[5] http://wiki.webmproject.org/alpha-channel