The function match only on two of the tree constructors of the type t.

To do so, I'm requiring features like allowing 'a instead of a full name in locally abstract types to have a restricted polymorphism ([< `A | `B ]) instead of a full polymorphism ('a) in the final signature.

Additional Information

I don't really need this, but it's an experimentation and I think it would be great to have it.

Also, I don't know if it's possible theoretically. If not, I'm interesting in papers on the subject.

Note that for this to work with polymorphic variants, we would probably need to support refining constraints on polymorphic variant types. This means supporting functions like the following, which add a constraint to a locally abstract type and then refine that constraint to a smaller type:

This is a well-known limitation of locally abstract types.
In theory there is no problem (implementation may still be tricky), but the current syntax for locally abstract types makes it hard.
The syntax type 'a... you propose is indeed one I thought about, but this would mean introducing yet another new syntax for types.
The problem with (type a = [< `A|`B]), or rather (type a = private [< `A|`B]) is that is does not allow mutual recursion between private types.
Let's say that all this is under consideration, but with no easy solution in view...

Sure, I considered that too, but this gets really heavy.
let f : type a = private [> `B of b] and b = private [> `A of a]. b -> a
The advantage of
let f : type 'a 'b. ([> `A of [> `B of 'b] as 'a] as 'b) -> 'a
is that is lighter.
Also, this is not fully compatible with the current short-hand
type a b. ...
which should then be seen as an abbreviation for
type a and b. ...
Not very coherent.

I may end up with the heavier syntax anyway, rather than creating something completely new.
This still requires heavy changes in the implementation of locally abstract types.