0005221: Gram.token_info abstraction is too restrictrive, could be moved to Token

Description

I'm having a type regression under 3.12 due to a formerly public type which is now private in Camlp4 interface.

My use case is to compose the Gram1.Entry.of_parser and Gram2.parse_tokens_after_input, to define a grammar entry from the action of a grammar entry of a *different* grammar. This was possible in 3.10/11 as both functions relied on the following type:

(Token.t * Loc.t) Stream.t

As both grammars relied on the same Lexer module -- of course it doesn't make sense to apply one grammar to an other input stream if the lexers are different -- the Token and Loc module are shared, the types are compatible and this works fine.

At 3.12 the type was changed to:

(Token.t * token_info) Stream.t

where token_info is a private type of the `Gram` module. Two private types of distinct modules are incompatible, so my code doesn't compile anymore.

I suggest the token_info type definition could be moved into the Token module. I don't think its semantic place it necessarily in the Grammar module, it's mostly a per-token information. If it was defined in Token, even abstractly, different grammars sharing the same Token and Loc module could communicate `token_info` again.

This change wouldn't break anything as it would only expose more sharing of the `token_info` type. Of course, for backward-compatibility, an alias inside Gram (type token_info = Token.token_info) would be fine.

Additional Information

Attached is a simple example file exhibiting the problem: it compiles under 3.10/11, but not under 3.12. It is actually faithfully reproduces my use case in the Macaque project.

I committed a fix in the version/3.12 branch. I prefered to keep the token_info type close to the grammar code. However thanks to applicative functors, I've written the type in such a way that the type only depends on the location type.