I'm not sure what the issue is.
In [sig type t end with type t = [`T of t]], the last t refers to int, so clearly they do not match.
So I suppose the problem is in the error message ?
For unification errors, we detect name conflicts and print internal ids in that case. This could be helpful here too.
Note that the error message you get with -short-paths is slightly better:
Error: Signature mismatch:
Modules do not match:
sig type t = [ `T of t ] end
is not included in
sig type t = [ `T of int ] end
Type declarations do not match:
type t = [ `T of t ]
is not included in
type t = [ `T of t ]

Note that in the signature, t is expanded as int, but not in the type itself.
I think this is due to the fact the definition is not added to the reference environment in the last case; this could be improved.

pr6634.diff is a first attempt at fixing the problem, using the new "type nonrec" keyword.
The current version only works if the original definition is the first one in its recursive group, and gives the following error message:

Error: Signature mismatch:
Modules do not match:
sig type t = [ `T of t ] end
is not included in
sig type nonrec t = [ `T of t ] end
Type declarations do not match:
type t = [ `T of t ]
is not included in
type nonrec t = [ `T of t ]

A full(?) solution would put all the changed definitions at the top of the signatures, merging them in a single type nonrec.
Of course, one could be concerned about the noise involved in doing so.