After further analysis I think the correct solution would be to backport the fix proposed by @jphalip there which is to always allow reference to the primary key of the model instead to avoid documenting to_field_allowed() and expose a security foot-gun.

Since the admin doesn't currently work with many to many through a model pointing to a non-primary key field (see #23862) the only remaining edge case would be dealing with dynamically generated inlines with a foreign key pointing to a non-primary key field. For this case I suggest we follow comment:1 suggestion to document that get_inline_instances() should always return instances of a subset of the classes defined in inlines.

Since the initial to_field_allowed patch introduced many regression I'll try to get feedback from the developer mailing list before committing to this solution.