For simplicity reasons I would like to specify a number (of interfaces) of messages by predefining the parameter and result type. So that if you work with message X you know what the params and result type are.

Such as

public interface MyMessage extends Message<String,String> {}

That works fine, now I would like to have a common base class which can implement any of the messages.

will not work, as the compiler complains that P and R are not bound.
Introducing them as additional parameters works again, but at that moment the resulting class will not be cast-able to BaseMessage or the desired MyMessage.

I would like to have a generic factory method, which would be given the message class, the parameter and the result object and will return an instance of BaseMessage which implements the MyMessage interface. Like in the follwing

I am afraid such a generic factory is not possible, because you can't instantiate generic types like

T a = new BaseMessage<T>();

You have to tell the compiler the exact generic type parameters.

What you can do is create instances of each concrete message type, and store them in a map with the corresponding class tokens as key. Then the factory method can look up the right message and either return it directly, or (if you need separate instances) use the message object as a prototype to create the instance to be returned.

We have a somewhat similar design in our current project, with the difference that the concrete subclasses are each existing classes, with a distinct name and (unfortunately largely - but not fully - duplicated) implementation. So we can use standard classloading and Spring to instantiate them.

You haven't specified how the factory decides which implementation class to use, or whether the implementation classes need any custom initialization logic or parameters. Assuming simply instantiating the registered implementation class using the default constructor is sufficient, you can do: