I'm planning on using scripting for automated UI testing. Main application is written in c#, and I want it to be scriptable, so I can do everything end-user can do, but programmatically.

I'll avoid opinion-based questions like "What do you think of software that provides an interface for scripting, like VBA macros in Excel?", or "Can this be future of all programming, big and small?" although some inside would be great.

Here goes the question:

Which is most suitable to the purpose of building such an interface for your own application, dll-based approach or by parsing own scripting language?

If you are going to create a gui recorder, you should generate the script actions directly from your application. This way you are able to generate the most meaningful scripts and the scripting interface can be made user-friendly.

It is not an option to have some generic script recorder that listens to all UI events and tries to generate a script from them.

I don't think there's any good reason to invent your own scripting language. There are several languages that are designed specifically to be embedded in other applications (eg: tcl, lua). Save yourself the trouble of inventing yet another language and instead consider adapting one of those.