If you see a popup, this means that Negotiate failed. The whole point of waffle is so the user never sees a popup. But if you do see one, then the domain credentials is what you want to enter. There're two things that will happen from here - either the client
is still trying to Negotiate, in which case it will try to call LogonUser locally and send a ticket to the server, or it gave up and does Basic auth, then the server will then call LogonUser, which tries the local SAM and then Active Directory.

Ok. So for the dialog to work, the credentials to be entered have to corespond to an existing AD/etc account. I guess I'll go and setup a test-environment to check this out.

I assume that when a client, which is not connected to a domain starts the negotiate process, he will get the un/pw dialog. Here the user will enter his AD credentials and the authentication is tried with this username against the AD (and trusted domains)
to which the server is connected. Is this right? If so, does a user specify his domain by entering the username like "DOMAIN\user" ?