if winvnc run as service system (started with windows before user logon)
new_rc4.key could be found on %systemroot%\system32 or %windir%\system32

if winvnc run as application.
new_rc4.key located on %programfiles%\ultravnc

anyway,
is better to create the key from the ultravnc viewer.
new_rc4.key always generated where the vncviewer started.

rename new_rc4.key ---> rc4.key and copy to place where winvnc.exe exist (exemple, folder of UltraVNC but not subfolder )

rc4.key must be strictly identical at both side (for server and viewer)

dsmplugin settings are not dynamic.
after change settings to winvnc running as service systemneed a reboot or net stop/start uvnc_service (=>uvnc 1.0.4, 1.0.5, 1.0.5.6, 1.0.6.4)need a reboot or net stop/start winvnc (<=uvnc 1.0.2)or close/open winvnc if running as application for apply change.

same release of dsmplugin at both side (viewer and server)

OS vista, known limit:

vncviewer <1.0.5.6
if vncviewer is installed on %programfiles%\UltraVNC\
require vncviewer run as administrator, then viewer can success connect with msrc4plugin-122.dsm (or msrc4plugin.dsm)
(bug of dsmplugin logging 1.2.2)

fixed with uvnc 1.0.6.4 and msrc4plugin.dsm 1.2.4

this not a problem if vncviewer +msrc4plugin.dsm +rc4.key are on folder like C:\test where read/write allowed for any users

remember hint Vista OS:
folder %systemroot%, %windir%, windows and %programfiles%, program files,
read/write access is allowed only with run as administrator even you are administrator of computer !!!----------

So did anyone ever find the solution to this problem, as I am having the same issue and have tried many different things, but no luck yet, any help would be appreciated. (same version and keys on both sides, both boxes checked, etc...)