No, that should work fine to limit access. About the only issue that comes to mind to consider is the remote api, as you would need to allow access to that if you use it (eg. via roundcube ispconfig plugin on a remote server or similar).