To answer your question specifically, you can use the max_recipients parameter of the Requests Dialog to stop the user from inviting more than a particular number of friends

Note that if you're giving some sort of bonus for users accepting the requests you should also implement logic in your own app which tracks the number and IDs of the sent requests to corelate them later when processing the acceptances, as users could conceivably trigger the dialog themselves via javascript or the popup dialog and send more requests than your app allows