Me and my friend want to sell game server-side software, but we want some sort of control, over use of that software.

For example, we want some sort of serial key system, so only verified/paid systems, can run our software. We are concerned, that software will be exploited/resold/distributed freely and we wont get any income from that.

How should we approach this problem? Is there any ready solutions? How about h-/crackers, is there any medicine against tinkering code and removing/explore/exploit our serial key system? Do we need to encrypt in some sort of way our server side software, so it will be hard to deal with? What about key/license server side solutions?

Not distributing any code, server is compiled C++ executable, running on windows machine.

Target audience is individuals, not companies. This is first time, we have no experience in selling software, but we have individuals, that are willing to pay - we just don't want our server end up cracked and distributed all around the globe, for at least year or two, after release of first version.

If it's 1), have you ever considered using the power of the law to your advantage? Basically sue the pants off any company who uses your software without permission.

If it's 2), you have a wide variety of options available. You can tie access to an API key which you sell, or issue certificates which allow access to your servers.

Since you said the software is intended for server side use it probably won't apply to you, but attempting to add DRM to any widely distributed software is an exercise in futility. Any DRM scheme can be bypassed given a determined enough attacker.

You should have a look into internet activation and 'call home' DRM schemes, but you probably have some sense that that's not exactly foolproof.

The fact is, if it's software running on a machine that someone else controls, that someone else can apply all sorts of tools to break whatever scheme you come up with.

You may find success by enlisting the client machines for help - set up a CA and issue a certificate as a 'license file' to each server. Have the clients check that key for revokation as they connect and you have a pretty robust system.

I think I forgot to mention, that we are not distributing any code, server is compiled C++.
–
DeeleFeb 13 '13 at 11:09

The second half is accurate in that case - consider some sort of call-home DRM or distribution via a marketplace? You could even set up a CA and have the client check the server cert against it, then you can revoke it if there's trouble.
–
Bob WatsonFeb 13 '13 at 11:22

You should look at YubiKey.com, it's possible to configure their devices to act as a dongle of sorts.

The way I understand it, each YubiKey has two slots on it. One of the slots (#2) can have custom authentication on it. You also have the ability to alter the method in which the custom authentication works.

In addition you should check out SLP Server at http://www.inishtech.com . They provide a secure sandbox for your application, and a few ways to prevent copying your data.