Why aren't you using encryption libraries that are already available? If you're implementing an existing algorithm, there are publicly accessible test suites. Yet, only external review of your code by multiple pairs of professional eyes (cryptographers/security specialists) can provide any measure of the implementation's strength. However, the usual piece of advice would be Don't roll yer own!
–
Deer HunterJan 5 '13 at 6:42

2 Answers
2

Yes, it is called writing research papers and getting them peer reviewed.

There is no reason you should ever write your own encryption algorithm unless you are in the field of academic research, and then you will still not "use" your algorithm for at least 4 years (that's how long it took to choose SHA-3 between the time when the algorithm was submitted to when it it was selected as the official standard) while other people look at it.

You will make mistakes that you will not even know are mistakes (do you know what a timing attack is? does your algorithm protect against it?). If someone with resources ever wanted to attack your algorithm do you think something you came up with could stand up to a team of PhD Crypto Analysts looking at it with a fine tooth comb for a week, a month, a year?

I agree that do not go for a new encryption algorithm if you want to use it for practical purpose. Since you have not mention what type of cipher you are developing i.e stream cipher, block cipher as there are different classes of attacks. For a research prospective there are number of attacks through which you can evaluate the security of your encryption algorithm.

Linear cryptanalysis

Differential cryptanalysis

Chosen plaintext attack (attacker is given ciphertext of his choice of plaintext)

Known ciphertext attack

Correlation attack

Side channel attacks

Weak keys attack

As you see there different variation of attacks which not actually test your algorithms but actually test the its physical implementation i.e. power analysis