I'm a "normal" programmer (oxymoron, I know). I have no experience with hacking, but I need to get some information from some knowledgeable people on this subject.

I have a somewhat complex mathematical algorithm consisting of about 1000 lines of C++ code. If I compile and optimize it, how difficult will it be for someone else to reverse engineer it and discover how the algorithm works? Will it take a few days, or a few months, or a few years?

Are you concerned about someone to understand how your algorithm works, or to get the actual (equivalent) code you wrote ?From a technical point of view, you shall imagine someone with a debugger, that execute your program step-by-step and see the low level instructions as well as the changes to the data structures. The complexity of guessing how the algorithm works depends on your algorithm in this case.

Anyway I think you are more concerned about someone to use some automated process to de-compile your software and get your code or something equivalent. In that case you can consider to use some code-obfuscation technique.

I'm concerned with someone gaining enough understanding of how the algorithm works that they could replicate it themselves. Honestly, if I had the original C++ code for 1000 lines of something like the triangulation code, it would take me a couple of days to figure out the nuts and bolts of what all was going on. If I had to look at the assembly code, though, game over! However, there are a lot smarter people than me out there, and I know decompilers can get you farther than simple assembly code.

So , for a similar 1000-line piece of code, what's a good estimate for the time it would take to figure out the compiled algorithm?

n0on3 - you mentioned obfuscation. My understanding of obfuscation was that it was primarily helpful for interpreted code (Java, VB, etc.), since it mangled function and variable names. Code that's been compiled to binary machine code should not have any meaningful names left, right? It's all just push/pop/move/etc. statements, right? Can an obfuscator still help?

Along those lines, can you recommend any good software protection programs? I've tried VMProtect, but whatever it does to my program makes it look like a virus afterwards to some of the major AV packages.

I'm sorry, i can't recommend any software, i just read about these techniques in the past but never had the need to use them. Anyway, compiled code is as you said all about instructions, and what you don't want is someone to look at them, or inspect with some tool, and guess what your algorithm is like. What i thought was that maybe you can still trade something, like changing your implementation that probably match common programming best practices, inserting real instructions that makes your routines harder to understand or don't match what automated tools recognize, whitout changing their results. Obviously this is algorithmically useless, pure computational overhead and decrease performance, and since you were talking of optimized code you won't like this, but maybe you can find an acceptable tradeoff, if deny reverse engineering is really important for some reason.

First off I have to caveat this advice with the statement that I am a Reverse Engineering newbie. I would say based on what you want accomplished "keeping someone from replacing the algorithm" and the complexity of the code in question that successful reverse engineering would depend mostly on your implementation. Is this algorithm code going to be part of the main program executable or a separate file? If it is included in the main exe and fully integrated into your program honestly very few people are ever going to be able to replace that algorithm by reverse engineering. I am sure some experts would be able to but it would require significantly altering the flow of the executable using assembly code. Now on the other hand if you put the algorithm in a separate file using a debugger it would be fairly easy to trace the program flow and change algorithms. Of course it makes it a lot harder for you to ever upgrade your algorithm if you integrate it into your main executable.

@Neophyte: I think the sample you posted can be reversed within a few hours - a few days, as assumed before. There were much stronger protections already reversed, so this shouldn't be too hard, though it really depends on the knowledge and experience of the reverser.