"And just when you thought the whole Stuxnet/Duqu trojan saga couldn't get any crazier, a security firm who has been analyzing Duqu writes that it employs a programming language that they've never seen before." Pretty crazy, especially when you consider what some think the mystery language looks like "The unknown c++ looks like the older IBM compilers found in OS400 SYS38 and the oldest sys36.The C++ code was used to write the tcp/ip stack for the operating system and all of the communications."

Most of the time, it's relatively easy to see which programming language was used, by taking a look at the calling conventions: are parameters passed using the stack, or by using registers... If the stack is used, are they pushed from left to right, or right or left... etc...

Sure you can modify your compiler to change your calling conventions, but it would make it impossible to call external libraries + there is no real benefit (= it doesn't result in better code). Also, since C compilers compile to native code, it's still possible for a reverse-engineer what the code is doing, despite the modified calling convention.

I doubt those guys wrote their own compiler. They probably used some more obscure programming language for that piece of code, whatever the reason might be...