$\begingroup$"And i decided to create my own protocol about that." what motivated this change of mind? Generally, it's very dangerous and irresponsible to invent your own crypto protocol.$\endgroup$
– ThomasMar 10 '13 at 10:23

$\begingroup$I think custom protocol is unknown to cryptanalyst. And for this reason , it will be more secured. And SSl protocol is very flexible , and has many cihpers . I want to create custom protocol which will use one symmetric chiper and for key exchange RSA will be used . And this custom protocol will not be flexible . I dont invent anything :) i will use ssl protocol scheme i think , but i think about to store private keys where . In that situation i think to store private key in server. But i dont know advantages or disadvantages of it$\endgroup$
– Taleh IbrahimliMar 10 '13 at 10:50

$\begingroup$Secure protocols are designed to be secure even if they are known. Security through obscurity is not real security, at least not in the cryptographic sense. However, you are right that the flexibility of SSL/TLS is a valid security concern (in the sense that it complicates both the protocol and the implementations). Many existing SSL/TLS implementations will allow you to restrict the set of allowed protocol versions and cipher suites. Do that, and ISTM you will have accomplished what you want.$\endgroup$
– Henrick HellströmMar 10 '13 at 13:46

4

$\begingroup$"And for this reason , it will be more secured" a protocol that wasn't analyzed is much more likely to have undiscovered weaknesses than one that has been analyzed a lot. So this argument makes no sense.$\endgroup$
– CodesInChaosMar 11 '13 at 10:16

$\begingroup$"I think custom protocol is unknown to cryptanalyst. And for this reason , it will be more secured." If the protocol is unknown to cryptanalysts, then there is no way to know how secure it is. By contrast, SSL was carefully constructed by industry experts to be as secure as humanly possible and has been extensively analyzed and tested. The odds that you implement something more secure than SSL are essentially zero. Most likely, it would turn out to be horribly broken (as early versions of SSL turned out to be).$\endgroup$
– David SchwartzMar 11 '13 at 12:36

Your mobile device and your server already have SSL implementations. Concentrate on using it correctly.

Your comparison of SSH and SSL shows that you don't really understand what's going on. Specifically, with SSH, there are two unrelated key pairs in play: the one for server authentication (private key on the server, sent to clients and stored there), and the one for user authentication (private key on the client, stored on the server before it can be used). (The client machine's private key can also matter but only when doing host-based authentication which is quite rare.) See What is the difference between authorized_key and known_host file for SSH? for a more detailed explanation. In the case of SSL, there is no user authentication, only authentication of the server machine (and optionally of the client machine, but that's not normally done when you browse an https website).

The server will have a private key. Your mobile device needs to have the corresponding public key to ensure that it's talking to the right server. You can include the key with your application, but make sure you're able to update the key in case you want to connect to a different server. If the server needs to authenticate the mobile device or the user of the device, you will need to have a private key on the device, or a password, or some other authentication mechanism.

Understanding these aspects is the really important part. Figure out who needs to authenticate with whom, who should trust whom and who can trust whom. This will tell you which public keys you need to copy or validate. Use SSL, and make sure you use it right: the devil is in the public-key infrastructure.

I think you're doing this for fun, and to learn a thing or two about programming crypto primitives in practice. To answer your question I believe the public key should stay on the mobile device but if you're using multiple clients then you may need multiple key pairs. As others have pointed out, chances are you won't end up with much security. In any case post your final protocol here or on Security.SE (make sure you follow question guidelines though) and these guys will take it apart ;)

$\begingroup$And to make myself clear I'm not trying to discourage you from writing your own protocol, quite the contrary. Just don't rely on it for anything serious and you'll be fine.$\endgroup$
– rathMar 19 '13 at 9:21