I hope this is not a chicken-egg problem or reinventing the wheel but here goes.
I have a Java application that needs to access a password protected file (actually during the application startup).
The password protected file's path that it is supposed to be accessed is in a configuration file and the encrypted password is also in this configuration file.
So the problem is how to decrypt the encrypted password to be able to access the file?
I could just hardcode the decryption key in my code and be able to decrypt it.
Pro:

It works

Con:

I can not use it since the code is not obfuscated (can not be for various reasons) and the decryption key can be found

The password can not change since the decryption would hardcoded to the application

5 Answers
5

You need to store a "password" (or password to decrypt the password, or password to decrypt the password to decrypt the password, etc ad infinitum), somewhere.

So, either you can store it

on the computer, but then it can be deobfuscated and read (if your program can do it, any program can do it)

or in the user's head (harder to get at programatically, but has its own set of problems - 12345, Post-It notes, etc.)

or in a device that the user takes with them (likely to get lost/stolen, more expensive)

Depending on what you're protecting (a bank vault, or lolcat pictures?), these approaches can be combined or used separately. You also should take into account the encryption strength, in addition to password strength - "no point in using a vault door on a garden shed which has a window in the back".

+1 good answer. I would add that if you store the password in a file on the computer, it can be protected via a file ACL. If this is a service, the initialization could be run as a user which has read access to the file but after initialization, the app would run under a different user that doesn't have read access. In general, pure software solutions can raise the bar only so high.
–
Matthew RodatusAug 26 '11 at 16:20

2

"Storing secret information—data such as encryption keys, signing keys, and passwords—in software in a completely secure fashion is impossible with current PC hardware…The trick is to raise the security bar high enough to make it very difficult for anyone other than appropriate users to access the secret data." Writing Secure Code (2nd ed.) chapter nine. Microsoft Press.
–
Matthew RodatusAug 26 '11 at 16:20

@Matthew Rodatus: Or, to quote the old proverb, "you don't need to outrun the bear - just don't be the slowest one running from the bear." Good points; ACL can be useful, but since e.g. many people run WinXP as member of Local Administrators anyway, I was assuming full local privileges. That was, perhaps, too paranoid on my part.
–
PiskvorAug 29 '11 at 7:56

At its heart, this is a question about Control. You want to control access to this KeyStore file, so that only your application can access it but nobody else can. You don't want to put too few Controls on this, because that would open up the possibility of unauthorized access. Neither do you want to put too many controls on this, because that would be too cumbersome and too expensive to use.

To allow the application, and not one else, access to its KeyStore, you need to enable the application to assert an identity.

You have the following options:

Put the passphrase to the KeyStore in a configuration file and have the application read this on startup. This allows you to control the identity of the application instance by manipulating file system permissions (the application user can read, but not write, and nobody else can read). If your KeyStore is based on only the KeyStore file, consider putting the OS permission controls on the KeyStore file and doing away with the passphrase altogether. If your adversary can impersonate your application user, they have root on your system and you have much bigger problems.

When the application starts up, have someone type the passphrase to KeyStore on the console before the application loads the KeyStore. This obviously falls squarely into the 'cumbersome' category: it prevents automated startup, and it may be attacked by bribing the persons(s) who have to know the passphrase.

Use a Hardware Security Module (HSM) to back the KeyStore so that subverting file system security on the application server has to be combined with a physical attack on the hosting facility before the keys can be used. Same considerations as above for the security of the credentials used to log into the HSM. This is where I have to disclose that I sell HSMs for a living.

Use an HSM in combination with HSM-enforced access controls so that multiple operators have to provide a hardware credential (smart card) and type a passphrase on the system console before the application can start. This is the far end of 'cumbersome', but it protects against the bribery issue in 2. above.

Note that none of these are based on adding layers of encryption: it's all about enforcing Controls on access to the KeyStore contents, based on the identity of an application instance or authorization by one or more operators.

In the end, it all depends on how important that private key is, and what the consequences are when that private key is lost or compromised. Which Controls to use should be a business decision, based on your assessment of the risks and consequences.

If against processes running with the same or fewer permissions, then a java.security.KeyStore should allow the user to authorize your process to access the key with a password of their choosing. You still need to establish a trusted path for that authorization prompt.

If you're trying to protect against processes running as super-user or with the ability to intercept UI events for the entire windowing system, then you've got to give up on a password protected keystore.

Since the password protects the user's private key, before you use the private key, you must make sure that use is authorized by the user. The most logical way to ensure an access is authorized by the user is to ask them for a password. So it's hard to understand how this could ever be a problem.

This seems to be one of the perennial problems with security. You don't want the password to be in plaintext, because if I run it through JAD and see your connection string I will be least impressed. So storing it in the application is typically frowned upon, regardless of obfuscation or not. That being said the password has to be stored somewhere, typically as part of the configuration (properties file) or inside the database that the system uses.

With the properties file approach, it should be read only for the user that needs it, the unprivileged web server account that belongs to no groups and only has RX privileges or RW as need be. Now the issue here is that the second it gets read in by the application anyone can get a hold of it, but that is necessary.

With the database we have another set of issues, if the attacker compromises the unprivileged user account that is running the database, this should not be the web server, he can than gain access to the web server credentials thus compromising two systems for the price of one. This would be a bad way to get compromised as it proves the flaw in the initial system design.