Language Thread

SCIENCE Thread

How should I escape or cleanse user-provided passwords before I hash them and store them in my database?

When PHP developers consider hashing users' passwords for security purposes, they often tend to think of those passwords like they would any other user-provided data. This subject comes up often in PHP questions related to password storage; the developer often wants to cleanse the password using functions such as escape_string()(in various iterations), htmlspecialchars(), addslashes() and others before hashing it and storing it in the database.

2 comments

@legoscia 2016-04-14 16:20:43

Before hashing the password, you should normalise it as described in section 4 of RFC 7613. In particular:

Additional Mapping Rule: Any instances of non-ASCII space MUST be
mapped to ASCII space (U+0020); a non-ASCII space is any Unicode
code point having a Unicode general category of "Zs" (with the
exception of U+0020).

and:

Normalization Rule: Unicode Normalization Form C (NFC) MUST be
applied to all characters.

This attempts to ensure that if the user types the same password but using a different input method, the password should still be accepted.

@DavidS 2016-04-14 21:31:56

What would be an example of "the same password but using a different input method"?

@Margaret Bloom 2016-04-14 22:15:06

@DavidS, A super shiny North american Mac Book (that Joe used just before leaving) and a poorly internationalized Taiwanese internet café computer (that Joe is trying to use to download is flight back boarding card).

@DavidS 2016-04-14 22:46:15

Sounds jingoistic. :-) Thanks though.

@ruakh 2016-04-14 22:46:50

Hmm. If you do this, then you should also validate passwords to reject any that contain as-yet-unassigned characters. It would be terrible if a user uses NEWFANGLED SPACE, which your app doesn't recognize and therefore hashes as-is, and then you upgrade your Unicode Character Database and suddenly NEWFANGLED SPACE gets mapped to SPACE before hashing, such that (s)he can no longer enter a password that your app will hash to the old hash.

@Jay Blanchard 2016-04-27 19:39:32

Curious - how would a non-Ascii space get entered into the password if the application is UTF-8 all the way through?

@user3942918 2016-04-27 19:49:28

@Jay Blanchard 2016-04-27 19:59:09

Right @PaulCrovella, but if an applications and its database are setup to use UTF-8 all the way through why would you need any additional mapping?

@Reinstate Monica 2016-04-27 20:03:41

@JayBlanchard Because when you press a space bar on one machine and when you press it on another machine you might get two different Unicode code points, and they'll have two different UTF-8 encodings, without the user being aware of anything. It could be argued that this is a problem you wish to ignore, but RFC 7613 was borne out of such real-life issues, it's not a make-work recommendation.

@Reinstate Monica 2016-04-27 20:06:00

@ruakh Once you decide on handling passwords in a certain way, they must remain handled that way, or else things will break for existing use cases. If you intend to change the preprocessing method in the future, you should store it along the preprocessed and hashed representation of the password. That way, once you receive the input, you select the preprocessing/hashing method based on what you're comparing to.

@user3942918 2016-04-27 20:10:05

@JayBlanchard For the same reason you want to apply NFC - it's not always clear to a user that, for example, their phone's on-screen keyboard might be sending different codepoints that just happen to look the same as what they send via their laptop's keyboard. For a more visible example (and reason to apply NFC normalization) one might send ά as U+03AC while the other sends it as U+03B1 followed by U+0301.

@Jay Blanchard 2016-04-27 20:18:16

Thanks, those are good points to consider @PaulCrovella. I want to do some testing based on this.

@ruakh 2016-04-28 14:59:24

@KubaOber: NFC is guaranteed to be stable for future versions of Unicode, as is the "general category" of each character, provided the input consists only of already-assigned characters. So legoscia's approach is fine, even if you continue to add support for new characters over time, as long as you reject as-yet-unassigned characters.

@Jay Blanchard 2016-04-28 18:46:48

By "reject" do you mean they should be stripped out @ruakh?

@ruakh 2016-04-28 22:57:04

@JayBlanchard: Certainly not! That would cause the same problem. By "reject" I mean that you should show an error message.

@Jay Blanchard 2016-04-29 12:51:32

So, if I am understanding correctly @ruakh, the methodology you would employ is to analyse the proposed password for non-assigned characters and if it contains any of those reject the password and deliver a message to the user stating they cannot use certain characters in their password (I would tell them the non-assigned characters found in their attempt.)? I will try to find resources on the unassigned characters and try to determine what kind of UX impact that might have.

@Jay Blanchard 2016-04-14 16:08:28

You should never escape, trim or use any other cleansing mechanism on passwords you'll be hashing with PHP's password_hash() for a number of reasons, the single largest of which is because doing additional cleansing to the password requires unnecessary additional code.

You will argue (and you see it in every post where user data is accepted for use in your systems) that we should cleanse all user input and you would be right for every other piece of information we're accepting from our users. Passwords are different. Hashed passwords cannot offer any SQL injection threat because the string is turned into hash prior to storing in the database.

The act of hashing a password is the act of making the password safe to store in your database. The hash function doesn't give special meaning to any bytes, so no cleansing of its input is required for security reasons

If you follow the mantras of allowing users to use the passwords / phrases they desire and you don't limit passwords, allowing any length, any number of spaces and any special characters hashing will make the password/passphrase safe no matter what is contained within the password. As of right now the most common hash (the default), PASSWORD_BCRYPT, turns the password into a 60 character wide string containing a random salt along with the hashed password information and a cost (the algorithmic cost of creating the hash):

PASSWORD_BCRYPT is used to create new password hashes using the CRYPT_BLOWFISH algorithm. This will always result in a hash using the "$2y$" crypt format, which is always 60 characters wide.

The space requirements for storing the hash are subject to change as different hashing methods are added to the function, so it is always better to go larger on the column type for the stored hash, such as VARCHAR(255) or TEXT.

You could use a complete SQL query as your password and it would be hashed, making it unexecutable by the SQL engine e.g.,

SELECT * FROM `users`;

Could be hashed to $2y$10$1tOKcWUWBW5gBka04tGMO.BH7gs/qjAHZsC5wyG0zmI2C.KgaqU5G

Let's see how different sanitizing methods affect the password -

The password is I'm a "dessert topping" & a <floor wax>! (There are 5 spaces at the end of the password which are not displayed here.)

When we apply the following methods of trimming we get some wildy different results:

What happens when we send these to password_hash()? They all get hashed, just as the query did above. The problem comes in when you try to verify the password. If we employ one or more of these methods we must re-employ them prior to comparing them with password_verify(). The following would fail:

password_verify($_POST['upassword'], $hashed_password); // where $hashed_password comes from a database query

You would have to run the posted password through the cleansing method you chose before using the result of that in password verification. It is an unnecessary set of steps and will make the hash no better.

@Dan Bracuk 2016-04-14 16:18:08

If you hash the password with the trailing spaces, and buddy does not include them on his next attempt to authenticate, does he still get access?

@Jay Blanchard 2016-04-14 16:19:01

No. If he created his password with trailing spaces, which is allowed, he must use them on login @DanBracuk

@Dan Bracuk 2016-04-14 16:20:53

So your recommendation has a major disadvantage with respect to usability.

@Jay Blanchard 2016-04-14 16:22:47

How so @DanBracuk? If we allow the user to setup the password s/he desires, including leading/trailing spaces?

@I wrestled a bear once. 2016-04-14 16:24:51

That's why most things require you to enter your chosen password twice. If user added the spaces on accident they will figure it out before getting any further. If user did it on purpose than it's a non-issue.

@Margaret Bloom 2016-04-14 22:08:09

I didn't really get the whole point of this question-answer thing. The rule of thumb is to sanitize what the business requires to and to escape everything going in the db. Period. You have to escape the hash, nobody know how things will change in the future. Regarding the password cleansing, that's an hot topic. Google for example trims the password, I personally don't like it. But its help center surely handle more angry users than I do.

@DavidS 2016-04-14 23:05:55

@MargaretBloom, a rule of thumb is just a heuristic. We sometimes still need to think things through, like for passwords. You say "nobody knows how things will change in the future", but it seems if anything is going to change it's the way we escape data before we put it into the database, in which cases users would find themselves locked out when their passwords no longer match what we've stored. What is the danger in not escaping password hashes vs. the danger of escaping them?

@DavidS 2016-04-14 23:24:35

And maybe things are different in the PHP world, but in the Java world the rule of thumb isn't to escape everything going in the db, it's to use prepared statements. So much for rules of thumb.

@Steve Jessop 2016-04-14 23:25:39

Exactly: you will of course "escape the hash" in the limited sense of correctly passing it to a parameterized SQL query, where some code in your SQL connector may or may not do anything with it that corresponds to "escaping", you don't know and don't care. You just won't have to write any specific code to achieve that, because it's completely routine for all your SQL queries unless you've previously made some poor life decisions.

@Jay Blanchard 2016-04-15 12:06:24

The PHP world is trying to get to the prepared statements rule of thumb, but older database API's are wide-spread in the wild and have promoted bad habits @DavidS

@Louis Loudog Trottier 2017-02-27 20:40:53

What about filter_input() I personnaly filter the password input just to strip ASCII < 32.

@Jay Blanchard 2017-02-27 20:47:20

Are you filtering when hashing and when verifying @LouisLoudogTrottier? Even if you are you shouldn't because the password is made safe by the act of hashing it.

@Louis Loudog Trottier 2017-02-28 03:54:40

@JayBlanchard I filter the raw user input from a html form like so $password=filter_input(INPUT_POST,"password",FILTER_UNSAFE_R‌​AW,FILTER_FLAG_STRIP‌​_LOW) that "Strip characters with ASCII value less than 32". Then, it's processed by PHP (hash, verify) and passed to mysqli_statement. So ya the only filter i ever apply only strip ASCII les than 32 ei: NULL, [TAB], [New line], etc.. see asciitable.com

@Jay Blanchard 2017-02-28 13:38:25

Good point @LouisLoudogTrottier, but how often would a NULL byte character be entered into a passphrase? It would seem rather low odds for someone specify an octal in their passphrase.

@Louis Loudog Trottier 2017-02-28 14:14:43

i don't know either, maybe character encoding on a foreign language, miss configured machine, defect hardware or maybe just like the good old ALT-255 on windows. Or some od string like %00. Truly i don't know. I'm just paranoid (or OCD) that way. On the other hand, if anyone does used those low ascii, why not just strip them away to prevent the infinitly small chance that a NULL goes through. i appreciate the reply.

@Jay Blanchard 2017-02-28 14:16:52

I am going to perform some testing and, if warranted, I will update the answer with this information @LouisLoudogTrottier

@Louis Loudog Trottier 2017-03-01 03:52:11

I've made a fiddle a ran some test on your string and some other random string made with my fr-ca keyboard (±@£¢¤£¦¬¬¤³¦²½³½¾§¶[]}{~l­¯µ) and successfully password_verify() and password_hash() from a filter_input and it works. I don't know how or with what tools we can send null bytes so i didn't post my results. Seems like all other function failed comparing 'RAW' exvept filter_input(). I've dump the code on a sandbox => sandbox.onlinephpfunctions.com/code/… and can run it on phpfiddle. Geetings.

@Louis Loudog Trottier 2017-03-01 03:59:33

Problem occurs, and we have funny behavior (from a funny input) if we force $_POST by adding $_POST['upassword']="\01234567 "; bewteen line 2 and 3. Most of them return 'Good', trim fails (beacuse of the trailing spaces) and filter_input 'become' NULL. Maybe my way of testing is wrong or innacurate but seems like my test conclude that it is even worst to use filter input. Now we still need a way to strip NULL bytes from going to password_hash().

@Vir 2019-08-09 13:59:35

@JayBlanchard I am trimming my all user data including Password. So after reading this discussion and answer, I think I should not trim Password. Also I have read your article here jayblanchard.net/proper_password_hashing_with_PHP.html It is really very well explanation of password_hash() and password_verify() like easily understandable. Please share such more articles like this in easy language. Beast explanation I ever found of password_hash() and password_verify(). Thank you very much.