I thought about that, and tried upper/lower casing some parts of old passwords accepted by the popularity/bad password/anti-entropy filters, or substituting letters/numbers, which while bad should produce an entirely different hash (hashes being what they are). Still, A/D figures it out.

Either

A). they're using a bad hash (good hashes won't be 'similar'),

B). they're unencrypting old passwords and inspecting them for patterns/similarity

If this setting is enabled, user passwords meet the following requirements:

The password is at least six characters long.

The password contains characters from at least three of the following five categories: [ ...]

The password does not contain three or more characters from the user's account name.

None of this has anything to do with partial matches against password history. The only one that remotely has anything to do with that is (surprise surprise): Enforce password history

Which does: determines the number of unique new passwords a user must use before an old password can be reused. The value of this setting can be between 0 and 24; if this value is set to 0, enforce password history is disabled. For most organizations, set this value to 24 passwords.

This is for complete password comparisons. Again, not partial comparisons.

red-moon's company is running some non-standard software that's performing this additional security check.

Mind you, there is a "Store passwords using reversible encryption setting" - but it's still not leveraged for this kind of check. I wouldn't be surprised, though, if that setting is being used in combination with some third-party software vendor providing an additional layer of network login security.

My online banking converts the password to lowercase .... They also require the password to be between 6-12 characters made of entirely of letters and numbers. That's right, none alpha-numeral characters are forbidden. I feel like I should complain.

I've often noticed that the more important the password is, the shorter it has to be. My password pattern for 2011 is two common words stuck together, suffixed with 2011, followed by another 2-symbol sequence. Last year, it was the initial letter from each word from a verse of a favorite hymn. In either case, the password easily exceeds 16 characters - which works fine for reddit, slash dot, Facebook and so on - but not one of the financial institutions I do business with can cope with a password that exceeds 16 characters. Some of them won't let you exceed 8.

American Express was like this about a year ago, I wrote them a nice letter explaining that I didn't feel comfortable using a sub 12 digit password that couldn't have special characters in it and a month or so later they changed their password policy to allow longer passwords with special characters. It may have been a coincidence or it may not have been, but what do you have to lose besides a couple of minutes to write them a letter explaining your concern.

I should really write a letter to my bank. Passwords must be entirely numeric and I believe between 4 and 12 digits. The closest thing they have to making up for it is requiring your security question on an unregistered computer, which is still not a secure alternative for passwords, as the answers are all stored in plain-text (you can view the answers from your account when changing them)...

Think about the reverse: Assume this password is an illegal permutation then calculating the possible original passwords from it. If their hashes match none of the hashes of the actual original passwords, you're aokay.

I'd suppose they could store old passwords in plain text (for purposes of this discussion, I'll consider passwords with reversible encryption to be plain text). Keep only a hashed version of current passwords, and a plaintext list of old passwords. You get the old password when the user enters it to change to a new password.

Obviously, this is a bit of a security hole if people are reusing the same passwords in multiple places, but that's already venturing into risky territory.

I would imagine there is a check before digesting and checking. So something like:
IF uppercase & lowercase & number = true, generate hash, compare hash to previous hashes = VALID or INVALID previous hashes.

Hmmm... If that's true, then they would have to be storing the original passwords in clear. However, I'm not sure that it is - they require me to choose a new password every 90 days at work, but what I actually do is choose a new one each year, then append the 2-digit month at the the end of it each time I have to change it. For example, my passwords last year might have been "porkchop01", "porkchop04", "porkchop07", etc.

Another way of doing this is splitting your password into two and hashing each part separately. If one of the parts is the same then your password isn't different enough.

Of course this comes with certain risks. We now have a known length for the first part which makes it much easier to crack if we get hold of the hashes. And if we can get the first half then it helps us guess the second.

They could store old passwords in clear. The password changing process then would be:

Ask for current password

Ask for new password

Verify current password (via comparing hashes)

Compare new password to old passwords

Store hash of new password to DB

Store old password into DB (in clear text)

Note: I'm not familiar at all how passwords work in AD so I could be completely lost here. But that process would be relatively secure (no currently valid password is stored in clear), and still allows checking for password repetition.

If you're talking about network log on passwords, AD uses kerberos see "Client Authentication". If you're talking about website passwords it could just be pulling from a cache.
*found confirmation, clarified answer.