Posted
by
timothy
on Saturday August 07, 2010 @02:00AM
from the everything-has-a-downside dept.

jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."

Since you run memcached maybe you can answer this. From what I've read it seems like an alerted (or just savvy) admin could easily detect this kind of futzing and (if you're lucky just) inject honey directly onto your machine. Attempting to maliciously write to memcached seems equally problematic. You'd never know if you were successful or just seeing what someone wanted you to see. How off base am I?

The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.

I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.

They likely don't have security personnel. I wouldn't be surprised if there were some "just get in the way" or "security can be added later" or "who cares" or any number of other statements made that clueless developers later on regret.

I doubt most admins look and if memcached was open it's virtually a feast of access. You wouldn't necessarily find it easy to jump from memcached to the rest of the system but you could gather a lot of information and alter data going to users and internal systems. Depending how memcache is being used it could be a pretty big exploit. The hardest part would be finding the right data keys. All mine are md5 hashes of a per site secret token combined with the given key so pretty difficult but I'm sure the kind

that was due to the slow rate of terminals at ~8 chars/second back in the day. copy would have taken you 2 times as long to type, and don't forget each char had to be sent to the computer and then sent back to your display. meaning typing copy took at minimum 1 second at 8 baud.

Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.

All this is is stupid admins doing stupid things story and those are dime a dozen.

That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too).Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers;-)

A good tool is designed in such a way that the correct use is the easiest. Does memcached, for example, default to only accepting connections from the local host and require other IPs to be explicitly added? That would be trivial to implement and, if done, would require the admin to implement something like the correct security policy. In contrast, defaulting to accepting connections from anywhere means that the admin can use it incorrectly without needing to do any thinking, but needs to think before using it correctly.

It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.

I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

That's like saying knives should be designed to be dull, because users are idiots, and will cut themselves.

It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.

Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair strai

I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

Perhaps utilities like rm should be rewritten to use -i (or similar) by default then? Or how about we ought to remove them outright since they might be used incorrectly?

"Any piece of software can be used "insecurely" if configured incorrectly"

Not that you don't have a point, but you are doing nothing but supporting the argument you tried to debunk.

Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused

I think that in your rush to rebuke the parent with scathing sarcasm, you have misinterpreted their point. I don't see anyone suggesting the software should install in a secure, perfectly functional manner, with no configuration whatsoever. The only quote I saw was saying that it's preferable for the software to be installed without a working configuration, so the administrator must explicitly configure it for their purposes. If they fail to configure it securely in the process, that's their own failing. Wh

Seriously? Your point is that Gentoo doesn't do enough hand-holding? This is a distribution that has built its reputation around NOT doing excessive hand-holding.

That having been said, Gentoo does leave services that have security implications unconfigured, but I'm having trouble understanding how memcached is such a service. This isn't a system configuration issue, but a network layout issue.

Finally, there is this:

The admin may not be aware that memcached is not designed to be exposed directly to the internet.

True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

If you're that oblivious, how the hell is making you fill out a config file going to help you?

Because the config file section that needs to be modified has comments explaining the risk? Not everyone is going to go and read external documentation on a piece of software before installing it. If you're going to do that, why even use a distribution in the first place? Any documentation you read is l

True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

I can think of a number of network-facing software packages in the *nix world that are usually configured, by default, to listen on all interfaces, because that's usually what the end user wants. There are some particular issues I'd like to point out, but if you're feeling exceptionally annoyed you may

The Gentoo example appears to install a functional but insecure configuration by default. If the user simply starts the service (or possibly reboots the server without doing further configuration and it auto-starts at the next boot), then tests their application, they will be overjoyed that it's working and move on to their next task. The admin may not be aware that memcached is not designed to be exposed directly to the internet.

Insecure is in the eyes of the beholder. If the machine is on a firewalled net

It's pointless to design for the biggest idiots you can find because they keep designing better idiots. Memcached isn't an app for average users. It's a programmers tool. If people that should know better are still idiots then let the damage be on their heads. Why waste resources trying to save idiots from themselves.

I say we let people self destruct early so that we can stop wasting resources. Let bad developers get themselves fired, let drug users overdose, etc. Eliminate the chaff instead of coddling eve

Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.

If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.

You secure memcached the same way you secure any cache: by restricting direct access. All you have to do is put a proxy between the memcached server/instance and the internet and the problem is solved. If you don't have command-level access to the memcached instance you can't mine it, period.

Many very large websites don't bother to do this, which is very bad security practice, as the slideshow clearly demonstrates.

The weakness isn't really with memcached. To say memcache is insecure is like saying your d

You have to actively develop and deploy software which uses and trusts memcached before this is a security hazard.

In other words, while it might be nice if it was secure by default, you now need two groups of people to fuck up at the same time -- system administrators and software developers -- both of whom, as part of their job, are supposed to think about security.

Yes memcached defaults to only accepting connections from the local address. From memcached.conf:

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1

If you have not specified otherwise (e.g. if/etc/memcached.conf is missing), memcached listens on all interfaces yes. If you install memcached from a Debian-derived distros package repository, then it will create a default/etc/memcached.conf file with the above, default, configuration specified.

All this is is stupid admins doing stupid things story and those are dime a dozen.

I'll say. I have been looking at using memcached in conjunction with the mod_auth memcache module. I know it exists and have done very cursory investigation. I do not have a working prototype yet but I do know my predecessor built one a year or so ago. My very cursory glance already made it startlingly clear that this would be an issue, any competent sysadmin really should have spotted the same thing.

The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.

"netstat -lpn" lists a lot of stuff that is not exported, you can get closer with "netstat -tulpen" or whatever, but even then you only have a very very rough overview about what you are exporting. For example you could have samba (or apache or whatever) running, which you want, but have it export all of '/' instead of just the directory you want. There is no easy way to find that out without digging through a lot of config files.

If you're an admin -- particularly one for, say, PBS -- and you're setting up a server, don't you think it'd be worth asking how that server does its communication? And it's not like this is undocumented -- from the wiki page on setting up a server [google.com]:

By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

And maybe I am just "clever", but this was pretty much obvious to me within about two minutes of learning of memcached -- I'm amazed anyone actually ran it this way. Sorry, but this isn't a case of poor UI design -- if you're employed as a system administrator, i

NetworkingBy default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.

Here's an idea that won't impact performance: At startup, issue a big multi-line warning if the IP addresses that are getting bound aren't on a Private Internet [faqs.org]:

The Internet Assigned Numbers Authority (IANA) has reserved the
following three blocks of the IP address space for private internets:

Because in a lot of scenarios you might have a memcached server that uses public address space. You'd just use iptables to only allow ssh for remote management and then only allow your webservers to connect on your memcached port. No need for a big expensive hardware firewall. But I guess a warning wouldn't _hurt_ anything? Why not just "***WARNING - Only your webservers should be able to connect to memcache!"

The MS way usually is to pretend that it's easy. The docs consist of a leaflet that explains how to open your cd drive and place the cd the right side up, and how to handle the mouse to make the cursor move to the "Ok" button.

In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue.
This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them.
[fd: that's my name on the slides]

There's a deeper issue at play here as it relates to shifting apps and platforms away from your own hardware/networks. Developers are now often responsible for deploying apps onto cloud systems where they don't have experience with network-security or the tools for protecting network-based services, and this is an obvious difference from the traditional network/app split that occurs in most corporates. It doesn't help that memcached (by default) binds to * but they do make this pretty clear (also, remote enumeration of the cache is genuinely a debug feature).

Man pages help, but when the defaults don't aid developers we need to a rethink both of the software (memcached) and the systems were it's not running securely (cloud platforms).

The deeper issue that's exposed is that your mythical developers are ignoring the fact that most attacks come from inside. Mostly through rouge employees or compromised machines. They are operating on the fallacy that you can have one big happy subnet and everybody attached it is a team player.

I don't use memcached but it sounds like one should have your memcached server machines on their own LAN and multiple network cards in the client machines.

I refuse to give even the boss access. I've even designed systems that wouldn't give me access except under very specific conditions such as a second admin also entering their credentials from a known system during business hours. Some data needs to be treated with extreme care.

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1

Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)

It's slightly mad that software like this, which is designed without security, would use TCP per default, instead of local Unix sockets (access to which can be controlled with standard Unix filesystem permissions on the containing directory (careful about relying on permissions on the socket itself have any effect - not portable)). Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).

Sorry, I don't agree with that.Memcached is not meant for single-server configurations. It's meant to combine the memory of all connected servers, so that if you connect 10 servers with 4GB each, you get 40GB of cache. This won't work if you only allow loopback connections.The admin should block memcached connections on the outer interface. And set the default firewall policy to DENY to start with.

That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.

If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.

Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.

That's why you create your own packages and have a system to add your mods to the public versions of the packages. I don't custom compile everything but the components most key to what I'm doing are often custom compiled because the public packaged lack the functionality I need.

Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."

Right... because if someone exploits a security bug in your app (I know... your apps have no bugs) and is able to 'drop table users;' because the DB had no security mechanism, that would really rock. Instead, we have to suffer the tedium of only granting the privileges that the app needs and locking down all other functionality.

This isn't a security problem - this is operating by design. If you are a memcache user and this is news to you, you need to read more about the tools you are using. I bet you have security problems beyond this one.

Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.

I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.

It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.

Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.

I don't allow random systems to communicate with my database. It is only accessed directly from localhost. Any other system has to go through a predefined RPC function that sanitizes inputs and outputs and does a very specific task. Even that access is only granted to a few systems and never over the public network. I usually still use the built-in security too but I would consider removing it as I consider it almost useless.

Sounds like a great reason to use something like SSL between memcached and the webservers. Forward the incoming SSL connections to memcached back to the memcached port over a loopback address. Now you can have authentication and encryption, if you want it, without adding any complexity to memcached. That way if someone wants to run it locally on a webserver you haven't really added any complexity.

Because of MySQL injection attacks and other problems, it is nice to have different access rights for different parts of your site. So I would not recommend stripping out the auth layer, especially if you have more than 1 DB running.

I wouldn't recommend stripping out the authentication layer either, I merely said it's been done. That being said, if your app can suffer from SQL injection, you've got bigger problems. If your app is to the point of needing to strip out DB authentication for some extra performance (and if it is, you should probably be taking a different approach... such as memcache, the point of this whole discussion), you damn well better have all sorts of tests in place to make sure that you don't introduce problems tha

Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?

"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"

Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal b