Archive for the ‘security’ Category

While attending VUGX Rotterdam, which was a jolly time for the Varnish family, I have been pleasantly surprised by some little known but very useful new things available for your site.

Varnish Security Firewall

I’ll tell you about them in a moment, but I’ll start by plugging VSF: The Varnish Security Firewall, which is a really useful security framework for Varnish 4, which we recently made really, really easy to install. You can give it a spin right now to stop many of the most common attacks against your website today.

Shameless plugs out of the way, Varnish is a painless process to operate, but most sites out there need to admin web servers, they need to encrypt their traffic and they need to load balance, and all this together becomes pretty messy, requiring haproxy, nginx and sticky putty goo in a glorious sandwich.

SSL frontend with Hitch

One problem that keeps popping up is the lack of SSL support. No, Varnish is not going to link openssl before hell freezes over in PHK’s basement, because let’s be honest: it belongs in its own binary. Still, you can now replace your heavy-handed Nginx stack with the lightweight Hitch, a revamped stud, which even has rudimentary Let’sEncrypt support.

With the write-proxy = on option, we can even use the PROXY protocol to pass the client.ip to Varnish instead of trusting X-Forwarded-For, but we’ll have to update Varnish to accept it by copying /lib/systemd/system/varnish.service to /etc/systemd/system/varnish.service and editing ExecStart to include -a:8000,PROXY like so:

On the backend side, it’s fairly straightforward to write SSL-backends for Varnish using stunnel.

Varnish as a Web Server

For years we’ve been half-joking that what Varnish needs is web server capabilities, but little did we know that this has been around for years!

Today we can all enjoy Martin’s fsbackend vmod, which lets us serve (and cache) static files straight off the file system in a secure and fast way. It doesn’t come with much documentation though, so here’s how you use it to serve your sites with the standard “index.html”, some minimal trailing-slashes support and some content-typing:

This module is historically proceeded by Dridi’s efforts for Varnish 3 and 4, which by his own words was “a hack”, as well as the std.file(), which was not good enough for serving web pages.

Coupled with Hitch for SSL, this has allowed me to drop the Nginx sandwich completely on the static parts of my sites.

Regexes with sub-captures

Innate regexes are awesome in Varnish, but sub-captures are barely accessible and usually force you to copy and match a single string many times for multiple captures. This is both messy and not efficient. Enter Geoff from Uplex re vmod, which lets us access all subcaptures of a efficiently:

Consistent Hashing

If you’re running a really large site where you’re regularily pinning your server bandwidth you know that load-balancing and sharding is where it’s at. That’s where the stateless persistent hashing vmod that Niels from Uplex presented comes in.
When a frontend request hits your cluster, the vslp vmod looks up which cache is the master for that request, and then picks which backend owns the request in a consistent way across n servers. If you ever add frontends or backends, only 1/n+1 objects have to be rebalanced, giving you a limited increased load cost for adding new servers. The code has been in heavy production for the last couple of years!

This means you can keep adding frontends and backends to your pool, and you can round-robin load balance while scaling (near-) linearly.

Dynamic Backends

When running Varnish on EC2 with an ELB, it’s real tricky to keep the backend list up to date. Dynamic backend vmods will fix this problem Real Soon Now.

With consistent hashing in the Varnish core, and dynamic backends, we might no longer need haproxy nor any other messy load balancer, simplifying the stack for quite a lot of sites out there.

There are also rumours of a FCGI or even WSGI vmod, which will allow us to forgo the ancient dogmatic cows of web adminning entirely, and go Full Stack Varnish. Let’s just pray that day will come soon enough.

I’m not surprised, but then again I never am when it comes to this paranoia stuff. However, I do run a ton of SSH in production and know a lot of people that do. Are we all fucked? Well, almost, but not really.

Basically the nice and forward secure aes-*-gcm chacha20-poly1305 ciphers, the curve25519-sha256 Kex algorithm and Encrypt-Then-MAC message authentication modes are not available to those of us stuck in the early 2000s. That’s right, provably NSA-proof stuff not supported. Upgrading at this point makes sense.

Still, we can harden SSH, so go into /etc/ssh/moduli and delete all the moduli that have 5th column < 2048, and disable ECDSA host keys:

Note: Sadly, the -ctr ciphers do not provide forward security and hmac-ripemd160 isn’t the strongest MAC. But if you disable these, there are plenty of places you won’t be able to connect to. Upgrade your servers to get rid of these poor auth methods!

Kernel security is becoming more important nowadays, especially since the Linux kernel has grown so big and the platform is rife with misbehaving programs.

Some enthusiasts have illusions that the Linux kernel is somehow magically very secure, but those in the know will tell you there are quicker ways to gain root than sudo.Grsecurity is by far the best patchset for increasing the security of a system, whether it be a virtual LAMP box, a login shell server or a desktop machine. I have been tracking it and using it for years and consider it superior to the SELinux and AppArmor approaches.

It has only one major drawback: most distributions of Linux do not provide tested and up-to-date grsec-patched kernel packages, making Grsec-level security features nearly unobtainium for the common mortal.

So far I have built and tested kernels for Debian 7.5 and 7.6 Stable codenamed Wheezy. This is the standard, debian-patched kernel with added Grsecurity.

I have built separate packages which are suitable for servers, virtualized servers and desktops, and these have been tested on run-of-the-mill LAMP boxen as well as custom and well-trafficed shell servers, and of course my trusty desktops and laptops.

Furthermore I commit to also merging the patchsets and making available Grsecurity packages for Debian 8/Jessie and providing it all in a debian repo. I will then make this available in a repo so that people can easily add it to their setup.
I also commit to keeping these packages up to date on all the platforms I care about.

Ever since HACK.LU (where we spoke about VSF), Ebf0 and I have had quite some activity on PRADS, wonder why?

We really enjoyed the design of POM-NG, we find this little program quite inspiring and will keep in touch with GMsoft.

This might be the right time to announce PRADS-me! at prads.delta9.pl, a service to actively fingerprint your own self. Real useful even just for an IP check, geolocation or to see what types of fingerprints you are matching at any given time.

Jostein Haukeli at the University of Oslo Department of Informatics has written a paper on “False positive reduction through IDS network awareness”. We are excited about the prospect that our work is being used in data correlation, and we would like to see more event correlation stuff done in a scalable context.

In other news, PRADS is about to be baked into the next release of the Security Onion network monitoring linux distro. Version 12.04 beta already comes with PRADS included (replacing old-timers sancp and pads) but it did require some bug-squashing from our end. You know what that means? 0.3.2-rc1 was tagged in the tree recently. That’s right: a new PRADS release is coming up Real Soon Now.

Hi all,
I know it’s been a while since my last post.
There is lots to talk about, but let’s start afresh with something short and sweet.

Let me paint a picture for you:
There is something broken in the most holy and secure part of your network. You need to call support to have them look at it. The support rep wants console access, but you can’t give them axx to your holiest cream pie.
They offer to take over your desktop with a java rootkit app like TeamViewer, GoToMeeting or WebEx.
You decline. You need to stay in control, but show them what they need to see, that and only that.
How?

Let me be clear on the problem statement:
Read-only shell access to the most secure host, which is not available over the wire, viewed by multiple parties at the same time.

Here’s how to do that with SSH, screen(1) and some foo,
with ssh->chroot->rbash->readonly multiuser screen->reverse ssh->openvpn:

You will need a linux server in an “unsafe” zone which is exposed to your support rep on the internet or thru VPN.

Setup multiuser screen on the unsafe box. There are lots of guides for it, but the short and sweet of it is: unsafe# chmod +s `which screen`; chmod 755 /var/run/screen Indeed, this increases the attack surface, and therefore we call this box the unsafe one.

Win! Now you can reverse ssh back to the secure zone and let $user on the unsafe box read the terminal without being able to access anything but what you show her.

Bonus: Add `screen -r $youraccount` in $user/.profile and $user will drop straight into locked screen, and remember that multiuser screen is read-write-execute for all accounts that are addacl’d
so you might want to chacl before enabling the $user account login.

And there you have it, superparanoid reverese secure-unsecure remote shell viewer.

However, it seems that most people are interested in running CPM on older distributions, realistically the stable distribution codenamed squeeze is a favorite, as well as the Ubuntu LTS release 10.4 codenamed lucid lynx.

File us a ticket if you run into trouble with these packages or need cpm working on some other distribution.

CPM is a simple, paranoid password manager for the console with some cool features that make it stand out:

* data files can be encrypted for more than one person
* data files are signed by the last person who saved it so forging data files is not possible
* data files are en- and decryptable directly by gpg and gzip
* the application memory is protected from paging, core dumps, ptrace attacks and runtime environment
* data is validated using an internal DTD
* several passwords per account are possible to store
* it’s possible to handle several data files, each encrypted for different people
* cracklib checks of password strength and warnings about weak passwords
* user definable hierarchy with unlimited depth
* long comments for any node in the hierarchy
* password generator
* only one password visible at a time
* searchable database from the command line
* user definable search patterns (e.g. user@hostname)
* several hits can be displayed at once (e.g. several accounts per host)
* conversion scripts for Password Management System (pms), Password Safe and CSV files

I’m not going to tell you about DigiNotar, whose file of bankruptcy this month held shock for no one after recently having lost the keys to the grand vault, in which the government held much stock. Though I have many comments upon the sophistication of the player that so thoroughly owned the most trusted agencies of the digital age….

The cracker hardly needed them skillz, considering it has been a challenge to keep that whole corrupt industry accountable. The trouble with the central authority system is that even if only one of the keys is compromised, the system is broken and gives no assurances whatsoever. No warning bells either. Just a sweet silent man in the middle, passing along all the best parts to his lover.

I’m not going to mention the fatal flaws in certificate revocation that became embarrassingly apparent when the damage was done.
What’s hardly the matter since this kind of thing is bound to crop up, that hole in TLS was deemed unexploitable – now there’s a Titanic if I ever saw one. Un sinkable. Too fat to die.

SSL is an open book for those who dare to look, and it’s got more than a couple old bugs. It’s okay though, we can patch it, they will say. Dare to look the other way!
Not that you need those anyway, since there are some really nice sslsnarfing techniques out there that entirely forgo attacks on SSL as “too inefficient”.

But I say nay! Unacceptable. There is another way.. and we’re already doing it! We sign our own signatures and we back each other’s signatures.
Now that’s business, something that the companies on your CA trusted list were painfully aware of when they laid down the law of the code and put themselves on the trust list. Yet still ca-cert is not on your trust list, and warning bells fly off on some of the most trustworthy sites- self-signed ones.

Just don’t ask them why or how, or anything that isn’t directly relevant. Do you even know what is on your trust list? You might just be surprised at what you can find.

# ls -al /etc/ssl/certs | wc -l
479

How many of these do you trust? How many of these should you trust? I’ll tell you: *none*.

We should not be adding static lists of central signing authorities to our systems. This is a brittle and dangerous system. We knew this, but hackers have now thankfully demonstrated it.
A better way is for every person (and by extension every browser) to keep their own list of signing certs, and to exchange these certs with their friends (automagically, if you like). Your friends lists can come out of a social network, any social network, and it will mean that any site that has been vetted by one or more of your friends will likely be safe for you to use as well. It’s even better than that, you can check certs from multiple friends and detect discrepancies.

That, my friends, is called the Web of Trust, and is a design that is heading in the right direction. convergence.io is doing something similar already to a Firefox near you, while GPG has worked like this for three decades!

It has to be simple. It has to be very simple. And it has to be chemically free of one word: ‘central’.

One real easy way to do this on linux would be using git and signed manifests. I already do this in gone to assure that only files on a manifest signed by a trusted key get installed.

Just a quick note pertaining to a previous post, namely the new evil that is firefox geolocation. This is new in firefox 3.5. Yes, it is opt-in and yes firefox does not track you but yes the servers you opt in to will track you and that my friends is one of the most serious misfeatures of our times, repeated again and again in stuff like Google Latitude, Android and Apple photo geo-tagging.
If you care about your personal security at all you do not want the internet tracking where you are, which is essentially what this amounts to.
Disable it now by going to the about:config location in your firefox, typing geo. in the search field and double clicking the geo.enabled line so that it says

If you didn’t know that just following a link could lead to loss of your identity, virus infection and the unknowing participation in denial of service sieges, distribution of child pornography and letting attackers break through your company’s firewalls (not to mention immanentizing the eschaton), you could be forgiven for not paying attention to the happy happy field of information security.

If you knew this rather distressing fact about the web of links, but thought you could defend with an up-to-date antivirus, a current browser and the avoidance of “shady” websites I would hate to prick your bubble by telling you regular honest websites are being used against us, browsers all have to run the all-singing-all-dancing-all-executing flash and jave and anti-virus is not only ineffective but doesn’t even target this type of attacks. Knowing these things might be a little distressing so I won’t tell you.

At least my bank is secure, you say, it’s got SSL and everything! Well, you’d be twisting my arm into telling you, embarassedly, that SSL as implemented in browsers is very neatly broken, that all we needed was one of the Certified Trusted Authority Parties to go bad for us all to wake up with a butt-ache, but we now have not one but at least three such bad parties, not to mention all the MiM magic and DNS trickery that you don’t want to hear about anyway.

I will tell you however that the only defense is two-pronged: not allowing websites to script – which is a pain – and damage containment, which is not exactly perfect.

Let us say you already knew all of this, but no longer cared because there was an even greater danger on the web: the total breach of containment of privacy that is social media and tracking cookies which all want to integrate and track your every move through the web so that usage and mood patterns can be profiled, tracked, bought and sold. Doesn’t matter, right? You’re logged in to Facebook, Linkedin and Google+ and get all your links from there, so you have your own personal filter which only shows you stuff you care about, turning your blind eye towards anything you need to know that comes from left field, suckering you into giving away your privates every time you hit “like” or “add to friends list”.

In a post-panopticlick world there is really only one lesson to be learned: care about your privacy, because noone else will give a damn. It’s not about whether you have anything to hide, it’s about the accumultion of your private info by crooks to use as they please!

Oh and crooks include the great people at Facebook, Microsoft and Google, that’s why I recommend disabling all tracking features that come in the guise of “better speed” or “increased security”. Pictures below show how to do it in chromium and firefox.

Ok that was Goggle’s Chromium – note all the unchecked- checkboxen… disabling prefetch, malware blocklists and suggestion services, all of which are sending far too much data to Google’s scaredy-ass all seeing eye. Aaaand here’s Firefox:

Mhm that’s pointing the browser at about:config, searching for prefetch and setting it to false. Yep.

Those pictures show you how to find and set better privacy features in your browser. Safari users are up shit creek, but unlike the Internet Explorer users, they at least have a paddle! Great stuff, what next?

Keep a “secure” browser to browse with that you don’t log into anything personal with.. and make this your default browser!

What is a “secure” browser? Let’s talk a little about that without insulting anyone’s intelligence, shall we?
First off, I’m putting the word “secure” in uhm qoutes, because uhm the browser will never be secure, be it ever so protected. Ok, moving on you want to be running noscript and or adblock and or disconnect and or noflash, depending on whether you are a Mac, Windows (but then you are at game over already) or Linux user with Firefox or Chromium (NOT IExploder, that shit is scary!).

All of these tools make it so the sites you visit don’t get to run code on your machine. The end result is that sites are marginally less usable, but that popup popunder popver poopop ads and scary tracker/botnet/mal stuff doesn’t run automagically. Here are some links:
– noscript
– adblock
– disconnect
– Flashblock
– Have you heard about sandboxing?

Chromium is getting the right idea by killing java and flash from the get-go. This should be the default, but you have to make it so!

You should definitely be cloaking your user-agent, as it is a useless yet very telling string your browser hoes out to all. Do this with the User-Agent Modifying Extension.

Also, you may find Torbutton and Foxyproxy interesting, the real privacy is in bouncing your traffic thru things that wash it for you. Putty or any other decent ssh client will create a proxy for you:

`ssh -D8080 me@myshellserver`

and you can point your browser’s SOCKS proxy settings to localhost:8080 putting all your traffic through your shell server!