Monthly Archives: November 2010

This month’s ACUG meeting was cooler than usual – instead of having one speaker talk on a cloud-related topic, we had multiple group members do short presentation on what they’re actually doing in the cloud. I love talks like that, it’s where you get real rubber meets the road takeaways.

I thought I’d share my notes on the presentations. I’ll write up the one we did separately, but I got a lot out of these:

Seems to me it’s one part REST, one part “making you have a DTD for your XML”. Which is good! We’re very interested in OData for our data centric services coming up.

Moving your SaaS from Colo to Cloud

Josh Arnold was from PeopleAdmin, now he’s a tech recruiter, but can speak to what they did before he left. PeopleAdmin was a Sungard type colo setup. Had a “rotting” out of country DR site.

They were rewriting their stack from java/mssql to ruby/linux.

At the time they were spending $15k/mo on the colo (not including the cost of their HW). Amazon estimated cost was 1/3 that but really they found out after moving it’s 1/2. What was the surprise cost? Lower than expected perf (disk io) forced more instances than physical boxes of equivalent “size.”

Flexible provisioning and autoscaling was great, the colo couldn’t scale fast enough. How do you scale?

The cloud made having an out of country DR site easy, and not have it rot and get old.

Question: What did you lose in the move? We were prepared for mental “control issues” so didn’t have those. There’s definitely advanced functionality (e.g. with firewalls) and native hardware performance you lose, but that wasn’t much.

They evalled Rackspace and Amazon (cursory eval). They had some F5s they wanted to use and the ability to mix in real hardware was tempting but they mainly went straight to Amazon. Drivers were the community around it and their leadership in the space.

Timeline was 2 years (rewrite app, slowly migrate customers). It’ll be more like 3-4 before it’s done. There were issues where they were glad they didn’t mass migrate everyone at once.

Technical challenges:

Performance was a little lax (disk performance, they think) and they ended up needing more servers. Used tricks like RAIDed EBSes to try to get the most io they could (mainly for the databases).

Every customer had a SSL cert, and they had 600 of them to mess with. That was a problem because of the 5 Elastic IP limit. Went to certs that allow subsidiary domains – Digicert allowed 100 per cert (other CAs limit to much less) so they could get 100 per IP.

App servers did outbound LDAP conns to customer premise for auth integration and they usually tried to allow those in via IP rules in their corporate firewalls, but now on Amazon outbound IPs are dynamic. They set up a proxy with a static (elastic) Ip to route all that through.

Rightscale – they used it. They like it.

They used nginx for the load balancing, SSL termination. Was a single point of failure though.

Remember that many of the implementations you are hearing about now were started back before Rackspace had an API, before Amazon had load balancers, etc.

In discussion about hybrid clouds, the point was brought up a lot of providers talk about it – gogrid, opsource, rackspace – but often there are gotchas.

Chris Hilton from Thoughtworks is all about the DevOps, and works on stuff like continuous deployment for a living.

DevOps is:

collaboration between devs and operations staff

agile sysadmin, using agile dev tools

dev/ops/qa integration to achieve business goals

Why DevOps?

Silos. agile dev broke down the wall between dev/qa (and biz).

devs are usually incentivized for change, and ops are incentivized for stability, which creates an innate conflict.

but if both are incentivized to deliver business value instead…

DevOps Practices

version control!

automated provisioning and deployment (Puppet/chef/rPath)

self healing

monitoring infra and apps

identical environments dev/test/prod

automated db mgmt

Why DevOps In The Cloud?

cloud requires automation, devops provides automation

References

“Continuous Delivery” Humble and Farley

Rapid and Reliable Releases InfoQ

Refactoring Databases by Ambler and Sadalage

Another tidbit: they’re writing puppet lite in powershell to fill the tool gap – some tool suppliers are starting, but the general degree of tool support for people who use both Windows and Linux is shameful.

Moving Software from On Premise to SaaS

John Mikula of Pervasive tells us about the Pervasive Data Cloud. They wanted to take their on premise “Data Integrator” product, basically a command line tool ($, devs needed to implement), to a wider audience.

Started 4 years ago. They realized that the data sources they’re connecting to and pumping to, like Quickbooks Online, Salesforce, etc are all SaaS from the get go. “Well, let’s make our middle part the same!”

They wrote a Java EE wrapper, put it on Rackspace colo initally.

It gets a customer’s metadata, puts it on a queue, another system takes it off and process it. A very scaling-friendly architecture. And Rackspace (colo) wasn’t scaling fast enough, so they moved it to Amazon.

Their initial system had 2 glassfish front ends, 25 workers

For queuing, they tried Amazon SQS but it was limited, then went to Apache Zookeeper

First effort was about “deploy a single app” – namely salesforce/quickbooks integration. Then they made a domain specific model and refactored and made an API to manage the domain specific entities so new apps could be created easily.

Recommended approach – solve easy problems and work from there. That’s more than enough for people to buy in.

Their core engine’s not designed for multitenancy – have batches of workers for one guy’s code – so their code can be unsafe but it’s in its own bucket and doesn’t mess up anyone else.

Changing internal business processes in a mature company was a challenge – moving from perm license model to per month just with accounting and whatnot was a big long hairy deal.

Making the API was rough. His estimate of a couple months grew to 6. Requirements gathering was a problem, very iterative. They weren’t agile enough – they only had one interim release and it wasn’t really usable; if they did it again they’d do the agile ‘right thing’ of putting out usable milestones more frequently to see what worked and what people really needed.

In Closing

Whew! I found all the presentations really engaging and thank everyone for sharing the nuts and bolts of how they did it!

Hey all, I just wanted to take a moment to share with you that our first cloud-based product just went live! LabVIEW Web UI Builder is National Instruments’ first SaaS application. It’s actually free to use, go to ni.com and “Try It Now”, all you have to do is make an account. It’s a freemium model, so you can use it, save your code, run it, etc. all you want; we charge to get the “Build & Deploy” functionality that enables you to compile, download, and deploy the bundled app to an embedded device or whatnot.

Essentially it’s a Silverlight app (can be installed out of browser on your box or just launched off the site) that lets you graphically program test & measurement, control, and simulation type of programs. You can save your programs to the cloud or locally to your own machine. The programs can interact via Web services with anything, but in our case it’s especially interesting when they interact with data acquisition devices. There’s some sample programs on that page that show what can be done, though those are definitely tuned to engineers… We have apps internally that let you play frogger and duck hunt, or do the usual Web mashup kinds of things calling google maps apis. So feel free and try out some graphical programming!

And it’s 100% DevOps powered. Our implementation team consists of developers and sysadmins, and we built the whole thing using an agile development methodology. All our systems are created by model-driven automation from assets and definitions in source control. We’ll post more about the specifics now that we’ve gotten version 1 done! (Of course, the next product is just about ready too…)

Why The Cloud Is More Secure Than Your Existing Systems

You can read the slides (sadly, the animations don’t come through so some bits may not make sense…). In general my premise is that people that worry about cloud security need to compare it to what they can actually do themselves. Mocking a cloud provider’s data center for not being ISO 27001 compliant or having a two hour outage only makes sense if YOUR data center IS compliant and if your IT systems’ uptime is actually higher than that. Too much of the discussion is about the FUD and not the reality. Security guys have this picture in their mind of a super whizbang secure system and judge the cloud against that, even though the real security in the actual organization they work at is much less. I illustrate this with ways in which our cloud systems are beating our IT systems in terms of availablity, DR, etc.

The cloud can give small to medium businesses – you know, the guys that form 99% of the business landscape – security features that heretofore were reserved for people with huge money and lots of staff. Used to be, if you couldn’t pay $100k for Fortify, for instance, you just couldn’t do source code security scanning. “Proper security” therefore has an about $1M entry fee, which of course means it’s only for billion dollar companies. But now, given the cloud providers’ features, and new security as a service offerings, more vigorous security is within reach of more people. And that’s great -building on the messages in previous sessions from Matt’s keynote and Homeland Security’s talk, we need pervasive security for ALL, not just for the biggest.

HTTPS Can Byte Me

This was a very technical talk so I’m not going to try to reproduce it all for you here. Read the white paper and slides. But basically there are a lot of things about how the Web works that makes HTTPS somewhat defeatable.

First, there are insecure redirects, DNS lookups, etc. before you ever get to a “secure” connection. But even after that you can do a lot of hacking from traffic characterization – premapping sites, watching “encrypted” traffic and seeing patterns in size, get vs post, etc. A lot of the discussion was around doing things like making a user precache content to remove noisiness via a side channel (like a tab; browsers don’t segment tabs). Anyway, there’s a lot of middle ground between “You can read all the traffic” and “The traffic is totally obscured to you,” and it’s that middle ground that it can be profitable to play in.

Tell Me Your IP And I’ll Tell You Who You Are

Noa Bar-Yosef from Imperva talked about using IP addresses to identify attackers – it’s not as old and busted as you may think. She argues that it is still useful to apply IP intelligence to security problems.

Industrialized hacking is a $1T business, not to mention competitive hacking/insiders, corporate espionage… There’s bad people trying to get at you.

“Look at the IP address” has gotten to where it’s not considered useful, due to pooling from ISPs, masquerading, hopping… You certainly can’t use them to prove in court who someone is.

But… home users’ IPs persist 65% more than a day, 15% persist more than a week. A lot of folks don’t go through aggregators, and not all hopping matters (the new IP is still in the same general location). So the new “IP Intelligence” consists of gathering info, analyzing it, and using it intelligently.

Inherent info an IP gives you – its type of allocation, ownership, and geolocation. You can apply reputation-based analytics to them usefully.

Geolocation can give context – you can restrict IPs by location, sure, but also it can provide “why are they hitting that” fraud detection. Are hits from unusual locations, simultaneous from different locations, or from places really different from what the account’s information would indicate? Maybe you can’t block on them – but you can influence fuzzy decisions. Flag for analysis. Trigger adaptive authentication or reduced functionality.

Mitigating Business Risks With Application Security

This talk was by Joe Jarzombek, Department of Homeland Security. Normally I wouldn’t go to a management-track session called something like this, when I looked at the program this was my third choice out of all three tracks. But James gave me a heads up that he had talked with Joe at dinner the previous night and he was engaging and knew his stuff, and since there were plenty of other NI’ers there to cover the other sessions, I took a chance, and I wasn’t disappointed!

From a pure “Web guy” standpoint it wasn’t super thrilling, but in my National Instruments hat, where we make hardware and software used to operate large hadron colliders and various other large scale important stuff where you would be very sad if things went awry with it, and by sad I mean “crushed to death,” it was very interesting.

Joe runs the DHS National Cyber Security Division’s new Software Assurance Program. It’s a government effort to get this damn software secure, because it’s pretty obvious that events on a 9/11 kind of scale are more and more achievable via computer compromise.

They’re attempting to leverage standards and, much like OWASP’s approach with the Web security “Top 10,” they are starting out by pushing on the Top 25 CWE (Common Weakness Enumeration) errors in software. What about the rest? Fix those first, then worry about the rest!

Movement towards cloud computing has opened up people’s eyes to trust issues. The same issues are relevant to every piece of COTS software you get as part of your supply chain! It requires a profound shift from physical to virtual security.

“We need a rating scheme!” Like food labels, for software. They’re thinking about it in conjunction with NIST and OWASP as a way to raise product assurance expectations.

He mentioned that other software areas like embedded and industrial control might have different views on the top 25 and they’re very interested in how to include those.

They’re publishing a bunch of pocket guides to try to make the process accessible. There’s a focus on supply risk chain management, including services.

Side note – don’t disable compiler warnings! Even the compiler guys are working with the sec guys. If you disable compiler warnings you’re on the “willful disregard” side of due diligence.

You need to provide security engineering and risk-based analysis throughout the lifecycle (plan, design, build, deploy) – that generates more resilient software products/systems.

Like Matt, he mentioned the Rugged Software Manifesto. Hearing this both from “OWASP guy” and “Homeland security guy” convinced me it was something that bore looking into. I like the focus on “rugged” – it’s more than just being secure, and “security” can seem like an ephemeral concept to untrained developers. “Rugged” nicely encompasses reliable, secure, resilient… I like it.

It was interesting, at times it seemed like Government Program Bureaucratese but then he’d pull out stuff like the CWE top 25 and the Rugged Software Manifesto – they really seem to be trying to leverage “real” efforts and help use the pull of Homeland Security’s Cyber Security Division to spread them more widely.

Why ha.ckers.org Doesn’t Get Hacked

The first LASCON session I went to was Why ha.ckers.org Doesn’t Get Hacked by James Flom (who with rsnake is ha.ckers.org). By its nature, it gets like 500-1000 hack attempts a week, but they’ve kept it secure for six years now.

From the network perspective, they use dual firewalls running the openBSD open source pf, which does Cisco-style traffic inspection. Systems inside have no egress, and they have the user traffic and admin traffic segmented to different firewalls sets and switches.

On the systems, they use chroot jails mounted read only. Old school! Jails are virtualization on the cheap, and if combined with a read only filesystem, give you a single out of band point of update, and you can do upgrades with minimal downtime. They monitor them from the parent host.

Rsnake has done a whole separate presentation on how he’s secured his browser – the biggest attack vector is often “compromise the browser of an admin” and not direct attack on the asset.

They went to WordPress for their software – how to secure that? Obviously code security’s a nightmare there. So they set up a defense in depth scheme where they check the source ip, cert, and user/pass auth at the firewall, then to admin proxy check source IP, path, htaccess user/pass, and finally do the app auth.

On-host WAF – custom, more of a “Web IDS” really, which feeds back “naughty people” to the firewall for blocking

For Apache – have your content owned by a different user, in their case there’s not even a user in the jail that can write to the files.

Use file ACLs, too.

Use case – they found an Apache flaw, reported it, and as is too often the case, Apache didn’t care. So they modded their pf to detect the premise of the attack and block it (not just the specific attack). (Heard of slowloris?)

Their ISP has been an issue – as they’ve moved their ISPs have shut them down out of cluelessness sometimes (Time Warner Business Class FTL).

They are moving to relayd for load balancing and SSL. The PCI rule about “stay encrypted all the way to the box” is dumb, because it would prevent them from doing useful security inspection at that layer.

A good talk, though sadly a lot of the direct takeaways would mean “go to FreeBSD,” which I would rather not do. But a lot of the concepts can port to other OSes and pure virtualization/cloud scenarios. And note how joining network security, OS security, and appsec gets you way more leverage than having “separate layers” where each layer only worries about itself.n

And may I just say that I love how Apache can be run “read only” – sadly, most software, even other open source software like Tomcat, can’t be. It all wants to write down into its config and its running directories itself, and it’s a horrible design practice and security risk. If you’re writing software, remember that if it’s compromised and it can write to its own exes/config/etc. you’re owned. Make your software run on a read only FS (with read/write in /tmp for stuff acceptable). It’s the right thing to do.