A page to show up #1 on Google when searching for "Jeremiah" (Currently #4). Only the prophet and TV show left! I have the edge, TV show is cancelled and the prophet isn't generating any new content.

The prophet, TV show, and that pesky Owyang guy going down!A page to show up #1 on Google when searching for "Jeremiah Grossman", and it FINALLY has!

Thursday, June 26, 2008

Can WAFs protect against business logic flaws?

“Web Application Firewalls (WAF) are a total waste of time/money because they can’t protect against business logic flaws!,” a common theme among the few, but vocal, seriously anti-WAF zealots out there. While there is some truth it’s also like saying car door locks are useless because criminals can break in by smashing the windows. Or car alarms are a waste because they don’t protect against carjacking. And steering wheel locks are lousy because the car is at risk to thieves with tow trucks. You see where I’m going with this. Every security measure has a particular purpose, limitation, and overall value to the owner considering what it is they’re protecting.

Sure, WAFs don’t defend against every logic flaw, or even every crazy form of SQLi or XSS. Just as white/black box scanners can’t identify every vulnerability and neither can expert pen-testers or source code auditors. A/V products don’t red flag every piece of malware. Anti-spam misses some junk mail. Yet we still utilize these solutions anyway because their value outweighs their limitations. And of course WAFs don’t replace vulnerability assessment (VA) or secure coding practices just as Nessus doesn’t compete with network firewalls or good segmentation practices. Therefore I recommend we ignore rash criticisms and keep an open mind into what WAFs can/can’t do, the value they may provide today, and consider how they made be improved – including being aided by VA intelligence (VA+WAF).

I’m going to keep my comments vendor agnostic. Perhaps some of the features described below are already included in some of the available WAF products. In fact I know they are and claim no novelty of any of these ideas (probably printed elsewhere), but I’ll leave it to the vendors to comment on their specific products capabilities. I think the readers here might be pleasantly surprised. My intent here is to explore a few of the more common business logic flaw examples we’ve all seen, assume we know where their location (VA), and attempt to hypothesize defense measures.

Business Logic Flaw examples

1) Rotating numbers in URLs, the classic case of Insufficient Authentication, Insufficient Authorization and Insufficient Process Validation where an attacker can gain access to data or functionality their user-level should not have. We’ve seen these issues countless times in order tracking systems, bank account screens, and even in online vote registration. I see at least two possible ways to prevent these types of business logic flaws with a WAF.

URL encryptionThe WAF inspects outbound Web page content, dynamically encrypts and replaces every URL directed to the website, and by extension decrypts them on the way back in – completely transparent to the web server or application. For example:

action

becomes…

action

or

action

URL encryption is powerful as it prevents URL parameter tampering and by extension protects against a wide range of attacks (XSS, SQLi, CSRF, etc.). No parameter tampering, no number rotation, no business logic flaw. Implementation is really tricky though because the HTML parser has to be perfect otherwise requests will be blocked when links are missed. Bookmarks and search engine indexing is also potentially disrupted. However, websites where most functionality is behind a login screen, such as banking sites, might not mind. Its also possible these side effects could be reduced by only focusing on the URLs known to be vulnerable (VA) instead of pursuing global enforcement. There is no need to encrypt URLs that aren’t vulnerable.

Session-State trackingUsers can be tracked from one page to the next so it’s technically possible for a WAF to know where they are in a particular flow and where they should be able to get to, or not. If an attacker were to rotate a number in a URL the WAF could be capable of determining if they should have been able to get it (UI-wise) from where they are. If they shouldn’t be able to, deny! Or perhaps a more forgiving threshold is in order so the may try 1, 2, or even 10 illegal URLs, but not more because that would surely be abnormal behavior. Scalability is biggest drawback here as increasingly large state tables are required for tracking. However, if you know a particular URL or parameter name has a problem with number rotation, WAFs can again be configured to focus and enforce controls only there.

2) Session hijacking by way of cookie tampering is another old school hack that has implications for Credential/Session Prediction, Insufficient Session Expiration, and Session Fixation. This issue doesn’t show up as much as it once did because most developers are using the native session handling APIs in their development frameworks as opposed to rolling their own. A very good thing.

Just like the previous example we can utilize some good ol’ on-the-fly cookie encryption/decryption that can be easily performed with a WAF. If an attacker is unable to modify their cookie to a valid value, and the WAF would know cryptographically, then session handling issues go away. You could even add some httpOnly, secure, and non-persistent flags if you want. You’d still probably have issues with Insufficient Session Expiration or Session Fixation, but we’re getting somewhere. The only drawback I can think of is if JavaScript or some other client-side language needed to read/write the original cookies values.

3) WAFs could also potentially be used to stop login brute force attacks or Insufficient Anti-Automation by including CAPTCHAs on-the-fly at various choke points. Again, thresholds would be neat. We could explore other examples, but I think you get the idea and this post is long enough. Well at least I don't want to write anymore. :)

Its important to understand that we’re at the very beginning of WAFs (or website VA for that matter), their deployments, which is also why there is so little field experience posted anywhere. We need an open community dialog so we can see where this technology can go and how it can be improved. - independent of the PCI 6.6 mandate. My point is I don’t think WAFs will be able to solve all of our web application security problems, or even all business logic flaws, and I don’t know of anyone who does. It certainly would be nice though to see what WAFs can do or be made to do. We won’t know unless we keep and open mind and try.

"Any fool can criticize, condemn, and complain, and most fools do." Benjamin Franklin

30 comments:

So you admit that having an incredibly immature and misunderstood technology as a compliance requirement is probably the worst idea in the history of ideas?

Yet at the same time, you continue to try and capitalize on it. You choose "one step forward; two steps back" instead of "request for comments", "benchmarking", and "show me the money". Maybe you misunderstood the meaning behind that last one.

I bet a million people have great ideas on how WAFs could work better. My suggestion: fugetaboudit. Fugetaboudit, it's a total waste of time, money, resources, talent, and you name it. Focus on the problems at the source and let's come up with process solutions, people solutions, or other non-technology solutions so that suffering organizations don't have to constantly sink their money, time, and souls into.

Security folks, by necessity, have a clever nature and like to find ways around systems. The fact that a proposed solution has weaknesses doesn't mean that it's totally useless, like a lot of us security geeks frequently imply. It just means that it's a tool with a particular purpose, and the architect has to understand that purpose and how it fits into the larger design.

Good topic for discussion. I think an ideal WAF should be able to protect against the most common web app vulnerabilities while "knowing" as little as possible about the applications in your environment. This gives you the benefit of a basic layer of protection across all your applications from common attacks such as SQLi, XSS, or anything that can be stopped via signature/pattern matching. It's a band-aid to give an organization time to fix applications against known or new vulnerabilities. I think this model presents the best trade off between cost and level of protection.

While I don't doubt that WAFs can help stop these types of business logic attacks, I hesitate to accept that it's a good return on investment once you cross a certain level of effort. Not only do you have to heavily customize the WAF for each application it's protecting - you also have to potentially reconfigure the WAF any time those applications change. It becomes yet another piece of software that has to be continuously developed and reconfigured in parallel with your other applications.

As always, your points are both timely and well taken. If WAFs cannot recognize business logic flaws then certainly neither can automated Vulnerability scanners or other automated assessment tools. the point is that the market is so immature that you will always need a human element to mitigate false positives and account for logic or other issues that the WAF/VA misses. Hence, the beauty of the Whitehat SaaS buinsess model.

I might also add what I feel to be the primary business drivers for WAFs (even with their shortcomings):

1) Zero-day patching -- buying more time to fix a vuln once it has been discovered

2) forensics visibility into the web application -- the ability to both watch and learn what your users are doing within the web application so you can make more informed decisions in the event of a compromise or security incident.

Granted, WAFs are not perfect and they admittedly have their short comings but so do secure source code analysis tools and Virus/Malware scanners. Actually, pdp architect recently referencd a quote by Giorgio Maone here (http://www.gnucitizen.org/blog/tomorrows-malware/) that is perhaps very timely for this discussion:

And if today’s malware mostly runs on Windows because it’s the commonest executable platform, tomorrow’s will likely run on the Web, for the very same reason.

Surely, we can all agree that fighting Virus and other malware with virus scanners is a losing battle but even so, we still fight the battle. With any luck WAFs will continue to mature but for WAF critics to make blanket statements that WAFs should be avoided becasue they are not 100% effective strikes me as being a bit misinformed in terms of the business case and other drivers for WAFs, in general.

Commercial WAF's are very expensive to purchase and even more expensive to maintain over time. WAF critics believe that money spent buying and maintaining a WAF would be better spent (and actually reduce overall risk more) on activities such as code review, manual pen testing, developer training and fixing the underlying code. I personally think Andre is right on - please note that Andre is also driving towards benchmarking and metrics, not just throwing out blanket statements.

Please note that most of the WAF supporters in the industry financially benefit towards their propagation.

Operational security folks who are responsible for the security of a public facing website would welcome duck-tape and chewing gum if would help them to block an attack or mitigate a known vulnerabiliy.

@Jim Manico -

Please note that most of the WAF supporters in the industry financially benefit towards their propagation.

Usually I do not post self esteem comments, but you asked the vendors to comment on their specific products capabilities, so here it goes... I'll try to keep feature marketing to the minimum and focus on the vision to solve the problems mentioned.

First, here is a statement that might shock few people who do not know me very well. IMHO, a standalone WAF is not the cure for all diseases. I do think that it's probably the best line of defense against web attacks and while Amichai and I sometimes argue about the amount of resources organizations should spend on fixing or trying to fix the root cause, it seems clear to me that from a time-to-fix vs time-to-exploit perspective, WAF is the best first solution. Integrating WAF with Vulnerability Assessment tools (aka WAF+VA) will improve overall security as well.

But even this is not enough…

At Imperva, we believe that application security does not start or end with WAF. Our vision has always been that in order to provide the most capable solution, one should look at the web application as well as the backend systems (i.e. databases in most cases) and how the system implements business logic. SecureSphere WAF is only one component of the solution.

Database Security, with activity monitoring and ability to understand the business logic of data usage are key elements of a complete solution. At Imperva, we call this Application Data Security / ADS (see also http://blog.imperva.com/mt/mt-search.cgi?tag=application%20data%20security&blog_id=2), Rich Mogull talks about something similar, calling it Application and Database Monitoring and Protection / ADMP (and he’s updated his thinking on it just recently in this post http://securosis.com/2008/06/25/the-future-of-application-and-database-security-part-1-setting-the-stage/ )

Some of the skeptics will point to the burden of learning the applications and business logic, especially in a dynamic business environment with rapid changes. Others might point that there's a need to correlate between multiple layers of protection and have a truly integrated protection mechanism that combines all the different protection methods. In an attempt to answer to those skeptics, you can find the some of “secret sauce” components of Imperva’s solutions: Dynamic Profiling http://www.imperva.com/technology/dynamic_profiling.html and Correlated Attack Validation http://www.imperva.com/technology/cav.html

For those with interest, I went into a bit more depth previously on the Imperva blog http://blog.imperva.com/2008/05/waf-defined.html .

Jim and Andre, you guys both have some valid points but let's take a specific hypothetical use case for the purpose of discussion:

example scenario:You are recently hired to oversee web application security for an organization where there was no one in that position before. You immediately find a bunch of stuff that needs to be fixed 'yesterday' and to be frank are perhaps a bit overwhelmed at the scope of the problem. Keep in mind that you are now fully accountable for all Layer 7 security events regardless of how or when they surfaced originally. What are you going to do?

I agree that you will (and should) focus significant time on "activities such as code review, manual pen testing, developer training and fixing the underlying code" . But to be fair, the number of internet facing issues in the web application are so great that it would take months (if not years) to address them all given that you are not allowed to pull all developers off of their current tasks just to fix all of the current web app exposures. In fact, you will likely be competing for developer resources to fix the problems with the business as the company must continue to exist and continually try to make the product better.

In short, you have no choice but to prioritize your efforts. Where will you get the most short term 'bang for the buck' with your efforts?

I submit that you get the greatest short term 'bang for the buck' by blocking as many of the issues that you can at the web application perimeter as soon as possible. This not only buys you the time to fix the issues in a structured and organized way but also also allows you to sleep at night knowing that your pants are effectively down to your knees.

Granted, this is not the case for everyone but in this example, I am hard pressed to find an alternative that does not support deploying WAFs as they currently exist in their current state of evolution.

For the record, I have no financial stake in any WAF vendor and do not in any way benefit financially for my views other than allowing me to keep my current position by offering a solution that effectively mitigates current (short term) web exposure.

@kyle, very well said. Its unfortunate so many inside infosec also lose the distinction between possible and probable as well.

@ryan, on the topic of business logic flaws, you might be right about the cost benefits there. If it turns out to be the case find, we can always go back to the problems we currently have. Maybe we can get lucky with a little bit of innovation. Who knows.

@cyberlocksmith, thanks for the kind words, but I also think you got the fundamental security ideas well in hand. :)

@Jim, Ryan makes a good point. ;) Please point out the person in this discussion who is completely unbiased. Me, I don't make a dime on WAF sales and their being purchased just the same. I just try to make their effectiveness a bit better where possible. Its what customers asked for.

@cyberlocksmith, the scenario you describe is extremely common. I'd wager more common than anything else out there (at least in my experience), perhaps as a direct result from PCI forcing people to at least look at the webappsec problem. Whatever the reason WAFs are an option to be considered as are other approaches. If they're right for you great, if not well, their mere existence or anyone else using them doesn't affect you.

some years ago i wrote an apache module which does something similar. You can find it here (Mod_antiTamper): http://www.wisec.it/projects.php?id=3(also mod_security have a patch for that)It was just a exercise in style but it does its work.Anyway the problem with this kind of approach is that it becomes quite impossible to use javascript to create links and dynamical hyper references.And that's not so good in 2k8 ...:(

@Stefano, who did you not pursue the work? Seems like it would still be viable for a lot of websites even in the face of heavy JS. Developers might be inclined to work WITH the parser to help it as well.

Also seems to me the bulk of JS generated links are for images and style sheets, things where file extension url exceptions could be made by the device.

> Please note that most of the WAF supporters in the industry financially benefit towards their propagation.

>>So says the secure coding consultant ;)

The big different is, I personally focus on education and defensive coding practices to assist organizations down the path of becoming self sufficient when it comes to Application Security Excellence. The WAF crowd, IMO, is trying to get organizations to spend ridiculous amounts of $ on products and services that provides little ROI and little risk reduction.

Take the following code from a real site I am having the pleasure of auditing...the problem is pretty evident. I tested the script on a handful of pages, including login scripts, update pages, and more...and nothing. It did work fine with the sample that came in the package.

Due to the very high failure rate I have experienced, I can't trust it...so, thanks for nothing MS.

@ Manico: right on. That's the key, by helping companies improve their SDLC, you are betting on them becoming reliant on their people and processes, not some technology that will become outdated again in 2-3 years (maybe even less).

@Jim, I'm not saying any of those efforts you specified aren't good value, but I have a question...

In your experience does conducting an average risk assessment, source code review, acquiring developers security tools, and conducting ongoing developer training (miss anything?), collectively cost less than purchasing//managing a WAF? If so, please provide some relative cost estimates so we may better understand where you are coming from.

I'd also argue that measuring a WAFs direct benefit is easier than "secure software" initiatives. You have a vuln one day and not the next. In my experience code fixes have a hard time doing the same.

And remember, WAF aren't implemented or managed by devs, but by IT security who previously were able to assist in the website security effort. I'd argue we have brought someone else that can help who wouldn't have otherwise been able to.

I think that you have to think of the usage of a WAF like a network firewall is used for the network.You can live without a network firewall because every server/workstation has a personal firewall or IP filter. But any IT security guy will argue that defense in depth is a good pattern, so a network firewall and DMZs are necessary.

For me, a WAF is an additional layer to protect a web site.First of all: it will provide you with a decent reporting of hack attempts (blocked or not) so that you have an idea of the need for a WAF, also to have some KPIs for management.

I have some clients that don't have the possibility to patch or rewrite all legacy applications (of which most in Classic ASP!) so a WAF could help in protecting (not securing) these apps.

I have seen some WAF solutions that provide protection that are not part of the development framework and would require a lot of manual effor to code it correctly. I think about hidden field encryption, cookie encryption, state management, brute-force detection, ....

"Companies are spending WAY too much money on network infrastructure defenses and way too little on things like awareness training, software security, and data leakage!

Studies from Gartner and NIST indicate that between 75-92% of security vulnerabilities are due to flaws in software; yet, they also report that over 90% of IT security spend is on perimeter security such as firewalls and IPS. This demonstrates two things to me:

1. CIO’s (who have buying power) don’t understand the real threats

2. CISO’s (who don’t have enough power to open eyes) may understand the real threats but can't do anything about it

There IS NO PERIMETER anymore. And most of the perimeter/network defenses are USELESS against the software problem and the insider threat."

@Jim, ahh this is good connection. I recognized the same issues and also identified that it wouldn't swap anytime soon. This type of awareness and budget reallocation will take time, perhaps 5-10 years from now. What's interesting about WAFs and the VA+WAF concept is that the budget dollars come from infrastructure, not necessarily dev or even IT security. So in that context the solutions you and I both talk about really don't compete time, people, or even budget wise.

A comment little away from what You have written here but still about WAF. Are there any solutions or if no, is it worthy to create such AV that would check files for malicious Javascript/PHP/etc code. For example, when somebody makes passive XSS the scanner will identify it and remove it. Such scanner would run on a web server and from time to time scan all files.

A quick point I'd like to see addressed... There are two distinct groups which hold a key stake in securing web-based applications: developers and security. While the developers (should) know the details of their web applications they typically fail in security understanding; on the other side of the coin security professionals typically completely fail to understand business applications (not for lack of trying) but know security well. Here you have two forces seemingly staring at each other across the chasm of business - both with one piece of the puzzle and neither able to make a good solution on their own. What you have is that each has to compensate for the fact that the other is lacking... which is a huge challenge.

Something else that sucks... you have limited time, money and resources so you're left with a 'good enough' solution - WAFs fill that void reasonably when you do a good enough job configuring the devices.

The point I'd like to see us address is this... at what point does the effort required to make a WAF work tip the scales of utility to where it would be better to simply employ that knowledge to secure the app from within... [inquiring minds want to know]

The "time investment" issue is somewhat of a misnomer. The person who would be responsible for fixing the code (developer) is not the same person responsible for managing the WAF (IT security). So just because time is invested in one, doesn't mean it distracts or takes resources away from the other. Even monetarily speaking this is normally the case.

My company has several hundred web applications. Most of which are being updated/changed frequently. We currently have a standard process of performing automated web application vulnerability scan to detect techincal flaws and manual testing for business logic flaws for every new web site. A WAF is being considered currently. However, I was not sure how much additional value this new investment will bring us? Another concern is that we may have to put in considerable amount of time to continuously fight false positves or making changes to the rules since we have large number of unique applications and they change often. Any suggestions?

Excellent question, but first, why is a WAF being considered at all? Secondly, how many people are dedicated to performing vulnerability scans and assessments on all those sites as they’re being updated? For hundreds (200?), I’m guessing at least 4-5 people or so.

WAFs provide value in several ways including compliance, visibility, virtual patching, and defense-in-depth. Since your VA program appears relatively mature, I’ll rule out compliance since that’s mostly PCI related and your procedures should have you all set for 6.6. Other organizations use WAFs with no intention of blocking traffic, but instead use the device monitor incoming attacks, application behavior, and system load – things standard Web analytics reporting don’t provide. While WAF tuning is always required to achieve proper visibility, false-positives aren’t going to block legit traffic in this scenario. This leaves virtual patching and defense-in-depth.

If your environment is such that code is frequently pushed with known vulnerabilities despite your best efforts to find/fix them ahead of time, then virtual patching maybe a source of great value. This in addition to websites already deployed with vulnerabilities that are unlikely to get fixed in an acceptable amount of time or at all, which is an extremely common occurrence. Now, since you know the vulnerabilities in your website ahead of time, FPs and management headaches can be extremely reduced by leveraging this knowledge. I’ve written extensively on the subject here, here, and here.

Finally, when at all possible, we try not to rely on a single layer of security (including our code) to protect our systems. This is another point where WAFs can play a role – but also the source of the most tuning and management time required since most will try to go with a positive security model for as much of the site as they can. Get up and go time is probably 4-6 weeks at least most sites, and then scales upwards should the site change often. Someone really needs to be dedicated in this function and be skilled in webappsec.

Jeremiah - Thanks so much for the quick response. These are very valuable advice.

Our current security policy requires all identified high risk findings to be fixed prior to go live and we are less concerned about these new ones. The reason we are considering WAF is that we still have about 35% of legacy sites that have not been through our security assessment process and we don't know whether or not they are vulnerable. In addition, our development team makes update to our web applications frequently. We have been pushing them to engage us whenever the site goes through a major upgrade but have met resistence due to the cost associated with our security assessment. We are hoping the blocking piece of WAF can buy us some time before these applications are assessed or retired. We like the idea of multi layer security. However, we are worried that turning this on will have a big impact on our large number of websites (about 300) due to false positves. Putting this in the monitoring mode first to establish a pattern may not be an easy task since our applications are changing frequently.

By the way, do you know if multiple WAFs will be required for the number of web applications we have?

First there’s the issue with performing ongoing vulnerability assessments on all 300 websites and keeping up with the code changes. I’ve always advocated that websites should be assessed on a weekly basis because even if there we no changes, that doesn’t mean the state of the webappsec art stayed the same. New attacks techniques are discovered all the time and a previously “secure” website might be open to them (i.e. XSS filter-bypass in ASP.NET).

Please forgive the shameless plug, but this exactly why we built WhiteHat Sentinel. It’s likely we’d be able to assess the bulk of your websites on a weekly basis for equal to or less than what’s being currently invested in time and money. We also offer integration features where dynamic WAF rules (virtual patches) are generated for F5s ASM and ModSecurity to defend against vulnerabilities (XSS and SQLi) we identify. Feel free to contact me directly if you want to learn more and/or get a demo.

The second issue is what WAFs to consider and what criteria is important. With a WAF hardware appliance (F5 ASM) you almost always need two devices for failover – traffic volume is also going to be a huge factor. Alternatively, you can go with a web server add-on like ModSecurity for Apache, which is well worth a look. There are “learning” WAFs that attempt to profile applications by monitorin traffic, and while set-up is made easier, initial baselines can easily take 2-4 weeks. And if the sites change every two weeks or more, people who’ve deployed them said it’s basically impossible to get them into block mode. This white paper should help you learn more about how WAFs function.

At the end of the day, someone from IT security is going to have to "own" the WAF to care and feed for it. It takes resources, but its better than getting hacked. :)

And the links I forgot to include in my last comment:http://jeremiahgrossman.blogspot.com/2008/06/why-most-wafs-do-not-block.html

How many WAF detractors posting here have actually had to go to the effort of remediating applications in production within an enterprise environment? Also, how are you dealing with legacy applications that the business is hopelessly dependent on with no migration path?

It isn't always as simple as fixing the root cause. Sometimes you need a mitigating control. This is where WAFs excel.