Shortly after the post, I was provided with a copy of the original
research and was able to give a
more thorough analysis of the paper. It is reproduced below, only altered
for better formatting.

After being provided a link to the original paper and reading additional
comments, I wanted to follow-up to my original post with more
thoughts. If you want the slightly more technical review, search down to
"methodology review". The paper in question is "Analyzing Websites for
User-Visible Security Design Flaws" by Laura Falk, Atul Prakash and Kevin
Borders. I strongly encourage more security professionals to provide
peer scrutiny to security research coming from universities.

As was pointed out, the research was done in 2006 (testing in Nov/Dec) but
the results are just now being published. Three people working on a study
on 214 web sites should not take that long to publish. To wait so long in
publishing research on a topic like this, one must question if it is
responsible, or more to the point, relevant. In the world of high end
custom banking applications, my experience consulting for such companies
tells me that many will do periodic audits from third parties and that
these sites get continuous improvements and changes every week. One of the
web sites I use for personal banking has changed dramatically in the last
12 months, making huge changes to the functionality and presumably
architecture, security and design. The results of a 2006 audit of that
site are probably most irrelevant.

As with most research papers, the lack of publish date in the header is
annoying. The abstract does not mention the 2006 to 2008 time gap between
research and publication either. This time difference is seen almost
immediately in the citation of Schechter et al, regarding people
"disregarding SSL indicators". The current releases of several browsers,
most notably IE7 and Firefox 3 make pretty big shifts in how the browser
handles and warns about SSL indicators. Each browser is considerably more
paranoid and will throw a warning over more discrepencies that each would
have ignored in previous versions.

On page 1, Prakash et al list the criteria for the categories of "design
flaws" they examined. As expected and mentioned in my previous post, the
design flaws they examined are not necessarily a vulnerability, and often
times do not put customer data at risk or they require additional
requisites to be exploitable. To look at one of their design flaws as an
example, consider the following:

Presenting secure login options on insecure pages: Some
sites present login forms that forward to a secure page but
do not come from a secure page. This is problematic because
an attacker could modify the insecure page to submit
login credentials to an insecure destination.

This summary of a design flaw is problematic in that it makes several
assumptions and/or does not fully qualify the attack vector. First, to
"modify" an insecure page being served from the bank to the user's client
(browser), the attacker would have to compromise the server (making this
attack moot) or conduct a Man-in-The-Middle (MiTM) attack. I assume the
latter is meant since the implication is that an attacker could not
effectively MiTM attack a page wrapped in encryption (SSL).

It is interesting that the lack of SSL encryption is chosen as a design
flaw with the notion that manipulation of an insecure page is the
preferred attack vector. Such an attack is considerably more difficult to
conduct compared to other threats (e.g. SQL Injection, Privilege
Escalation) and would essentially target a single customers. Many large
applications serving hundreds of thousands of users makes this trade-off
of mixed security pages for performance reasons, as the overhead of
encrypting all traffic can be costly.

Later in the paper when the team attempts to better define this design
weakness, they say:

Consider the case where the customer service contact information for
resetting passwords is provided on an insecure page. To compromise the
system, an attacker only needs to spoof or modify the page, replacing
the customer service phone numbers with bogus numbers.

Web pages can be spoofed regardless of the transport, so the presence of
SSL encryption means little to nothing. If his team is implying an
attacker "only need [..] modify the page", that would require compromising
the server or performing a MiTM attack. Again, this is not a trivial
attack by any means and in the latter, would affect one customer.

While this is only one of five design flaws Prakash's team looked for,
consider the third example which is the exact same design weakness:

Contact information/security advice on insecure pages: Some
sites host their security recommendations, contact information,
and various other sensitive information about their site
and company on insecure pages. This is dangerous because
an attacker could forge the insecure page and present different
recommendations and contact information.

This is the exact same issue as #2 in the list but just makes the
specification of the content on the page. Factor in that issue #1 will be
more prevalent in large organizations but a non-issue in smaller ones and
the criteria of five design weaknesses gets cut down from five to four,
with one that is likely not to be seen on some of the sites tested at all.

The paper quickly summarizes their findings before going in detail, before
concluding "Overall, only 24% of the sites were completely free of these
design flaws, indicating that some of the flaws we identified are not
widely understood, even among institutions where security is critical."
This assumption and conclusion is dangerous and irresponsible. The
implication that the presence of one or more of these flaws is indicative
of the site not understanding the threat is presumptious. With the example
given above about the high overhead of encrypting all content, some of the
"design flaws" may be business decisions and acceptable risk.

Prakash et al begin to demonstrate their lack of understanding of
client-server relationships and the transport mechanism for different
protocols. The following paragraph from page 2 immediately calls his
team's technical competence:

One of the most interesting design flaws we discovered is the
presentation of FAQs and contact information on insecure pages. In
the past, FAQs and contact information were usually sent through the
mail to the customer. It is not generally recognized that this
information should be protected. However, when this information is
presented online, the user becomes vulnerable to socialengineering and
offline attacks as a result of the information being displayed on an
insecure page.

Prakash's contention that unencrypted content delivered from a web server
to a browser is somehow different than when unencrypted content is
delivered from a mail server to a mail reader is silly. If an attacker has
the ability to MiTM attack a person, it isn't going to be limited to HTTP.
Sending that contact information via mail will result in a user deleting
it or maybe storing it in a folder. The first time the person needs to
contact the bank, they would check the web page for the contact
information. If said information is not available, it now further burdens
the bank as they may call a generic number and get transferred around
several times. This adds to customer frustration and causes bank employees
to spend extra time dealing with a customer that could have called the
correct number to begin with.

Prakash's team goes on to make more assumptions or not fully understand
the importance of how web clients behave. Without getting into a full
discussion on the philosophy of e-commerce sites adding mechanisms to
invalidate client-side vulnerabilities, the general notion that it should
be done if feasible seems reasonable. In this context, feasible means that
it doesn't overly burden the bank web site, does not impact performance
and is generally transparent to the end user. One example of this is a
Cross-frame spoofing issue that made it trivial for an attacker to use a
phishing attack to MiTM attack MSIE 6 users. Web sites can add a small
bit of javascript to help ensure that browsers load their pages in a new
frame and essentially mitigates this risk. This is a good example of how
many banks were helping protect customers, even though the vulnerability
was in the customer's software, not the bank web site. Prakash's team
claims:

Our work is similar in that some of the flaws that we consider impair a
user's ability to make correct security decisions. However, our work
differs in that the cause is not poor or confusing client-side
interfaces. Instead, the flaws originate in poor design or policy
choices at the server that prevent or make it difficult for users to
make correct choices from the perspective of securing their
transactions.

While a mismatched SSL certificate used to be virtually ignored in some
cases, new versions of popular browsers now behave differently in how they
alert users, giving them the ability to more easily make correct choices.
Claiming that this research is not impacted by "poor or confusing
client-side interfaces" is misleading. While the older browsers were not
necessarily confusing, they handled some situations regarding establishing
trust poorly.

The next area of technology Prakash's team doesn't seem to fully
understand is vulnerability scanners. In the paper his team says:

Network scanners, such as Nessus [11], and application-level
website scanners, such as AppScan [17], can be used to analyze for many
configuration and implementation bugs, such as use of unpatched
services and vulnerability to cross-side scripting or SQLinjection
attacks. As far as we are aware, the design flaws that we examine are
currently not identified by these scanners.

Both Nessus and AppScan will identify several vulnerabilities that
directly relate to the design flaws outlined. Both will give warning over
invalid or expired SSL certificates, AppScan will warn about mixed-mode
security pages and neither will perform tests for some of the design flaws
listed (#3, #4, #5) because no scanner in the world can do it.

Methodology Review:

On page 8 (of 10), the team gives very brief descriptions of their testing
methodology. The lack of description or their testing methodology
undermines significant portions of the research. For "Break in the Chain
of Trust", the paper says "Under no circumstance should an insecure page
make a transition to a securitysensitive website hosted on another domain,
regardless of whether the destination site uses SSL." This is an arbitrary
'rule' that is not widely accepted by anyone including the banking
industry. Many web pages are designed to act as portals that link to
additional features. The 'rule' as quotes from the paper would force large
bank organizations to consolidate all web resources on a single domain.
While that may be nice, it simply isn't feasible to many businesses,
especially ones with a large organization that includes multiple
companies. Linking from http://bigbank.com/ to https://regionalbank.com/
is perfectly acceptable and should use proper SSL certificates and
technology controls to help ensure the user ends up on the correct page,
loaded directly by the browser.

The second design weakness studied was "Presenting Secure Login Options on
Insecure Pages". The paper explains their methodology as ".. searched each
web page for the string "login". If the string was found, we searched the
same page for the strings "username" or "user id" or "password". If the
string .login. and .username. or .user id. or .password. were found on the
same page, we then verified whether the page was displayed using the HTTP
protocol. If this was the case, we assumed this site contained the design
flaw." The key word here being 'assumed'. There are scenarios where the
above methodology could easily generate a false positive. Even back in
2006, there were trivial ways to more easily determine the use of HTTPS
with certainty.

The third design weakness studied was "Contact Information/Security Advice
on Insecure Pages" and is perhaps the most technically lacking testing
method one could perform:

We searched each web page for the string "contact", "information",
or "FAQ". If those strings where found, we checked whether
the page was protected with SSL. If not, then we considered it to
contain the design flaw.

The mere presence of these words on a site do not mean they are in the
context of listing bank contact information. While 'contact' will
frequently link to a 'contact us' page, looking for 'information' or 'FAQ'
is absurd.

In the fourth weakness, "Inadequate Policies for User IDs and Passwords",
the team openly admits that their methodology may produce "optimistic"
results and that they had no way to verify their results "without
generating an account on the website". Heaven forbid they find a couple
hundred students at the university to participate by logging into their
personal banks and checking this in more detail. That extra effort would
have made this portion the only positive and accurate test. From the
paper:

Our count could be optimistic; some sites may require strong
passwords without stating an explicit policy. We had no obvious
means of verifying this without generating an account on the website.
Our count could also be conservative for sites that have poor
policies resulting in weak passwords. Thus, our results for this design
flaw should only be taken as a rough estimate of the extent of
this particular problem.

As before, the fifth design weakness was extrapolated using a glorified
'grep' of the web page, analyzing proximity of a few keywords and then
verifying the hits above an 85% threshold. And as before, this testing
methodology makes huge assumptions about the wording on the page, does not
positively account for HTML formatting that would impact the 'distance'
between words (especially in pages with frames) and does not begin to test
the functionality (see page 9, section 4.5).

Finally, the paper attempts to interpret the results of this poorly
conceived and improperly tested study. Table 1 on page 10 says that 30% of
the sites tested were affected by "Break in the chain of trust" but gets
contradicted (clarified) in the first paragraph of the results:

With automated tools, such as the one used in our study, false
positives are possible. To the extent feasible, we manually examined
the results to eliminate false positives from the reported data.
Our break-in-chain-of-trust data had a significant number false
positives. Our automated tool reported about 30% of the websites to
potentially use third-party sites in an unsafe way, but only 17% were
found to do so without giving some sort of notification to the user
about that transition.

Having such an admittedly large margin of error on the results based on
their own methodology should be an eye-opener in regards to the integrity
and accuracy of the results. Despite this revelation, the next paragraph
immediately cites the table with the 30% number and begins to make
conclusions based on the bogus numbers. They go on to further explain
their primary tool for gathering information (the wget tool) may not have
retrieved all of the information needed to properly assess the site.

Despite the weak methodology, 44% error rate on at least one test and
admitted errors, Prakash et al go on to say "We found that 76% of sites
have at least one design flaw." Such statements are certainly not factual
or even statistically correct based on the research presented.

Prakash followed up with a brief reply that was dismissive of my write-up,
and only called for more research of bank sites. What he failed to understand
is that more research of this nature, with huge gaps and a lack of understanding
of the subject matter, will only produce more statistics and articles that
have no relevance or meaning in our industry. Worse, they serve to provide
inaccurate information that could only be used as a basis for poor decisions
for people with an interest in information security.

Copyright 2008-2010 by Brian Martin. Permission is granted to quote, reprint or redistribute provided the
text is not altered, and appropriate credit is given.