Are the popular speed tests (cable, dsl, etc) that are available accurate? if so, how?
Also, it seems that they only measure download speed of one file and not responsiveness, latency, robustness, etc.
Does anyone know how it should be measured? any links to good articles on that topic?

8 Answers
8

I'd say most speed tests are flawedif you want to measure web browsing performance (which is what a sizable number of people will be doing). Almost all tests use Flash or Java, totally bypassing your browser HTTP stack. These tests also normally use a single large file, totally bypassing the benefits of persistent connections and HTTP/1.1 pipelining only some browsers support natively. If you do use Flash or Java to download single large files, then these tests may be more relevant.

Also, I've seen great variability in results across browser for some flash tests, which has to do with the way browsers handle flash animation, thus skewing results with no relation to network performance.

To make these tests better, the methodology needs to reflect:

That tests are tailored to what is being tested! Don't use flash unless you are testing flash network performance on single files. A browser test should use multiple resources (remembering CSS and JS can block successive downloads in some browsers) across multiple tabs to reproduce real circumstances. If you are testing torrent speed, you need huge numbers of small bandwidth HTTP/UDP connections.

Measuring variability. Most tests do something once, and a single sample is not sufficient to know something reliably. One needs to average across repeat tests, and critically, variance needs to be estimated. Is 180 really better than 150 when the variability is +- 100??? {answer: no}

Not sure of a perfect resource for this, but Steve Souder writes extensively on web performance at least:

The tests others have suggested all fall foul of these technical problems. They will give you a rough number that may help generally, but you should be aware of the limitations. Particularly fancy flash animations seem utterly useless, and as I've said actually can skew results significantly.

Good points about Flash. Speedtest.net looks pretty and it told me my ping was decent, but when I actually tried to play an FPS the lag was horrible.
–
hyperslugAug 26 '09 at 17:50

1

Small FYI on HTTP 1.1 pipelining: The big browsers either don't support it at all (Internet Explorer, Safari) or else don't have it enabled by default (Firefox). I think Opera is the exception.
–
Chris W. ReaAug 26 '09 at 19:19

A good connection is made up of key parts: Through put and responsiveness. There are more but I am sticking to the main 2 (in my opinion). Through put is how much data can be pushed or pulled through the connection (download/upload). Responsiveness is how fast does my request make it the destination and back. Both affect each other a bit.

Through put is a pretty easy test, just go to one of the sites listed here and it will test your upload/download speeds. Just remember that the connection you pay for from you ISP is an UPPER limit (most of the time). For example I pay for a 6mb down/ 1mb up connection. 6mb down/1mb up is the uppper limit of my connection. Some ISPs will have a lower limit. An example would be if an ISP says your connection will be between 3mb down and 6mb down. The other key part is the remote host that you are pulling from or pushing data too. If you connection is bigger, then you are limited by the remote host connection and effectively DoS the remote host.

Responsiveness is a bit hard to test. I am not 100% sure on the correct way to test it. commands like ping and tracert in window are a good start. They will give you a rough idea of how long a round trip to and from a remote host takes. This also takes into account the time the remote host takes to process your request and send it back so take this with a grain of salt. Responsiveness is not a big deal when surfing the web or chatting on line. It comes into play with applications like online games. A slow response time will cause you to "lag" and make the game less fluid to the point of unplayable. Online games rely on fast response over higher through put.

Other things to consider in evaluating a connection are Uptime, what ports are blocked, ISP policies regarding us of Bittorrent, IRC, SMTP, or other services.

I know this doesn't directly answer your question, but given the title of your question, there will be a large amount of google traffic hitting this page looking for speed tests.
–
JeffreyAug 26 '09 at 16:58

Speedtest is not particularly accurate these days. A lot of ISPs have a 'turbo mode' that works for a few seconds and doubles or triples your bandwidth for that very limited timeframe. speedtest is completely fooled by this, as are most other sites, and shows only your speed during this turbo mode. If you care about speed, you probably care more about sustained speed.
–
ChrisInEdmontonAug 26 '09 at 20:45

In addition to using just a speed test often it's helpful to find out packet loss as that can give the appearance of a slow connection as well. Under both Unix and Windows you can use the traceroute command, or if you want a nice GUI program you can try something like Ping Plotter.

The reason for this is that the Internet is made up of many connections, and your connection to your ISP is merely one of them. These speed test websites are convenient to use, but results may vary simply based on the Internet traffic between your ISP and that speed test webserver.

If you want to measure the speed from you to your ISP, then you need to use a server on their network.

On the other hand -- at some point you will want to connect to some server outside of your ISP. (Let's say, www.superuser.com) The connection from your ISP to the outside world is important too. A good ISP will have several connections to the Internet, so if there are problems in the AT&T network, you will still have connections with Global Crossing, and Level 3's networks.

To get an idea of this, you probably cannot do much better than to try various speed test websites, at different times of the day and days of the week, and see what patterns you notice.