Windstream's Downstream Throughput

140813.1942 - Windstream has a new record! I've had some form of online connection pretty much
continuously since the mid 1980s. Windstream had set the record for the single longest outage ever on 100418 (9 hours), but they now have blown past
that record DRAMATICALLY! See that blank spot from 140806.0300 to 140811.1300? That's a 130-hour outage!!! Now,
life in the country without TV or radio or news or the ability to make a phone call is not without its charms, and our job here is to
measure speed, not availability, so we ignore connection failures even if they are Windstream's fault, but if you think
130-hour outages in the 21st Century are unacceptable, you might want to look elsewhere for an ISP.
See the updates for a history, and see how it looked when we started.

This is Windstream's
downstream throughput (in Mbps, with a linear regression line) in Arroyo del Agua, New Mexico
(87012). At the lowest tier they promise 3 Mbps; here is what they deliver.
When I subscribed (I had no choice; they are the local telco monopoly and this
valley has no cable), I wanted the fastest tier, but I decided to try the lowest
tier first—to see if they could deliver on that—before paying for more.
They FAILED! I have compared numerous data-points with my neighbors, and if you're
paying for 12 Mbps, you're really getting burned: higher tiers seem to fare no better than lower tiers
at peak times.

--sorry, multi-chart navigation needs javascript--

J'Accuse!

I believe
the cause of this is that their backhaul
is radically oversubscribed but they are
too cheap and greedy to invest in the infrastructure necessary to provide their
customers what they promise in their—essentially fraudulent—sales literature,
where I don't remember seeing anything like this:

Expect actual broadband only when you're asleep or at work. When you're home and
awake expect something more like dialup. You remember dialup don't you? Welcome back!

In networking, for high quality throughput
you must build to peak demand, or at least devise a combination of infrastructure and
compensating strategies that adequately approximates that ideal. Windstream doesn't
even come close. The problem is that if your ISP does not care to give you what they
promise, and is disinclined to be honest about it, they can usually
dismiss speed test data as anecdotal or blame it on internal wiring. Before dealing with Windstream, I didn't think a
large telecom would have a systematic policy of trying to deceive its customers,
but if you call 'tech support', you'll get an amazing runaround; you'll be told to unplug things and restart things
by people who obviously have no idea what they're doing, and assume you're too clueless to notice.
You might be told that from 'there' your speed is fine, although if they are actually
testing something and not just faking it, their test packets must be getting a
much better QoS priority
than their customers'. A sign that they might actually be testing something is that
increasingly I got "Hmmm, yeah, that's not good..."
A sign that they're faking is that I've been told to unplug my ethernet cable and
then been told "Ah see, the speed JUMPED up!" while watching wireshark
across the NIC
showing that NO packets were going ANYWHERE.
You might be told that they'll send someone out, even though you say that's a waste of a field tech's time because
there is nothing to fix. You can point out that your gateway's diagnostics
report the Signal/Noise Margin and Attenuation between
you and the DSLAM are 25.1dB and 5.8dB respectively, that these are excellent numbers,
and that MLab's NDT
probe always produces variations on the following.

This connection is network limited 99.72% of the time.

Information: throughput is limited by other network traffic.

Information: Other network traffic is congesting the link.

The 2wire tester, which Windstream used to recommend
in their tech support hold message, often reports speeds of under 100 Kbps!
You can point out that all of this testing is moot, because your wiring
does not have emotional mood swings that cause radical
fluctuations in throughput. When probed, the network equipment claims to be OC-48,
which has nominal bandwidth of ~2.35 Gbps. Back in August 2009, when I first signed up, the throughput was
fine for a few days; then for a little while it seemed to be
reasonable during the day, sometimes peaking above 3 Mbps, becoming really bad only in the evening; by March of 2010 it was horrible all day long,
with occasional anomalous peaks at about 80% of what was promised, prompting me to create this page.
See updates for what transpired subsequently. If you ask,
you will not be told their oversubscription rate. Apparently Windstream thinks
it is more cost-effective to cheat and lie to their customers, and hire legions of untrained people to pretend
they are tech support than to replace their defective switches or slide some more fiber down that tube.
I have fruitlessly argued all of these points with those legions. Here therefore are my data. I
challenge Windstream to offer an alternative explanation to mine.

Design Notes

To produce this page, I wrote a Java client/server socket pair that attempts
to be as 'generous' as possible to the connection. Its results tend to be about equal
to Ookla's, and much more
generous than 2wire's. It doesn't include initial latency
and socket set-up in its calculations; it only starts the clock after the first (8K)
test packet is received, not sent. Its throughput on
localhost
is ~120 Mbps, on a slow (1.4 GHz) single-core Intel P4, and since the load is normally
split between the client and the server, we can pretty safely ignore CPU load
as a significant contributor to slow throughput in the sub-3 Mbps range (and indeed
CPU usage is empirically negligible). The client runs as a Linux cron job or a Windows
scheduled task, but can be invoked manually any time. The server runs on a low-load
backbone-connected Linux server. Each run produces a log entry on the client that is then sent to the server.
The server's log is rendered out to this page as a PNG by a PHP page that invokes
Google's pretty sweetChart API
(although a built-in regression line would have been nice).

Updates

100313.2330 - Added a regression line. We seem to be trending upward! Let's see if it's just the weekend...

100314.1207 - Flattened out.

100316.0100 - Changed the auto-check interval to every hour, then took a few hours off.

100316.2250 - Added the ability to split charts and scroll through them (look for side arrows).

100320.1714 - Today, for the first time, the single-chart mean is above 1 Mbps.

100322.1204 - The client was down for 3 hours last night, so Windstream got a free pass at what looks like a time of abysmal performance.

100329.1640 - ...I said great, as long as it's a fix and not just preferential treatment for my packets.

100330.1400 - Got a call from a 'level 2' person at Windstream; he said it was a 'known issue' that they were working on.

100410.0021 - It still all looks pretty much the same to me.

100420.1501 - Notice the big data gap on 4/18? Lines went completely dead throughout
the region at about 00:45 on the 18th. Interestingly, DSL was back up before voice, about 9 hours later at 10:00. Local-only voice
was back by 22:00. Long distance was up the next morning. I hope nobody in the area needed 911 for those 21 hours. BTW, the mean
is back solidly below 1 Mps, and that is ignoring the dead time on the 18th,
which, strictly speaking, should have counted as 9 zeros.

100507.1632 - Well...it's been over a month since Windstream offered to help me with 'my' problem, and
nothing has changed. I guess we can safely conclude that they have no answer to my challenge. Have you noticed that every
bill is higher than the last? This is the first time I have ever seen a utility do this: inexorably raise the bill
every single month, without delivering what they promise, and without explanation. Perhaps it's time to turn over my results
to the FTC or PUC, or whoever else is responsible for protecting citizens from this sort of behavior.

100521.1407 - Starting yesterday morning we look fixed! Is it time to retire this page? I'll let the
test run a little longer, then, as I promised Mollie on 100329, run it from a neighbor's. If the fix looks real,
we might be done! I feel like Ralph Nader
or Upton Sinclair. :)

110305.2111 - My neighbor's results were always suspicious-looking, as
if the fix had not been real, as if Windstream had somehow been giving special priority to my packets.
Now—suddenly—it looks as if Windstream has decided to stop doing that, so we're back! Here,
for comparison, are my results in May 2010, when 'the fix was in'—apparently just to shut me up.

110724.1931 - I was in Seattle for few weeks, and when I came back on July 6 Windstream was completely down for all but
5 or so of the next 72 hours. Their downstream throughput seems to have improved though. We'll see...

120227.2320 - That big gap starting 120206 wasn't Windstream's fault; I was away on the beach!

120626.2120 - The gap at the end of May WAS Windstream's fault: four whole days completely down!

131215.1800 - After being reasonable (though not great) for a while, peak-time
throughput has degenerated radically down to nearly 0 since December 2, 2013.

140623.0031 - After a few good months, peak-time perfomance started to degrade
in May, and now it's complete garbage. With a nominal speed of 12 Mbps, peak-time throughput is often less than 1
Mbps. Normally throughput starts to recover at about midnight, but at midnight tonight it was barely 0.5 Mbps with no load from me.

140705.1507 - Worse than I could possibly have imagined! From about 7:30 in the morning
until about 2:30 the next morning every day, with nominal 12 Mbps and approximately no load from me, throughput is around 1 Mbps with lows approaching 0.