West Virginia: Library’s $20,000 router cost more than the building

Congress wants answers about how broadband funds were spent.

Public libraries rarely own and operate Internet routers that are worth more than the building in which the library sits. Rather incredibly, however, that was the case for at least one library in West Virginia. The state has been pilloried for using federal funds to buy enterprise-grade Cisco 3945 routers at around $20,000 apiece and then installing them at every "anchor institution" in the state, no matter how small. That led to many ridiculous situations, though the most ridiculous of all may have taken place in the town of Marmet.

"The state installed a $22,600 router at the Marmet Public Library, which has a single Internet connection," according to Charleston Gazette. We covered the Marmet case earlier this week but didn't realize just how crazy the story was. Thankfully, the paper has the punchline: "The router cost more than the trailer that houses the Marmet Library, according to Kanawha County Commission staff."

Such stories have captured the attention of federal officials and Congressional reps. Staff from the House Energy and Commerce Committee wanted a hearing on whether American taxpayers are getting their money's worth from the billion spent on broadband projects. The West Virginia router story came up in a committee memo setting the stage for the hearing.

"At a May 2012 hearing, Communications and Technology Subcommittee Chairman Walden and Rep. John Shimkus raised questions regarding a series of press stories alleging West Virginia had bought expensive, enterprise-grade routers for tiny libraries with only a few computers, and that hundreds of routers were also sitting in warehouses yet to be installed," it read. "Following the hearing, they sent letters to West Virginia and the Department of Commerce Inspector General seeking additional information. Audits by the Inspector General and West Virginia each recently concluded that the purchases were excessive, with the IG estimating overspending of between $500,000 and $1.2 million and the state auditor estimating the waste at potentially $9 million."

At the hearing held this morning, Rep. Greg Walden (R-OR) opened by noting "the Department of Commerce Inspector General and a state auditor have both recently concluded that West Virginia overspent hundreds of thousands or even millions of dollars on enterprise-grade servers for small libraries with only a few computers."

The Department of Commerce sent an assistant inspector general, Ann Eilers, to the hearing today. She confirmed that a federal audit of the West Virginia program found numerous problems. Among them, the state "had not demonstrated that [broadband] funds used to purchase routers were spent cost-effectively."

She also noted the state "did not perform a study to determine the appropriate size router that would most effectively and efficiently meet" each site's needs. Instead, it just bought 1,164 routers capable of serving hundreds of concurrent users and sent them out to agencies. The predictable result was plenty of over-sized routers, as at Marmet, and routers that weren't a good fit for their deployment (the West Virginia State Police have only deployed two of the 77 routers they received due to incompatibility with the current police VoIP telephony system).

Nevertheless, the state is pushing back. When the Department of Commerce report first appeared last month, the governor's office told the Charleston Gazette, "We do not believe purchasing routers with enhanced capabilities for our 'community anchor institutions' was a waste of taxpayer resources. Instead, we view this project as an investment in West Virginia's future."

The larger picture

Still, despite the millions overspent on one particular piece of the much larger West Virginia grant project, outright fraud has been minimal, and waste generally appears to be controlled through federal audits and site visits. Lawrence Strickling, the federal official ultimately in charge of $4 billion in federal broadband expenditures, provided numerous examples to the committee of how the money was being spent. Here's one of the most compelling, from Maine:

The Three Ring Binder project—one of the first awards announced in December of 2009—which is supported by the Maine state government, the state university system, and a group of small telecom carriers, used $25.4 million in Recovery Act funds to build a 1,100-mile dark-fiber network across the state consisting of three interconnected fiber rings. Thirteen local carriers are now leasing that fiber to bring broadband to rural communities that, in many cases, previously had only dial-up service...

One of the 13 local carriers leasing fiber is Pioneer Broadband, which serves Aroostook County, a poor, rural county of potato fields and blueberry barrens where Interstate 95 literally comes to an end. Pioneer is leasing capacity on the Three Ring Binder network to bring DSL and even fiber-to-the-home to a string of remote towns that had no broadband whatsoever until now.

The University of Maine system will now be able to bring 10-gigabit connections to all seven university campuses to support big data-driven research and collaboration with other major academic institutions around the nation. The project is also turning on a 10-gigabit connection to the Jackson Lab, a genetics lab, so that it can exchange extremely large gene sequencing datasets with a new facility in Farmington, Connecticut.

Axiom Technologies is using a $1.4 million broadband adoption grant in very innovative ways in Washington County, Maine. It is transforming Down East Community Hospital—a 25-bed critical-care hospital in Machias, connected by the Three Ring Binder project—into a teaching facility for nursing students. The grant paid for video-conferencing equipment that allows nursing students to take necessary classes through a nursing college in Lewiston, nearly 200 miles away...

Axiom is also equipping local lobstermen and blueberry farmers with rugged wireless devices, broadband connections, and broadband training to help them manage extensive state data collection and reporting requirements. Axiom is developing software to move these tasks out of old-fashioned paper-and-pencil logbooks and into the electronic realm.

Ah, the story that keeps on giving. A few Ars posters even mentioned how the router was probably more expensive than the building. Didn't think they would actually be correct. Good to hear at least some of this Federal money was spent more intelligently.

There is just no need for any of the garbage Cisco sold them... They could have deployed sonicwalls at each location and done their VPN connections in a much more cost effective way. A small TZ unit would work at many of the very small branches, and you could throw one of the NSA series as a core or if needed the super massive unit...

Being a non-network person, what exactly are these routers doing. At home I hook into a WiFi box which connects to a cable modem which talks to my providers network. I assume somewhere in the cable companies wiring there are routers and such to direct data to me and my neighbors, the size and power is going to be determined by how many are served by each device and how much data is supposed to flow through them. At work, we have closets with boxes, do not know if they are switches or routers, that connect with other buildings with more boxes and eventually things get out into the wild.

What is the purpose of the Library router? Is it just covering the 2 or 3 devices in the building that might require being hooked up to the internet or are they using it to hook up the whole town? Is there a local internet provider? I can see a small school with 40 or 50 devices might need a router of some type, but a 1 room library does seem extreme.

It may not have been outright fraud in WV, but it was piss poor money management, poor planning/oversight and most likely slick talk from certain sales reps to those in charge of handling this. Carpet calling needs to take place on a serious level.

Seems likely to me that this is a case of a non-tech person asked to do something technical and just calling a vendor and asking for a solution. Of course the vendor suggested the most hilariously overbuilt solution possible with the highest profit margins, that's their job. It's the WV tech guy's fault for not doing his homework and laughing in his face.

Being a non-network person, what exactly are these routers doing. At home I hook into a WiFi box which connects to a cable modem which talks to my providers network. I assume somewhere in the cable companies wiring there are routers and such to direct data to me and my neighbors, the size and power is going to be determined by how many are served by each device and how much data is supposed to flow through them. At work, we have closets with boxes, do not know if they are switches or routers, that connect with other buildings with more boxes and eventually things get out into the wild.

What is the purpose of the Library router? Is it just covering the 2 or 3 devices in the building that might require being hooked up to the internet or are they using it to hook up the whole town? Is there a local internet provider? I can see a small school with 40 or 50 devices might need a router of some type, but a 1 room library does seem extreme.

It is my understanding that all the branch locations where connected together with VPN connections. so the router handled the internet/VPN connections and probably not much else... total waste of money - and there is NO way anyone could ever justify the costs...

Probably the saddest part of this whole story is that federal funds allocated to a pet cause were used to purchase an expensive router when the town has a crappy small library. The funds would have been far better spent in this town on better physical facilities, or a computer lab... or almost anything else to help the citizens of this town better educate themselves and improve their children's chances of getting job in a tech economy.

Seems likely to me that this is a case of a non-tech person asked to do something technical and just calling a vendor and asking for a solution. Of course the vendor suggested the most hilariously overbuilt solution possible with the highest profit margins, that's their job. It's the WV tech guy's fault for not doing his homework and laughing in his face.

Oh, I thought it was the vendor's job to put together a fair proposal that meets his client's needs.

I agree the WV tech guy didn't do his job... but neither did Cisco. Due-diligence fail all around.

Being a non-network person, what exactly are these routers doing. At home I hook into a WiFi box which connects to a cable modem which talks to my providers network. I assume somewhere in the cable companies wiring there are routers and such to direct data to me and my neighbors, the size and power is going to be determined by how many are served by each device and how much data is supposed to flow through them. At work, we have closets with boxes, do not know if they are switches or routers, that connect with other buildings with more boxes and eventually things get out into the wild.

What is the purpose of the Library router? Is it just covering the 2 or 3 devices in the building that might require being hooked up to the internet or are they using it to hook up the whole town? Is there a local internet provider? I can see a small school with 40 or 50 devices might need a router of some type, but a 1 room library does seem extreme.

From the previous stories, they are using these enterprise/high end routers for one, possibly two connection at the trailer library there and in other rural areas in WV. These are meant to be used in large corporate environments handling several hundred machines(I believe upwards of 600+). Installing that type of router in that location is akin to using an 18 wheeler to move a box of twinkies. Massive over kill.

This was at the state level. The federal government gave WV $20 million to update their internet infrastructure in the rural areas, then the WV state government made that money disappear faster than at a David Copperfield show.

I've been wondering about this - I suppose a cheapo consumer-level router isn't the best idea, but wouldn't something like a, say, $150 Netgear Prosafe or equivilent small-business router do the job just as well as some of the enterprise-level Cisco routers some people have been mentioning?

I've also seen stupid (though not on the same level) wastes of money at my public high school. For instance, they bought a ton of iPads (that don't get used very efficiently) around the same time they were begging to pass a measure to tax the community more so they could continue offering 7 periods as an option. Or, they got lots of teachers monitors with pen input, even though half of them don't even use them. Let's not mention how they got i5 machines with 6450s when they're only being used to run Chrome and Word (these replaced old crappy Pentium 4 machines after all). There's a lot of waste there that they could easily trim.

Seems likely to me that this is a case of a non-tech person asked to do something technical and just calling a vendor and asking for a solution. Of course the vendor suggested the most hilariously overbuilt solution possible with the highest profit margins, that's their job. It's the WV tech guy's fault for not doing his homework and laughing in his face.

Oh, I thought it was the vendor's job to put together a fair proposal that meets his client's needs.

I agree the WV tech guy didn't do his job... but neither did Cisco. Due-diligence fail all around.

Apparently the client didn't know their needs. Or bother with actually finding them out. Hilarity ensues. This actually might be laughable or at least serve as a useful lesson if not for all the wasted tax dollars.

I've been wondering about this - I suppose a cheapo consumer-level router isn't the best idea, but wouldn't something like a, say, $150 Netgear Prosafe or equivilent small-business router do the job just as well as some of the enterprise-level Cisco routers some people have been mentioning?

That's what we're saying. We're also saying someone thought $20,000 Cisco routers for everyone was a good idea, and it was more of a faux pas on the part of the guy who thought of this idea.

Though, I'd love to be a Cisco salesman right about now. Could use the commission.

I take it the sales droids and the purchases made sure that someone who was actually technically knowledgeable about the actual deployment was excluded from the meetings. Either that, or someone applies the worst case scenario of one place which actually needed the high-end router, applied the rule of "same equipment reduces maintenance costs", and applied it to every location, all in the name of "convergence." Combine that with "we got this nice grant, and we must spend EVERY SINGLE DOLLAR!"

I take it the sales droids and the purchases made sure that someone who was actually technically knowledgeable about the actual deployment was excluded from the meetings. Either that, or someone applies the worst case scenario of one place which actually needed the high-end router, applied the rule of "same equipment reduces maintenance costs", and applied it to every location, all in the name of "convergence." Combine that with "we got this nice grant, and we must spend EVERY SINGLE DOLLAR!"

You're making a big assumption that there actually were meetings . . .

I won't disagree that the router was massive overkill; but I have to object to "cost more than the building". Perhaps they got a good deal on the building, but I just did some mid-level renovations on my house; $20k for a to-code (even WVa code) building of that size (from scratch) would be an incredible steal. Labor and materials would easily be 50k or more, plus the value of the land and the other equipment needed.

So, this internet grant.... was this under the category of "It is essential that we deploy this network, otherwise, people are GOING TO DIE!!!!"? Or, was it under the category of "gee, it would be nice to have, but we can get by without it."? Somehow, I think it got started under a congresscritters idea of, "Gee, wouldn't it be nice to have this shiny Internet in my state, thus helping me stay in office."

I'm not sure what's worse, that this actually happened or every other comment has been to the effect of "Yeah, happens in [insert place, job, company, etc here] all the time."

Is it just me, or does anyone else find the squandering of funds so that the entity doesn't "lose" them asinine? I wish that people were sensible and they could say "You know what, we don't need $25 million this year, we just need $5 million". Then, I wish the government entity dealing out the money would actually have enough brains to believe the same organization when they say "Well, we didn't need any money last year, but this year we need $X amount because we had X, Y and Z happen". The government entity could do the research, determine if that is true, and give out funds accordingly. Instead, we have this retarded "use it or lose it" system/mentality where people waste money intentionally to establish a track record of "needing" money.

We need to totally redo the system of how people obtain resources from their higher, but frankly, I don't think the people in the organizations or the government can be trusted to just do the right thing. I don't think the people in the organization will truly and without bias say "this is what we truly need this year", and I don't think the government in turn will fully trust the organization that they are asking for only what they truly need. Sad really..

I won't disagree that the router was massive overkill; but I have to object to "cost more than the building". Perhaps they got a good deal on the building, but I just did some mid-level renovations on my house; $20k for a to-code (even WVa code) building of that size (from scratch) would be an incredible steal. Labor and materials would easily be 50k or more, plus the value of the land and the other equipment needed.

The "building" is a trailer -- as in "pre-fab" (no construction required). The only installation was setting it on the pad, and ensuring all connections (water, electric, and now Internet) were working.

I won't disagree that the router was massive overkill; but I have to object to "cost more than the building". Perhaps they got a good deal on the building, but I just did some mid-level renovations on my house; $20k for a to-code (even WVa code) building of that size (from scratch) would be an incredible steal. Labor and materials would easily be 50k or more, plus the value of the land and the other equipment needed.

They were basing this on the current value of the trailer. Trailers depreciate. Actually it was a fairly cheap shot on their part. The building was usable and will stand for quite some time. Most small towns can't afford even that level of structure.

I won't disagree that the router was massive overkill; but I have to object to "cost more than the building". Perhaps they got a good deal on the building, but I just did some mid-level renovations on my house; $20k for a to-code (even WVa code) building of that size (from scratch) would be an incredible steal. Labor and materials would easily be 50k or more, plus the value of the land and the other equipment needed.

The building looks essentially like a single-wide manufactured home. Such structures are built basically on an assembly line and therefore are fairly economical. Less than $20k actually is pretty reasonable.

I won't disagree that the router was massive overkill; but I have to object to "cost more than the building". Perhaps they got a good deal on the building, but I just did some mid-level renovations on my house; $20k for a to-code (even WVa code) building of that size (from scratch) would be an incredible steal. Labor and materials would easily be 50k or more, plus the value of the land and the other equipment needed.

Are we looking at the same picture of a single wide trailer that was at the top of this article? If the picture is of the actual one room library I would say that you probably work for the West Virginia government accounting office based on your post.

Money-management is often lacking in government, there's little to no impetus to spend wisely, the perversity of the system often encourages you to make sure you spend it all, for fear of having your budget cut next time around.

There's normally some level of accountability that helps with some of the absurd spending, like in this case.

My question is, when they sell the router are they free to use the money for other projects?