Posted
by
Soulskill
on Friday May 11, 2012 @02:50PM
from the your-tax-dollars-at-work dept.

DesScorp writes "The Charleston Gazette is reporting that the state of West Virginia has purchased hundred of enterprise class routers from Cisco at over $22,000 dollars apiece via federal stimulus money. The stimulus cash was intended to spread broadband coverage. The problem is that the routers are overkill, and are being placed in small schools and libraries with just a handful of users. The West Virginia Office of Technology warned that the purchase was 'grossly oversized' for the intended uses, but the purchase went through anyway. Curiously, the project is being headed up not by the state's usual authorities on such matters, but by Jimmy Gianato, West Virginia's Homeland Security Chief. In addition to the $24 million contract signed with Verizon Network Integration to provide the routers and maintenance, Gianato asked for additional equipment and services that tacked an additional $2.26 million to the bill. Perhaps the worst part is that hundreds of the routers are sitting in their boxes, unused, two years after the purchase."

I've been visiting with my parents here in WV and saw that story in the local paper a few days ago. I have to believe that someone had a buddy getting a commission, because that's how it generally goes here. I remember seeing this map [westvirginia.com] a couple weeks before and can't help but think it'd be a better option for spreading broadband.

Why not just mandate that all telephone companies MUST offer DSL to any customer that asks (in the same way government mandates companies must provide phone service). Instant broadband coverage to everybody who wants it.

Because then people scream 'communism' and rewrite history to pretend that the regulation that resulted in everyone having phone access didn't work and didn't provide a massive economic boost to the country.

Though it would be far form 'instant', a massive amount of infrastructure needs to be built, but there is a game theory element to it where telcos are generally hoping one of their competitors makes the investment instead.

If the phone company is going to charge me $1000/yr for DSL, and place a new CO just for me, then they better be able to get several hundred others in my neighborhood to also get service from the same CO. There's no way that my $1000/yr will pay for it.

If a mandate went in that all companies had to provide DSL to all possible customers, I guarantee there are some people who would be told that their service would costs thousands per month, because of their location. Now, you may think that this is easy to solve, by just price-fixing the cost also. If feel this way, then you should consider voting for Jimmy Carter this year.

FYI, you can use repeaters to overcome this. Centurylink has used a few repeaters to get DSL out in my area for only $50 a month. Grated the speed is only 1.5mb but I will take that over satellite any day. I live 12 miles from town or around 9 miles from the closest DSLAM.

It's funny how the DSL providers will swear up and down for years that they can't get DSL or fiber into your area. Then one day, they hear that the city itself is planning to roll out city-wide broadband, so the DSL company is "suddenly" able to serve your area, and sues the city for unfair competition.

Because as soon as you get 18000 feet from the CO, you hit a loading coil, which kills DSL. Your mandate would require installing (and powering) DSLAMs essentially within 3 miles of any customer. Plus the back haul routers, power feeds, and other ancillary gear. Add it up, spread it around, you'd blow through the money spent on those WV routers pretty quick.

Which is not to say WV got the right routers for its needs. The 3945 ISR is an enterprise class machine, with capabilities to do things the WV libraries would never need. Cisco makes a number of SOHO routers that would have been perfect for what WV wanted for a lot less. (And in the article in Ars Technica, a Cisco sales rep essentially said so. That article said it was a reseller - Verizon Network Integration - who sold WV these routers, not Cisco themselves.)

Note that he didn't say he was 20,000 feet from a DSLAM, he is 20,000 feet from a CO. It's quite common for Telcos to run fiber to a neighborhood and then install a DSLAM. In my old neighborhood they ran fiber to the neighborhoods in the early 90s because they were running out of copper pairs. Back when modems and fax machines were booming. To upgrade to DSL they just upgraded the cabinets/huts with DSLAMs.

OTOH, just because you are close enough doesn't mean they can provide DSL. In the mid 90's I lived

I've seen this exact thing happen with just about every school that I had contact with while serving as Tech Director of a k-12 school district.I believe that people forget where the govt gets their money from and just think that it's free money.

It doesn't even have to make sense, believe me if schools could get hundreds of chauffeured limos for free to replace their buses, they would do it, because after all it's "free", and for some reason it doesn't matter how stupid of an idea it is.

In my case, I had to fight to get equipment that was reasonable for the job in place of millions of dollars worth of equipment that was overkill in the extreme for all the schools i was in charge of. We actually had meetings where we were yelling at each other because I thought putting a $60,000 switch into a school that only had 200 students was a waste of tax payers money. I was just told repeatedly that if we were going to get it for free, or almost free that we should take as much as we could get.

The government is the only place I can think of where the people who spend the money are the same people who can arbitarily decide how much to take from their customers (taxpayers) without any recourse.

Of course, in such a system, such abuses are going to transpire regularly.

A more interesting question I haven't seen asked: Is it possible DHS asked for more pricey equipment and that the schools complied because the higher-end units implement more of the latest monitoring and security support? CALEA and other such measures.

Some of the cheaper units may not allow DHS to tap or to disable systems as easily or quickly. Each newer generation seems to add more of this sort of capability to the switches.

(I can't speak authoritatively to broadband switches, but I can speak to cellphone networks and their policy enforcement and AAA services, where this sort of thing is definitely always getting more capable without much public fanfare).

State and federal spending rules are designed to be penny wise and pound foolish. They'll imprison a contractor who charges 5 hours of lunch breaks to a contract but won't even fire an employee who wastes several millions of dollars in a spending spree so ludicrous that no reasonable person would have charged forward on that. So the Verizon contractor who skips an hour a day but costs the tax payers a few thousand dollars at the most is more likely to get prosecuted than the high ranking government employee who just spent $25m when $2.5m (parts and labor) was likely the true ceiling for legitimate costs.

sounds an awful lot like the ATM tech who went to prison a yr or so ago for replacing real $ w/counterfeit while all the wall st executives who replaced real $ w/securities they knew couldn't possibly generate the required cash-flow over their life who've not only not been indicted but have gotten to keep all their comp for their "performance"

Well, there's a difference between incompetence and fraud. Billing extra hours to increase your own paycheck is a pretty clear case of fraud, but unless you can prove that somehow the person who ordered the routers got any personal kickback from it you can't really say the same about those. It is of course possible, but I've also seen cases where my biggest question is who thought it was a good idea to hire that person to sit on a budget. But everything can happen when the stupid hire the really stupid.

The most hilarious part is when Gianaro defended it in the name of " equal opportunity"' : "A student in a school of 200 students should have the same opportunity as a student in a school with 2,000 students."

WTF? Does he really thing the technology works like that...the bigger the router, the bigger the opportunity?

Clearly, you havent seen the packets that come out of these routers. Theyre so crisp, so clean, the ones so sharp, the zeros so full bodied and round....

I used to be a skeptic, but one day I broke down and decided to try the Cisco experience. I hooked my comcast modem's ethernet port up to a Cisco 3900, and that to a Catalyst 6513, and now I never have to experience low quality on youtube simply because someone only uploaded at 320p. Everything is so much cleaner, the sounds more audible, the content more enjoyable, even the slashdot comments have become wittier.

And that was the purpose of Porkulus. To piss away the better part of a Trillion dollars in the belief that just throwing such a huge sack of cash at the economy would somehow fix things. Of course it failed. But does anyone on the left admit that? Sure! Idiots like Paul Krugman insist that it failed because they didn't flush twice as much money down a rathole and that it isn't too late to flush some more.

Of course all too much of it would up taking backroads into the pockets of politically connected/favored people and organizations. And that was the actual goal.

This is not much compared with the Homeland Security debacle. Our little town was essentially forced to by trunking radios because that's what 'everybody else' is using. We have all of 6 VHF channels on the island. We don't need trunking. But now everybody carries these dipshit, overweight Motorola monsters that require a $20K (I kid you not) station to program them. Their only major advantage of the new ones is that they're so heavy they can be used for self defense.

We were forced to get a 'Police boat' to keep us safe from der Terrorists. Fine, we have a large harbor system and we're on an island. But the only ones you could get with DHS money (which had to be spent on the boat) were designed for nice urban harbors that didn't have things like sharp volcanic rocks everywhere and that weren't designed for heavy water.

So the boat sits in the stall with it's side bladder half inflated (another rock) while the old beat up aluminum skiff does the actual work.

No, Byrd was good at being on committees and refusing to sign off on anything that he could possibly get built in WV. There is a long list of government facilities that really have NO business being in WV, but they're here. My favorite example is the United States Coast Guard's Operations Systems Center. West Virginia, being land-locked and all, is an obvious choice for a base that supports a sea-faring service. This USCG station is directly adjacent to a massive IRS facility. In Fairmont, WV there is some NASA IV&V stuff as well as some NOAA facilities. Not to mention CJIS (the largest division of the FBI) in Clarksburg. Sugar Grove may be too old to be Byrd's doing, but the rest are relatively recent. I'm sure the list goes on; these are just the one's that I've personally dealt with.

nope, economic stimulus only works when money changes hand again and again so the economy grows. Routers sitting in a box don't do that. Routers installed do a little of that. Routers installed and gainfully used can do a big amount of that, but not by using a $22,000 router to do a $150 router's job.

The money isn't sitting in a box. I'm sure the money is changing hands and growing. It's just that it never gets back around to the taxpayers who still owe for it. Don't worry about a huge lot of money not being used --- there are a select few people making good use of it.

The money might be changing hands downstream but the gp's point is correct - it's only economically positive if both parties get something worthwhile in the transaction in question. An overpowered router plugged in and routing is still a beneficial transaction just inefficient. An overpowered router sitting in its box is just a waste.

#1 - Juniper - just as good IMHO far cheaper (better in some ways)#2 - Many router distributions [wikipedia.org] are just as good and FAR FAR cheaper. They could have bought an awesome overkill machine with a pile of multi-port NIC cards and still bought a lot of tech for the school with the money left over.

They didn't even get near the level of price-comparison, unfortunately.

Given both vendor's likely flexibility on pricing in the face of large orders(small margins on my goods are better than selling nothing because you bought the other guy's stuff, and the marginal cost of a fancy router is substantially smaller than its sticker price), as well as the portion of the bill that was absorbed by miscellaneous options and config and integration and whatnot(which, given that those were handled by another contr

While both of your points can be valid depending on the situation, I think it's stepping around the key point of the article. It doesn't really matter whether you choose a slightly less expensive Juniper system or if you home brew something, if at the end of the day you spec out a $15,000 server to host that router distribution, you're still paying *way* too much for routing services at a site that hosts less than 10 devices.

I've dealt with the exact same challenges that this Gianato says he was trying to avoid by simply buying the same model for everywhere. It's a ludicrous strategy, especially when choosing the 3945 as your standard. Using 1900 series Cisco gear would still be overkill for most of these sites, and would cost 10%.

Finally - it seems to me like the government is paying full list for their gear. Even small businesses get SOME discount from Cisco and their resellers, who the hell actually pays list? We're not even a big shop and our discount is at least 30-40% depending on what we're purchasing.

For an extreme example, see the train to the nowhere (desert) in California. That's right. It just stops rather than continuing on to Las Vegas.

And for the Most extreme example, see the ghost cities of China where the government is builiding cities to "stimulate" the economy and the cities are almot completely empty. http://www.youtube.com/watch?v=rPILhiTJv7E [youtube.com] Government stimulus == waste, not stimulus. The free market allocates money better (and when the money gets wasted, it's usually some rich fat cat who wastes the money, not the taxpayers).

From an economic perspective, money saved is money wasted -- if it's not changing hands, it's not helping the economy. So, which rich fat cat is helping our economy, the one you say is "wasting" money, or the one who is sitting on his nest egg, not letting it trickle down?

"And for the Most extreme example, see the ghost cities of China where the government is builiding cities to "stimulate" the economy and the cities are almot completely empty."
So basically what you're saying is that you shouldn't create demand by filling the supply first?... or is it just when governments do it?

A decade ago I worked a contract for a small school district in Texas, installing server. The servers were several years out of date - purchased with a federal grant for millions of dollars. They then say in a warehouse until the district got YET ANOTHER grant to install it. Maintenance? Not unless they get another grant because no one there had a clue.....

I find an extreme bias in Network Shops that have been indoctrinated with the CCXX mentality: If it doesn't come from Cisco, it's no good and most of the time they buy too much gear!

Cisco makes great stuff and they do have "small" gear too for this, looks like someone put in specs that were way overkill or that the competitive bidding process was not followed. That's common in government where you really don't have skilled people coming up with the technical specs, which in this case were probably done by somebody at VZ..

Well when a vendor says it's overkill and specific models and configurations are specified in a public project, then something is fishy.

When a government needs a truck they say "must be 4 wheel drive, able to carry 500 lbs and get at least 15mpg in the city." Not "I need a Chevy Silverado." If they have compatibility needs they can say "Must be able to integrate with a Verizon MPLS network" not "The router should be a Cisco 3945" That should and apparently did raise flags but nobody listened in the WV pr

Wow, a 3945 router to serve as a T1 endpoint? Whoever spec'd and approved that should be fired, no question! I just ordered a half dozen routers for just this purpose, a 1921 with T1 interface for under $1k each.

I have seen plenty of branch offices and banks with using Cisco 3800 series devices, the 3945 predecessors.

Whether or not the device was overpaid for is a different question - I wouldn't be surprised if they used some 8A competition limiting factors that jacked up prices, or, if it included the actual installation and smartnet maintenance costs.

It might be private IP (MPLS, Frame relay, etc). Lot's of municipalities build their own private WAN and then have a couple of Internet connections for access to the Internet. This allows you to build and deploy lots of services "On Net" securely. It's very common practice, same as mid-large enterprises.

People who manage school budgets are not unlike the people that manage home budgets: they don't get much credit for saving money, except for the credit they get is for how they spend the money that they have saved. There unfortunatly is a tendency to avoid splurge/waste all that money that was diligently saved. Example: look, I saved enough money to send us on a expensive vacation! Look what I bought with this stimulus money!

Also, schools (like many businesses), are prime targets for product and service slamming attack by unscrupulus vendors. Even in the best of times, purchasing groups for school districts and many businesses aren't really experts at what to buy, or even how to negotiate deals. They often aren't much better than the typical minimally-informed car buyer who goes into a car dealer and expects to buy a car and only does it once every 5-10 years. The car dealer gives them an over-inflated price, lets the purchaser negotiate it down so the potential purchaser can feel good, they buy the product and a few more marginally-valuable goodies that have super-high profit margins as add-ons at the last moment. If the purchaser doesn't play ball, they've wasted all the time and go to the next pre-qualified vendor that does the exact same thing to the purchaser, until eventually either the purchaser gets lucky and finds a honest vendor, or they just get tired and buy something that is sorta what they want/need.

Why does this happen more to businesses and schools than individuals? It probably doesn't, it just seem like that because of reporting. Joe-average (or Jane-average) consumer has this happen all the time to them (esp if they don't care too much about money, or maybe they didn't earn the money, but got it from their spouse), but you don't see it on the news. Many people buy stuff because it's "cool" or they got a free gift bag, money is often not a criteria. However many times, the motivation boils down to you can't show people the money you save/earn/found unless it makes a splash and if you feel the need to show the splash to show your worth (to your boss/spouse/friend), it's easy to fall into this trap and vendors know it and they have a product/price point for every amount of splash you want to make.

so your saying I can get small schools to buy huge enterprise class equipment, at retail prices, and not install them! Most publicly funded institutions get huge discounts on this stuff, so there is even more profit to be made.

That was such a rigged process we went through. We even had the governor sign our petition that was submitted to the fed (promising matching funds and loans) to extend broadband to TRULY rural and unserved (not underserved, UNSERVED) areas and lost out to the big boys who went and did stupid stuff like this.

Take a look at the bio [wv.gov] of the guy who is the proximate cause of this debacle. He's got quite a solid background in public safety, but in 2009 when the money bomb dropped he had no experience whatsoever in procuring and managing technology. So why didn't they hire somebody who knew what he was doing? Because they were required to spend the money right away. You can't hire somebody in government right away. It just doesn't happen. But you *can* hire a contractor or vendor.

I've seen this before. You give a local or state agency with little or no experience with technology a bundle of money to solve some pressing problem like bioterrorism, and you order them to spend it on technology *immediately* or lose it. They don't have time to figure out how to spend the money reasonably because they've got to get the purchase orders cut *right away*. You've basically handed them a golden hot potato.

If you remember the big debate over the fiscal stimulus, the people you'd have expected to vote against it were grumbling, but they voted for it, provided that the money was channeled into "shovel ready" projects. Think about the assumption behind that, which is that the anticipation of income in the near future has no stimulative effect on current hiring or private spending. I actually think that's backward. People are more likely to invest their own money if their is money coming down the pike; if it has to be spent right now they aren't going to hire or invest, they're just going to pass it on.

At the time I thought the "shovel ready" emphasis was a recipe for fraud and abuse, because I'd seen the golden hot potato effect at work in the post 9/11 rush to spend money on homeland security. I saw agencies that were competent at their job and well-intentioned, but chronically underfunded suddenly find themselves with a big pot of money to spend on things they had no experience with. Now how do you think *that* was likely to go? Under the circumstances the only way to get rid of the golden hot potato was to hand it to a contractor who had the experience and administrative capability to absorb a lot of federal money quickly. It's a specialized skill; not every vendor has the accounting infrastructure to suck up hundreds of thousands or millions of federal dollars overnight with all the bogus "controls" attached to it.

I'm convinced the golden hot potato effect is no accident. Somebody always makes a ridiculous profit off these things. The ultimate cause of this problem isn't the guy who's handed the hot potato. It's politicians doing their cronies a favor buy turning a federal grant into something that can't possibly be spent wisely.

The routers alone cost the state $7,800 each, but "add-ons" -- additional equipment that came with the devices -- boosted the price tag by $14,800.

"It's like buying a car," Gianato said. "You get a lot of options with the car."

An online Cisco retailer was selling new 3945 series routers for $5,800 last week. The routers have a list price of $13,000 each.

Cisco was the lower of two bidders for the $24 million router sale. Hebron, Ky.-based Pomeroy bid $24.8 million for the 1,064 Cisco routers.

State officials requested that the devices include a "T1 interface card" that would allow schools, libraries and other sites to use the high-capacity routers with their existing copper-wire T1 broadband connections -- while waiting to hook up to fiber optic cable.

All certainly true. However, small libraries don't need all of that functionality. They could probably get by with a WRT56. It's only a cost savings if you need the functions in the first place.

And I don't even think that the argument that all of the routers should be the same makes any sense. When you have libraries ranging from one room to a five story building, there isn't going to be a one size fits all.

State officials requested that the devices include a "T1 interface card" that would allow schools, libraries and other sites to use the high-capacity routers with their existing copper-wire T1 broadband connections -- while waiting to hook up to fiber optic cable.

The adapter cards added $1.08 million to the purchase price.

Instead of, say, keeping the old routers, and buying a Cisco 1800 for less than $1000? There HAS to be a illegal commission somewhere in there... $22 million stupid?

He sounds like he just doesn't understand how this works, He seems to think a $22,000 router would somehow faster or better than a $500 router even if only 4 people are connected to for basic web browsing.
FTA: Gianato said putting the same size router in every school was about "equal opportunity." "We wanted to make sure a student in McDowell County had the same opportunities as a student in Kanawha County or anywhere else," he said. "A student in a school of 200 students should have the same opportunity as a student in a school with 2,000 students."

PoE is (in my experience) basically only used for door access control card readers and biometrics, and IP cameras, and copper Ethernet extenders, and 802.11 wireless access points, and pro audio gear.

Meanwhile, the only VoIP phones I've ever installed needed a local 5V wall-wart and did not support PoE (would've been nice).

Your mileage plainly differs.

That said: It's easy to segregate PoE devices, because there typically are only so many of them, and it's also ridiculously easy to add PoE into any existing infrastructure -- especially if the only thing using it is a bunch of bandwidth-efficient telephones. Cross-connect cables and human time are cheaper, all day long, than superfluously providing PoE to every port.

The point is they're getting far more than they need. Imagine we're talking about a library with, say, 15 public terminals and a few office computers. If each computer is just browsing low-bandwidth sites like a webmail service then a really cheap ($50) router could just about handle it. Step up to everybody watching YouTube vids and you might need to spend a few hundred dollars to keep things reliable. The systems they got were suitable for a university or large office building, what they needed was "a

Yes Cisco is the best for routing equipment, there is no denying that. My big problem is which Cisco equipment they chose to buy. 3945s are edge routers and have the price tag to match.
They could have easily gone with 1921 or even an ASA 5505 for 10% of the cost and still had the reliability provided by Cisco gear.

For $22,000 they could have bought 44,000 WRT54gs with DD-WRT on them, flashed them all with the same firmware/config, and if anything went wrong just threw the malfunctioning one away and popped in one of the 39,999 spares.

Theres a point at which "reliable" is no longer enough to justify the pricetag, especially when dealing with a 4-user scenario. And its not like there arent oodles of Cisco products for way less money that can handle T1 and come with the "legendary" cisco name, for instance a 1800 router, or if youre feeling particularly spend happy a 2900.

cisco will sell you as expensive router as you're willing to pay for. there's no upper limit.

of course those routers do actually nothing for spreading broadband coverage.. washington should slap them for misusing the funds.

"Looking at how technology evolves, we wanted something that was scalable, expandable and viable, five to 10 years out. We wanted to make sure every place had the same opportunity across the state." - fucking dimwits. 22k in it equipment budget spread over 10 years would have done wonders to some libraries and schools. buying 22k routers without immediate use for them -if one actually had looked at how technology evolves- is stupid.

Washington should keep its mouth shut, since this is an in-state affair.

People seem to sadly forget the 10th amendment and that there are matters that the fed has no say in.

Not if you are using federal funding. The hand that feeds you always has a say on how to use the money (or not giving you any). I'm sure there is clause in accepting the funding to not abuse or mis-appropriate it.

(even though I know you're not serious, I'll still answer the question)1. capacity - route more packets. switch more MAC addresses. hold larger routing tables. gigabit and 10-gigabit2. reliability - your airport extreme is not even six-9's, nor can it support any sort of hot failure.3. expandability - add different short distance and long distance fibre interfaces though GBIC. (10GBASE-SR/LR/LX4/ER/etc)4. administration - support routing and configuration protocols. support self-healing networks.

The only key piece of information you need to know is that they are Cisco routers; that explains everything.

Not really - Cisco may be expensive, but it doesn't need to be *that* expensive. If they really do want to get 5 - 10 years of lifetime out of them, going Cisco is not the worst choice.

You may say "Bah, just buy a Netgear and replace it when it fails", but schools and libraries often don't have the staff to do that, in 2 years their current IT Admin is not going to have any idea how the router was set up by their previous admin.

Wow! Thats an enormous waste of money!They make $22,000 routers? What could they possibly do that like an Airport Extreme can't? heh.

Screw that. I just told my boss about this story and he imploded. We've been deploying MikroTiks to manyremote sites and they never bat an eye (although we sometimes prefer to run OpenWRT on them instead ofdealing with RouterOS). At around $80 a piece, they could have saved nearly $24 million dollars. Addsome additional cost for something cheap that interfaces with their T-1's. Ah, they should have got all thefiber ready and then bought something much cheaper that would interface with it. $24 million

What we're forgetting is that this was a grant from the federal govt. They had to spend however much they got, or they coudln't get another grant again.

I used to live on Air Force Base, and they would repave the roads every year to make sure every dime was spent. Then they took out all the roads, and tore down the houses to build a park. I'm sure they eventually tore out the park, and rebuilt the houses, and continue to repave the roads every year.