QUICK LINKS

Auditor says $8M wasted on routers

CHARLESTON, Feb 11, 2013 (The Dominion Post - McClatchy-Tribune Information Services via COMTEX) --
The state wasted $8 million buying oversized routers for its broadband expansion program and used an illegal purchasing process to get them: That's what the legislative auditor told legislators Sunday afternoon.

The state may have wasted up to an additional $6 million buying blanket upgrades for all the routers, without examining whether every site needed the upgrades, Legislative Auditor Aaron Allred told members at a joint Government Operations and Government Organization meeting.

But the governor's office defended the purchases as a vision for the future, and said the purchases were legal.

In 2010, the state Office of Technology and Broadband Technology Opportunity Program (BTOP) Grant Implementation Team bought 1,164 Cisco model 3945 routers to deploy at anchor institutions -- schools, libraries, hospitals and public safety centers -- as part of its BTOP program.

There are five levels of routers, ranging from Level 1 for home use to level 5 that serve as the core of the Internet, Allred's audit report explains. The Cisco 3945s are level 3, suitable for large businesses and capable of handling anywhere up to 200 simultaneous external users and from 700 to 1,200 phones via Voice Over Internet Protocol (VoIP).

Cisco's own engineering standards show that the routers were too big for most of the anchor sites, the audit says.

With the upgrades, the routers cost $22,600 apiece.

The audit determined that only one of the state's 105 public libraries has 15 phone lines. A few have from three to eight; 84 have one or two lines. By purchasing appropriate routers, the state would have saved $2.8 million.

Routers were poorly deployed to state schools, Allred said: 36 of 57 schools with enrollment exceeding 750 -- which could have used the larger routers -- didn't get them. Those include Morgantown and University high schools in Monongalia County, along with Cheat Lake and Mountainview elementaries.

State Police has 67 detachments, with staff levels ranging from one to more than 20 -- but nowhere near the 200 a single 3945 router can handle. State Police received 77 routers, and to date only two have been hooked up because they aren't compatible with the State Police telephone system and require further upgrades that will costs another $84,768.

In many cases, Allred said, routers were poorly deployed. The city of Clay in Clay County, for example, received seven routers -- all concentrated within a half mile of each other -- for its population of 491 people.

In another case, the Marmet Public Library, housed in a tiny modular building with two doors, got a router while nearby Riverside High, with 1,244 students, didn't.

Funds spent on the unnecessary hardware, Allred said, could have been used to string more fiber optic network across the state: 104 miles for each $5 wasted. Also, maintenance costs may exceed the cost of buying replacement routers. While 75 State Police routers are still sitting in boxes, Cisco has already announced the end of life for the router fans.

Allred cited two possible causes for the misspent funds. One, the state relied on the goodwill of the Cisco sales team pitching the routers. "Cisco sales representatives and engineers had a moral responsibility to propose a plan which reasonably complied with Cisco's own engineering standards. ... Cisco's representatives showed a wanton indifference to the interests of the public."
Two, the state never did a needs assessment to see what routers would be right for each anchor site.

The audit makes three recommendations:
Determine whether Cisco should be disallowed from future contract opportunities with the state;
The Office of Technology should conduct a needs study and see if any of the routers can be redeployed.

The Office of Technology should determine if the useless upgrades on the State Police routers can be exchanged for upgrades to make them compatible with its current VoIP system.

Illegal purchase
Allred said the state used a complicated and questionable "secondary bid process" to buy the routers. A secondary bid process -- which isn't defined in state code or Purchasing Division rules -- allows the state to use an existing contract to purchase subsequent goods and services of the same type from pre-approved vendors.

Because Cisco and three other vendors selling Cisco equipment were all preapproved, the BTOP team reasoned, and Cisco won the BTOP contract over one other bidder, the BTOP team didn't need approval from the Purchasing Division for the $24 million contract -- even though the law requires all purchases exceeding $25,000 to go through Purchasing.

Allred cited several other rules that appear to preclude the use of the secondary bid process for the BTOP program. "The process used to purchase the Cisco routers was not equal, fair or consistent with the intent of the purchasing statute."
The audit recommends that the state stop using the secondary bid process unless and until the Purchasing Division decides to seek legislative approval for it.

The governor's response
Rob Alsop is the governor's chief of staff and the governor's new designee to the Broadband Deployment Council -- which does not oversee the BTOP program.

He admitted the routers are too big for most of their sites. "It was for expansion and a dream of the future," he said. The Grant Implementation Team was looking five and 10 years down the road. That's why they didn't bother with a current needs assessment.

While it may have seemed counter intuitive to send the routers to tiny schools and libraries while bypassing many big schools he said, those schools already have good internet capacity. Again, they were looking to the future.

The secondary bid process was legitimate in this case, he said, because Cisco was one of four approved vendors selling Cisco equipment, and won a bid over one other competitor.

And while it appears the state way overspent, Cisco actually offered a 51 percent discount off its usual price for the 3945s. If the state had mixed and matched routers, the discount would have been about 25 percent less -- offsetting any potential savings.

Sen. Evan Jenkins, DCabell, asked Allred about Alsop's comments on the future vision. Allred answered, "I think they far exceed the future needs." Technological advancement and hardware shelf life give a router a lifespan of about seven years. The money could have been spent on getting out more fiber network.