Palo Alto Weekly

News - June 3, 2011

Palo Alto's fiber dreams dealt a blow

Consultants agree a fiber system for all residents isn't feasible

by Gennady Sheyner

Palo Alto's decades-long dream of expanding its fiber ring to bring high-speed Internet to every home in the city should be deferred, if not abandoned altogether, because of high costs, questionable demand and fierce competition from existing telecommunications giants, two consulting firms have concurred in separate reports.

The reports from Columbia Telecommunications Corporation (CTC) and Tellus Venture Associates were commissioned by the city as part of its effort to create a new business plan for its 41-mile "dark fiber" ring — a network that the city built in the late 1990s that currently serves 68 customers. While both consultants recommend extending the fiber ring to new areas and building new infrastructure to support it, each concludes that a citywide fiber system — known as "fiber to the home" or "fiber to the premise" — would not make business sense in the current economic climate.

"There is no compelling case for providing fiber service directly to residents at this time," consultant Stephen Blum of Tellus concluded in his report. "Palo Alto is served by large incumbent retail video and broadband service providers that enjoy decisive competitive advantages resulting from economies of scale.

"Comparable municipal ventures have failed. It is unlikely that a City-run or supported residential fiber service would be able to achieve enough market penetration or subscriber revenue to support itself in the near term."

The findings, which the city's Utilities Advisory Commission reviewed Wednesday night, are the latest blow to the city's long-held ambitions to spread fiber-based Internet access to its famously tech-savvy masses. The city's earlier attempt to partner with an Internet consortium on a fiber-to-the-premise initiative fizzled in 2009, when the consortium's financing collapsed. Palo Alto also joined more than a thousand other cities in applying for Google's ambitious Fiber to the Community project, which aims to hook up an entire city to ultra high-speed Internet. Kansas City ultimately won the Google prize.

The new studies are sure to disappoint proponents of a citywide fiber system. But the reports provide an array of recommendations to the city and its Utilities Department for improving the city's small but lucrative fiber service, which is projected to generate an estimated $3.3 million in the current fiscal year.

The CTC report, which evaluates ways to expand the existing network, recommends a two-phased approach to widening the ring. The first phase would entail building new "access points" at nine existing electrical substations to entice private companies to work with the city on dark-fiber initiatives and to support various other wireless services. This could entail building new cellular towers, which CTC recommends making at least 75 feet tall, at the substations. These facilities would be leased to a variety of telecom companies and would help the city meet the spiking demand for wireless coverage.

This proposal will almost certainly prove a tough sell in Palo Alto, where two recent cell-tower proposals attracted intense opposition from residents at the proposed sites. In one case, AT&T was forced to pull its application for a 50-foot tower at St. Albert the Great Church after a group of residents in the Crescent Park neighborhood pressured the church to step away from its partnership with the telecom giant.

The CTC report acknowledges that its proposed initiative "will not be welcomed by all" but argues that this approach is "both a responsible form of stewardship of (City of Palo Alto Utilities) facilities and communications assets and a reasonable way to address a highly charged urban problem."

"We are aware that an initiative of this type is not without political challenges," the CTC report states. "This is especially true with regard to the aesthetics of cell towers.

"Everyone wants to have reliable citywide cell coverage for voice and high-speed data connectivity, but hardly anyone wants a cell tower located next door, or in fact even within the neighborhood.

"By developing a proactive cell tower placement program within the confines of existing electric substations, (City of Palo Alto Utilities) would in effect be blending the common aspects of facilities everyone needs and leveraging the common characteristics of both media."

The second phase in CTC's proposal involves building 88 "access nodes" throughout the city. Each of these nodes would be able to provide fiber access to about 250 homes and businesses. This "fiber to the neighborhood" initiative would cost about $5 million to build and, if all goes well, entice a private operator to build the "last mile" of the network to each home. The entire fiber project would cost between $40 million and $60 million, depending on the type of system deployed.

The high cost of building a citywide fiber system has deterred the City Council in the past from taking on the project without partners from the private sector. According to the Tellus analysis of market conditions, such a system would not be a financially feasible project for the city to take on.

In its report, the firm lists several cities, including Alameda and Provo, Utah, where citywide Internet initiatives had failed. Both Tellus and CTC also emphasized the dominant role of Comcast and AT&T in Palo Alto's broadband market — a tough obstacle for the city's fledgling operation to overcome.

The Tellus report urges the city to instead focus on its core customers — high-tech firms and telecom companies. Tellus evaluated various parts of the city where the existing fiber ring could be extended and singled out the area around East Meadow Circle (home to Space Systems-Loral and Dell Computers) as the "best immediate prospect" for such an extension. Other potentially lucrative areas for expansion are areas along El Camino Real and Sand Hill Road.

Blum wrote in his report for Tellus that the broadband business model is changing rapidly and that a fiber-to-the-home (FTTH) system could "eventually" become economically viable in markets such as Palo Alto.

"For the present though, the broadband sector's turmoil and uncertainty make FTTH system investments less attractive," Tellus concluded. "The current state of the broadband market does not support a business case for a third, overbuild residential broadband system in Palo Alto."

The city's utilities officials are expected to use the two new studies to put together a business plan for the fiber service by this fall. Jim Fleming, the city's project manager for the fiber utility, wrote in his report that staff will further analyze the reports' recommendations and reach out to customers in areas the consultants had identified as "underserved." Staff also plans to "evaluate the feasibility of constructing cellular towers at some or all of the electric substations," Fleming wrote.

Posted by I Want Internet, a resident of Palo Verde
on May 31, 2011 at 6:22 pm

Comcast and AT&T may be incumbents but their offerings are absolutely horrible. Internet is unreliable in Comcast's case, and slow in AT&T's case. Both companies have horrendous customer service. The best internet service I've ever had is with Speakeasy, using AT&T's twisted pair lines, but the speeds are abysmal (because of AT&T's lines).

I for one am more than happy to pay $100/month for fast, reliable, non-shared Internet access of 20-30 Mbps. Just give me a simple pipe with fast and reliable bits. That's all I ask. Nobody offers that now.

Posted by Greg, a resident of another community
on May 31, 2011 at 7:17 pm

Why do you need fast internet or wireless coverage? Just go to one of your dozen libraries!!!

Posted by More!!, a resident of Community Center
on May 31, 2011 at 7:34 pm

Hi Greg,
I can go to the library and use the internet. But what about carrying my big screen TV to the library.
If you are stuck in time and not progressive your definition of Internet is nothing but browsing.

Posted by Chris, a resident of Community Center
on May 31, 2011 at 7:54 pm

I signed up for Comcast business class service for my "home office" about 6 months ago (about 13Mbps for $75/month), and have not had a single complaint since. Works just great, and good customer service. I'm guessing that their business class organization is completely separate from their consumer operation.

Unrelated: I live 2 blocks from St. Albert the Great, and am in a complete cell phone dead zone (at least for T-Mobile). I was looking forward to that tower, and am sad the project got killed. Hopefully they manage to get a tower in at the local substation, as that is nearby too.

Posted by I Want Internet, a resident of Palo Verde
on May 31, 2011 at 8:49 pm

@Chris, I have business class Comcast also. If you have any applications that need consistent always-on connectivity, it still doesn't work. Try running a ping test in the background 24/7 and you will find huge variations in ping times and complete dropouts (lost packets) in short spurts all day long. Even browsing varies hugely in speed. I have made probably about 30 calls to Comcast Business Class Internet tech support, the local techs know me by name (and have given me their emails and cell numbers), and they know when I call about a problem it's usually because something is actually wrong with their network, because I reliably pinpoint it.

Cable is fundamentally a poor medium for reliable service because it's shared; if a neighbor unplugs a cable in their house and doesn't terminate the dangling cable, your line gets noise on it. Eventually Comcast needs to send a tech out to climb the pole and figure out which house is causing the noise, then try to convince the residents to let them in to fix the source.

Posted by pat, a resident of Midtown
on May 31, 2011 at 8:53 pm

What a surprise. This city has been having this "dream" since at least 2004.

Problem is that Palo Alto has too many dreams and not enough business plans.

Posted by JA3+, a resident of Crescent Park
on May 31, 2011 at 9:08 pm

"Comparable municipal ventures have failed."

Ample reason for the City of Palo Alto to drop further pursuit of this idea.

Although not listed in the original article, this is likely a link to the CTC and TVA reports mentioned in the story: Web Link

Overall, I'm disappointed with the quality of the work in both reports. Although CTC claims expertise and "over a decade of experience working with local governments nationwide on cellular towers issues", they appear to have little understanding of cellular communications infrastructure. One of CTC key recommendations is that the City build 100 foot monopole towers on the site of 9 electrical substations and lease space to cell phone providers. Unfortunately, none of these sites provides an adequate "fall zone" for such a structure. While there is no doubt structural some engineer who might sign off on a "self collapsing" design, placing such a tower next to a critical electrical distribution point in a major earthquake zone is, to put it kindly, insane. Generally, fall zones are 110% of the tower height with the requirement that the tower fall completely within the area or property lines of the site. This is why cell companies love placing towers in churches - the fall zone problem is mitigated by the supporting church structure.

CTC claims that each 100 foot tower could be built for approximately $75,000. Although the costs vary widely, it's helpful to know that the 140' ECOMM monopole next to the Los Altos police station cost roughly $350,000 Note, a fraction of this cost was used for radio equipment and the tower cost doesn't vary linearly with height. But, CTC's estimate is irresponsibly optimistic.

Both reports make much of the "4.9 GHz Public Mobile Safety Band" and opportunity to use the monopole site at electrical substations to build out a public safety network using this band. This must be old material borrowed and recycled into this report from previous work. All interest in public safety mobile broadband is focused entirely on the new 700 MHz spectrum which became available with the switch to digital TV. Public safety organizations are now lobbying Congress for the adjacent "D Block" to enlarge their spectrum footprint in this band. The consensus is that the 700MHz spectrum will be used for "public safety only" LTE network. In fact, the SF Bay Area is currently the only place in the country with an LTE testbed using this spectrum. There is little interest in 4.9 GHz anywhere in the country, now or in the future.

Posted by Nayeli, a resident of Midtown
on Jun 1, 2011 at 9:01 am

When we moved here a couple of years ago, I was shocked when AT&T could only offer the lowest tier of DSL at our home.

We had ordered the highest level, but it never worked. So, we called the AT&T techs.

After a week of speaking with local and phone AT&T techs, we were finally told that the lines in Palo Alto are just "old." They were only able to support the lowest tier of bandwidth (768 Kbps). A year later, we were told that they had finally improved to where we could purchase the next tier up (1.5 Mbps)

I thought that it was ironic that we could get the highest level of DSL from the rural home that my husband and I rented in Texas during grad school (located several miles outside of town), but we could only access the lowest tiers of service right off of Alma St. in Palo Alto -- the heart of the Silicon Valley.

We considered moving over to Comcast. However, we have Comcast Digital Cable and we see periodic hesitations and pixelations that my husband believes are caused by bad bandwidth.

We were hoping that Palo Alto would be a leader in faster internet. After all, we are the home of HP, Facebook, Stanford and other major tech giants.

Posted by pat, a resident of Midtown
on Jun 1, 2011 at 10:13 am

Thanks, radioguy, for your critique of the consultant's report.

One of the many problems with Palo Alto hiring consultants (for seemingly every project) is that no one on staff has the expertise to judge the consultants. This became obvious to me years ago when the city re-did its website for $250K. The result was a disaster and it was clear from employees' defense of the site that they didn't know diddly about websites. It took years  with the help of a citizens' committee  to make some basic fixes.

JA3+ notes that the fact that "Comparable municipal ventures have failed" is ample reason for the city to drop the idea.

These failures were known and pointed out to the city council many years ago, but folks at City Hall continue to believe that Palo Alto is "special" and has nothing to learn from other lesser communities. It's part of the Kool-Aid culture.

Posted by Darwin, a resident of another community
on Jun 1, 2011 at 10:24 am

I've been incredibly happy with Comcast, both with their product and with their customer service. The only gripe I have is that it's expensive.

Posted by I want internet also!, a resident of Downtown North
on Jun 1, 2011 at 10:47 am

I completely agree how shocking it is to be in the "heart of Silicon Valley" and have such poor bandwidth. AT&T along with Comcast have horrific customer service. Let's face it if we behaved the way they do, we'd likely be fired from our jobs. I've had both Comcast & at&t(DSL) and finally have settled on AT&T because 1)I don't have to keep calling them every 3-12 months to get a new "deal" to bring the price down and 2) for the price I'm settling for the the intermittent service. Maybe the other cities have failed in this venture but one reason is because the telecommunication giants already have spent their (your) money on infrastructure decades ago and the city has to spend the money NOW. not very fair is it? So we settle for suboptimal service. I mean really, I'm paying AT&T 30 bucks a month to run a couple of electrons down a wire? come on!

Posted by Julius, a resident of Monroe Park
on Jun 1, 2011 at 12:13 pm

I think we need a "public option" for Internet connection. Comcast and AT&T have a fudiciary responsibility to their shareholders to maximize duopolistic pricing up to the threshold of antitrust enforcement.

Internet access should be like water access. Why not fund it out of City revenues and make it free at the basic level? Higher-bandwidth connections could still generate revenues.

Posted by pat, a resident of Midtown
on Jun 1, 2011 at 1:38 pm

Julius: Why not have the city provide "free" water, gas and electricity too?

City services are not "free." We're all paying for them with our tax dollars. And in case you hadn't noticed, the city doesn't know how to run a business or make money.

Posted by the_punnisher, a resident of Mountain View
on Jun 1, 2011 at 1:58 pm

Palo Alto ( and most of the South Bay ) has some of the oldest operating switchgear that was originally put in by the original Western Electric Bell System.

The dirty secret is that your " Baby Bell " has no intention of upgrading the infrastructure; they keep the existing setup operational by swapping ( semi ) operational line amps at the CO. I had a long talk with the VP of Operations ( a hardware techie ) of one of the " independent " providers. This talk was the result of getting the same lousy DSL service ( yes, the problem was on the pole and the techs were the same lousy ones I had to deal with earlier ).

They have to rely on the same equipment and service and are just as frustrated at the aging equipment problem.

You have a perfect trunk setup for F/O, the CALTRAIN right of way, but your service provider will not spend the $$$ to use it.

The PUC SHOULD get involved, but too many are in the pockets of the people they are supposed to regulate.

Qworst finally has a bunch of dark fiber; they will hook it up when they feel the rates they are allowed makes it profitable.

When the $$$ are enough, your Baby Bell MIGHT make the investment. Until then, think of Ernestine's typical response: " We don't have to, we are the Phone Company "...

Posted by jardins, a resident of Midtown
on Jun 1, 2011 at 3:31 pm

If the city does bring in fiber to more residences later on, I for one want the right to choose whether or not to be connected. I don't trust the city not to hike the rates for fiber after a while--just the way we're now being punished for minimizing our garbage output.

Posted by Fiber is for Digestion in Palo Alto, a resident of Ventura
on Jun 2, 2011 at 1:50 am

Ironically, Palo Alto's municipal administration is not entrepreneurial. There has *never* been support for fiber from City Hall - never. Sure, they hire consultants, but the consultants either come up with lame business plans or suggest that there is too much "risk".

There was a time when Palo Alto had a chance to deploy fiber for universal Palo Alto access; that time is past. The lack of ingenuity (of past City Administrations - including the current one - and City Councils)has taken its toll. We're now in a structural resetting mode because Silicon Valley is no longer the goose that lays all the golden eggs. Competitive advantage? Not so hot any more.

Sure, Palo Alto will remain prominent due to its proximity to Stanford, and the nostalgia attached to the "startup game" (Palo Alto is no longer "startup central").

That said, Comcast, ATT and Verizon are thieves; they pay off our government officials to guarantee their oligopoly pricing models. Look who just became the President (or CEO) of Comcast - a highly placed *sitting* FCC official. Does that surprise you?

City Hall? Forget City Hall taking on any risk at all. Remember, the City Manager governs at the pleasure of City Council. Do you really think that Palo Alto's Executive will risk a cushy $250K+ job and benefits on something that *might* go wrong? Seriously folks, what are you smoking? Same goes for School Superintendents.

Go out and look around - see what top Education and Municipal officials are up to. What do they invent? Invention isn't their job. Maintaining the status quo - along with their plush salaries - is their job. That goes for those who surround them in the higher up circle.

Past City Councils can also be put to task for this loss; they were slow to act when we had the impetus and cash to try something. They and the accompanying Administrations didn't want to rock the boat.

Can you get 805 of Palo Altans to vote for fiber - like they have done for school bonds and the library? Those are the kinds of numbers that City Administrations like, because it means that there is a clear mandate w/o going out on a limb - even a thick limb. I mean, what are you gonna do if you lose your $250+K job to a boondoggle that nobody asked for. Get real, folks!

Eventually, perhaps America will permit overseas bandwidth providers to tap into our markets. Don't hold your breath. Until then, America will continue to decline in broadband market penetration relative to most of the rest of the developed world; American politicians will keep taking money form the telcos to make that happen; and, American municipalities will hide under the sheets because they all know that they're screwed, long-term, re: the cushy ride that they've had for the last 30-40 years. It's the latter - assumed guaranteed - good times that lulled everyone into a sense of "this is going to last forever". Well, those chickens are now coming home to roost.

What does all this mean? It means that we're not as smart as we thought we were - and were way less adaptive than we thought we were. Eventually, as constraints increase and we continue into slow free fall, some areas of government and our citizenry will venture out to take real risks again, because there won't be as much mythic power and status to loose (and it is "mythic" because it's just not there any more - I mean who outside of Palo Alto really thinks that Palo Alto is the locus of power these days. Not anyone (of import) that I know - and I get around.

The only way you're going to see fiber in this town is to find someone with cash to put that cash down on the table, to start the whole thing up. Why not tap Mark Zuckerberg; maybe you can catch up with him on his way back from his latest pig or goat slaughter?

Posted by Fiber is for Digestion in Palo Alto, a resident of Ventura
on Jun 2, 2011 at 1:52 am

meant to say "80% of Palo Altans"

Posted by Free-The-Market, a resident of Another Palo Alto neighborhood
on Jun 2, 2011 at 8:22 am

> "There is no compelling case for providing fiber service directly to
> residents at this time," consultant Stephen Blum of Tellus concluded
> in his report. "Palo Alto is served by large incumbent retail video
> and broadband service providers that enjoy decisive competitive
> advantages resulting from economies of scale.

This has been true for a long time. Sad that no one on the City Council could see this by themselves. Kind of makes one wonder what good a City Council this ignorant is to the well being, and prosperity, of a town like this one?

> Comcast, ATT and Verizon are thieves; they pay off our
> government officials to guarantee their oligopoly pricing models.

Well .. maybe. But if the Telcos are "theives" for "paying off government officials" .. what does that make government officials who take their money? And wouldn't these be the same government officials that would be in charge of a government-provided telephone/cable system?

> Can you get 805 of Palo Altans to vote for fiber - like they
> have done for school bonds and the library?

Firstly, the "voters" passed those revenue bonds--not the residents. Generally, less than 25% of the residents voted for these expenditures. Business owners were given no votes. Upwards of 50% of the residents are renters, who pay no property taxes directly, and another 20% of the voters are "grandfathered-in" seniors, who pay less than $1,500 a year in property taxes--leaving the newer residents who pay $10,000-$30,000 a year to actually pay for these public projects.

Fiber-to-the-Home is nice, but very expensive. One can be sure that the people pushing it will benefit handsomely, while others will end up paying the bill.

> fiber in this town ..

Well, maybe .. but the reality is that Palo Alto has become a retirement community .. at least for the foreseeable future, and older folks just don't need a Internet service at 1-GaZizzabyte/sec. There are possibilities that might make this sort of thing desirable, but we are seeing strong competition for providing these services with smart phones. Fiber means "being home" to use it. Cell phones allow people to move about--which is a lot more desirable to being home to use your telephone and have Internet access.

The Telcos can provide VDSL to most of their customers--offering up to 100 mbps easily. But the market needs to be there .. and for most people--Video on Demand seems to be about the only use they can find for paying for that sort of bandwidth ..

FTTH may someday be available to new housing projects, and could be easily retrofitted into apartment buildings .. but there will need to be a huge shift in the way people think about how they spend their money in order to justify this shift. To be specific, America needs to want to buy on-line, rather than in local stores. This, of course, is disruptive. We've seen the decline of local book stores with the current level of service available to consumers. We will see a decline in the need for local libraries with the shift to digital distribution. With increased bandwidth, we could see very intelligent applications emerge, which might allow us to purchase clothes on-line, by using images of ourselves as models, and the rendering software showing us in the clothes are interested in. What will this do to local economies, and tax bases?

Palo Alto will survive without FTTH. Most of the businesses already have been offered City-run Fiber services, and about 99.9% have said: "No Thanks!".

Getting the City out of telecommunications oversight/regulation, and letting the private sector do its job will be the way for us to get higher bandwidth sooner.

Posted by Resident, a resident of Another Palo Alto neighborhood
on Jun 2, 2011 at 9:02 am

Internet service is really behind the times in Palo Alto.

We have extended family in other parts of the world who have far superior choices and various options for charges. We have an either/or with no real options for upgrades.

Fast internet is now a utility as important as water, power, sewage and trash removal, and less important than land lines or cable tv. Everywhere else seems to understand that except Silicon Valley.

Modern tvs, security systems, and even cars, are internet ready but not available to us due to the poor internet infrastructure.

I don't care about the excuses. We are behind 3rd world countries once again when it comes to 21st century technology and transport availability.

Posted by Free-The-Market, a resident of Another Palo Alto neighborhood
on Jun 2, 2011 at 9:43 am

Here's an example of the kinds of interference that the Telcos have in "big government" towns like Palo Alto--

Who cares where the PEG channels are located? Virtually no one watches them anyway.

---
Each Comcast customer pays $0.88 a month, generating about $327,000 for the Community Media Center each year, according to Cavallo. In 2006, the city received franchise fees of $581,000, a city report states.
---

Comcast customers have every right to be angry about having to pay "government" (and its "country cousin" the Media Center for services that are of value to very few people.

One wonders if the fact that AT&T is not considered to be a "Cable" company where its data/video services are concerned (ie--forced to pay the City to do business here), that maybe there would not be as much opposition to its U-Verse service (that comes with higher data rates), as there has been by the unaccountable City of Palo Alto Government.

Posted by Paul, a resident of Downtown North
on Jun 3, 2011 at 9:43 am

"letting the private sector do its job will be the way for us to get higher bandwidth sooner."

So what's stopping AT&T and Comcast from installing high bandwidth lines? Why, the Private Sector, of course. The investment would take a big hit on their current balance sheets, and Wall Street doesn't like big hits on balance sheets. It's much safer and short term profitable to stonewall the customers and collect monthly fees. After all, where else you gonna go? Who else has the infrastructure?

Anybody remember all those promises about the benefits of deregulation? Nice, weren't they?

Posted by anon, a resident of Downtown North
on Jun 4, 2012 at 5:57 pm

You all should just get Clear WiMax. We just got it to supplement inet access at a small business in downtown PA, and are getting consistent 9000 kbps download speed at $50/month. This even endures steady for large (1 GB) downloads, some providers will lower the speed as the download proceeds (they advertise a slower speed, but I think we must be near a tower.)

Also, previous experience at home with Comcast has been great. Upgrade your service to the "Blast" level, I get around 30 Mbps, sometimes 36. Isn't that enough? What do you guys want? They also offer higher levels for more $ of course. I do not notice outages or slowdowns at all, but perhaps my neighborhood doesn't use it as much - previously at a different address I did have cable-neighborhood related slowdowns.

Posted by Pat, a resident of another community
on Dec 30, 2012 at 10:39 am

I would suggest that everyone interested in the future of Palo Alto as a leader read Susan Crawfords new book Captive Audience. Here is a youtube of her describing the book.

Posted by Under the radar, a resident of Midtown
on Dec 31, 2012 at 11:41 pm

AT&T is now on the board of the local Chamber of Commerce. Hold on to
your wallets.

Shiyama Clunie, a new Director, is the Area Manager for AT&T's External Affairs department, where she is charged with maintaining AT&T's position as a good corporate citizen in the communities it serves. Shiyama is currently leading AT&T's effort to improve wireless service in Palo Alto, an important goal for Palo Alto businesses and residents alike.