Public safety radio failed most recently following president's visit last month.

On Tuesday, the San Francisco Chronicle reported that local officials, in collaboration with investigators from the Federal Communications Commission, have now found the culprit in the Oakland Police Department’s ongoing public safety radio woes: AT&T.

Last week, FCC and the City of Oakland notified the mobile network that its towers were interfering with the radios, but the problem got significantly worse when a police car found itself within a quarter-mile of a tower. (That is, according to David Cruise, the city’s newly hired public safety systems adviser, as reported by the Chronicle).

"If the officer is in an area close to one of their cell sites, essentially the cell site overpowers their radios," he told the paper.

A failing $18 million investment

Since July 2011, the Oakland Police Department has been using a new $18 million police radio system that has suffered numerous failures—one major occasion was as recently as last month, just after President Barack Obama departed town for a fundraiser.

Police and local officials have been scratching their heads trying to figure out what’s gone wrong. The Bay Area city, home to 400,000 residents, has had problems with its understaffed police force over the last year (remember Occupy Oakland?). A basic issue like radio communications certainly doesn’t help things.

In late July, one local columnist thundered: "It's time for city officials to bite the bullet, eat the cash loss, and invest in a reliable, inter-modal radio system before someone gets hurt—or killed."

Worse still, Oakland boasts the largest population in the East Bay region and has declined to join a network of 40 police and fire departments that are working on building a unified public safety radio network. So for now, the radios also don’t work in hundreds of buildings around the city, including the downtown basement of the Oakland Police headquarters.

AT&T shuts down 2G frequencies

As a result, AT&T shut down its 850MHz (2G) frequency on 16 cell sites around the city, which it has used for nearly two decades in the city.

"AT&T is working closely with the City of Oakland to understand if the issues they’ve raised are connected in any way to AT&T’s network," wrote Seth Bloom, a company spokesperson, in an e-mail sent to Ars.

"In the interest of public safety and as a cautionary measure while we’re looking into the matter, we have temporarily taken some 2G frequencies out of service at some cell sites in Oakland. We continue to operate on other 2G frequencies and our 3G and 4G service throughout the area is unaffected."

Many mobile phone networks can cause interference with existing public safety networks, which take priority. That’s why some networks pay to "re-band" their networks to frequencies farther away from public safety ones, which is what Sprint Nextel did in 2011—paying $10.5 million for the privilege.

Cruise also told the Chronicle that while the two companies involved in installing Oakland’s new radio system, Daily Wells Communications Inc. and Harris Corporation, checked for interference, they may not have done a thorough job. Those two vendors also did not respond to requests for comment.

"Unless they drove close to a cell site, they wouldn't have seen this," Cruise said.

Promoted Comments

Doing any meaningful interference testing with mobile systems (what would be equate to site surveys for fixed installations) means, well, going mobile! Either that didn't happen or it was low-balled, and maybe even pencil-whipped. If they'd found interference that was beyond their control, it should have at least been reported, and could possibly have been dealt with in a more timely and less risky fashion. Or not. Either way, the providers should be held accountable for whatever role they had in missing such a major issue, in an area that is such a major system design consideration. Or, we could smile and blame AT&T, which I admit is very, very tempting.

This article really needs a lot more explanation, so I'll do what I can.

Whatever interference requirements and the terms of its study are subject to the original Request For Proposal (RFP, assuming it did go out to bid), and something the City and whomever they used as a consultant would have specified. If there isn't an interference analysis done (the frequencies they use are probably existing prior to the new system), there's usually some clause that says the vendor will look into and examine possible means of mitigation. The actual frequencies they're using are here (but I haven't verified with FCC licensing): http://www.radioreference.com/apps/db/?sid=6077

Usually, the concern for interference is other sites that utilize the same frequencies within a particular radius and not cell sites in the same frequency range. There is a dedicated public safety potentially (851-866), but I'm not certain I buy the interference argument. According to Wikipedia, what comes closest to it with 824–849 MHz is used for uplink (which means cellular device to base station). The other band they use is 869–894 MHz, and anything bleeding over that badly from that far is malfunctioning. There's something that doesn't seem right to me, but the blurb that Oakland put out is too brief and doesn't have any actual information. There's nothing in the GSM frequency information that points it to a reliable source of interference, though FCC and AT&T are saying they were. In regards to rejecting possible interference, there are abilities for the system to overcome interference, both in terms of digital modulation and receiver sensitivity. The problem occurs when the noise floor is too close to the actual signal, and there are so many variables that it's hard to even identify if the problem is actually interference or user error.

HOWEVER, coverage testing should have noticed that something was odd, depending on if inbound and outbound testing was performed and how many channels are affected (and which side was seeing the interference, most likely outbound). This is also something that the RFP specifies. I really can't speak more without knowing about the actual RFP.

Doing any meaningful interference testing with mobile systems (what would be equate to site surveys for fixed installations) means, well, going mobile! Either that didn't happen or it was low-balled, and maybe even pencil-whipped. If they'd found interference that was beyond their control, it should have at least been reported, and could possibly have been dealt with in a more timely and less risky fashion. Or not. Either way, the providers should be held accountable for whatever role they had in missing such a major issue, in an area that is such a major system design consideration. Or, we could smile and blame AT&T, which I admit is very, very tempting.

Gee, according to FCC regulations, the radio equipment that the police bought is supposed to reject unwanted interference. Clearly the AT&T towers have precedence since they were there before the new radio system. I'd say the police need to make the radio manufacturer fix their hardware.

Gee, according to FCC regulations, the radio equipment that the police bought is supposed to reject unwanted interference. Clearly the AT&T towers have precedence since they were there before the new radio system. I'd say the police need to make the radio manufacturer fix their hardware.

I am inclined to agree with this point; I wonder if the vendors doing the installation told the city about the problems--my guess, they probably did.

As easy and as much fun as it is to bash AT&T, this obviously is probably with the vendor supplied equipment. Even if AT&T were operating on frequencies they were not licensed for, they only notice it after deployment?

As a result, AT&T shut down its 850 MHz (2G) frequency on 16 cell sites around the city, which it has used for nearly two decades in the city.

This of course suggests that Oakland is not the only city in the country where the specific police radio system could have come into conflict with other service.

Home radio devices, such as an FM radio, wifi router or smartphone , are unlicensed and the user must shut them off if they interfere with others' radio services (such as the police radios). Licensed radio devices, such as AT&T's, however, have specific limits on the amount of energy they can radiate outside their licensed frequencies. Either AT&T's signals exceed those limits, or the police radios' receivers are inadequate for their intended use.

Absent independent engineering review, I'd call it a coin flip. But as an Oakland resident, I'd want the police radio able to work around even the occasional misbehaving device; this is what “no excuses” is really about.

So AT&T willingly powering down its own 2g band doesn't count as lifting a finger?

Barring some weird malfunction with multiple towers, it sounds like someone involved in the process of buying this radio system bungled their job very badly: how on earth do you not notice you're overlapping with 2g when testing a radio in a city?

So AT&T willingly powering down its own 2g band doesn't count as lifting a finger?

Barring some weird malfunction with multiple towers, it sounds like someone involved in the process of buying this radio system bungled their job very badly: how on earth do you not notice you're overlapping with 2g when testing a radio in a city?

Acknowledged.

I still perceive this is very one-sided. The government schmoes will lose every time, even when there is a legitimate public interest. How do we balance?

This article really needs a lot more explanation, so I'll do what I can.

Whatever interference requirements and the terms of its study are subject to the original Request For Proposal (RFP, assuming it did go out to bid), and something the City and whomever they used as a consultant would have specified. If there isn't an interference analysis done (the frequencies they use are probably existing prior to the new system), there's usually some clause that says the vendor will look into and examine possible means of mitigation. The actual frequencies they're using are here (but I haven't verified with FCC licensing): http://www.radioreference.com/apps/db/?sid=6077

Usually, the concern for interference is other sites that utilize the same frequencies within a particular radius and not cell sites in the same frequency range. There is a dedicated public safety potentially (851-866), but I'm not certain I buy the interference argument. According to Wikipedia, what comes closest to it with 824–849 MHz is used for uplink (which means cellular device to base station). The other band they use is 869–894 MHz, and anything bleeding over that badly from that far is malfunctioning. There's something that doesn't seem right to me, but the blurb that Oakland put out is too brief and doesn't have any actual information. There's nothing in the GSM frequency information that points it to a reliable source of interference, though FCC and AT&T are saying they were. In regards to rejecting possible interference, there are abilities for the system to overcome interference, both in terms of digital modulation and receiver sensitivity. The problem occurs when the noise floor is too close to the actual signal, and there are so many variables that it's hard to even identify if the problem is actually interference or user error.

HOWEVER, coverage testing should have noticed that something was odd, depending on if inbound and outbound testing was performed and how many channels are affected (and which side was seeing the interference, most likely outbound). This is also something that the RFP specifies. I really can't speak more without knowing about the actual RFP.

I'm surprised nobody has commented yet, that this is quite possibly VERY OLD 2G cellular equipment - it could quite simply be that the antennas aren't working quite properly anymore, or that the radios themselves are out of whack.

I work with (low power) RF networks for a living, and I can tell you - *weird* things just sort of... happen. Especially when your antennas are hanging 180ft+ up in the air.

I'm surprised nobody has commented yet, that this is quite possibly VERY OLD 2G cellular equipment - it could quite simply be that the antennas aren't working quite properly anymore, or that the radios themselves are out of whack.

I work with (low power) RF networks for a living, and I can tell you - *weird* things just sort of... happen. Especially when your antennas are hanging 180ft+ up in the air.

If that happens, then they're not doing the maintenance that they should be doing, and they should also be getting reflected power from the station.

There is a dedicated public safety potentially (851-866), but I'm not certain I buy the interference argument. According to Wikipedia, what comes closest to it with 824–849 MHz is used for uplink (which means cellular device to base station). The other band they use is 869–894 MHz, and anything bleeding over that badly from that far is malfunctioning. There's something that doesn't seem right to me, but the blurb that Oakland put out is too brief and doesn't have any actual information. There's nothing in the GSM frequency information that points it to a reliable source of interference, though FCC and AT&T are saying they were. In regards to rejecting possible interference, there are abilities for the system to overcome interference, both in terms of digital modulation and receiver sensitivity. The problem occurs when the noise floor is too close to the actual signal, and there are so many variables that it's hard to even identify if the problem is actually interference or user error.

I don't think it necessarily implies a malfunction. Rather, it could be a large disparity in power levels between the interference source (base station) and the wanted transmitter (handheld radio), coupled with the receiver (another handheld radio) being in closer proximity to the interference source.

RF filters aren't magical devices that perfectly reject all the frequencies they're designed to filter out. Their interference rejection capabilities are dictated by RELATIVE change in frequency, i.e., a roll-off of X dB per octave. If your information is correct, the base station's allocation goes down to 869 MHz and the public safety band starts at 866 MHz - although 3MHz seems like a big gap, it's really only a 0.3% relative change in frequency.

Similarly, RF signal spectra aren't brickwalls. You can achieve a very sharp taper down to -30 or -40 dBc with proper pulse shaping, but even that may not be good enough when the interfering signal is much stronger than the wanted signal.

tl;dr - This might all be due to bad planning, entirely predictable with proper analysis.

Ars' and other articles seem to indicate (but are not entirely clear) that AT&T has temporarily shut down its 2G GSM 850 airlink in Oakland but has left up its 3G W-CDMA 850 airlink. If this is correct and it has halted the public safety interference issues, then AT&T may very well be at fault. Narrowband GSM channels at various frequencies within ATT's Cellular 850 MHz A block license could be creating intermodulation products that are falling within public safety's 800 MHz spectrum.

[transformers]I HAVE AN EMERGENCY PENTAGON CALL! .... NO I DON'T HAVE A CREDIT CARD!

Sir the attitude is not going to speed things up any at all. Now, I'm going to ask you to speak clearly into the mouth piece.

I'M IN THE MIDDLE OF A WAR!

Would you be interested in our premium package?

NO I'M NOT INTERESTED IN A PREMIUM PACKAGE[/transformers]

Pfft, quote a good movie: Operator, this is Major Mandrake at Burpelson Air Force Base. I would like to place an emergency call to Merkin Muffley at the Capitol. That's right, the President ...Try the War Room.

He smiles, hopefully, at COLONEL GUANO's sinister face.

MAJOR MANDRAKE That's right, operator, the President... Operator, how much will that be for the first three minutes?...Two dollars and twenty-five cents?

He quickly counts his change and sees it's not enough. He starts fumbling through his pockets. No more change.

MAJOR MANDRAKE Look, operator, can you make this a collect call? That's right, Major Mandrake from Burpelson...They aren't allowed to?...Tell them it's terrifically important...Just a second...

He opens the door and addresses COLONEL GUANO.

MAJOR MANDRAKE Colonel, they aren't allowed to accept any collect long-distance calls. The operator

Ars' and other articles seem to indicate (but are not entirely clear) that AT&T has temporarily shut down its 2G GSM 850 airlink in Oakland but has left up its 3G W-CDMA 850 airlink. If this is correct and it has halted the public safety interference issues, then AT&T may very well be at fault. Narrowband GSM channels at various frequencies within ATT's Cellular 850 MHz A block license could be creating intermodulation products that are falling within public safety's 800 MHz spectrum.

AJ

The thing is, inter mod is rarely regulated adequately and so at&t could be quite legal. Not that it should be, but the FCC has not been good at dealing with it. As others pointed out the rfp was either not followed or was poorly conceived and thus the testing failed. Its going to take FOI requests to know for sure.

This still reeks of incompetence or malfeasance on the equipment provider's end, before even putting up any equipment they needed to make sure there would be no interference, sure it's easy to blame AT&T but it should have been found a long time ago, not in the middle of a Presidential visit

The City of Oakland communications people should have done a site study as well. It sounds like though that the police radios Oakland are using are getting "de-sensed" on the mobile receiver side. Communication is getting blocked by the 2G cell tower transmitter every time it transmits. If AT&T shuts off their 2G system and the problem goes away for those police officers in close proximity to the AT&T cell site, then that's the issue. Selectivity on the portable and hand-held radios does not match the cell site's tuned cavities or other filter devices.

AT&T has already announced the shutdown of 2G in a number of other cities. They've likely run into this issue already in other cities.

Sounds like Oakland is low-balling the communication system also. How could a system that they have to engineer (or contract for the engineering) actually be cheaper than a system being installed in all of its neighboring municipalities? Economy of scale ought to bring the price down.

Contractor for Oakland did less than "pencil-whip" the numbers, they likely pulled the numbers out of their wazoos! Site surveys done right, cost a lot of money. Doing the site surveys are likely more expensive than the equipment itself. Site surveys are also the easiest to skimp on to save the profit in a low-ball bid. That's money the contractor/salesman wants in his pocket.

The thing is, inter mod is rarely regulated adequately and so at&t could be quite legal. Not that it should be, but the FCC has not been good at dealing with it. As others pointed out the rfp was either not followed or was poorly conceived and thus the testing failed. Its going to take FOI requests to know for sure.

Unless the public safety radios have selectivity/sensitivity problems, the issue almost has to be AT&T intermodulation products falling within public safety spectrum range. Since public safety is using new radios, it is operating in its rebanded 800 MHz spectrum, which has moved from adjacent to the Cellular 850 MHz A block license to below all ESMR 800 MHz licenses.

For reference here is a rundown of the spectrum (in uplink x downlink format):

Public Safety 800 MHz:

806-815 MHz x 851-860 MHz

AT&T Cellular 850 MHz A block:

824-835 MHz x 869-880 MHz845-846.5 MHz x 890-891.5 MHz

The AT&T Cellular A block downlink (which seems to be at issue, not the uplink) and public safety spectrum are separated by a minimum of 9 MHz (e.g. 860 MHz vs 869 MHz). So, first adjacent channel interference cannot be the issue. Something else has to be the cause.

One thing to keep in mind is that the issues at hand are not constant, otherwise they would have had to scrap the whole system since it did not work. Inconsistent issues are hard to pinpoint to a direct issue because you need to be able to reproduce the issues. Some situations that is easy, in something like radio frequencies not so much.

Another thing to remember is that validation of any form has a tendency to be on a smaller scale than that of the deployment. As a result additional issues can appear after deployment than during validation. Validation of any form is unfortunately about reducing the number of issues encountered during deployment, not about making a flawless product because that will never happen.

I agree that this sounds like inter-modulation. If it is, it's unlikely to be "just" AT&T contributing to the problem. Intermod is a notorious pain in the ass for RF, and really requires a site survey to properly diagnose.

Did they publish a location of one of the problem areas? We could use the FCC website and do the intermod calculations ourselves. The most likely culprits are sharing the same tower, maybe with not enough vertical separation.

ATT will need to go out with some spectrum analyzers and make sure that they are in compliance with rules regarding out of band emissions. They have to be so many db down from the main signal.

If they are not in compliance then the problem is indeed on their end.

If they are in compliance then the problem the problem is likely that the police receivers are being overloaded. That is not an ATT issue provided they are running at or below their power limit.

The city would need to fix the problem with their faulty radios.

It's a pretty easy problem to test for. Go up near the cell with a high quality receiver tuned to the police frequencies and see if you have the same issue. If the problem is caused by out of band emissions from ATT then the problem will still be there, regardless of what you are using for a receiver. If the problem goes away then the receivers that the police are using are the problem.

I think that it's unlikely that ATT is the problem, more likely the problem is caused by the police receiver being overloaded by a strong signal from the ATT cell. It's also likely that the signal from the police transmitter in that area is not very strong.

I place my bet on a faulty police radio system.

Keep in mind that not only does ATT transmit in the 800 mHz region they are also receiving there as well. Thus they have to make sure that their transmitters are very clean so that they don't plug their own ears.

Thus, if ATT were actually causing a problem, they would likely cause more of a problem to themselves than to anyone else and they would need to fix that problem or people wouldn't be able to make calls unless they were very close to the cell.

Whatever interference requirements and the terms of its study are subject to the original Request For Proposal (RFP, assuming it did go out to bid), and something the City and whomever they used as a consultant would have specified. If there isn't an interference analysis done (the frequencies they use are probably existing prior to the new system), there's usually some clause that says the vendor will look into and examine possible means of mitigation. The actual frequencies they're using are here (but I haven't verified with FCC licensing): http://www.radioreference.com/apps/db/?sid=6077

Usually, the concern for interference is other sites that utilize the same frequencies within a particular radius and not cell sites in the same frequency range. There is a dedicated public safety potentially (851-866), but I'm not certain I buy the interference argument. According to Wikipedia, what comes closest to it with 824–849 MHz is used for uplink (which means cellular device to base station). The other band they use is 869–894 MHz, and anything bleeding over that badly from that far is malfunctioning. There's something that doesn't seem right to me, but the blurb that Oakland put out is too brief and doesn't have any actual information. There's nothing in the GSM frequency information that points it to a reliable source of interference, though FCC and AT&T are saying they were. In regards to rejecting possible interference, there are abilities for the system to overcome interference, both in terms of digital modulation and receiver sensitivity. The problem occurs when the noise floor is too close to the actual signal, and there are so many variables that it's hard to even identify if the problem is actually interference or user error.

Let's put aside the blame; I'd like to ask a more adult question:Is there a sensible reason that we still have dedicated public safety frequencies?What I have in mind is: why doesn't public safety use the same infrastructure as everyone else? Meaning they get the same coverage advantages, the same improvements in performance, etc etc.

The obvious rejoinder is that they need to be able to get through even when the public is making a bazillion phone calls. But this is basically a QOS issue, and one that is already solved. My understanding is that GSM already has QOS in place to ensure that certain phones can punch through, no matter how overloaded the system, for the use of telco engineers.

Obviously there are a variety of minor issues one can argue about. Maybe public safety wants push-to-talk functionality. Easy enough to add. Maybe they want an alternative parallel name space, so prank callers can't call public safety "phones". Not trivial, but again doable. But it seems that rather than the current incoherent idiocy we have now, with fragmented bands and the public safety people re-inventing the efficient digital wireless wheel at massive cost, there is scope for a grand bargain. They give up at least some of their frequencies to the telcos, and in return the telcos give them highest priority on the network and more or less unlimited minutes/data for free.

We don't, for example, have a separate public safety electrical grid, or public safety water pipes. Why does it make sense to have dedicated public safety comms infrastructure?

I don't have enough knowledge to have a strong opinion about this either way. But it does seem to me that what we have here is an awful lot of "this is the way it was done in 1930 so this is the way it should still be done today".

"Worse still, Oakland boasts the largest population in the East Bay region and has declined to join a network of 40 police and fire departments that are working on building a unified public safety radio network."

LOL... licensed radio traffic in the LA and SF regions of California is a cluster... the reflections due to buildings and landscape, sheer number of users, people who ignore licensing requirements...

I am amazed anything works out there. armandh2 and secretmanofagent hit the nail on the head. Inter-mod issues are huge.

Pray tell, where is this intermodulation happening?

I presume you're talking about non-linearities producing power output at a whole range of frequencies, but with the strongest being either odd multiples of the frequency (not relevant, AFAICT) or simple sums or differences of multiples (that being a really weird coincidence to get much power into the set). Intermodulation external to the electronics — in the antennas, nearby buildings, etc — depends on some non-linear element somehow, such as a semi-conducting solder joint. Yes, they exist, but aren't likely to produce much of ANY real-world interference.

Unfiltered, out-of-band higher-order harmonics from ordinary digital modulation of signals? Yeah, I could see that AT&T might be pumping out crud in the safety band. The front end of the receivers being overwhelmed by inadequate selectivity against legal nearby signals before the chosen signals are down-converted to the IF band that has more than enough selectivity? Ditto.

Is there a sensible reason that we still have dedicated public safety frequencies?What I have in mind is: why doesn't public safety use the same infrastructure as everyone else?

Maybe in your neck of the woods you have never had a dropped call, or your carrier has ridiculously over-provisioned cell towers so that you always have 5 bars.

I had a dropped call at the biggest supermarket in North Oakland (one of its nicer areas) just this week. One bar when I looked, and the person working there said “you can't make calls here on any network.”

Bad luck? I don't think you could find a single ten-mile-radius area anywhere in the state of California where you could get the necessary signal strength and lack of congestion to support life-critical communications. What's good enough for day-to-day use is dangerously inadequate for public safety. Your cause of more efficient allocation of the spectrum won't be helped by unwarranted assumptions about what's needed.