KQED, one of the largest public media companies in America, was hit by a ransomware attack in July. The disruption was barely visible to the public, but behind the scenes, employees struggled with Rube Goldbergian workarounds while tech staff worked long months to rebuild a more secure system.

The Crippling Ransomware Attack on a San Francisco NPR Member Station

Originally published Oct. 31, 2017

Lesley McClurg is trudging through the KQED newsroom, oozing distress.

McClurg, who in the course of her career as a radio reporter has conducted interviews amid raging fires and torrential floods, has finally met her match: an office printer. She has just spent an entire hour trying to get her radio script to print, a task that took the cumulative analytical skills of three nearby co-workers plus a guy from IT.

“And now I need to go on the radio and sound like I’m calm and relaxed and NPR-friendly,” McClurg tells me. “But, really, I need to scream.”

Sponsored By

And so went Day 33 of the Great KQED Ransomware Attack.

This summer, criminals somewhere in Russia, Ukraine or maybe down the block, hacked into KQED’s computer system, installed malicious code that encrypted the station’s files, software and servers, and demanded money for their safe return. The perpetrators, as is routine in ransomware attacks, demanded payment in the form of the cryptocurrency bitcoin—if the victims ever wanted to see their data alive again. The message, rife with grammatical and spelling errors (cold comfort to no one but the copy editors), included directions on how to recover the data in “3 Easy Step” (sic), with step one being the transfer of funds.

'It's not some teenager sitting in his garage. These are effectively organized crime.'Elizabeth Weise, USA Today

The chaos commenced on June 15, as KQED’s employees began suddenly reporting computer crashes en masse. The IT department at the San Francisco-based NPR and PBS station decided to play it safe—they shut down the entire network to stop the malware’s spread.

The results were difficult to face: Phones went dead; the internet vanished; reporters’ audio interviews disappeared. And it meant the only way the head of IT, John Reilly, could get a message out to each of the 503 full- and part-time employees to keep their hands off their computers was to post handwritten signs at every entrance and on every desk.

KQED, one of the largest public media companies in America, has been hobbling its way to recovery ever since. What was it like? Think of a really boring episode of “The Twilight Zone”—or better yet, “Black Mirror.” Or getting stuck in an absurdist satire about human dependence on technology. Everyone had their particular breaking point, triggered by one Rube Goldbergian workaround or another.

One afternoon, a KQED marketing employee attempted to send out a meeting agenda in a single email to some 60 people. Our temporary email system wouldn’t allow grouping of multiple addresses into one alias, so each destination had to be entered one at a time. Any tiny problem with the syntax—an extra period here, an omitted semicolon there—grounded the message until you could figure it out. Four hours, that job took.

Ransomware—it’s a growth industry. Last week, a new strain called "Bad Rabbit" hit computer systems in Russia, Eastern Europe and beyond. This year, a posse of federal agencies, including the Justice Department and the CIA, issued a report describing ransomware as the “fastest growing malware threat, targeting users of all types—from the home user to the corporate network.”

And in another study, anti-virus software maker Symantec estimated 463,000 ransomware attacks worldwide in 2016, a 36 percent increase from the previous year.

Entrepreneurial hackers also have innovated “ransomware as a service,” in which they peddle kits to criminals who are less technically adept.

“It’s not some teenager sitting in his garage,” USA Today tech reporter Elizabeth Weise told KQED. “These are effectively organized crime; they have offices, they have office workers who come in.”

Ransomware Survival Tip: Triage

You can't fix everything at once, so think like a triage doctor and prioritize essential functions. KQED priorities: Check and secure critical information (membership data was off-site, so was not breached), pay staff, pay bills, and keep daily radio news on the air. What had to wait: Networked printing, high-speed internet and access to stored documents and audio.

Over the last two years, researchers from New York University and Google tracked about $25 million in victim payments. Not bad, but not the global drug trade either. Estimates of monetary gain for perpetrators of the WannaCry ransomware scourge, a worldwide attack last May on computers running the Microsoft Windows operating system, were remarkably low, even though the virus disabled more than 200,000 computers, including those of FedEx and England’s National Health Service.

That does not, however, take into account the cost in lost sales and business interruptions, which can be considerably higher. The U.K. company Reckitt Benckiser, maker of Clearasil, Lysol and Airborne, estimated a $117 million loss resulting from the Petya ransomware attack in June.

In KQED’s case, the attackers demanded 1.7 bitcoins per computer (roughly $2,500 at the exchange rate of the time) or—act now!—just $27,000 for the lot.

KQED briefly considered paying up, said Chief Technology Officer Dan Mansergh, but abandoned the idea after the FBI laid out the risks. They were: being seen as an easy mark; inclusion of code in the decryption key that would set the stage for future intrusions; and failure of the perps to keep their part of the bargain to set your data free.

KQED has yet to fully calculate the attack’s cost, but much of it will be in IT. In fiscal year 2018, the company is allocating $475,000 for new network protections. Those funds will pay for four new IT employees, plus more sophisticated anti-virus software.

As for the cost in employee productivity and stress? A lot.

‘Worst Stress of My Life’

The attack took place on a Thursday. The next day, news reporters and editors at KQED, the most-listened-to radio station in the Bay Area, could access the internet only through personal hot spots on their cellphones. Reporters read stories into their recording decks to get on the air and didn’t use sound bites; no one crumpled cellophane to emulate the sound of a fire, but it was still old school.

How to Make It Harder for Malware to Shut You DownFor 18 days before the ransomware attack actually commenced, the implanted malware sat dormant in KQED’s computer system, probably sniffing out usernames and passwords. When the malware got hold of an account with the ability to enter anywhere and do anything, it propagated throughout the network, infecting machines and locking up files. Read more...

The station's weekly current affairs television program, KQED Newsroom, had to abandon its original production plans and run a previously aired special show. And when the television studios were still out the following week, the staff took production out of the building, to a makeshift studio at UC Hastings College of the Law and another at the Electronic Frontier Foundation for a segment on ransomware.

Meanwhile, IT collected and wiped clean every computer that the malware, called Samas, had so much as breathed on. But that was nowhere near the end of it. To stop the infection from spreading, IT had taken the network down. No phones, one rickety internet router, no access to documents or audio, and the only way to print was to walk your laptop to a printer and plug in the cable. And after the malware was eradicated, the network did not come back up.

“We could not just put it back the way it was, because there were all kinds of security holes,” said Michael Kadel, KQED’s network systems engineer.

So the staff went to work creating temporary systems to keep things functioning until a more secure infrastructure could be built. They worked long hours, long being a euphemism for 12-, 16- and 20-hour shifts at a time.

“Worst stress of my life,” said IT director Reilly.

Kadel agreed. “A solid two months of just terrible, terrible stress.”

Mina Kim, afternoon drive-time news anchor and Forum host. 'Initially, we all had this attitude, the show must go on. After that surge of adrenaline and can-do attitude subsides, you’re just exhausted. It’s workaround fatigue.' (Arash Malekzadeh/KQED)

Kadel said he’d once suffered through cancer. “This was way worse.”

Those views are not necessarily unheard of in these types of situations, said Jonah Silas Sheridan, a Bay Area computer security consultant for nonprofits.

“It’s a big deal for security practitioners,” Sheridan said. “How can you secure someone’s systems when you are in a place of sleeplessness and arrhythmia yourself?”

Meanwhile, the radio news team and their engineers went full-on MacGyver to rig a system that kept the news on the air. In normal times, reporters and editors were fully networked, with universal real-time access to every story, revision and audio clip. Now, every story had to pass through three or four computers. Audio had to be uploaded to one site, then downloaded somewhere else. Cellphones and the instant messaging app Slack were conscripted into the workflow. The motto: Whatever works.

As weeks turned into months, the newsfolk reported on the congressional health care debate, the move to end DACA, the total solar eclipse and the eruption of far-right rallies in Charlottesville and other cities, all using a system that took three to four times the usual amount of time.

Ransomware Survival Tip: Ease Work RulesDo what you can to ease stress and tech difficulties by relaxing work rules. KQED staff often found it more productive to work in neighborhood cafes or at home, where they had high-speed internet. And be clear with staff about how long it will take to resume normal operations.

The audience most likely didn’t notice. But the staff found the relentless workarounds, jury-rigging and duct-taping exhausting.

“Initially, we all had this attitude, ‘The show must go on,’ ” said Mina Kim, the afternoon drive-time news anchor. “After that surge of adrenaline and can-do attitude subsides, you’re just exhausted. It’s workaround fatigue.”

I asked John Reilly, who’s done a lot of consulting in his career, if he’d ever seen an organization experience the level of disruption KQED had. “No, not through an attack,” he said.

How could it have happened? And how much at risk are other companies?

Convenience vs. Security

Security experts like to recite an aphorism to describe the basic dilemma of their profession.

The safest computer, they say, is one sitting in a metal box in a locked room, switched off and unconnected to anything. Such a computer, goeth the Parable of the Completely Safe Workstation, is impenetrable.

And utterly useless.

“You do have to make trade-offs of usability and convenience for security,” said Paul Guthrie, a vice president for PSC, the security firm hired by KQED after the attack. “And sometimes the pendulum swings too far.”

KQED Director of IT John Reilly and Network Systems Engineer Michael Kadel. Much of the burden for rebuilding the network in a safer way fell on them. (Arash Malekzadeh/KQED)

Guthrie is one of the tech and security people both inside and outside KQED who agree the company’s pendulum over the years had strayed quite a way over into convenience territory.

Kadel and Reilly, who had been the head of IT for a year and a half when the attack hit, were frank about the vulnerabilities the ransomware exposed.

“Honestly, I’m surprised this didn’t happen long ago,” Reilly said.

Kadel put it this way: The culture of KQED, a multifaceted media organization that produces a huge amount of radio, TV, educational materials and websites, tilted toward easier collaboration. If someone wanted to put the video from a news interview into the video editing system, for instance, and then share the audio with radio reporters, they could. And they could do that from the same computer they used for daily email.

But security people know this type of one-big-network-for-all is risky. Their job, after all, is to keep malicious hackers from turning the place into “The Day the Earth Stood Still.” And when systems connect to each other like they could at KQED, malicious actors can more easily infect the entire infrastructure.

Another problem: The station’s IT department allowed employees to download and install free software from the web. IT folks call this the “granting of local admin rights.” Many in the industry also call it “A Catastrophe Waiting to Happen.” Free software is often laden with viruses, worms and other nasty extras.

Ransomware Survival Tip: Establish a Walk-Up IT Help DeskShield your IT staff from the deluge of help requests, while providing a clear avenue for employees to get help. At KQED, IT set up long tables in an open area and staffed a walk-up help desk all day. KQED Chief of Staff Jason Black laid down the law in a companywide memo: 'Individual requests, side doors, hallway nudges, note passing, private Slacking, hand and/or smoke signals to IT or Systems Engineering are not helpful right now.'

Kadel said one reason IT gave the staff free rein was due to a lack of IT employees to oversee the process.

“You can’t visit 600 people to do a Flash update,” he said. “You have to let people do their own Flash update.”

Lots of consultants say companies routinely shortchange IT. But Kadel may have a point. At the time of the attack, KQED employed nine IT employees to service roughly 503 workers. That means IT represented less than 2 percent of the organization’s workforce.

A 2016 survey of media and entertainment companies by the research firm Gartner found that IT workers made up an average of 6.5 percent of employees, more than three times the level at KQED. Across all industries, the number is 5 percent.

Finally, KQED used standard anti-virus programs to protect its system. But hackers know how to get around these by altering their code. In a test run by KQED’s security consultant, just 10 out of 60 off-the-shelf anti-virus programs recognized the variant of Samas that hit the company.

“If somebody wishes to put out a new variant, it takes a little bit for the anti-virus vendors to catch up,” said Guthrie.

KQED is now plugging all these holes, redesigning its network and all of its media systems. It’s also deploying much more sophisticated anti-virus software that looks for processes that are outside the normal behavior, shutting them down. Among the new IT employees coming on board: a full-time security pro.

What Kept Working

When everything went down, the safest place for data turned out to be the cloud. KQED lost no member or donor information, because it wasn’t stored in the network. And as the attack short-circuited system after system, KQED’s network of websites kept on keeping on. How so?

“We’ve moved most of KQED.org to be hosted in the cloud, so it’s sitting on [Amazon] servers,” said Tim Olson, KQED’s chief digital officer. “It’s more disaster recovery, more efficient for everybody.”

'In some ways I view this as a dress rehearsal. Because we’re in an earthquake zone.'John Reilly, KQED

The cloud is “certainly a strategy that a lot of smaller organizations employ, because it’s a lot of work to keep your systems patched,” said IT consultant Sheridan. And big cloud services like those of Amazon, Microsoft and Google spend tons of money to pay for cybersecurity. “That being said, you’re now in a very deep trust relationship with someone else, Sheridan said.

But in terms of maintaining access to data after the attack, the cloud was the silver lining. When everything went down and all of KQED’s storage systems became inaccessible, there sat my audio interviews, my budget numbers, my everything, safe and snug on Google Drive.

Talking About It

Below the hard truths about security vs. convenience now adorning whiteboards at KQED are a few about survival, too.

“We've learned a lot,” said Reilly. “People have realized that we weren't prepared in a lot of ways. But then again, in some ways I view this as a dress rehearsal. Because we’re in an earthquake zone.”

What’s being done to fight back against the spreading threat of ransomware across the world?

Companies and individuals dealing with a ransomware attack may find help at NoMoreRansom.org, an initiative of the Netherlands’ police, Europol and the computer security companies Kaspersky Lab and McAfee. The website includes decryption tools for dozens of known ransomware infections, as well as advice on how to prevent an attack in the first place. (Update Nov. 15: The U.S. Dept. of Homeland Security has banned federal agencies from using Kaspersky products due to alleged ties to the Russian government and intelligence services.)

Ransomware Survival Tip: Use the Cloud and Your Cell NetworkSlack, Google Drive, Dropbox and cellphones can substitute for networked communications and make transferring files easy.

One thing every expert recommends: Back up like crazy.

“The number one thing to do is back up your data,” said Alexander Garcia-Tobar, co-founder of ValiMail, a San Francisco company that authenticates email in order to prevent intrusions like ransomware.

“If you’re backing up on a daily basis or even on a weekly basis, you back up to before the known infection and you cross your fingers.”

Guthrie said high-end tools designed to keep things running in the event of a catastrophe can be a lifesaver. But they are expensive.

“There’s network backup and restore systems that are very good and very fast and very expensive,” said Guthrie. “Smaller companies tend to do what they can with the resources they’re given.”

KQED decided to go public with our tale as a kind of public service, so that other organizations can learn not only from our mistakes but also from the way we kept functioning with minimal disruption to the public, despite maximum disruption internally.

Sponsored By

Sheridan said opening up can only help.

“You might be surprised at how many different kinds of breaches, how many kinds of incidents, have happened in other organizations, and continue to happen,” he said. “We would be well-advised as a community to figure out a way to talk about these things without shame. We need to learn from them collectively.”