Antivirus products aim to prevent hackers from getting into a computer. But hackers often get in anyway these days. So Mr. Dye is leading a reinvention effort at Symantec that reflects a broader shift in the $70 billion a year cybersecurity industry.

Rather than fighting to keep the bad guys out, new technologies from an array of companies assume hackers get in so aim to spot them and minimize the damage.

Network-equipment maker Juniper Networks Inc. JNPR +1.52% wants customers to place fake data inside their firewalls to distract hackers. Shape Security Inc., a Silicon Valley startup, assumes that hackers will steal passwords and credit-card numbers so seeks to make it difficult to use the pilfered information. FireEye Inc. FEYE +0.61% created technology that scans networks for malicious-looking computer code that made it past the first line of defense. FireEye recently paid $1 billion for Mandiant, a small firm led by former Air Force investigators who act like cyber-Ghostbusters after a data breach.

Symantec seeks to join the fray this week. It is creating its own response team to help hacked businesses. Within six months, the Mountain View, Calif., company plans to sell intelligence briefings on specific threats so clients can learn not just that they are getting hacked, but why as well. Symantec also is developing technology to look for more-advanced malicious software inside a network that mimics offerings from its rivals.

The company needs a turnaround. Revenue fell in each of the past two quarters, though profit rose because of cost cuts. The company, which reports earnings Thursday, forecast revenue of $1.62 billion to $1.66 billion for the quarter through March, down at least 5% from a year earlier. The company in March fired Chief Executive Steve Bennett, the second time in two years it had ousted a CEO.

Mr. Dye, who has spent more than a decade with Symantec, says it was galling to watch other security companies surge ahead. “It’s one thing to sit there and get frustrated,” he says. “It’s another thing to act on it, go get your act together and go play the game you should have been playing in the first place.”

Symantec pioneered computer security with its antivirus software in the late 1980s. The technology keeps hackers out by checking against a list of malicious code spotted on computers. Think of it as an immune system for machines.

But hackers increasingly use novel bugs. Mr. Dye estimates antivirus now catches just 45% of cyberattacks.

That puts Symantec in a pickle. Antivirus and other products that run on individual devices still account for more than 40% of the company’s revenue. Specialized cybersecurity services for businesses account for less than one-fifth of revenue and generate smaller profit margins. It would be impractical, if not impossible, to sell such services to individual consumers.

Ted Schlein, who helped create Symantec’s first antivirus product, describes such software as “necessary but insufficient.” As a partner at venture-capital firm Kleiner Perkins Caufield & Byers, Mr. Schlein invests in new cybersecurity companies that compete with Symantec.

Mr. Dye says Symantec’s Norton security suite has evolved beyond antivirus software and already looks for suspicious activity that may come from previously unseen viruses. It also includes, among other things, a password manager, a spam blocker and a tool that scans a user’s Facebook feed to guard against dangerous links.

The company has no plans to abandon Norton but will find revenue growth in its new product lines, he says. “If customers are shifting from protect to detect and respond, the growth is going to come from detect and respond,” Mr. Dye says.

Other traditional antivirus makers such as McAfee, Intel Corp.’s security unit, have moved in the same direction. Michael Fey, McAfee’s chief technology officer, says there is typically a two- to three-year lag on developing the technology Symantec seeks to create. “They haven’t been part of the thought-leader group for some time,” Mr. Fey says.

International Business Machine Corp. on Monday plans to unveil its own security suite that looks for irregular behavior in computer networks.

If Symantec has an opening, it is that no security company has determined how consistently to defeat the most ambitious hackers from China, Iran and the former Soviet bloc. Hackers linked to Iran last spring breached the digital perimeters of energy companies and one of the U.S.’s five biggest banks but were caught before moving further into the systems. The incidents were notable because the two industries have among the best private-sector cyberdefenses.

Cybersecurity firms also want to help discern the most serious threats from the less serious. Before Target Corp. was breached last year, FireEye security equipment alerted the retailer to suspicious activity. But the company decided it didn’t require follow up. Former Target employees say the team lacked the resources to pursue all threats.

“What do we do with all the things that we’re 60% sure are a problem?” Mr. Dye says. Analysts say Symantec’s software runs on so many machines that it may be able provide more guidance on which hackers can be ignored and which are truly a problem.