Sooner or later it happens to every network administrator: the discovery of compromised security, leaked secrets or electronic trespassers. Once it’s happened, what next?

Introduction|~||~||~|The adage runs that there are two kinds of network professionals; those who have experienced a disaster, and those who are about to. That holds true for a number of fields - backup, security, you name it.

This article is aimed at those in the unfortunate position of making the transition from the one to the other - those who have discovered that their network is not quite as sacrosanct as they would have liked. (What this article is not, is a discussion of tools crackers use to gain access to networks, nor does it cover specific applications used in defence. Those applications will be covered in depth in future features; the focus here is on principles and techniques, not nuts and bolts.)

That discovery can take a number of forms. One of your admins asks why there's an account called "grimble" with full admin rights on the NT server. Someone notices that a departmental server's processor utilisation is unusually high, and none of the active processes seem to be the cause. A long-distance call turns up, sourced from the RAS-server's dial-back line. Whatever the event is, the conclusion is obvious - someone has playing in your backyard without permission. That is never a pleasant discovery. You are now faced with both the onerous task of securing and sanitising the network, as well as the unpleasant job of informing management that corporate data, company secrets, customer information, financial records and user files are all potentially compromised. Ideally, no-one ever lands in the hot-seat in that way, but it happens to all of us sooner or later, and this article is for the occupant of that chair.

This article assumes several things. Firstly, that the network has been compromised, and that intrusion has been detected. Second, that the attack is serious enough, or complex enough, that standard security procedures are unlikely to be sufficient. While it may normally be enough to simply tighten up the firewall rules, a competent cracker will Trojan system binaries, modify your kernel and patch custom modules into network services. There are more ways of concealing code in a compromised OS than I can usefully describe in one article, so let's assume we can't reliably establish exactly where the cracker has been. Assuming something is on the loose in your network, and out of control, where do you go from here?

||**||Control the panic|~||~||~|Before we get onto the physical act of securing the network, there is an entire side of the operation many administrators neglect entirely, thereby causing the company even more damage, and that is the PR angle. People talk - you can't keep the users in the dark because they are, after all, being affected. You have to have a consistent message to tell business partners, customers, and even the press should they hear of the incident and look for a story. There needs to be a firm and reassuring message behind which you can stand. "We became aware of a security breach and have taken steps to prevent further incursions. There appears to be no damage to data, but we are carefully evaluating our systems to ensure no traces remain" is a great deal better than "Oh man, we were cracked hard. The admins have no clue, my data is all over the 'net and we're haemorrhaging customers. What a disaster." Don't try to cover up. Make sure your users understand, and that they understand what to tell outsiders. At all times remember that one of those users may be the person responsible! By keeping the users in the loop, they will be more receptive to the tighter security measures you will have to introduce to prevent repeat incidents.

Once you have any possible panic under control, you can put the network under the microscope and start looking for the intruder.

Start by backing up the system. An attacker, wanting to cover his tracks, can easily plant logic bombs that will erase disks, crash systems, and do other damage. Once discovered, you have only a short window of time before the attacker knows the game is up. Disaster recovery and security are closely related fields, but there's no reason to destruction-test more of your network than necessary. When backing up the system, don't forget the users. Trojans on PCs can cause users to lose data that may be as damaging as if the server goes down. Back up everything in sight. Hopefully, you'll never need these backups, and under no circumstances should they be used to run a restore unless data is lost during the security response activities - data in those files should be considered compromised, and restoring it may well open the same holes that you have just closed, allowing the cracker back in, and this time he'll be even harder to detect.

While it is tempting to slam the door shut as fast as possible, there are risks in that. The attacker will immediately know that he has been discovered, and may take action. In fact, as discussed, he may have left time-bombs in his wake to cause damage, or to attempt another attack, in the event of his exclusion from the network. For example, an application can be set to listen on a port, and if no data is received, assume that the penetration has been discovered. A canny attacker will use existing ports, and even existing applications, to accomplish this. A module added to the open-source web server Apache can easily do this job, or a modified version of an ftp daemon, mail client, SMB agent - all of which a network administrator might expect to be running, all on standard ports. Not only will that not show up on a port scan, it won't even generate any unusual traffic, unless you are keeping an exceptionally close eye on your logs.

||**||Do the legwork|~||~||~|Before raising the drawbridge, do as much investigation as possible. Scan the network for ports open on machines where they should be. Look for traffic between machines which have no reason to communicate. Check the logs for signs of failed intrusion attempts within the network which may have preceded successful penetration. You need to establish as clear a picture as possible about what the attacker is using your network for.

Obviously there is a compromise here - if the attacker is actively damaging data, taking time to further establish the pattern of activity is going to be harmful. In that case, you may need to bite the bullet, shut out the attack and trust to your backups if a time-bomb does go off.

What can you trust? Unfortunately, almost nothing. Even if you think the attacker only has access to a single machine of specific subset of the network, you can't be certain. Any competent cracker will ensure he has several points of entry, so that if one is closed, another remains available. If you lock the front door but leave the back door open, the attacker will just unlock the front door from the inside, or open a window. You have to assume every node on the network is suspect, paranoid though it may sound.

Even the most basic of Windows PCs can be used to effect entry later through unconventional means. Your shiny new firewall and IDS may well never know about it until it's too late. For example, a Trojaned version of notepad.exe could be placed on users' machines, looking and acting just like Microsoft's own text editor. When a specific set of characters is detected in a text file, that executable can launch an attack, give the attacker entry to the network, scan the network and mail the results back - anything at all, just as if the cracker was sitting at that PC. Even the simplest text file is suspect (and what do we tell users? "Only text email attachments are safe!") And I am not being alarmist - these tools are out there, readily available to crackers.

Assuming nothing is safe, you need to recover systems back to a situation where either they are to be trusted, or where the damage they can cause is limited. That may mean reinstalling systems from scratch, or reverting to a last-known-good version if you have file-checking tools such as tripwire.

Tripwire (and others like it) maintain a database of file attributes (size, signature, etc) and are able to check for changes to the file. By restoring to a version that matches the original, you can be fairly sure of a minimum of unauthorised changes. Using such tools is a good idea anyway: you will probably detect serious intrusions a great deal faster than you would otherwise.

||**||Keep it clean|~||~||~|Once you have individual systems recovered, you want to keep them clean. Bear in mind that once you roll back to the way it was before the attack, you have also restored whatever vulnerability was exploited in the first place. You have two options now; one is to perform a revamp of security, patching the OS, installing IDS, tightening up the firewall and developing a set of policies to keep the network secure. All good security practice, and if this article is relevant to you for the wrong reasons, it's probably long overdue. The second option is the "honey pot" trap, involving leaving a tempting target vulnerable to the attacker to lure him in, thereby establishing his modus operandi and possibly the source of the attack as well. The risk of this can be minimised by placing the target outside the DMZ created by your new security measures. Obviously you don't want to use an active server for the target - assume the target is a write-off that will be thoroughly compromised, and wait for something to happen. This is a technique usually best tackled by security professionals or auditors; it is too easy to inadvertently allow the honey pot to become a springboard into the rest of the network.

Double firewalls

The goal here, in the first stages of recovery, is to get the core of the network as clean as possible. That will mean at least a second ring of security - firewall, IDS, monitoring - outside of which lies the rest of the network (users, remote access devices, and the rest), which is in turn protected from the Internet by your existing mechanisms. There is no point in trying to patch all of your security at once - as explained, there is nothing you can trust - everything on the network must be examined and overhauled, and that is a mammoth task. It just isn't possible to do it all at once without effectively turning the company off for a week or more.

Start at the centre, and work outwards, but be as sure as you can that the core is stable - any vulnerabilities, and you will be back to square one. At this stage, it is more important to detect an attack than to outright prevent it. Why? Because it's better to log and monitor all traffic and spot an attack than to install a firewall and trust it to protect you from everything. That is naïve, and often the cause of the original incident. Make sure your IDS is up to scratch, and keep a close eye on it. Because many rootkit packages modify logs and logging programs, wrappers that forward logs to isolated machines for analysis are a good idea.

||**||Housekeeping|~||~||~|Once the core is secure, you can start on the laborious task of cleaning up the rest of the mess. This is where it starts to get nasty, because a lot of what is involved is going to meet with resistance from users. Anyone can install a firewall, deploy anti-virus and anti-vandal software, but it's at the desktop level that you will meet resistance, not from the use of new technology, but from the enforcement of security policy. Unfortunately, that is inevitable. Most security breaches occur not because of a lack of security mechanisms, but because the policy was weak, and the policy is usually weak because it is too much effort to enforce. Sadly, that is just not good enough - if you don't want to find yourself in the position of recovering from a severe network penetration, you absolutely must have a strong security policy.

There is no point applying the new security framework over a penetrated system, however. Does that mean reinstalling every user's software? Ideally, yes, but that may not be necessary if you are willing to accept the possibility of sporadic user downtime (which is common enough anyway). Move important data to the newly cleaned core servers (but make sure the data is clean! Macros hidden in Microsoft Office documents could put you right back where you started), and hope the security around the core will protect you from attacks from users' PCs. Note that this is just another iteration of the first step I described - backing up the system.

Check for common tools

Because of the vast number of existing tools for cracking desktop systems, it's quite likely your attacker will have used one or more of them. Many can be detected by anti-virus agents and standard port-scans, so it's worth a look to see if any known backdoors or Trojans are running on the network. An nmap sweep of the network can show many interesting services you didn't know about; it's worth running regularly, but be aware that your IDS will go berserk if you start running scans from arbitrary PCs!

One of the aspects of tightening up security that users are going to dislike is the process of changing passwords, but it is essential. Not only should passwords be changed (and length, complexity and expiration issues enforced), any service which permits log-in details transmitted in clear text (and hence available to a traffic sniffer) should be eliminated. This may mean migrating some applications to more secure versions, such as ditching telnet in favour of SSH. Remember that some password may have deeper implications than users know - a POP3 password may in fact also be a password to a shell account on a Unix server. Just because the user didn't know that does not mean an attacker will not gleefully seize the opportunity. Clearly the Unix administrator should not have configured the system that way and should identify the non-shell user logging in, but that is another topic.

While you're down there at the user's PC, take a look for suspect files on the drives. Large files that may be encrypted virtual disk volumes, zip files with passwords, binaries in odd places, unidentified services running in the background (remember that no respectable cracking tool will appear in the task manager - you'll need a system analyser to spot them) and so on. If there are signs of tampering, isolate that machine immediately and disinfect it, which may mean a reinstall, and may imply that other machines have been similarly breached.

Take stock

Once the initial panic is over, you can step back and take stock of what you have. The goal here was not to build and deploy a security policy, it was fire-control in the face of disaster. Once the immediate fire is out, attention can be diverted to prevention. As always, prevention is far better than cure, as admins who have weathered a security breach will know. During the clean-up operation, you'll have effectively conducted a low-level audit of your security policies and backup architecture, so you should be in a position to take the next step, drafting and maintaining policies that will make the next attackers job a great deal more difficult. You'll never be able to lock the attackers out completely - 100% security is as much a myth as 100% server uptime - but you can get close.

||**||Insider trading|~||~||~|It is a sad fact that 80% of network attacks are sourced from inside the network. That takes two forms - employees with a grudge seeking retribution of some kind, or outsiders who have managed to compromise a machine inside the network using a backdoor (such as Sub7 or BackOrifice) or a Trojan, and are manipulating that machine somehow, taking advantage of its position inside the firewall to further penetrate the network.

Because of this possibility, it is obviously not acceptable to simply have a firewall protecting the perimeter of the network. Inside attacks are the most dangerous of all - and you need to keep them at arms length just as the rest of the Internet.

Either way, the conclusion is inevitable; you can trust neither your users nor their PCs. Passwords should change regularly, traffic should be monitored and software on users' PCs should be watched (do your users really need ICQ? IRC? This is software that is easily exploitable, yet finds its way onto many corporate networks because the admins are not exercising control over those desktops.

Segment core services on the network; keep a clean core behind a filter. This adds two benefits - it ensures that there is no miscellaneous traffic overhead clogging the network core, and it provides a further level of logging and filtering.

All network segments should be monitored anyway, with an updated IDS keeping an eye on suspicious activity.

Remote users are especially at risk - connections to the corporate network by VPN or RAS are dangerous because you have effectively lost control of the end node. Extra security, both in authentication and in the services available to the remote users, needs to be applied.

Too paranoid?

It's not uncommon for a security consultant to be dubbed "too paranoid". After all, they are usually recommending spending money on something that hopefully will never be called upon to actually do anything. They advocate unfriendly user policies and tight control, alienate users, and generally seem to want to turn the easy-going networks we are used to into a boot-camp.

That's all true. Is it unjustified? Not necessarily. Compare security to a physical danger, and it all starts to make more sense. If you discover a gas leak in a building that could lead to a catastrophic fire, do you hope it'll never happen? Put up a notice saying "please don't strike any matches" and hope employees will get the hint? Of course not - you take immediate action, and thereafter take steps to prevent any future reoccurrence. That's all part of a fairly basic workplace safety policy which many big companies have. Security is no different. You need to recognise the dangers inherent in lax security, and react quickly when incidents do arise.

To a security professional, there is no such thing as "too paranoid" or "too secure". To an attacker, there is no such thing as "unbreakable".||**||