Deploying ISS RealSecure in a Large Scale Environment, Part Two: Manageability and Reporting

Machine Translations

Welcome to the second half of our series on deploying ISS' RealSecure IDS product. The previous article focused on some of the initial thoughts and practices of actually getting your infrastructure out there. This article will expand on that by helping you to manage your deployment, as well as to provide our ideas on event response and executive level reporting. You will find this half to be much more detailed in terms of actual practical knowledge you can use today.

Manageability

When RealSecure is used in a smaller environments, it is easily managed. The console (Figure 2) allows you to watch events in real time over an encrypted channel. Updating an agent or engine's properties is as simple as right clicking and making changes. However, to deploy on a large scale you must take several steps into consideration:

Policy tuning - ensure you are not missing important events or gathering useless data

Data collection - how to collect the data into a manageable format

Event response - what to do if you detect hack attempts or a breach

Figure 2. The RealSecure Console (Click to view a larger image)

Policy Tuning

In a lab environment, it is easy to guess at what type of policy you require. However, once RealSecure is deployed on a production network your "default" policies will most likely need serious tuning. False positives will quickly fill up your database, causing queries to slow down and increase network traffic between engines, agents and consoles. You will need to balance the data collection versus streamlining policies, as too many signatures left out will leave you vulnerable to an unmonitored attack.

Enabling and Disabling Policy Signatures

There are several ways of tuning your policies. By enabling all signatures (or most, if you know there are some which do not apply) and watching the console you can get an idea of which events are important. However, a better test is to do the following:

Choose a heavily trafficked segment (engine) or heavily used server (agent) for testing.

Apply a liberal policy to the engine or agent.

Allow the machine to gather data for a period of time. You can monitor how many events are in the remote machine's database by right clicking and selecting "maintain log" or using the EngineMgr.exe -a loginfo command.

Ensure the rsntclientlog.mdb file is empty on your Console machine. Note that "empty" does not mean deleted. ISS can provide you with a skeleton database file that just has the table structures in it if you do not have one. The file should be about 235k.

Perform a database synch with the console or EngineMgr.

You can now run the console's reporting tool to see what your top 20 events are. This will help you trim down unnecessary signatures. After you have done this several times on different segments and on different hosts, you will be ready to push your policies out to your machines.

Different Policies for Different Environments

Unless you have a boring infrastructure, you are going to find you will be maintaining a number of different policy files. Depending on a machine's role (e.g. File and Print, Domain Controller, Web Server) you will have different signatures enabled. You do not want to watch for scans of port 80 on a Web server because these ports are always in use. Also, you will most likely want to monitor program execution on a NT domain controller but not necessarily on an application server. The same concepts hold true for network engines as well. During a recent large-scale RealSecure deployment we needed nine policies.

Data Collection and Detector Management

As your systems are gathering data, it must be collected into a central repository. The local detectors (agents/engines) have a database that will fill over a period of time, and events will be lost. To prevent this, the data must be collected into a central database server on a regular basis. Unfortunately, a single console and its associated database cannot handle having a very large (500+) number of detectors reporting to it. To combat this, ISS provides SAFEsuite Decisions as a central data warehouse facility. It uses a combination of a SQL server, data collection agents and reporting tools to collect and analyze the data. This gives you the ability to store all events over a greater period of time to improve your ability to provide both a greater number of reports, and to perform historical trend analysis.

Getting the data from the detectors to this reporting infrastructure is where the deployment is at it's most challenging. A standard RealSecure deployment provides two methods of collecting data - The RealSecure Console (a GUI), and EngineMgr - a command-line interface. The console provides both real-time event monitoring and data collection. EngineMgr only does policy, database and informational functions (no real-time monitoring). You will find you must use both of these if you are to have any hopes of maintaining a consistent and stable infrastructure that is flexible enough to allow quick policy modifications during crisis situations.

The RealSecure Console

The RealSecure console actively monitors each detector and pulls the remote database after a threshold has been hit. Unfortunately, it can be cumbersome to work with in a large environment. Every time the GUI is started each detector must be added individually for monitoring/collecting. Also, the number of machines that each console can process is very limited in number. In an environment with 1000 detectors, you could need anywhere from 20-30 consoles just to collect the data. These numbers could vary depending on how heavily trafficked the detectors are and how powerful the console boxes are. This is due to both the high memory utilization of the console itself, and the always-limiting 1GB limit of Microsoft Jet databases. It is also important to note that there is no provided way of correlating the databases of multiple consoles without the SAFEsuite Decisions product.

Another limitation in the current version of the console is the inability to push policies out to multiple machines simultaneously. In order to update each machine, it must be selected, the policy selected, and then applied. It would be nearly impossible to do this to hundreds of machines in a timely fashion even with a staff of 10. This should be fixed in the next release of the product, but for now you must work around it.

EngineMgr.exe

This section explains in detail what we have found to be the most efficient and useful way to manage your enterprise. By following what we explain here you should be able to manage 1000+ detectors from a single machine. The command line equivalence to the GUI is a programmed called EngineMgr.exe. Since it is a command-line program, it can be programmatically used to collect data or make changes. EngineMgr supports over 18 commands that mimic almost every function of the GUI. For the purpose of day-to-day operations the following commands will be used:

-acqmaster: Acquires master for a detector

-relmaster: Releases master

-getdb: Pulls data from the detector

-applypolicy: Publishes a new policy

-clearlog: Deletes the remote log (database) on the detector

-applyengineprop: Defines the responses (email, SNMP, etc.) for a detector

Once you have policies in place, the -getdb option is to be used on a regular basis to collect data. During one RealSecure implementation we configured two consoles to pull data from 300 engines and agents. One console was used for to poll network engines and critical agents. This totaled 51 machines - 22 engines and 29 agents. They continuously gathered data and it was pulled into the SafeSuite database. The second console pulled from 250 agents in the same fashion. The total time required for a console to cycle through all detectors varied between 20 minutes to 2 hours depending on the time of day. During production hours more data was collected and more time was necessary to pull. If the number of records held on the detectors is increased you could add more to a console's responsible list.

Sample EngineMgr Algorithms

The following algorithms will give you an idea of what commands we ran to manage our environment. The "@" and "$" symbols below represent an array and a variable, respectively.

Step 1 - Copy the console's public key, add the engine, and acquire master of all remote machines:

Step 2 - Apply your policies and response files. Note - you will need to use some type of logic to differentiate which group of servers gets a particular policy. The Policy and Response files must reside in the C:\Program Files\ISS\RealSecure3.x\Policies directory on the machine running EngineMgr. We chose to use a delimited file with different columns to organize the engines. Most of your machine will use the same response file (to signify email address, SNMP destinations, etc.) For example:

Aside from collecting data, you will need to react to certain events in a timely fashion. If you are watching the console, you will have all the information you need. You also have the option of sending an email, SNMP message, or running an external program. If you use SNMP inside your company, you could use this for notification of important events. By the way, if you are an OpenView shop there is a nice plug-in that ISS provides to fit into the HP framework. Likewise an email could be sent to a pager or other address. For monitoring of networks, a TCP Kill can be used to terminate the session. One last option is to run an external program. You could use this for just about anything - a proprietary notification system, configuring of network resources, etc.

If you choose to use the EngineMgr approach for managing your infrastructure, then you will most likely not be deploying a great number of consoles. In this case, you will need to rely on a real-time event response. We like to call this the full "console-less" deployment. We have found the best method is to configure your policies to send all of your "high priority" events to this real-time system and then to generate daily reports on the rest of the data. This real-time system will hopefully be something that is sent directly to your pager or cellular phone, allowing you to react immediately. This also applies even if you have consoles, but maybe not someone sitting in front of them on a 24x7 basis.

Reporting

RealSecure, when combined with SAFEsuite Decisions, allows you to analyze detector data in many formats. Some example report names include:

Attack Analysis - Top 10 Common Attacks

Attack Trends - Top 5 by hour. Includes separate options for weekend/weekday breakdown Coordinated Attack from a single source

Top 20 events

Event Priority

There are over 25 reports included with Decisions. How you will use them depends on your particular environment and needs. By keeping previous reports you can compare different time periods to help identify new attackers.

In addition to providing forensic data about attacks, overview reports such as "Attacks per Check" (Figure 3) are ideal candidates to give to other departments so they have an idea of the types of data being collected. This keeps the Security group visible, enhancing their image while not giving away any critical data that could be misused. Often internal network and user account problems will show up on such a report and can be used by systems administrators to rectify the situation. When problems are solved, money is typically saved because network and system performance is improved.

The best part about SAFEsuite Decisions is it is fairly standard in terms of interface options. You can easily create your own reports using the Crystal interface. This allows you to really get specific reports based upon your environment and your specific needs. As an example, we had a deployment where a daily report of all interactive logons onto a specific set of Windows NT Terminal Servers was needed to help keep a tab on security, as well as to supply data to the charge-back facilities. It is a flexible and almost necessary add-on to your deployment.

When you combine all of these products into a single deployment, you can really get a complete security monitoring solution from a single vendor. This allows you to be both pro-active and re-active in terms of your overall site security.

Known Issues, Tips & Tricks

There are several known challenges that can perturb someone deploying RealSecure in an enterprise. One of the main limitations of the software is its use of a Microsoft Jet database to hold data on the console. As we explained earlier, data is pulled from the detectors and stored into an MDB file. Due to limitations with Jet, this file cannot exceed 1 gigabyte in size. As data is collected into the MDB, the file must be replaced or purged on a regular basis. Some customers have implemented a method of pulling data directly into a SQL database via an alternate DSN, but this is not supported by ISS at this time. Part of your getdb routine can check the file size, and when it sees it has reached a particular threshold it can be replaced.

Another issue is the lack of port options in EngineMgr. In order to communicate with agents and engines on the other side of a firewall, you must configure one port for each machine to communicate with it. For example, you may configure port 1500 to communicate with web server 1, 1501 to communicate with web server 2, etc. However, EngineMgr only allows you to use the standard ports so some type of port redirection must be done. You can usually finagle a "plug" type of gateway on your firewall to get around this. Again, your mileage may vary.

One other consideration is the use of RealSecure Network Engines on a switched network. In order to view all traffic on a segment, port spanning must be done at the switch level. Your network operations group must be involved and be part of the project plan before cooperating. If you have a deep switch fabric you may find that you must deploy more engines or start creating additional VLAN's to replicate your data to. This could be a significant engineering effort so plan for it in the beginning. This is not just an issue with ISS, but all network based IDS. It is mentioned here for completeness.

Finally, again another general problem not one associated with RealSecure, is encrypted data. NO signature will be effective against network streams that are encrypted, such as HTTPS. The only chance you have is with the host agent as only the destination host really sees the data in it's unencrypted format.

Conclusion

The market for intrusion detection systems is still in its early stages. If you have chosen to deploy ISS RealSecure in your organization we think you have made an excellent decision. Be prepared to work with the software and expect greater functionality as it matures. With proper planning you can provide a first-rate level of protection for your company.

Richard Reybok currently manages a Wall Street financial firm's global security engineering team where he puts his ninja security skills to the test. He has over six years experience in information security technologies and over ten years in the industry itself. When not defending the free world, Rich relaxes by spending time with his lovely wife Jennifer and two beautiful daughters, Samantha and Kaitlyn.

Michael Engle currently manages computer security and incident response for a large financial firm on Wall Street. He has over 8 years experience in the industry. In addition to computing as a hobby, he enjoys scuba diving, skiing and traveling the world with his fiancée Diana.