Charlotte, N.C., Automates Incident Logs

As an information security engineer for Charlotte, N.C., Rusty Agee is responsible for maintaining the integrity of the network for 6,500 users spread over 100 locations. He and a small team oversee traffic coming into the network from all city departments. Fire stations, police satellite buildings, utilities and solid waste facilities, the transportation department, engineering, administrative functions and more all fall under his purview.

Until recently, the city used an incident management system that satisfied state regulations for storing incident data. But the system was difficult to manage and labor-intensive when it came time to retrieve information – in short, it had outlived its useful life.

“We were compliant because we had the data for a year, but it was of little or no use to us because it literally took hours to comb through just to pull the data back online and then several more hours to comb through it,” Agee explained to Government Technology.

After considering several solutions, IT security staff went with Boulder, Colo.-based security information and event management provider LogRhythm. According to Agee, this was the right choice for Charlotte to get quick access to log information, and easily-digestible reports that were simple to analyze.

Dave Anthony, vice president of customer care at LogRhythm, explained that the company is gaining traction in the government market given growing concerns about cybersecurity threats. One of the first questions they get from many public-sector customers, according to Anthony, is “‘Where have I been hacked already that I didn’t even know about?’”

What they find, more often than not, is surprising.

“When you start looking for something for the first time, you’re almost certain to find it,” Anthony said. "And most of our customers, when they start looking, they do find something -- and it was something they didn't expect.”

The city of Charlotte bought the LogRhythm solution at the end of 2010, and began using it in early 2011. The deployment itself lasted about one week, and then Agee began the process of fine-tuning the amount of logs that it produced to eliminate duplication and make sure reports only contained needed data. He describes this tuning process as ongoing.

In fact, he suggests that other agencies considering a similar event manager should spend time on the front end considering the volume of logs it will generate and ensure that they scale it for their organization.

“Make sure you size it appropriately for the incidents and the amount of logs you’re going to have going through it, because I think you'll be surprised at how many logs you’re actually getting,” Agee recommends, adding that he didn’t expect the volume of logs he got from domain controllers, for example.

The new incident system positions Charlotte for compliance with a diverse set of requirements that come from covering such a broad range of city service areas. Law enforcement networks need to satisfy CJIS (Criminal Justice Information Systems) standards, for example, while PCI (Payment Card Industry) guidelines must be observed for departments like transportation that process payments using customer credit cards.

In Charlotte, IT security staff primarily uses data generated by LogRhythm to investigate the source of an infected machine using geo-location features. They can look at the Internet activity of a compromised computer, for example, and trace the traffic back to a specific website. Another major use is to monitor privileged account access.

“If someone that has domain admin rights puts somebody else in domain admin rights, we want to know about that,” Agee explains, adding that the city’s policy is to limit the number of privileged accounts according to its role-based security program.

In all, Agee estimates that the system generates about 1 million logs per day, most of which come from the city’s 14 firewalls. Between 10 and 15 incidents requiring user interaction are investigated on a daily basis. The system’s reporting capabilities help IT staff justify a proactive, rather than reactive, stance on security spending.

“We can show what type of events we’re getting hit with and use that for a justification of keeping our security at a level that we feel like it needs to be,” Agee added.

Government Technology editor Noelle Knell has more than 15 years of writing and editing experience, covering public projects, transportation, business and technology. A California native, she has worked in both state and local government, and is a graduate of the University of California, Davis, with majors in political science and American history. She can be reached via email and on Twitter.