Join members of the IBM Security QRadar Team on August 24th for the next QRadar Open Mic webcast where we talk about Log Source Extensions. During this session, we will be discussing how log source extensions work, how to write extensions, tips, and taking your questions. This session is not only an opportunity to learn about log source extensions, but also to provide feedback directly to developers, support, and product management about your experiences. During this session we will not be troubleshooting specific customer tickets or regex issues, but teaching and talking about extensions.

When to use log source extensions (Custom Event Properties vs using an Extension)

Looking at an example payload

LSX structure, limitations, & requirements

Identity

Creating QID Map Entries (qidmap_cli.sh)

Mapping events

Discussion & Questions

Note: This topic list may change depending on the volume of questions and how the discussion goes. Since this is a live presentation, we may divert from the slides to discuss topics from the online chat.

Why operational changes are made, without any customer (or, apparently, even support team notification? (The intentional, and significant, reduction of the number of supported capture groups, which broke numerous of our LSX's, with no visibility to why they suddenly would not work.)

Why does the upload engine not properly vet the LSX when it exceeds the newly reduced capture group limit?

Why does nothing else in the system generate alerts for any pre-existing LSXs, that exceed the newly capture group limit? And/or, why is that not vetted as part of each upgrade script, to look for this problem, and alert the customer to this issue?

How are you proposing that we test our RegEx? We previously utilized the Custom Properties dialog, which is presumably JRE, considering the Custom Properties expressions are supposed to be codified in JRE. I've now been told by QR support that the RegEx engine utilized for LSX was also recently replaced (maybe at the same time that the capture group limit was drastically reduced), and that is not (now) compatible with JRE.

Why are the docs not kept current? I had to open a ticket to find out whether it supported IPv6 variables, and what their names were. The only docs that I have, and can find, are years old.

Why is this component such a step-child? The sales people have always pitched this as the solution that makes QR so flexible, yet in almost all cases where I've ever called technical support, they quickly retreat to the "it's not supported" line, and start telling me that I'll have to contract for professional services. That might be understandable if the product was WELL documented, and FULLY documented, but as noted in a previous question, the docs don't appear to ever be updated, and also don't even explain all of the existing parameters. There are many variable names used in the LSX construct that we are simply left to GUESS about the underlying logic by which they actually function. The match-groups being a perfect example of this, where there are no examples of using "more than one", and when I tried to get clarification, the "professional services" response was all that I received. I actually got one of these working, but purely by trial and error.

Why operational changes are made, without any customer (or, apparently, even support team notification? (The intentional, and significant, reduction of the number of supported capture groups, which broke numerous of our LSX's, with no visibility to why they suddenly would not work.)

Why does the upload engine not properly vet the LSX when it exceeds the newly reduced capture group limit?

Why does nothing else in the system generate alerts for any pre-existing LSXs, that exceed the newly capture group limit? And/or, why is that not vetted as part of each upgrade script, to look for this problem, and alert the customer to this issue?

How are you proposing that we test our RegEx? We previously utilized the Custom Properties dialog, which is presumably JRE, considering the Custom Properties expressions are supposed to be codified in JRE. I've now been told by QR support that the RegEx engine utilized for LSX was also recently replaced (maybe at the same time that the capture group limit was drastically reduced), and that is not (now) compatible with JRE.

Why are the docs not kept current? I had to open a ticket to find out whether it supported IPv6 variables, and what their names were. The only docs that I have, and can find, are years old.

Why is this component such a step-child? The sales people have always pitched this as the solution that makes QR so flexible, yet in almost all cases where I've ever called technical support, they quickly retreat to the "it's not supported" line, and start telling me that I'll have to contract for professional services. That might be understandable if the product was WELL documented, and FULLY documented, but as noted in a previous question, the docs don't appear to ever be updated, and also don't even explain all of the existing parameters. There are many variable names used in the LSX construct that we are simply left to GUESS about the underlying logic by which they actually function. The match-groups being a perfect example of this, where there are no examples of using "more than one", and when I tried to get clarification, the "professional services" response was all that I received. I actually got one of these working, but purely by trial and error.

I cannot speak much to the capture group limit, however to address some of the questions.

For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.

The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.

I cannot speak much to the capture group limit, however to address some of the questions.

For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.

The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.

>>>I cannot speak much to the capture group limit, however to address some of the questions.<<<

Some change that was made in the last year or so, apparently to resolve some performance issue. That's understandable, but not documenting the change (not even the tech support guys knew about it), or even (to this day) generating an error for LSX uploads that exceed the limit, are the real sins.

>>>For Testing Regex's I've always used the Custom Properties Dialog, it uses a Javascript engine vs JRE so there are some minor differences that I've seen when attempting to use look-ahead and look-behind assertions. but it works in 99% of the cases.<<<

I guess the question here is are those minor differences well documented, in current LSX documentation? When an LSX doesn't work right, you have no debugging capability. Minor differences that result in parsing failures, result in failures that then result in painful trial-and-error exercises. Again, the failure to document the inconsistencies result in unwarranted pain and suffering by those of us attempting to develop the LSX on the end-user side. Coding complex variants of RegEx expressions can already be classified as a "black art", IMHO, and not having visibility to some syntactical discrepancy that will fail, blindly, is unforgivable. (I speak from experience. I've toyed around with different parts of a RegEx for hours, trying to figure out how to change it, before stumbling upon some innocuous modification that suddenly made it work, even though ALL of the variations worked fine in the Custom Properties dialog.)

>>>The regex engine in LSX has not changed that I know of, with 7.2.5 I have not seen any changes with the LSX's I work on. I will check with dev to confirm this and update this post if necessary.<<<

Sorry, I was incomplete in what I said. It has changed in some previous release, during the last year (apparently at the same time that the capture group "upper limit" was reduced). Again, that was communicated to me by support, from what he was told by development; all adjunct to the discovery that my LSX failures were the result of changes made by development. (And that discovery only occurred because, in my second attempt to get someone to help me, "one brave support tech" was willing to spin up an older version of QR, so that he could verify my claim that my broken LSXs had previously all worked fine. And the "bug" from that finding, resulted in the development response… "oh yeah, BTW, we significantly reduced the capture group limit.")

FWLIW, I actually find that online format to be far more difficult to peruse than is the "flat" PDF version, but also, related to my original concern, even this version makes no mention of the IPv6-related variables, nor does it delve into the more complex capabilities that would seem to exist with the whole custom DSM environment. And shortcoming like that, make we wonder "what other things" might be missing, beyond what I'm already aware of. What is frustrating for me is that, obviously, someone went to a lot of trouble to codify some (seemingly) elaborate functionality in this whole LSX construct, but then, because someone is either unwilling to properly document all of its varied features, or has deemed those to be only within the purview of professional services, all of that coding effort is mostly wasted, as 99% of us don't have any idea how some of those more elegant, enhanced features work (except thru guesswork, and trial-end-error testing).

You asked for questions. You seem to be doing this presentation to encourage a wider-scale use of the "magic of LSX", but if nobody at IBM/QR is willing to address how poorly this has been documented and supported, then my prediction is just a lot more "ill will", especially when those customers call, and the first words out of the support team's mouths are "we don't support LSX, you'll need to engage prof. svcs. if you want help with it.".

All, again, FWLIW. No need to reply. I do appreciate your attempt to offer some help.

Sorry for the delay. As mentioned in the open mic, I wanted to make sure I discussed this in detail. There are two potential options here for resolving this issue.

Option 1

I think that you might be able to use the Syslog Redirect protocol with your Universal DSM. In a nutshell, what the Syslog Redirect protocol does is take a value (such an IP or hostname) from the Syslog payload and injects this value in to the packet header. The packet header is the highest priority for QRadar when determining the event source. In this case, you are going to want to look at the event payload to see if the hostname or IP is represent there that you can use. You have a slightly different issue than we normally see because there is a non-standard value in your header. This solution should work just fine for you and option 2 is likely not required.

Below is a generic payload I created to represent what your event might look like. This is an anti-virus payload, so yours might look very different than my example. In this case, we would want to try to replace 14>Aug 17 14:33:17 025696: myhostname.example.com header with a new header that contains the value Local:10.10.10.10, which is the true IP of the device that sent the event.

Note: There might be multiple IPs or hostnames in an event payload. You are going to want to select the best option to represent the device that sent the event.

How to configure your log source

IMPORTANT: For the Log Source Identifier field, you will notice that I list "Source IP" generically when adding the log source. This is because with a non-standard header I'm not sure what QRadar thinks the event source IP or hostname really is. You should put the Packet IP address of the system sending the event in the Log Source Identifier field. Then assign your protocol TCP/UDP and upload/apply your LSX. Your LSX would then need to be adjusted to identify the proper values from the event payload since there is now going to be a new syslog header. Your appliance must also be configured to sent to port 517 TCP or 517 UDP.

Notice in bold where the event above now has a new syslog header created by QRadar. Syslog redirect injects a new packet header before processing the event. Your LSX would then need to be crafted to identify the proper values from the event payload.

These are the values you can put in your LSX as discussed during the open mic:

EventName

EventCategory

SourceIp

SourcePort

SourceIpPreNAT

SourceIpPostNAT

SourceMAC

SourcePortPreNAT

SourcePortPostNAT

DestinationIp

DestinationPort

DestinationIpPreNAT

DestinationIpPostNAT

DestinationPortPreNAT

DestinationPortPostNAT

DestinationMAC

DeviceTime

Protocol

UserName

HostName

GroupName

NetBIOSName

ExtraIdentityData

SourceIpv6

DestinationIpv6

Any other fields else that is important data from the payload that you are going to want to capture is going to be a custom event property in QRadar.

Option 2

The other option is to send these payloads to an intermediate server that is running syslog-ng, then use syslog redirect. You can configure an instance of Syslog-ng to listen and route these packets to QRadar. When you forward the Syslog event, you can configure the Syslog-ng server inject it's own header.

This would mean that this is what gets forwarded to QRadar: [Syslog-ng header 1][Syslog non-compliant header2][event payload]

However, the IP in the "redirected" header will have the Syslog-ng servers address (unless the config is specalized). So, then you apply syslog redirect to the event and get this, which is messy, but it also works:

Q1. Is there a way to map events to qids pre-emptively via the terminal?

A1: No. Unfortunately, there is no method of mapping events without using the QRadar user interface. You can create your custom QIDs and import them to QRadar before you create your LSX, however, there needs to be events in the QRadar user interface to map these events. This might be something that we could add as a possible API endpoint. You would need to open an RFE to make a feature request around being able to

Q2. Is there any available documentation for other similar back-end commands?

A2: The documentation covers all supported command-line parameters. There is an "unofficial log source extension guide" that shows how to use a script called logrun.pl. This script allows you to take payloads from events and replay them through QRadar. This can be helpful to generate events for LSX testing without having to wait for events to stream in from the event source. If you have the full event with payloads in a file, you can use logrun to replay events through QRadar. This will generate duplicate events, but might save time without having to wait or log in to the event source to trip a specific event you are working on parsing. Logrun is in /opt/qradar/bin and it has a help interface through typing --help. The unoffical guide is discussed in this post: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014970193 . This is the only instance I can think of a script that is not in the official documentation that might be helpful.