Detection & Response rules are designed to allow you to push out a detection rule and with a custom action in record time.
Think of them like AWS Lambda or Google Cloud Functions. They can be added and removed with single operations
and they become immediately available/running as they are set.

A D&R rule has two components:

the detection part of the rule is a simple expression that describes what the rule should match on.

the response part of the rule describes the list of actions that should be taken when the rule matches.

The REST interface expects those rules in their native JSON format (as described below), but UIs to generate this
format are available if you are uncomfortable with the JSON.

Detection and Response rules support a few namespaces. Initially you do not have to worry about using them since by default
operations on rules use the general namespace.

However, if you plan on having multiple groups of people accessing DR rules and want to maintain some segmentation, then
namespaces are for you. An example of this is an MSSP wanting to allow their customers to create their own rules without
giving them access to the MSSP-maintained sets of rules.

Beyond the general namespace, the main other namespace is called managed (as in MSSP-managed). Currently, operating
on namespaces other than general can only be accomplished using the REST API by providing the namespace
parameter in the relevant queries.

It is possible to set an experation time for D&R rules as well as False Positive rules. The expiration is set by providing
a expire_on paramater when creating/setting the D&R or FP rule. The value of the parameter should be a second-based
unix epoch timestamp, like expire_on: 1588876878.

Once that timestamp has been reached, the rule will be automatically deleted. Note that the exact precision of the expiration
can vary, the rule could effectively remain in operation for as long as 10 minutes past the expiration.

Targets are types of sources of events that the rule should apply to. The D&R rules apply by default to the edr target.
This means that if you omit the target element from the Detection component, the rule will assume you want it to apply
to events coming from the LimaCharlie agents. Other targets are available however.

In the case of the log target, path references apply to JSON parsed logs the same way as in edr DR rules, but rules on pure text logs requires using the
path /txt as the value of a log line. The log source matches the log's source string, and the log type matches the log's type string.

When running D&R rules against Windows Event Logs (target: log and log source: wel), although the External Logs Service may ingest
the same Windows Event Log file that contains some records that have already been processed by the rules, the LimaCharlie platform will keep track of the
processed EventRecordID and therefore will NOT run the same D&R rule over the same record multiple times.

This means you can safely set the Artifact Collection Service to collect various Windows Event Logs from your hosts and run D&R rules over them
without risking producing the same alert multiple times.

For most Windows Event Logs available, see c:\windows\system32\winevt\logs\.

Each logical operation in this component is a dictionary with an op field. Complex logical evaluation is done
by using two special op values, and and or. These will take a rules parameter that is a list of other logical
operations to perform.

Here is a basic example of a rule that says:
When we receive a STARTING_UP event from a linux sensor, and this sensor has the tag test_tag, match.

The "event": "SOME_EVENT_NAME" pattern can be used in all logical nodes to filter the type of event being
evaluated. You can also use an "events": [ "EVENT_ONE", "EVENT_THREE"] to filter in certain types of events. When a detection is generated (through the report action), it gets fed back into D&R rules with an event_type
of _DETECTIONNAME. This can be used to compose higher order detections.

A recurring parameter also found in many operations is the "path": <> parameter. It represents a path within the event
being evaluated that we want the value of. Its structure is very close to a directory structure. It also supports the
* wildcard to represent 0 or more directories, and the ? wildcard which represents exactly one directory.

The root of the path should be event or routing depending on whether you want to get a value from the event itself or
the routing instead.

Note that many comparison values support a special "lookback" format. That is, an operation that supports comparing
a value to a literal like "system32", can also support a value of "<<event/PARENT/FILE_PATH>>". When that value is
surrounded by "<< >>", it will be interpreted as a path within the event and the
value at that path will replace the "<<...>>" value. This allows you to "look back" at the event and use values
within for your rule.

It is possible to store some pieces of state on a per-sensor basis for the lifetime of a sensor (single boot).
This state is called "variables". Variables have a name and a set of values associated with them. Values can
be associated with a variable name at run-time using the add var (and del var) response.

The add var action supports an optional ttl value which is the number of seconds the value should be
set for, like ttl: 3600.

Other rules may then use the values in a variable as part of the detection component.

Like the "lookback" feature, variables can be used in the is, contains, starts with and ends with operators
by surrounding the variable name with [[ and ]], like [[my-var]].

For example, these rules looking for unsigned execution from external drives (like USB).

Check to see if a value is greater or lower (numerically) than a value in the event.

They both use the path and value parameters.
They also both support the length of parameter as a boolean (true or false). If set to true, instead of comparing
the value at the specified path, it compares the length of the value at that path.

The matches op compares the value at path with a regular expression supplied in the re parameter.
Under the hood, this uses the Python 2.7 re module with findall, which means the regular expression
is applied to every line of the field (if the field is multi-line), which enables you to apply the regexp
to log files.

The string distance op looks up the Levenshtein Distance between
two strings. In other words it generates the minimum number of character changes required for one string
to become equal to another.

For example, the Levenshtein Distance between google.com and googlr.com (r instead of e) is 1.

This can be used to find variations of file names or domain names that could be used for phishing, for example.

Suppose your company is onephoton.com, looking for the Levenshtein Distance between all DOMAIN_NAME in
DNS_REQUEST events, compared to onephoton.com could detect an attacker using onephot0n.com in a phishing
email domain.

The operator takes a path parameter indicating which field to compare, a max parameter indicating the
maximum Levenshtein Distance to match and a value parameter that is either a string or a list of strings
that represent the value(s) to compare to.

Looks up a value against a LimaCharlie Resource such as a threat feed. The value is supplied via the path parameter and
the resource path is defined in the resource parameter. Resources are of the form lcr://<resource_type>/<resource_name>.
In order to access a resource you must have subscribed to it via app.limacharlie.io.

Generally the D&R rules operate in a stateless fashion, meaning a rule operates on one event at a time and either matches or doesn't.

To be able to perform D&R rules across events to detect more complex behaviors, you can use the with child and with descendant
parameters. Note that those parameters can ONLY be specified on a rule operator specifying the event: NEW_PROCESS since they only
apply to the relationship between an event and a process.

Both the with child and with descendant parameters are effectively the same except for the "depth" of the relationship they cover.
The child specifies that the target state (described below) must apply to the direct children of the matching process. The descendant
specifies that the target state must apply globally to any descendands (children of children) of the matching process.

The value of a with child (or descendant) is simply another (stateless) rule operator. The logic defined in this rule operator describes
the set of conditions that must match, not a single event, but the collection of events that are children (or descendants) of the matching
process.

Here is an example of a stateful detection looking for a "cmd.exe" process that has a child "calc.exe":

The above example is looking for an outlook.exe process that spawns a chrome.exe and drops a .ps1 (powershell) file to disk:

outlook.exe --+--> chrome.exe
|
+--> New .ps1 file

On top of containing stateless rules, an operator underneath a with child (or descendant) can also contain another operator with another with child (or descendant).

Another parameter comes into play if you want to define a set of operators underneath a with child to be "stateless", meaning where you want all the operators to apply
and match with single events (like the classic stateless D&R rules). This parameter is is stateless: true. Simply add it to the operators at the root of the logic you
want to be applied statelessly.

Finally, a node in stateful mode under a with child or with descendand can specify a count: N parameter. When specified, you specify that the given node must be
matched N times before it is considered successful. So setting count: 3 in a node looking for a event/FILE_PATH ending in cmd.exe will mean that we want to match
only if we see 3 instances of a cmd.exe in that context to match. An example usage of this is to set count: in a matches operator looking for a set of processes
which would result in detecting a "burst" of matching processes from a parent (like: if a process starts more than 3 cmd.exe, alert). Adding a within: Z parameter
to the count: N limits the count to where the first and last event in the count is within a Z seconds time window.

The lookup can also use certain APIs in their lookup, such as VirusTotal. Note that for the VT API to be accessible, the
organization needs to be subscribed to the VT API Add-On, and a valid VT API Key needs to be set in the integrations
configurations.

As shown in the example below, a metadata_rules parameter is also valid for the lookup operation. It can contain
further detection rules to be applied to the metadata returned by a lookup match. In the case of VT this is a dictionary
of AntiVirus vendor reports (here we test for more than 1 vendor reporting that the hash is bad), while in the case of a custom
lookup resource it would be whatever is set as the item's metadata.

To activate VirusTotal usage, you must subscibe to the VirusTotal API in the Add-On section. Then you must set your VirusTotal
API key in the Integrations section of the limacharlie.io web interface.

VirusTotal results are cached locally for a limited period of time which reduces the usage of your API key and saves you money.

Also note that if your API Key runs out of quota with VirusTotal, hashes seen until you have quota again will be ignored.

The lookup can also use certain APIs in their lookup, like IP GeoLocation. Note that for the IP GeoLocation to be accessible, the
organization needs to be subscribed to the api/ip-geo API Add-On.

As shown in the example below, a metadata_rules parameter is also valid for the lookup operation. It can contain
further detection rules to be applied to the metadata returned by a lookup match. In the case of ip-geo this is a dictionary
of the geolocation data returned by the IP GeoLocation data provider, MaxMind.com.

The format of the metadata returned is documented here and looks like this:

Use an external detection rule loaded from a LimaCharlie Resource. The resource is specified via the resource parameter.
Resources are of the form lcr://<resource_type>/<resource_name>. The external operation only supports Resources of
type detection. The external detection replaces the current detection rule, which means it can be combined with other
detection logic using the and and or operations.

Only accessible for the target: log. Scans the relevant original log file in the cloud using the Yara signature specified.

The Yara signatures are specified as a LimaCharlie Resource of the form lcr://<resource_type>/<resource_name>. Currently
the main source of Yara signatures are the Yara Sources specified in the Yara Service. If your Yara Source is
named my-yara-source, the LC Resource would be: lcr://service/yara/my-yara-source.

The yara operator scan the log file at most once. This means it can be used both as a simple "scan" detection like this:

op: yara
target: log
resource: lcr://service/yara/my-yara-source

Or it can also be used as part of a more complex D&R rule evaluation like this:

The sub domain extracts specific components from a domain name. The value of sub domain is in basic slice notation.
This notation is of the form startIndex:endIndex where the index is 0-based and indicates which parts of the domain to keep.
Examples:

0:2 means the first 2 components of the domain: aa.bb for aa.bb.cc.dd.

This action sends the task (as described here) in the command parameter to the sensor that the event
under evaluation is from. An optional investigation parameter can be given, it associates the given
identifier with the task and events from the sensor that relate to the task.

Reports the match as a detection. This means that the content of this event will be bubbled up to the Detection Output
stream. Think of it as an alert. It takes a name parameter that will be used as a detection category and an optional publish
parameter that defaults to true. If set to false, the report won't be published to the Output stream.

This last distinction about the publish parameter is important because the detections created by the report action
get fed back into the D&R rules so that more complex rules may handle more complex evaluations of those.
Setting publish to false means that this detection is only really used as an intermediary and should not be reported in and
of itself. When fed back, the event_type is set to _DETECTIONNAME.

A "non-published" rule stays within the D&R system only (as stated above), but sometimes we also want Services
to get a notification of the detection without having the detection recorded to Outputs
or retention. For example, a Service may want to listen for the CONNECTED event to do
something and it makes no sense to record this detection after Services have been notified. To
accomplish this, you can simply prefix your detection name with __ (double underscore).

The priority parameter is optional. If set, it should be an integer. This integer
will get added to the root of the detection report.

The metadata parameter is optional. It can be set to any data which will be included
in the detection generated in a field named detect_mtd at the root of the detection.
This can be used to include information for internal use like reference numbers or URLs.

These two actions associate and disassociate, respectively, the tag found in the tag parameter with the sensor. The "add tag" operation
can also optionally take a "ttl" parameter that is a number of seconds the tag should remain applied to the agent.

Perform an asynchronous request to a service the organization is subscribed to. A service
request contains two main component: the name of the service, like dumper, and the arbitrary
content of the request to this service in the request value. The request content
will vary depending on the service (see the relevant service's documentation).

Sometimes users install a sensor on a VM image by mistake. This means every time a new instance
of the image gets started the same sensor ID (SID) is used for multiple boxes with different names.
When detected, LimaCharlie produces a sensor_clone event.

We can use these events to deduplicate. This example targets Windows clones.

The typical use case for FP rules is to add exceptions from some detections that are cross-cutting (for
example ignore all detections from a specific host), organization-specific exceptions (like ignoring
alerts relating to a custom piece of software used in an organization), or suppressing errors from
managed D&R rules you don't have direct access to.

An FP rule is structured with the same format at the detection component of a D&R rule. The main
difference is that the rule applies to the content of a detection, as can be seen in the Detections
section of the web app.

Most stateless operators from the D&R rules are available in FP rules.