osquery For Security

Introduction to osquery — Part 1

Osquery is a tool that was developed at Facebook that allows you to query security, reliability, and compliance based information about the Linux and OSX based systems in your environment. When it comes to securing a Linux and/or OSX network environment, it’s hard to beat a tool that’s easy to install, open source, and completely free.

If you’re an IT or security engineer working with a fleet of Linux and/or OSX systems, osquery should be your first choice of software to install on those hosts. In my experience, it’s rare to find a monitoring or security tool that both engineers and security engineers can agree upon. Osquery fits the bill because it provides resources for both engineering and security teams.

Osquery isn’t like other vendor security tools. It doesn’t try to hide itself deep in the operating system or prevent itself from being uninstalled. It also doesn’t place any new restrictions on the users or system. In fact, it does the complete opposite by enabling users to easily gain more information about the system.

Osquery includes an interactive query console/shell (osqueryi) and a daemon (osqueryd) that allows osquery to run in the background, schedule queries, and aggregate logs.

If you’re fairly new to osquery or wondering what an enterprise deployment might look like, keep on reading. This post contains an overview of how to create an osquery config, centralize the log output, and start creating effective searches and alerts.

Building a Config

Osquery’s configuration file (often named osquery.conf) contains the configuration options and queries that osqueryd uses when it runs.

We’ll use the code sample below as our basic config. By naming it osquery.conf and placing it in /var/osquery, we ensure that it will be picked up by the default config_path settings.

As you can see in lines 3–7, we’ve only scheduled a single query under the “schedule” heading. That query is scheduled to run every 60 seconds. That’s quite often, but it will help us populate our log file. If you’d like to see what information that query will log when it runs, open up the interactive osqueryi application in a terminal:

Although we added one query to our configuration file, we also included 3 query packs (lines 9–13). Query packs are just JSON config files that contain additional queries. Think of it like importing software libraries. Query packs make it easy to categorize your queries and also allow you to keep a short and tidy config. If you want to see all of the queries that are scheduled to run from our config (including the packs), you can use: “SELECT name FROM osquery_schedule;”.

If you want to view or change the queries that will be running from the packs, you can view them in /var/osquery/packs. Queries listed inside the packs allow you to change the interval for how often each query will run and disable certain queries if they’re unwanted. For the sake of this post, we’re going to leave them all enabled.

Debugging Your Config

Osqueryd won’t run correctly if there are problems with your configuration file. Here’s a few ways to debug it:

Launch osqueryi in verbose mode and point it to your config using the config_path argument. If you see any initialization lines containing “Error reading config”, you’ve got a problem.

Configuring Inputs

We want to collect the file that contains the query results as an input, and that file is located at /var/log/osquery/osqueryd.results.log. Optionally, if you wanted to include any error related information, you could also include the osqueryd.WARNING and osqueryd.ERROR logs as well. For now, we’ll stick with results.

In /Applications/SplunkForwarder/etc/apps/SplunkUniversalForwarder/local, append the following lines to inputs.conf:

Server-side

Grokking osquery Logs

The #1 thing to understand about the way osquery writes logs is that it writes differential logs.

To understand this more clearly, let’s use the usb_devices table as an example. If we have a query that logs all usb_devices like so:

SELECT * from usb_devices;

you might expect that query to log all present usb devices each time it runs. That will not be the case. After osquery has recorded all of the initial usb devices from the first time the query executes, it will only log devices that have been added or removed from the table since the last execution.

However, if you encounter a situation where you want to see the full set of results each time a query is run, take a look at snapshot logs.

osquery + Splunk = ❤

Let’s look at some of the most efficient ways to dig through this data. Since the data is already in JSON, we don’t have to worry about field extractions.

Let’s start by examining all of the queries that we have results for, using the “name” field.

We can recreate a nice table view by using Splunk’s built in table command: