Planning a journey

We're planning a journey, and like any real journey we need to know who's coming
with us, have an idea of where we want to get to, and what some of the choices along
our way will be. We need to make some plans before we start. This journey is not
a journey to a place, but the journey we took to learn how to build Splunk®
apps in the real world.

This chapter introduces the companies that will collaborate on this journey,
the people from those companies who will come along with us, and most importantly
the Splunk apps that we intend to build as the goal of our journey. This chapter
outlines some of the key features and functionality that we plan to include in the
finished apps and introduces some of the issues we are considering right at the
start of our journey. The chapters that follow enable you to follow along as we
continue on this journey to build our Splunk apps.

A journey

This guide describes the journey undertaken by the project team and is intended
to give you an insight into how to set about developing Splunk apps, to recommend
some good and proven practices, and to provide a useful reference when you come
to build your own Splunk apps. However, like on any real journey, we make mistakes,
have arguments, and change our minds along the way; so in addition to showing you
how best to do things, we highlight the pitfalls and issues that we encounter, and
the solutions we find. We do not intend this guide to cover everything related to
developing Splunk apps, it focuses on the specific issues we faced in building our
two sample apps. However, our two sample apps are representative of the typical
apps Splunk customers do build.

The primary objective of our journey is to deliver applied engineering guidance
along with Splunk apps that act as reference implementations. We hope this guidance
will help to lower the entry barrier, reduce the time to market, and decrease the
risk for you, our Splunk customers and partners. The following five principles characterize our guidance philosophy:

Proven: based on real-world experience, and validated by field
engineers and partners.

Authoritative: offers the best available advice, in context.

Accurate: technically validated and properly tested.

Actionable: provides the steps to success.

Relevant: addresses a real world problem.

Each chapter of this guide covers a specific theme such as UI design or data
onboarding and each follows the basic chronology of the journey as we develop our
apps. The chapters also include commentary from various experts, descriptions of
some of the team's working practices where they shed light on what happened, and
discussions about why we made particular decisions. We hope this guide will mature
through a dialog with you, our audience of Splunk app developers.

Our organizations

Three different companies collaborate on this journey: Splunk, Conducive, and
Auth0. Conducive already uses Splunk Enterprise and has previously developed Splunk
apps; Auth0 is new to Splunk Enterprise. Both Conducive and Auth0 have ideas about
how they can use Splunk apps to add value to their own products and services and
want to partner with Splunk to pursue those plans further. This means that the journey
includes two separate development projects for two separate Splunk apps: one for
Auth0 to enable insights into how users are interacting with the Auth0 service,
and one for Conducive to build a Splunk app to assist in monitoring secure document
repositories (for more information, see
Our apps below). We built both of these Splunk apps using Splunk simple XML. The next chapter,
Platform and tools: a kitbag for our Journey, discusses why we chose this specific technology in more detail.

Conducive

Conducive
is a system integrator with one of the focus areas being standards compliance.
Their user audience includes business managers, compliance officers, auditors and
investigators.

Some of the specific goals that the Conducive team has for their involvement
in the project are to learn how to:

Implement auto complete drop downs that can lazy load a large number of
options.

Perform form validation prior to running a search.

Use pivots that enable a user to use drag and drop to manipulate the columns
displayed on a dashboard.

Detect deviations from the norm across multiple variables.

Incorporate predictive and statistical analytics.

Handle outliers.

Design an app and dashboards that enable non-technical users to create reports
and configure alerts.

Auth0

Auth0 is
an Identity-as-a-Service infrastructure provider. Their user audience is primarily
developers.

Some of the specific goals Auth0 has for their involvement in this
journey are to enhance their monitoring and reporting functionality with a Splunk
app.

Their learning objectives include:

Understanding a structure of a Splunk app.

Using the Splunk'search processing language (SPL™).

Building and testing modular inputs.

Performing data validation when configuring a modular input.

Utilizing Simple XML to build a basic set of panels and dashboards and to
refine them iteratively.

Packaging and publishing a Splunk app.

Our panel of experts

A panel of experts from a number of roles follow the journey, and the following
chapters include their comments on the development effort. The roles these experts perform are:

A developer, expert in using Splunk and developing
Splunk apps. He wants to see a well written Splunk app that follows
proven practices.

An architect who wants to see a well-architected
solution.

A UX designer who wants to see an engaging,
intuitive, easy-to-use UI for the app.

A tester who wants to ensure the highest possible
quality for the reference implementations.

A system administrator who wants to know how
to deploy and monitor the apps, and understand how to configure them.

A representative of the business who knows
how the apps will be used and their expected benefit to the business.

A release manager who wants to ensure that
the apps are packaged in a way that makes them easy to deploy and to
update in the future.

A security expert who wants to ensure that
the apps comply with any security requirements in the target environment.

A performance tester who wants to ensure that
the apps performance is acceptable with production loads.

Our apps

During the journey, the team develops two apps, one with Conducive and one with
Auth0. This book focuses primarily on the Conducive app but discusses the Auth0
app where it provides a contrast or highlights different issues. The following two
sections in this chapter provide overviews of the apps as we envision them at
the start of the project.

Pluggable Auditing
System (PAS)

The PAS Splunk reference app is intended to enable an organization to monitor
a highly secure document repository such as one hosted on Box.com. Conducive wants
the organization to use the app to see who has looked at, updated, modified, deleted,
or downloaded documents in the repository. The app should enable the organization
to be both reactive and proactive in its approach so that in addition to being able
to see what happened, the app notifies the organization of potential security issues
when it detects unusual behavior associated with either a user or a document.

Potential users of the app include anyone with responsibility for protecting
sensitive data. We expect the end users of the app to be non-technical business
users such as:

An investigator assigned to protect data or to discover the identity of
someone who is stealing data. An investigator is typically from the compliance
or auditing side of the business.

A manager who needs to know what his or her employees are doing. This is
not necessarily to monitor productivity but to identify unusual behavior.

These end user roles are typically non-technical, and they should be able to use the app without technical help: an intuitive UI is important.

In addition to the end users, there is a technical role for configuring the app
with rules for triggering alerts specific to the organization and its requirements.
Some organizations may configure these rules once when they deploy the app, others
may have the requirement to be able to update these rules in response to changes
in the business environment or to emerging threats. This may require some basic
technical knowledge of Splunk Enterprise. The app should include a set of basic
default rules to get started.

The app should include the following five key elements:

An Overview Dashboard to provide summary details of most reports
and to highlight areas that require further investigation.

A Reactive Analysis capability to determine the activities of one
or more users of the document repository and the geographic location where they
performed those activities. The app should also be able to correlate those activities
across different users, documents, or data sets.

A Rules-based Proactive Analysis capability to apply a set of predetermined
and custom rules to the audit logs and reference data to search for potentially
malicious activity. The organization can add custom rules as required to
search for evolving and newly identified undesirable behaviors, or to filter
out changes in behaviors that the organization no longer considers to be undesirable.

A Statistical Proactive Analysis capability to apply statistical
analysis to the audit and reference data to establish various behavioral norms.
This capability should include the functionality to apply weighting factors
to the different data elements in order to differentiate false positives and
noise from normal behavior.

A Workflow Solution with capabilities such as triggering incidents
for review, categorizing them by type, assigning tasks to the relevant personnel,
capturing details relevant to the investigation in an unalterable manner, and
escalating incidents.

These were our goals at the start of the journey. The finished apps do not incorporate everything we envisaged, primarily because we ran out
of time and resources needed to implement everything in our backlog.

Auth0

At the start of the project, Auth0 is using Splunk Enterprise to monitor
some basic events related to users of the Auth0 services. For example, whenever
a new user signs up with Auth0, the Auth0 service sends event data to a Splunk REST
endpoint. This event data includes information such as the event type, the target
app, the client IP address, and the userid. Auth0 then display this information
in a simple dashboard.

Auth0 would like to build a Splunk app that will replace the current solution
and have the following features:

Identifying and tracking 'slipping away' users. Auth0 assign all users a
status (new, active, very active, slipping away, all but deleted) and they send
out a boilerplate email to slipping away customers asking why they are leaving.
Auth0 would like to use rules in Splunk Enterprise to identify slipping away
users.

Sending emails to all users of a specific connection type. For example,
if Facebook make a change to one of their APIs, Auth0 wants to send a notification
to all users of the Auth0 Facebook connector.

Auth0 are also considering making a Splunk app available to their customers to
enable them to track business metrics such as:

If the customer has a portfolio of applications, who is accessing which
ones?

Which users are encountering problems?

Suspicious log-ins and other anomalies.

You check the details of your Splunk Enterprise license if you are planning to provide your customers with reports and data from your Splunk
Enterprise installation.

The app should display this information in way that's attractive to customers.

Beginning our journey

The following chapters describe what happened on our journey, with each chapter
focusing on a high level theme. Each chapter covers the issues the team encountered
in tackling the various stories and use cases, how they resolved those issues, the
decisions they made, and the lessons they learned. You'll also learn more about
the specifics of the Splunk apps we have introduced in this chapter. Some chapters
will also include sections that describe how the team works or arrives at particular
decisions.

The following flowchart illustrates the typical workflow for developing
a Splunk app. Not all projects will follow these steps exactly, but they provide
a rough outline of the key stages for most Splunk app projects:

To start, look at a sample of the real business data and begin to understand
what information is available.

During the development process use simulated data so you are not reliant
on a third-party system while you are building the app. Typically, this helps
you develop and test faster. The chapter "Platform
and tools: a kitbag for our journey" discusses this.

Create a skeleton app using the Splunk Enterprise web UI or the command-line
tools. You can then add dashboards and visualizations to this app. This is discussed
in the chapter "Platform
and tools: a kitbag for our journey."

Our team has a variety of technical skills relevant to the project that include:

JavaScript, XML and knowledge of Model-View-Controller web development.
These are all relevant to building our apps.

C# and Python. These are relevant to building our tests and connectors/modular
inputs.

Some of the team also have experience of building Splunk apps using technologies
such as Splunk SPL and the simple XML development model for apps.

Production notes

Note that code samples and screenshots shown in the following chapters do not
necessarily match the code and UI in the completed apps. The samples show the code
and UI at that particular point in the journey, and in some cases they may be modified
at a later stage in our journey. In particular, some screenshots and code sample
refer to the warum app; this was our original internal code name
for the app that was changed near the end of the project to pas.

The majority of the screenshots in this guide were taken on a Windows machine,
in some cases where there is a significant difference we show a *nix screenshot
as well.

Questions?

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »