Packaging and deployment: reaching our destination

During this stage of our journey, we explore the options for packaging and deploying
the PAS apps. We investigate how we can package the multiple apps that make up our
solution in a way that makes it easy for a customer to deploy them in their own
environment, and how to facilitate updates to the apps after the customer has deployed
them.

Packaging and deploying the PAS app

The following screenshots show the result of the deployment process for a user
in Splunk Enterprise. The first screenshot shows the home page in Splunk Enterprise
with the main Splunk Reference App - PAS in the list of installed
apps. The second screenshot shows the details of all the installed apps, which includes
the main Splunk Reference App and the various add-ons (Splunk
Reference App - HR info, Splunk Reference App - Application Add-on,
Splunk Reference App - Database Add-on, Splunk Reference
App - Files Add-on, and Google Drive Data Provider Add-on).
Notice how the add-ons are invisible and that not all the PAS apps share the same
version number.

The remainder of this chapter describes how we packaged the apps to achieve
this result.

Alternatives

The most complex part of packaging the apps was deciding how to handle the multiple
Splunk apps that make up the PAS solution. Typically, a Splunk app is packaged into
an .spl file that a customer can either download and install manually or
discover on the Splunkbase
website. From Splunkbase, Splunk Enterprise downloads and installs it automatically.
Unfortunately, a single .spl file only enables installing a single
Splunk app, and the PAS solution is made up of multiple Splunk apps.

An .spl file is a zipped tar archive that contains
all the required files for the app.

We considered the following options:

Creating an installer inside a Splunk app
Taking this approach, we could package all the dependencies inside the main
app and include logic that installs them as and when they are required. The
Splunk App for VMware plans to use this approach in the future.
A variation on this approach is to have a dedicated installer app that sets
up the main app and all of its dependencies, and then redirects requests to
the main app it installed. The
Splunk App for Enterprise Security uses this approach.

Using an external installer
Using this approach, we could create a script that automatically installs an
app and its dependencies. Alternatively, we could use a third-party installer
tool to build the installation scripts.

Ask the customer to install the app and its dependencies manually
from a single ZIP file
Taking this approach, we could package the main app and all the add-on apps
in a single ZIP archive. We would then ask the customer to unpack the archive
into the etc/apps/ folder. The
Splunk App for VMware currently uses this approach.

Ask the customer to install the main app and each dependent add-on
individually
Taking this approach, we would add detailed installation instructions to the
documentation explaining how the customer should obtain and install each of
the apps individually. Optionally, we could add logic to the main app that detects
missing dependencies and then prompts the customer to install them. The
Splunk App for Microsoft Exchange uses this approach.

The following chart plots each of these options showing how ease of implementation
maps onto ease of use for the customer.

While it may be possible to check programmatically for missing dependencies in
the same Splunk Enterprise instance, it is more difficult to check for missing dependencies
when you expect that dependency to be installed on another machine such as a Splunk
Enterprise forwarder. In this scenario, you can programmatically run some searches
in your main app that look for data that the dependency provides. If you don't find
the data, the main app can then prompt the customer to install and configure the
missing dependency. The
Splunk App for Microsoft Exchange uses this approach.

Overview
of the solution

The approach we decided on for the PAS app is a hybrid approach. We package the
main app as a standard Splunk app in an .spl file. In the splunk_pas_ref_app.spl file, we also include the other add-on apps in a subfolder along with a set of scripts that the customer runs to install them. The advantages of this approach for us are:

It is relatively simple for us to build the .spl file and the installation
scripts.

We can deploy the .spl file to
Splunkbase
where customers can find it and then install it in their own environments.

After the customer has installed the main app, it is relatively simple to
run the install script to configure the add-on apps.

The disadvantages of this approach are:

Installation is a two-step process. First, install the main PAS app in the
standard way from Splunkbase. Then run the appropriate script to install the
add-on apps in the local Splunk Enterprise environment.

The scripts assume that you are deploying all the apps on the same Splunk
Enterprise instance. The user must edit the scripts or install the add-on apps
manually in more complex environments that have multiple Splunk instances (such
as forwarders) on multiple machines.

Details of the packaging solution

This section describes the important details of how we package and deploy the
PAS app. The high-level steps are:

Make sure that you organize all the correct files into the correct folder
structure.

Create an .spl file.

Upload the .spl file to Splunkbase.

After you have uploaded the .spl file to Splunkbase, customers can discover
your app and install it in their own environments. The following screenshot shows
the PAS app ready for a customer to install from the
Splunkbase website:

Including the add-on apps and scripts

The main PAS app has the standard folder structure of a Splunk app. For example,
it contains these folders: bin, default,
metadata, and static. Therefore, the customer
can deploy the main PAS app in their environment in the same way as any other Splunk
app. We also include all of our add-on apps in the subfolder appserver/addons/
as shown in the following partial directory listing.

Splunk Enterprise does not locate and configure these add-on
apps in this location. Therefore, in the bin folder in the main PAS app we include
two scripts that create symbolic links in the etc/apps/ folder that point to these
subfolders containing the add-on apps. The following install-addons.sh script creates
the symbolic links in OS X, Linux, or *nix operating systems.

If testing an app on different versions of Splunk Enterprise
on the same box, install each version in a dedicated folder (perhaps, adding
a suffix for the version, for example, splunk_630). Then, it's easy to switch
between different versions by resetting SPLUNK_HOME.

The following install-addons.ps1 PowerShell script performs the same task in
Windows:

We signed this PowerShell script to make it easier for the customer
to run the script securely without weakening the execution policies.

The following screenshot shows the details of the symbolic links the script creates from the Terminal window in OS X:

Both scripts fail gracefully if you attempt to run them a second time. They will
not override existing apps or symbolic links.

Promoting add-ons to their own packages

The JIRA alert and Google Drive add-ons that we have built in support of the PAS app v1.5 have utility outside of this project. We would like to highlight them and share them with the community, while still using them in their original context within the PAS app. To this end, we have split them out into their own repositories and have linked them into the main PAS repository by use of Git submodules. In this way we can still include the apps in the main PAS package for easy deployment with our script, but we don't need to duplicate any code. We can work on that functionality with a greater focus and remain confident that we are always packaging the latest version of that code.

Since our script only creates links if they don't already exist, our users can also install either of the new standalone modules before installing the PAS app with no ill effects.

We also publish these two add-ons as standalone packages on Splunkbase.

Versions and other metadata

To enable updating the apps at some future time, it's important to add version
numbers to both the main PAS app and the add-ons. We do this in the app.conf files for each app. The following example is from the main PAS app:

The package stanza contains metadata related to app upgrades,
and the launcher stanza determines how the Launcher page in Splunk
Enterprise and the Splunkbase website display information about the app. For more
information about the contents of the app.conf file, see "app.conf"
in the "Admin Manual."

Tips and useful resources

The page "Step
8: Package your app or add-on" in the "Developing Views and Apps for
Splunkweb Manual" contains lots of useful resources related to packaging an
app including:

What should be in the app.conf file.

How to add an icon and screenshots on Splunkbase or in the Launcher. These
files must have particular resolutions and you must place them in a particular
location.

How to tar and zip your app on different platforms. Note that you must manually install
the gnutar tool referenced on this page on OS X.

There are also Splunk CLI commands that can be used for packaging. The latter will ignore all auxiliary git-related files:

splunk package app appname

git archive

When you are creating the .spl file, it's a good idea to check
out a clean copy of your app from your source control system and use that. This
removes the risk of accidently including any temporary file or files created
by Splunk Enterprise at runtime. However, you should be sure not to include
files related to your source control system such as .git and .gitignore in the
.spl package file.

When you tar and zip your app, you must add the folder that contains all your
app resources as the root folder in the package.

For the PAS app, we have chosen to include in the .spl file the indexes.conf file that specifies the location of the pas index files. This is a convenience to let a customer downloading and installing the PAS app on a single machine get started quickly with minimal manual configuration steps. In a distributed production environment, an administrator typically needs to configure the indexes on specific machines.

When you upload your .spl file to Splunkbase, it validates the
app folder structure and app.conf file. However, the Splunkbase team must approve
the app before it becomes visible on Splunkbase. This usually takes about 24 hours
to complete.

During the submission process you must provide various pieces of metadata. These
include the app name, a short description, license details, and contact details.
For more information about the submission process including naming guidelines and
approval criteria, see "About
submitting content" in the "Working with Splunkbase Manual."

Since the last major release of Building Splunk Solutions (which coincided with Splunk Enterprise 6.3.0), Splunk has released a major step forward in Splunk app packaging and deployment: a beta version of the Splunk Packaging Toolkit. While the Packaging Toolkit is currently only available as a CLI tool, you should be aware of its potential to significantly improve the Splunk app packaging and deployment experience.

The Packaging Toolkit is a tool for authoring, packaging, and validating a Splunk app in a way that simplifies app management, including installation, configuration, and updating. With minimal additional development work, the Packaging Toolkit disambiguates configuration and code, and partitions, or repackages, relevant pieces of an app into deployment packages. Each deployment package can be based on either physical workloads (forwarders, search heads, indexers, etc.) or logical groups of inputs.

Note: The Splunk Packaging Toolkit is still under development. It is currently available with API and command-line interface (CLI) support only, but development of a user interface is planned.

New concepts

The Packaging Toolkits introduces a few new concepts to the Splunk app creation process:

app.manifest: The Packaging Toolkit relies on the app.manifest file, a JSON file, to keep track of what components go where and what dependencies are required. Even when an app manifest is not provided, the Packaging Toolkit can derive one from an app's .conf files and other Splunk app artifacts. You don't have to change your current apps to take advantage of partitioning across search heads, indexers, and forwarders. When you run the Packaging Toolkit's generate-manifest command, it auto-generates a list of physical partitioning details and it stubs out several additional sections. To enable logical partitioning capabilities and dependency resolution, you must augment the app manifest with additional details. For more information about the app.manifest file, see The app manifest.

Partitioning: Partitioning is the act of taking a Splunk app's source package, examining it, and splitting it into a minimum of three deployment packages. Each deployment package contains only the information, configuration, and corresponding code that is relevant to the type of deployment node for which it is intended. The Packaging Toolkit can automatically partition existing Splunk apps into deployment packages for search heads, indexers, and forwarders. This is partitioning across physical boundaries. Partitioning among logical groups must be done by developers at design time. Later, at deployment time, admins can map logical groups to collections of server classes or nodes. For more information about partitioning in the Packaging Toolkit, see Partitioning.

Input groups: An input group specifies a named group of data inputs as defined in the inputs.conf file. Each group typically targets a specific forwarding environment. Input groups can be selectively deployed to specific server classes. For example, you could define an input group that targets Active Directory domain servers on specific versions of Windows. You define input groups in the app.manifest file, in schema members that begin with inputGroups. For each input group, you can define required dependencies, included data inputs, defined stanzas in inputs.conf, and other descriptive information. For more information about input groups, see Input groups.

Dependency packaging: The Packaging Toolkit, when it packages an app for distribution, gathers all dependencies listed in the app.manifest and packages them into a .dependencies directory in the top level of the package. When the app is deployed as normal in Splunk Enterprise, that directory will be ignored. When installed from Splunkbase in the Splunk Cloud self-service installer, all needed dependencies will be installed with the app.

Typical workflow

As a Splunk app developer, you'll typically use the Packaging Toolkit by doing the following:

To the app.conf file, add a [package] stanza with an app ID (id) that matches your folder name, and a [launcher] stanza with author and version values.

Generate an app manifest from an existing app package using the generate-manifest command and a syntax such as the following:

slim generate-manifest -o <app_folder>/app.manifest <app_folder>

The Packaging Toolkit parses the package's .conf files to validate the app configuration. The default generated manifest has basic information already filled out, such as the app name, author name, description, and so on.

PAS-specific uses

We decided to revise the PAS app to take advantage of the new capabilities of the Packaging Toolkit. Here are some PAS-specific changes we made:

Dependencies

The PAS app has several apps it depends on, particularly to define the data generation for the demo app. In the app.manifest file for the main pas_ref_app, we defined each of those dependencies.

Folder naming

For the Packaging Toolkit to find the PAS project, the app directory needs to match the name given in the app.manifest file, which in turn needs to match the id value in the [package] stanza in the app.conf file. Note that if you supply a info.id.group value in the app.manifest file, you must name your app directory with the group name as a prefix, separated by a hyphen. For example, if your app name is myapp and your group is mygroup, your app folder will need to be named mygroup-myapp.

Dependency repository

The Packaging Toolkit uses a repository to resolve local dependencies. By default, this repository is placed under the app's root directory, but in many cases, it may make sense to manage your repository explicitly. In the makefile provided in the PAS repository, we use an out/repository directory. We then build each of our apps so that all the dependencies are built before the main PAS app.

We decided to make a simple but significant addition to the PAS app for version 1.5.2: the PAS app now contains a makefile. In fact, it contains two makefiles: makefile (for *nix clients) and make.ps1 (for PowerShell clients). Both files are at the main PAS app's root directory, and both rely on the Splunk Packaging Toolkit for full functionality.

The reasoning for adding a makefile is twofold:

To avoid the error-prone necessity of manually creating a directory structure and placing app files and dependencies in the appropriate locations by hand.

To add the ability to package the app so that it is optimized for a local standalone Splunk Enterprise instance (make standalone_package), for a distributed or clustered deployment (make partition), for a one-click Splunk Cloud install (make package_oneclick), or for all three (make package_all).

For a full list of make targets, run make help from the PAS app's root directory.

Note: Before trying any of the package building or validation target options, you must install the Splunk Packaging Toolkit. For installation instructions and more information, see Overview of the Splunk Packaging Toolkit.

make target options

To see what the makefile does, open it in your text editor of choice. Following are extended descriptions of the makefile target options. Run these make commands from the root directory of the PAS app for them to work properly.

make help

Shows a help message, which displays a list of target options. Simply running make will also yield this list.

make clean

"Cleans" the app by force-removing the working directory for optional dependencies and removing the /out directory.>/p>

make validate

Validates the app by first building it (make package_oneclick), and then running the Packaging Toolkit's validate command on it. The validate command verifies the contents of the manifest of a Splunk app and the app's dependencies. For more information, see slim validate in the Splunk Packaging Toolkit documentation.

make package_oneclick

Packages the app for a one-click Splunk Cloud install. The package_oneclick command runs the Packaging Toolkit's package command on the app, and prepares the app for installation on Splunk Cloud. The resulting .tar.gz package is stored in /out/release/oneclick. For more information, see slim package in the Splunk Packaging Toolkit documentation.

make partition

First builds the app, and then partitions the app source package into a set of targeted deployment packages based on deployment configurations that are defined using the Splunk Packaging Toolkit. The configurations are based on three main Splunk clustered deployment workloads: indexer, search head, and forwarder. The resulting packages are stored in /out/release/partitioned. For more information, see slim partition in the Splunk Packaging Toolkit documentation.

make standalone_package

Packages the app for a local Splunk Enterprise instance. The resulting package is stored in /out/release/standalone.

make package_all

Builds three app packages: one for a local Splunk Enterprise instance, one for a one-click Splunk Cloud install, and one that has been partitioned into a set of targeted deployment packages. This command simply runs the following commands one after another: standalone_package, package_oneclick, and partition. The resulting packages are stored in /out/release.

make run_in_docker

Managing authorization and permissions

We include two roles in the PAS app to let an administrator manage
access to the functionality on the Setup dashboard. Any users that
an administrator adds to the pasadmin role will be able to save
the PAS configuration values on the Setup dashboard. Any users
that an administrator adds to the pasuser role will only have permissions
to read the configuration data created by a member of pasadmin
role. For more information about how the Setup dashboard uses the
App Key Value Store (KV Store) to persist global configuration settings for the PAS app, see the chapter "Working
with data: where it comes from and how we manage it" in this guide.

You can view the capabilities we assign to these two roles on the Access
Controls settings page in Splunk Enterprise. However these pages are large,
so here we show just the content of the authorize.conf file where the
settings are saved:

In this file, you can see that we add the pas index to the list
of indexes searched by default by members of the built-in admin
role. The custom pasuser role inherits from the built-in
user role and also searches the pas index by default.
The custom pasadmin role inherits from the custom pasuser
role and also searches the pas index by default. The pasadmin
role is also authorized to accelerate data models and edit user settings.

The administrator can add any existing indexes to the
pasuser and pasadmin roles. Events from customer
indexes will show up in the PAS app, provided they are tagged correctly.

When you define a new role, you should always inherit from an
existing role and then add the required capabilities.

To restrict access to the PAS configuration settings, we set the permissions
on the KV Store collections (ri_setup and ri-setup-coll) that
contain the configuration settings. The following screenshot shows the permissions
for the ri-setup-coll KV Store collection, where you can see that
we give the pasadmin role read and write
permissions, and the pasuser role read permissions
only:

Preparing the roles and permissions for packaging

When we defined our custom roles and permissions, we used the Settings
pages in Splunk Web UI. To include these settings it's important to copy them
from the location where the Splunk Web UI saves them, to the correct files that
we include in the .spl package.

When we create a new role from the Settings page, Splunk Enterprise
saves the configuration in the file etc/system/local/authorize.conf.
We copy the PAS app roles to the file authorize.conf in the default folder in the PAS app. We also remove some unused attributes (such as cumulativeRTSrchJobsQuota
and cumulativeSrchJobsQuota) from the role definitions to
keep them tidy (see the content of authorize.conf shown above).

When we change the permissions settings through the Settings
page, Splunk Enterprise strong the configuration in the file local.meta
in the metadata folder in the PAS app. This file is not included when
we package the app. We must copy these permissions settings into the
default.meta file in the metadata folder that is included when we package the app in the .spl archive file. We modify the contents of this file as shown in the following snippet from the default.meta file:

Here we are using the [ ] stanza notation to apply the permissions
to all knowledge objects in the PAS app. It's possible to set more granular permissions
by overriding the permissions in a stanza that relates to a specific knowledge object.

Programmatically enabling and disabling an add-on app

The PAS app has an optional dependency on the Eventgen app that can generate
sample data for the searches in the app. We are planning to deploy a version of
the PAS app to a cloud-hosted sandbox environment to let customers explore the app
without installing it in a local Splunk instance. However, to avoid consuming unnecessary
processing resources in the cloud we want to disable the Eventgen app by default,
but let a customer enable it if they want to view pseudo-live data in the PAS app.
To achieve the goal we considered four options:

Programmatically enabling/disabling the Eventgen app using the JavaScript
SDK
When we investigated this option, we discovered that the JavaScript SDK does
not support enabling and disabling apps from client-side code running in the
browser. We would need to run this code server-side, and this would require
a major change to the PAS app.

Programmatically enabling/disabling the Eventgen app via REST API call
Due to time constraints, we decided not to investigate this option any further.
However, it's possible that this option is not viable using client-side code
because it may be prevented by design for security reasons.

Programmatically enabling/disabling the underlying modular input of the
Eventgen app
Again, due to time constraints, we decided not to investigate this option any
further and opted for the simplest approach possible.

Programmatically detect the status of the app and inform the user
This is the simplest approach and is the one we chose. The next section describes
this approach in more detail.

Programmatically detect the status of an app and inform the user

This approach, which is the simplest, is the approach we take in the PAS app.
We detect the state of the Eventgen app using the code in the following snippet,
and display an appropriate message to the user on the Setup page
based on the current state of the Eventgen app. The code retrieves all the installed
apps using mvc class, checks that the Eventgen app is installed,
and then checks the status of the Eventgen app.

If we detect that the Eventgen app is not installed, we show the customer a link
to download and install it. If we detect that the Eventgen app is installed but
disabled, we show the customer a link to the Manage apps page in
Splunk Enterprise where the customer can manually enable the app.

Managing
app upgrades

We plan to ship enhanced versions of the PAS app in the future and customers
should be able to install the new version over the top of the current version. When
we package the PAS app, we include two important pieces of information in the app.conf file: a version number such as 0.91 and the check_for_updates configuration
value. These enable Splunk Enterprise to detect when a new version of the app is
available on Splunkbase and then notify the customer.

When we upload a new version of the PAS app package to Splunkbase, the web site checks that we are uploading a later version. The site also
gives us the option of keeping the new version hidden while we complete our
testing. When the tests are complete, we can mark the new version as the default
version for customers to install.

Note that from the perspective of the Splunkbase website, the package containing
the PAS main app and add-ons only contains a single app (the add-ons are installed
by the customer) and therefore instances of Splunk Enterprise will only detect when
the version of the main app is incremented. If we want to distribute new versions
of the add-on apps independently of the main app then we need to package each
add-on as a separate .spl to upload to Splunkbase. For example, we created a separate .spl file for our Google Drive add-on app and uploaded it to
Splunkbase to let customers upgrade it independently.

App certification

When submitting an app to Splunkbase, you can request an optional app certification by Splunk. Certified apps recognition constitutes high quality, including a security review, sufficient documentation, and user support. Certified apps will receive special visibility and promotion across Splunkbase to reach more users worldwide.

Questions?

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »