Introduction

The Shotgun Pipeline Toolkit is not only a collection of Apps and Engines maintained by Shotgun Software -
it is also a framework and a development platform which makes it easy to quickly build custom
pipeline tools! This document outlines the workflows in both these cases and also explains some of the basics
around app development.

All apps and engines maintained and released by Shotgun Software are open source and you can find them in Github

The Starter App Template

When starting a new app, we recommend that you take a look at our Starter App!
and take a look at our starter app! This is a "blank" app designed to be used as a starting point when you
build new apps. By using this app you get all the standard toolkit boilerplate code set up for you.

Forking an existing app

Because all the Shotgun Toolkit Apps are open source, it is also possible to use their source code as a base for
further app development. If you find that one of our default apps is missing a crucial feature, you can
clone it in github and continue your own development! We love pull requests, so if you feel you have made a
change that can benefit others, don't hesitate to feed it back to us as pull request. We may then fold
it back into the main version of the App.

If you think that the feature that you feel is missing is something that would be generally useful, we would love
if you would reach out to us via support@shotgunsoftware.com to hear your thoughts - perhaps this modification
is something that we already have planned to do, or know that some other client has done, or maybe there is another
way to achieve the same behaviour! We love to know what clients feel are missing from the current versions of our apps.

Creating a new app from scratch

This section goes through how to create a new app using our starter template. We assume that
you are familiar with github and git workflows but please note that you can do Toolkit development
even if you are not using git as your source control solution.

Create a development sandbox

Before you do anything else, we recommend that you set up a development sandbox by cloning the project
configuration. This will result in a separate configuration where you can do development
and make changes (and break things!) without affecting anyone one else on the production.
You can find detailed instructions about configuration cloning in the following Document:

Forking or downloading the starter app repository

The easiest way to get started is to fork our starter app. You can find it here.

If you don't want to use git source control just yet, you can just download the files from github as a zip,
and unzip them somewhere on disk. It is easy to add it to git (or other source control packages) at a later point.
If you already know that you need a git repository, you can also fork the starter app repo to your own
git server and then check out the contents into your local dev area. Either way, the goal is to have the starter
app code in your local development area so that you can start doing development.

Adding parameters and frameworks

At this stage, if you know that you will be using frameworks or have specific manifest parameters that you need,
this is a great time to get these set up! This is because in the next step, when we install the app into an environment,
the system will take care of all the configuration for you automatically.

For example, if you plan to use the widget and shotgun utils frameworks for your app, add the following
to the info.yml manifest:

# the frameworks required to run this appframeworks:-{"name":"tk-framework-shotgunutils","version":"v2.x.x"}-{"name":"tk-framework-qtwidgets","version":"v1.x.x"}

Minimum required framework versions

If there is a required minimum version for a framework, the minimum_version setting can be used in the info.yml manifest:

# the frameworks required to run this appframeworks:-{"name":"tk-framework-shotgunutils","version":"v2.x.x"}-{"name":"tk-framework-qtwidgets","version":"v1.x.x","minimum_version":"v1.5.0"}

The above will ensure that v1.5.0 of tk-framework-qtwidgets is available for the app to use. If it is not, the app will not be loaded at startup time and an error will be printed to the console. When the app is updated using tank updates, any configured frameworks not meeting their required minimum versions will be automatically updated along with the app.

For more information about frameworks and how they can be useful, check out the following links:

Add your new repository to an environment

Now let's get this App up and running inside of Toolkit!

With your development sandbox there will be a specific tank or tank.bat binary that you can use to address
this particular configuration. Open a terminal and navigate to your dev sandbox. Using the local tank command,
run the install_app command. When you install an app, you need to choose an environment and an engine to install
it into. The engine is
the application you will be running, so either tk-maya, tk-nuke etc. The environment is a
collection of tools that you want to run against a specific work area. In our default configuration,
when working with Shots, the environment is called shot_step and when working with Assets,
the environment is called asset_step.

For example, if you have your new app code located in /Users/manne/dev/tk-multi-mynewapp, you run the
following to install it:

Any frameworks that you app requires will be automatically installed by the install command.
Now if you start up maya for the environment you just installed into, you should see the starter app
appearing on the menu!

Anatomy of the Template Starter App

The app entry point and menu registration can be found in the app.py file.
This is where you typically set up your classes and get things initialized and
get menu items registered.

The manifest defines all the different settings that this app requires when it
is installed. Settings are often useful if you want reusable apps and you don't want to
hard code any values in the app itself.

The dialog.pycontains the logic and event callbacks etc. that produces the main app window.

The dialog.py loads in an auto generated widgets file residing in the UI folder. This file
contains all the base layouts and properties of the dialog UI.

In the resources folder, the dialog.ui file is a QT Designer file which you can open up and
use to rapidly design and define the look and feel of the App. Once you have made changes you
have to execute the build script which will convert the ui file to python code.

Do development

You are now all set to do development! As a convenience, as soon as you have one more more dev
items in your configuration, Toolkit will automatically add a Reload and Restart option to the menu
inside of maya, Nuke etc..
Clicking this will reload your configuration and code and then restart your engine. This means that
you can iterate quickly: Start maya once, then each time when you have made code or configuration
changes that you want to try out, simply hit the reload and restart button and your changes will be
pulled in. Note that if you have any UIs active on screen, these will not automatically update, but
you have to go in and re-launch the UIs from the menu.

Do testing

When you want to test your code, you can easily invite other users to your dev sandbox by adding
them to the pipeline configuration entry in Shotgun. As soon as you have added a user, they will
see new entries on their menus inside of Shotgun.

They can then go in and for example click the launch maya menu entry associated with your dev
sandbox. When maya starts up they will be running it from your configuration and seeing your
in progress work, just the way you are seeing it. Note: Make sure they have access to see your app code too,
otherwise the app will not load.

Preparing your first release

When you feel ready to create your first release, if you haven't already, we recommend that you start using source
control to make sure that changes are tracked. Toolkit supports git (http://git-scm.com/) out of the box and we
strongly recommend using this. The rest of this document assumes that are familiar with git and have it set up on
your machine.

Now you may already have a git repository set up for your app. If you haven't, it is time to create that and push
your first commit. Toolkit git repos need to contain just a single app -- basically your git repo should have the
same structure as the starter app repository where you got the original code from.

When you do development, toolkit uses a dev location in the environment file, something looking like this:

location:{path:/Users/manne/dev/tk-multi-mynewapp,type:dev}

This instructs toolkit to load the app code directly from disk in the given location, which is great for development,
where you want to change the code all the time.

For released software, we want to ensure that things are versioned so that they can be upgraded safely and easily.
All Toolkit's built-in apps uses the toolkit app store to track updates and releases, and they will have a location
tag that looks something like this:

location:{name:tk-multi-setframerange,type:app_store,version:v0.1.7}

This allows toolkit (for example the tank updates command) to check when updates are available, update and maintain
configurations in a very safe way. Whenever a new version is available, toolkit will automatically download the code
and place it in a local "app cache" on disk and ensure that users have access to it.

For in-house tools, you can use the same principle, but instead of using the toolkit app store you can use tags in git.
So for your new app, create and push a new tag to git. Note that if you have cloned our starter app, there are already
tags defined. We strongly recommend that you use Semantic Versioning when creating your tags ( see http://semver.org ).
Toolkit will use these version numbers to try to determine which version is the most recent, and
by following the convention vX.Y.Z your app versioning will be compatible with Toolkit.

Installing changes into the production config

Once you have created your first tag in git (eg. v1.0.0), we can safely install this in your production config.
This means that everyone on the production will get access to the new tool. Installing the
app is done using the install app command. Make sure that you are using the tank command that
belongs to the primary production config when you run the install command:

This will find the highest version tag in git and install this into the environment. Once installed, you can
simply run tank updates, and if new tags have been created, you will be prompted if you want to upgrade.
The workflow is now identical to the one which happens with official app store apps.

Modifying an existing app

Rather than starting from an empty starter template, it is sometimes necessary to add a minor
feature to an existing app, for example one of Shotgun Software's standard Apps. When you work with
a modified version of an app, you typically want to 'track' against the source app and make sure
to regularly pull in changes and bug fixes.

When you do this type of development, you pick up the parent code, then apply some of your changes,
then release this to your pipeline. Your release effectively consists of the base version of the app
PLUS your applied local changes. We recommend that you add a version suffix to the existing version
number. This will work seamlessly with Toolkit and is relatively straight forward. The following
workflow illustrates how to proceed:

You for the parent app and create your own repository. With the fork you get all the git tags.
The latest tag is called v0.2.12 and the master branch basically equals the contents in this tag.

You apply your changes and commit to your master. You now basically have v0.2.12 PLUS your changes.
When you release this to production you need to create a tag. Name the tag v0.2.12.1, to indicate
that your code is based on v0.2.12 and it is the first release.

Now someone finds a bug in your modifications. Fix the bug and tag up and release v0.2.12.2.

A number of important bug fixes have been released in the parent repository. Pull them down to
your repository. The most recent tag is now v0.2.15 due to releases that have happened in the
parent repository. Now merge your changes with master and test. You now basically have parent
app v0.2.15 merged with your changes. Tag up v0.2.15.1.

The tagging scheme outlined above guarantees that the toolkit updates work correctly and makes it
easy to quickly see what code each tag in your fork is based on.

Documentation Resources

Apart from this document, we also have a number of other documentation resources available
that can be useful when doing app development:

The Apps Configuration reference documentation outlines the various ways you can set up
parameters and configuration for your app.

Interactive Access to Toolkit

If you are simply interested in playing around with Toolkit in a more interactive fashion,
you can easily do this inside of for example Nuke or Maya once you have started the Toolkit engine
and can see the Shotgun menu. You can access all of toolkit via the currently running engine, which
you can retrieve like this inside Maya or Nuke's Python shell:

importsgtkengine=sgtk.platform.current_engine()

Once you have access to the engine you can reach most other aspects of the currently loaded
environment:

# A Shotgun API instanceengine.shotgun# The toolkit API instanceengine.sgtk# The current work area objectengine.context# All the apps currently loadedengine.apps

If you are in a command shell with access to the tank command, you can open up an interactive
Python session easily by running the tank shell command. Toolkit will try to start up as much
of the system as possible for you, but depending on your context, you may get a limited set of
accessors:

>tankShotABC123shellWelcometotheShotgunPipelineToolkit!Fordocumentation,seehttps://support.shotgunsoftware.comWillsearchacrossallShotgunProjects.-FoundShotABC123(Project'Chasing the Light')-Usingconfiguration'Primary'andCorev0.14.214-SettingtheContexttoShotABC123.-StartedShellEngineversionv0.3.4-Environment:/mnt/software/tank/chasing_v2/config/env/shot.yml.-Runningcommandshell...----------------------------------------------------------------------Command:Shell----------------------------------------------------------------------WelcometoShotgunPipelineToolkitPython!2.7.1(r271:86832,Jul312011,19:30:53)[GCC4.2.1(BasedonAppleInc.build5658)(LLVMbuild2335.15.00)]Runningondarwin-AtkAPIhandleisavailableviathetkvariable-AShotgunAPIhandleisavailableviatheshotgunvariable-Yourcurrentcontextisstoredinthecontextvariable-Theshellenginecanbeaccessedviatheenginevariable>>>

The Toolkit shell is handy when you want to do quick debugging or run little script snippets.

We actually don't have a standard for release notes at the moment outside our standard apps. The standard apps take advantage of the github wiki feature and this wiki page is processed and translated into the toolkit app store version page. Here is an example of a github version history and here is the equivalent version history in the app store.

With a project managed purely in git, it is harder to settle on a standard for storing the version history. Github provides both a wiki and their (semi-)new release feature, which looks really cool and which we may start utilizing a bit more. But for a studio git server, neither of those facilities would be present.

Perhaps we should start having a file as part of each app which contains the version history and move all our release note wikis pages into this file. Do you think this would be a good solution?

Think using the wiki pages on github is great, and super easy to update so in my opinion that is enough. Although I added a page release notes page for an app, and tried updating through the tank shell, it didnt seem to get the information. Are there some special formatting in the release notes wiki that the shell updater reads?

The process which converts the release notes wiki page into an update summary when you run tank updates actually only currently works for toolkit app store apps (and involves a couple of steps!). I really like this idea of supporting / moving release notes in a file and not just on a github wiki page - I think thin is probably the way to go for the future! We really want to encourage clients to write apps, and we want the same set of features available to them as is available to us when we write and deploy apps! I will log a feature requests and pass it on to our product team for triage and consideration!

ERROR: The configuration for engine 'tk-maya' located in the environment file'Z:\Tech\shotgun\configs\env\shot_step.yml' has areference to another file ('@settings.tk-maya.shot_step'). This type ofconfiguration arrangement cannot currently be automatically modified - pleaseedit it by hand!