DevCamps Documentation: Master

The camp system relies on a single master database to hold the known camp users,
camp types,
and existing camps.
All camps within the system are registered within this database,
along with the type of camp,
the version control system used for that camp,
the owner,
etc.
The camp system will not function without this database.

Connectivity info for this database must be provided to the camp system.
The camp master database configuration file,
camp-db-config,
serves this purpose.

It is expected to live at $camp_base/camp-db-config.

Entries consist of the form $key:$value,
one per line.

The following key/value pairs are expected to be provided by this file:

The password to use when connecting to the master camp database.
If your database is configured to allow access without a password,
then you may leave this option out; however,
that is typically not recommended,
since the connection attempts will happen from different UNIX user accounts.

Your camp type may specify arbitrary paths that should be copied/symlinked into camp instances via the copy-paths.yml file. As the extension suggests, this is expected to be a YAML file.

The structure of the file is as follows:

Each path to copy (file or directory, it matters not) gets a "document" within the YAML structure. This essentially means that you start each path's entry with the standard "---" YAML document header. The result of this is that Camp::Master sees the path entries as an array.

Each path entry is a hash. Each path hash must provide the following key/value pairs:

May be provided with a single scalar entry or an array of such entries; each should correspond to an rsync/tar-style "--exclude" pattern, which will be provided to the rsync call that performs the copy. This naturally only comes into play if copying the source rather than symlinking.

If provided will assume that the source location is on a remote server. Value should be the protocol used to connect to the remote location, 'ssh' is the only valid value at the moment.

Because the YAML stream is converted to an array, the paths are processed in order of specification within the stream. Therefore, you can have dependencies between paths and they'll work out as long as you arrange them in the proper order in the file.

When a new camp instance is created, each path is processed in order. Per path, the camp creator is offered the opportunity to specify whether the camp should receive a full copy or a symlink of the source path in question; the default response will be noted based on the default_link setting for that path. However, if the always option is set for the path, the user is not given the choice and the default simple happens automatically.

Here's an example of a file:

# The /var/www/html docroot will be symlinked to $camp/htdocs by default,
# but the user is given the choice to override with a copy.
---
source: /var/www/html
target: htdocs
default_link: 1
# The /var/cgi-bin/foo.com script directory will by copied (since no default_link
# is specified) by default to $camp/cgi-bin. The user is given the choice to
# symlink instead.
---
source: /var/cgi-bin/foo.com
target: cgi-bin
# The /var/www/image_repository is symlinked to $camp/image_repository; the user
# is not given a choice about this, because always is set.
---
source: /var/www/image_repository
target: image_repository
default_link: 1
always: 1
---
source: remote@endpoint.com:/var/www/image_repository
target: image_repository
default_link: 1
always: 1
allow_errors: 1
remote_source: ssh

Each camp type has a subdirectory under the $camp_base, and configuration information provided at this level applies only to the camp type in question, overriding any conflicting settings coming in from the base camp level.

The configuration information is parsed at the base camp level first, then the type-specific level second.

In each level, the local-config file is processed. This is expected to consist of key/value pairs of the form:

some_key:some_value

Empty lines and lines starting with the pound character are ignored, meaning that you can embed comments in your configuration files.

All values specified in these files undergo token substitution, such that any configuration variable can be substituted into a value by using the token:

__CAMP_VARIABLENAME__

For instance, if the local-config file has a line:

my_variable:some_useless_value

then a subsequent line:

other_variable:__CAMP_MY_VARIABLE__foo

will result in the "my_variable" key having value "some_useless_valuefoo".

These variables are used both by Camp::Master itself, and in the various templates that Camp::Master renders at camp creation time; the token substitution logic is the same in each case. This allows your template files to undergo rendering and have camp-specific information substituted in, localing the file to the camp being created (which is the entire point of the camp system).

Prior to parsing the base level and camp level configuration files, a number of configuration entries are determined automatically by Camp::Master.

Specify location of configuration file under "httpd_path" and include that in command strings.

For example, "conf/apache2.conf", used when executable has been compiled with a specific full path. To detect need for this option see httpd -V output and check for "SERVER_CONFIG_FILE" with a value similar to "/etc/apache2/apache2.conf" (any path beginning with "/").

Rendered Apache ProxyBalancer member configuration directives suitable for placement directly within a virtualhost's <Proxy balancer://...>...</Proxy> container. Defaults to using three mongrel listeners with ports incremented by one starting at the mongrel_base_port.

When using an application server other than those natively supported in camps, set this to the command to initialize your application server. This will run only once during camp creation. For example (using a custom init script):

The base database server logfile path; for Postgres, this is in fact the logfile to be used; MySQL uses multiple logging paths ordinarily, so it can be used as a prefix for a full path with MySQL (it probably could be used for all logging too).

All of the above variables are calculated for you. This calculation is one of the first things performed by Camp::Master; therefore, they can be safely overridden in your base or type configuration files as appropriate for your camp deployment.

A few variables must be provided by your configuration files in order for the camp system to function properly:

Space-separated list of paths to SQL files that should be run upon preparing a camp's database server. You may specify as many as you need, but at least one should be provided.

The paths may be absolute or relative; if relative, they are expected to be relative to the camp type's directory.

Each script specified in the space-separated list is run within its own db client shell; they are executed in order of specification. No assumptions are made for transaction handling; if you want transactions, put them in the scripts themselves.

It's important to know the context in which these run, which differs between database server types due to fundamental structural differences between those servers and how they organize things:

In Postgres, each script is run via a connection to the "postgres" user's "postgres" database. Therefore, your scripts need to provide \connect information in order to make changes against a different database.

Scripts listed in db_source_scripts are normally processed verbatim. List again here any scripts here that you would like to first have preprocessed with variable token substitution. This is useful if, for example, you would like to reference the camp number or camp owner's name in the script.

The paths should be specified identically to how they're listed in db_source_scripts or they will not be tokenized.

Space-separated list of paths to SQL files that should be run upon preparing a camp's MySQL database server, prior to processing the roles for that database; this gives an opportunity to initialize the database to a known set of users/permissions, for instance, or initalize things in other ways needed prior to role creation.

Sometimes when working in a camp a developer has SQL scripts that are in-progress and are needed when a database is refreshed. These SQL files should be placed in a standard directory in each camp and db_working_files_dir should be the path to this dir relative to the camp's path. Files in this dir that end in '.sql' will be executed in the camp's database in lexical order when the database is created or refreshed.

Allows specification of a number of seconds to sleep between starting the database server at camp creation time and processing the roles and db_source_scripts; necessary for UNIX socket connections in which the server needs a few seconds to start up and initialize its socket file.

Sane values appear to be in the 5-15 second range. Defaults to 5 seconds; set to 0 or blank to turn off.

Space-separated list of database names that are expected to live within your camp's database server.

This was formerly used to set up Postgres password access to each database in your user account's .pgpass file, but now each database user's password is applied to all databases ("*") there. It is still up to you to configure Postgres to allow appropriate access.

When using LVM snapshots for databases, you can specifiy the initial snapshot size. Defaults to 3G. When running the command `camp-lvm resize` or `camp-lvm resize_all`, camp snapshots may also be increased by this same size if their usage exceeds 50%.

A space-separated list of directories that are expected to reside under your camp (naturally, these paths are expected to be relative to the camp directory itself). These directories will be cleared out if you rebuild an existing camp (with the mkcamp script), which is the main reason they need to be provided to Camp::Master.

The name of the catalog for your camp's Interchange deployment. Specify one, even if you have multiple catalogs; the catroot variable is set based on this. For the majority of End Point Interchange deployments, multiple catalogs are a non-issue.

Some optional variables may also be provided to affect the functioning of your camp deployment:

The default database to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the database used for db_source_scripts import on MySQL.

Specifies the default version control system to use for the camp type. This is optional, but is certainly highly convenient for camp users and makes things easier to manage. The main reason to use a different version control system would be for using git-svn in a camp against a Subversion repository.

If this is set for a camp type, then it is not necessary for a user to specify their VCS type when creating an instance of that camp. The value of vcs_default_type has no bearing on existing camps; each extant camp's original VCS type is recorded in the camp database and will not change without heroic efforts on the part of the camp's owner.

Use this to specify the SCM repository path. This is not specific to VCS type and has been preserved for legacy purposes. SVN and Git repository paths can be specified in type-specific manner using repo_path_svn and repo_path_git, respectively; when those are used, they override the repo_path value.

There is no default value for repo_path, so don't count on it being present in your code unless you are quite sure of what you're doing.

Use this to specify the main project Subversion repository path; is is assumed that it will be accessed via the "file://" protocol, meaning that you need only provide the path to the repository (and any subdirectories within that repository).

When determining the Subversion repository location, Camp::Master consults, in this order:

repo_path_svn

repo_path

Default: type_path . '/svnrepo/trunk'

There is no default value for repo_path_svn, however, so do not count on this being available for token substitution unless you explicitly set it in the relevant local-config. The default repository location described above is enforced through logic, but not within the master configuration hash.

Use this to specify the main project Git repository path; no assumptions about the path/URL are made, so you are free to specify a file path (which is most likely as of this writing), a git://* URL, an rsync://* URL, etc.

Similar to determination of Subversion repository location, Camp::Master consults the following items in order when finding a camp type's Git repository location:

repo_path_git

repo_path

Default: type_path . '/repo.git'

There is no default value for repo_path_git, so you cannot count on its presence unless you know what you're doing and are working with a camp type that definitely specifies a value for it.

Options to pass literally to the "git clone ..." call when building a new camp and cloning from the source Git repo. See the "git-clone" man pages for valid options. There is no default, meaning that the effective default behavior is invoking git-clone without options.

Git is efficient in its use of disk space, but by using the various options like local, shared, or reference can potentially be even more efficient and speedy.

Specifies a branch to track and checkout after cloning. Use this in the case that your Git repository's active branch differs from the branch the camp type needs to work with. This would likely be the case if you're using a single master Git repository to track several lines of development; one camp type would work with the default branch "master", while other camp types would need to specify alternate branches.

This assumes that you're using a remote name of "origin". Since git_clone_options allows you to specify a different name for your clone, git_branch introduces some extra semantics: if the value of git_branch contains a forward slash ("/"), the word characters preceding the first slash are treated as the remote name.

If you use git_clone_options "-o foo" to specify a remote name of "foo" rather than "origin",

you can have your camp type check out branch "bar" via git_branch of "foo/bar". However, if you're using the default clone remote name of "origin", then git_branch "bar" would suffice to checkout branch "bar".

This also means that if the desired branch has forward slashes in the name, you'll need to include the remote name as the first portion of your git_branch value. So, if you want a branch named "releases/1.1", you would need a git_branch value of "origin/releases/1.1".

Rendering and installing arbitrary files into a camp from templates is a critical aspect of the camp system. Configuration files for Apache, Interchange, Rails, etc. should be reduced to templates, with things like hostnames, file paths, port numbers, etc. replaced with camp-system tokens of the form described in the section regarding CONFIGURATION VARIABLES. At camp creation time, Camp::Master will parse each template, performing token substitution, and install each parsed template into the appropriate location within the new camp.

The files to parse are specified within the camp type subdirectory's camp-config-files file. Specify one path per line; blanks and lines starting with the pound character are ignored.

All the files specified will undergo the described token substitution and installation into the new camp. The paths specified in camp-config-files are expected to be relative to the camp type directory's 'etc/' subdirectory, and also reflect the target path of the file when copied into the new camp itself. Thus, a file at /home/camp/some_type/etc/blah/foo.conf would be registered in camp-config-files with a path relative to etc/, or "blah/foo.conf", and would be installed at /home/some_user/campNN/blah/foo.conf after parsing.

Also, some assumptions are made about what files will be included if your camp uses Interchange versus Rails:

If your camp-config-files file specifies interchange/bin/interchange, then it is assumed to use Interchange.

If your camp-config-files file specifies rails/.../config/mongrel_cluster.yml, then it is assumed to use Rails.

Note that these are not mutually exclusive. It is theoretically acceptable for a single camp to employ both app server types. If you are bursting with curiosity and sadly deficient in sanity, by all means try this out.

While this can be expected to vary between deployments, a number of files are obvious candidates for inclusion within this templating scheme:

rails/.../config/database.yml

interchange/interchange_local.cfg

catalogs/.../catalog_local.cfg

httpd/conf/sites/some_site.conf

Anything that relies on file paths, ports, domain names, etc. should be abstracted out into such templates. A common pattern (implied by interchange_local.cfg and catalog_local.cfg above) is to encapsulate such details within "local" configuration files that are included from master configuration files; the master configuration files can stay in version control and have no need to vary between camps, production, etc., while the "local" configuration files are small, containing only what is absolutely necessary, and vary in a controlled way between environments by virtue of being managed by the camp system.

Typically speaking, use of Git for a particular camp type implies non-use of Subversion, and the opposite also holds. However, there is no reason that all types cannot be made available for use within the same camp type; the Git software suite comes with '''git-svn''', which can be used to track a Subversion repository via a Git repository. Therefore, all options can be available for a camp type provided the administrator takes the trouble to arrange it. That arrangement takes place largely outside of the camp system itself, and would instead involve post-commit hooks within one or both repositories to ensure that a change in one repository is pushed to the other.

Camp::Master ultimately makes no assumptions about your camp type's VCS offerings; if a camp user attempts to make a new camp using some version control system, Camp::Master will attempt to do it, and if the requested version control system hasn't been configured for the camp type, an error will result. Configuration options to tell Camp::Master about your VCS setup are fairly straightforward; see the configuration variable information for:

default_vcs_type

repo_path

repo_path_git

repo_path_svn

While no assumptions are made, the strong recommendation is to use Git. Subversion is slow, inefficient in its use of disk space, relatively inflexible, bad at branching, etc.

When creating a new camp using Git as the VCS type, Camp::Master assumes an SVN-like flow of operations; the camp type specifies where the Git repository lives, and the new camp will result in a new repository that tracks that central repository. Consequently, the operations used by Camp::Master tend to be clone, pull, and push operations, for remote repository use. When you're working within your Git-based camp, you of course need to keep this in mind, knowing that a checkout or a commit are only relative to the repository of that camp, and that you ultimately need a push to commit your stuff to the central camp repository.

At this time, Camp::Master doesn't provide any tools for creating a Git camp using an alternate repository as the repository to track; this is something that can be done using git directly, after creating a new camp. Create the camp with Git as the VCS choice, then use git directly within that camp to point the camp at an alternate remote repository.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see: http://www.gnu.org/licenses/