Software List

Ohio Supercomputer Center (OSC) has a variety of software applications to support all aspects of scientific research. We are actively updating this documentation to ensure it matches the state of the supercomputers. This page is currently missing some content; use module avail on each system for a comprehensive list of available software.

Availability & Restrictions

The available programs are ABAQUS/CAE, ABAQUS/Standard and ABAQUS/Explicit. .

Usage

Access

OSC's ABAQUS license can only be used for educational, institutional, instructional, and/or research purposes.

Only users who are faculty, research staff or students at the following institutions are permitted to utilized OSC's license:

The Ohio State University

University of Toledo

University of Cincinnati

Miami University of Ohio

Wright State University

Users from additional degree granting academic institutions may request to be added to this list per a cost.

To obtain validation please complete and return the "Academic Agreement to Use ABAQUS," located in the Software Forms Page.

Setup

Use module avail abaqus to view available ABAQUS modules for a given machine. To select a particular software version, type: module load software-name. For example: To select ABAQUS version 6.11-2 on Oakley, type: module load abaqus/6.11-2

Using ABAQUS

Example input data files are available with the ABAQUS release. The abaqus fetch utility is used to extract these input files for use. For example, to fetch input file for one of the sample problems including 4 input files, type:

abaqus fetch job=knee_bolster

abaqus fetch job=knee_bolster_ef1

abaqus fetch job=knee_bolster_ef2

abaqus fetch job=knee_bolster_ef3

Token Usage

ABAQUS software usage is monitored though a token-based license manager. This means very time you run a ABAQUS job, tokens are checked out from our pool for your tasks usage. To ensure your job is only started when its required ABAQUS tokens are available it is important to include a software flag within your job script's PBS directives. A minimum of 5 tokens are required per a job, so a 1 node, 1 processor ABAQUS job would need the following PBS software flag: #PBS -l software=abaqus+5. Jobs requiring more cores will need to request more tokens as calculated with the formula M = int(5 x N^0.422), where N is the total number of cores. For common requests, you can refer to the following table:

Cores (nodes x ppn each):

1

2

3

4

6

8

12

16

24

32

48

Tokens needed:

5

6

7

8

10

12

14

16

19

21

25

Batch Usage

When you log into oakley.osc.edu or glenn.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the 4000+ processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution.

Continuing with the above example, assume that you have fetched the four input files above into your work directory (where you submit your job, represented by $PBS_O_WORKDIR). A batch script can be created and submitted for a serial or parallel run.

Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory ($PBS_O_WORKDIR) to $TMPDIR, and copy your results back at the end of your script. Running your job on $TMPDIR ensures maximum efficiency.

The keyword interactive is required in the execution line abaqus job=knee_bolster interactive for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword interactive in this case simply tells ABAQUS not to return until the simulation has completed.

The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. Although, it is better practice to designate the main input file explicitly by adding input=<my_input_file_name>.inp to the execution line so that it looks like abaqus job=knee_bolster input=<my_input_file_name>.inp interactive

Define nodes=1 (1<=ppn<=8 for Glenn and ppn=12 for Oakley) for a serial run.

If ppn>1, add cpus=<n> to the execution line, where n=ppn. This is how it should look like: abaqus job=test input=<my_input_file_name1>.inp cpus=<n> interactive

Further Reading

See Also

ACML - AMD Core Math Library

The AMD Core Math Library (ACML) is a set of numerical routines tuned specifically for AMD64 platform processors (including Opteron™ and Athlon™64 ). The routines, which are available via both FORTRAN 77 and C interfaces, include BLAS, LAPACK, FFT, RND, and many others.

Availability & Restrictions

OSC supports use of ACML on the Glenn cluster.

VERSION

GLENN

OAKLEY

3.6.0

X

3.6.1

X

4.0.1

X

4.3.0

X

4.4.0

X

There are no restrictions on the use of the ACML library.

Usage

Set-up

To configure your environment for use of ACML load the “acml” module appropriate for your compiler:

Compiler

Module Command

GNU Fortran

module load acml-gfortran

Intel

module load acml-intel

Portland Group

module load acml-pgi

Portland Group, Multi-Core

module load acml-pgimp

This step is required for both building and running ACML applications. The default version is 4.0.1.

Building With ACML

The Glenn ACML modules will automatically configure your environment to locate appropriate include files and libraries. When the ACML modules are loaded the following environment variables will be set:

Variable

Use

$OMP_NUM_THREADS

Number of OpenMP threads to be used. Set to 1 in all ACML modules except acml-pgimp, which sets it to 4. Should be set by the user AFTER loading the module if a different value is needed.

$ACML_CFLAGS

Compiler flags: include path to be used

$ACML

Linker flags: libraries to be used

Usage with the Portland Group Compiler, C code

To build a sample C-code that uses the ACML library with the Portland Group Compiler on Glenn, follow the example steps below after logging into the system:

See Also

AMBER

The Assisted Model Building with Energy Refinement (AMBER) package contains many molecular simulation programs targeted at biomolecular systems. A wide variety of modelling techniques are available. It generally scales well on modest numbers of processors, and the GPU enabled CUDA programs are very efficient.

Availability & Restrictions

OSC has an "Academic Teaching Advanced " license for ANSYS. This allows for academic use of the software by Ohio faculty and students, with some restrictions. To view current ANSYS node restrictions, please see ANSYS's Terms of Use.

For academic users not on the OSU campus, OSC must pay an additional fee to Ansys to validate their use. For this reason, such new users must have their department or PI confirm their funding for ANSYS licenses.

The following versions of ANSYS are available at OSC:

Version

Glenn

Oakley

11.0

X

11.0 update

X

13.0

X

X

14.0

X*

14.5

X

X

* Commercial users should use the "ndem" module.

Usage

Set-up

Use of ANSYS for academic purposes requires validation. To obtain validation please complete and return "Academic Agreement to Use ANSYS." This can be obtained from your site consultant or from the files ansys.pdf, ansys.ps, oransys.txt located in the Academic Agreement Forms.

ANSYS supports X11 Windows and 3-D devices. PostScript and HP-GL output are available through the DISPLAY program.

Using ANSYS

The ANSYS commands and utility programs (see below) are located in your execution path. ANSYS is normally started with a module load command for the specific version, e.g. ansys110; or use only ansys for the default version (ansys110 on Glenn).

module load ansys

Following a successful loading of the ANSYS module, you can access the ansys commands:

ansys <switch options> <file>

The ansys command takes a number of Unix-style switches and parameters.

The -j Switch

The ansys command accepts a -j switch. It specifies the "job id," which determines the naming of output files. The default is the name of the input file.

The -d Switch

The ansys command accepts a -d switch. It specifies the device type. The value can be X11, x11, X11C, x11c, or 3D.

The -m Switch

The ansys command accepts a -m switch. It specifies the amount of working storage obtained from the system. The units are megawords.

The memory requirement for the entire execution will be approximately 5300000 words more than the -m specification. This is calculated for you if you use ansnqs to construct an NQS request.

The -b [nolist] Switch

The ansys command accepts a -b switch. It specifies that no user input is expected (batch execution).

The -s [noread] Switch

The ansys command accepts a -s switch. By default, the start-up file is read during an interactive session and not read during batch execution. These defaults may be changed with the -s command line argument. The noread option of the -s argument specifies that the start-up file is not to be read, even during an interactive session. Conversely, the -s argument with the -b batch argument forces the reading of the start-up file during batch execution.

The -g [off] Switch

The ansys command accepts a -g switch. It specifies that the ANSYS graphical user interface started automatically.

ANSYS parameters

ANSYS parameters may be assigned values on the ansys command. The parameter must be at least two characters long and must be a legal parameter name. The ANSYS parameter that is to be assigned a value should be given on the command line with a preceding dash (-), a space immediately after, and the value immediately after the space:

OSC Batch Usage

ANSYS can be run on OSC systems in interactive mode or in batch mode. Interactive mode is similar to running ANSYS on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Batch mode means that you submit the ANSYS job with a batch script by providing all the ANSYS input file(s) needed to run the simulation when resources become available.

Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. You can submit a simple one processor interactive batch job with X11 forwarding with the command qsub -I -X. The intention is that users can run ANSYS interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in batch mode.

Batch jobs can request mutiple nodes/cores and long compute time, up to the limits of the OSC systems (refer to the computing pages). Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Example

To run interactive ANSYS, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. For example, the following line requests one node (the default), one core, for a walltime of one hour, with one ANSYS license.

qsub -I -X -l walltime=1:00:00 -l software=ansys+1

This job will queue until resources becomes available. Once the job is started, you're automatically logged in on the compute node; and you can launch ANSYS and start the graphic interface with the following commands:

module load ansys
ansys -g

Batch Mode

For a given model, prepare the input file with ANSYS commands (named ansys.in for example) for the batch run. Assume the solution will need 30 hours and 1 processor. The following batch script would be needed for the serial applicaiton:

To run this job on OSC batch system, the above script (named submit_ansys.job) is to be submitted with the command:

qsub submit_ansys.job

To take advantage of the powerful compute resources at OSC, you may choose to run distributed ANSYS for large problems. Multiple nodes and cores can be requested to accelerate the solution time. Note that you'll need to change your batch script slightly for distributed runs.

For distributed ANSYS jobs using one node (nodes=1), the number of processors needs to be specified in the command line with options '-dis -np':

Notice that in the script above, the ansys parallel license is requested as well as ansys license in the format of

#PBS -W x=GRES:ansys+1%ansyspar+n

where n=m-4, with m being the total cpus called for this job. This line is necessary when the total cpus called is greater than 4 (m>4), which applies for the parallel example below as well.

For distributed jobs requesting multiple nodes, you need to specify the number of processors for each node in the command line. This information can be obtained from $PBS_NODEFILE. The following shows changes in the batch script if 2 nodes on Glenn are requested for a parallel ansys job:

Further Reading

See Also

Altair HyperWorks

HyperWorks is a high-performance, comprehensive toolbox of CAE software for engineering design and simulation.

Availability & Restrictions

HyperWorks is available to all OSC users without restriction.

The following version of Altair Hyperworks can be found for the following environments:

Version

Glenn

Oakley

Statewide

10.0

X

X

11.0

X

X

12.0

X

X

Usage

Using HyperWorks through OSC installation

To use HyperWorks on the OSC clusters, first ensure that X11 forwarding is enabled as the HyperWorks workbench is a graphical application. Then, load the hyperworks module:

module load hyperworks

The HyperWorks GUI can be launched then with the following command:

hw

State-wide access for HyperWorks

For information on downloading and installing a local copy through the state-wide license, follow the steps below.

NOTE: To run Altair HyperWorks, your computer must have access to the internet. The software contacts the license server at OSC to check out a license when it starts and periodically during execution. The amount of data transferred is small, so network connections over modems are acceptable.

Usage of HyperWorks on a local machine using the statewide license will vary from installation to installation.

If you have already registered with the Altair web site, enter the e-mail address that you registered with and your password and skip to step #5.

If you have not registered yet, click the link that says "Click here to register now". You will be prompted for some contact information and an e-mail address which will be your unique identifier.

IMPORTANT: The e-mail address you give must be from your academic institution. Under the statewide license agreement, registration from Ohio universities is allowed on the Altair web site. Trying to log in with a yahoo or hotmail e-mail account will not work. If you enter your university e-mail and the system will not register you, please contact OSChelp at oschelp@osc.edu.

Once you have logged in, click on "SUPPORT" and then "SOFTWARE DOWNLOADS"

In addition to downloading the software, download the "Installation Guide and Release Notes" for instructions on how to install the software.

IMPORTANT: If you have any questions or problems, please contact OSChelp at oschelp@osc.edu, rather than HyperWorks support. The software agreements outlines that problems should first be sent to OSC. If the OSC support line cannot answer or resolve the question, they have the ability to raise the problem to Altair support.

Further Reading

See Also

BLAST

The BLAST programs are widely used tools for searching DNA and protein databases for sequence similarity to identify homologs to a query sequence. While often referred to as just "BLAST", this can really be thought of as a set of programs: blastp, blastn, blastx, tblastn, and tblastx.

Availability & Restrictions

BLAST is available without restriction to all OSC users.

The following versions of BLAST are available on OSC systems:

Version

Glenn

Oakley

2.2.17

X

2.2.23+

X

2.2.24+

X

X

2.2.25+

X

X

2.2.26

X

If you need to use blastx, you will need to load one of the C++ implimenations modules of blast (any version with a "+").

Usage

Set-up

Setting up BLAST for usage depends on the system you are using. On Glenn, load the biosoftware module followed by the BLAST specific module:

module load biosoftw
module load blast

On Oakley, just load the BLAST specific module:

module load blast

Then create a resource file .ncbirc, and put it under your home directory.
If you are using the legacy blast program, the contents of the file contains at least two variables DATA and BLASTDB:

If you are using the C++ implementation of blast program, the contents of the file contains at least one variable BLASTDB:

[BLAST]
BLASTDB="/nfs/gpfs/PZS0002/biosoftw/db/"

Upon start, BLAST will read this file to get the path information it needs during BLAST searches. Without this file, BLAST will search the working directory, or whichever directory the command is issued from.

Using BLAST

The five flavors of BLAST mentioned above perform the following tasks:

tblastn: compares a protein query sequence against a nucleotide sequence database dynamically translated in all six reading frames (both strands).

tblastx: compares the six-frame translations of a nucleotide query sequence against the six-frame translations of a nucleotide sequence database. (Due to the nature of tblastx, gapped alignments are not available with this option)

We provide local access to nr and swissprot databases. Other databases are available upon request.

Further Reading

See Also

BLAT

BLAT is a sequence analysis tool which performs rapid mRNA/DNA and cross-species protein alignments. BLAT is more accurate and 500 times faster than popular existing tools for mRNA/DNA alignments and 50 times faster for protein alignments at sensitivity settings typically used when comparing vertebrate sequences.

BLAT is not BLAST. DNA BLAT works by keeping an index of the entire genome (but not the genome itself) in memory. Since the index takes up a bit less than a gigabyte of RAM, BLAT can deliver high performance on a reasonably priced Linux box. The index is used to find areas of probable homology, which are then loaded into memory for a detailed alignment. Protein BLAT works in a similar manner, except with 4-mers rather than 11-mers. The protein index takes a little more than 2 gigabytes.

Availability & Restrictions

BLAT is available without restriction to all OSC users.

The following versions of BLAT are available at OSC:

Version

Glenn

Oakley

34

X

Usage

Set-up

To initalize the Glenn system prior to using BLAT, run the following commands:

module load biosoftw
module loat blat

Using BLAT

The main programs in the blat suite are:

gfServer – a server that maintains an index of the genome in memory and uses the index to quickly find regions with high levels of sequence similarity to a query sequence.

gfClient – a program that queries gfServer over the network, and then does a detailed alignment of the query sequence with regions found by gfServer.

blat –combines client and server into a single program, first building the index, then using the index, and then exiting.

webBlat – a web based version of gfClient that presents the alignments in an interactive fashion. (not included on OSC server)

Building an index of the genome typically takes 10 or 15 minutes. Typically for interactive applications one uses gfServer to build a whole genome index. At that point gfClient or webBlat can align a single query within few seconds. If one is aligning a lot of sequences in a batch mode then blat can be more efficient, particularly if run on a cluster of computers. Each blat run is typically done against a single chromosome, but with a large number of query sequences.

Further Reading

See Also

Bowtie

"Bowtie is an ultrafast, memory-efficient short read aligner geared toward quickly aligning large sets of short DNA sequences (reads) to large genomes. It aligns 35-base-pair reads to the human genome at a rate of 25 million reads per hour on a typical workstation. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small: for the human genome, the index is typically about 2.2 GB (for unpaired alignment) or 2.9 GB (for paired-end or colorspace alignment). Multiple processors can be used simultaneously to achieve greater alignment speed. Bowtie can also output alignments in the standard SAM format, allowing Bowtie to interoperate with other tools supporting SAM, including the SAMtools consensus, SNP, and indel callers. Bowtie runs on the command line." (http://bowtie-bio.sourceforge.net/manual.shtml)

Availability & Restrictions

Bowtie is available to all OSC users without restriction.

The following version of Bowtie are available on OSC systems:

Version

Glenn

Oakley

0.12.7

X

Usage

Set-up

To configure your environment for using PAUP, run the following command:

module load paup

Using Bowtie

On the Glenn Cluster bowtie is accessed by executing the following commands:

module load biosoftw
module load bowtie

bowtie will be added to the users PATH and can then be run with the command:

bowtie [options]* <ebwt> {-1 <m1> -2 <m2 | --12 <r> | <s>} [<hit]

Below are definitions for some of the main optional arguments:

<m1> Comma-separated list of files containing upstream mates (or the sequences themselves, if -c is set) paired with mates in <m2><m2> Comma-separated list of files containing downstream mates (or the sequences themselves if -c is set) paired with mates in <m1><r> Comma-separated list of files containing Crossbow-style reads. Can be a mixture of paired and unpaired. Specify "-" for stdin.
<s> Comma-separated list of files containing unpaired reads, or the sequences themselves, if -c is set. Specify "-" for stdin.
<hit> File to write hits to (default: stdout)

Performance:

-o/--offrate <int> override offrate of index; must be >= index's offrate
-p/--threads <int> number of alignment threads to launch (default: 1)
--mm use memory-mapped I/O for index; many 'bowtie's can share
--shmem use shared mem for index; many 'bowtie's can share

Errors

The following scripts fail due to an ftp error: make_e_coli.sh, make_a_thaliana_tair.sh, and make_c_elegans_ws200.sh. The following scripts fail to obtain all of the fasta format files prior to bowtie conversion and fail: make_galGal3.sh, make_hg18.sh, make_h_sapiens_ncbi36.sh, make_h_sapiens_ncbi37.sh, make_mm9.sh, make_m_musculus_ncbi37.sh. The follow script does not work properly on the Glenn Cluster: gen_dnamasks2colormask.pl.

Further Reading

Cambridge Structural Database

Introduction

The Cambridge Structural Database (CSD) contains complete structure information on hundreds of thousands of small molecule crystals. The Cambridge Crystallographic Data Centre (CCDC) is a suite of programs for CSD search and analysis. The Cambridge Crystallographic Data Centre Home Page has additional information.

Version

Several versions of the software package are available at OSC. Search for csd in this list:

module avail csd

There are also a very limited number of licenses for the web version WebCSD; contact oschelp@osc.edu for availability.

COMSOL

COMSOL Multiphysics (formerly FEMLAB) is a finite element analysis and solver software package for various physics and engineering applications, especially coupled phenomena, or multiphysics. owned and supported by COMSOL, Inc.

Availability & Compatibility

COMSOL is available on the Oakley and Glenn clusters.

The versions currently available at OSC are

Version

GLENN

OAKLEY

3.4

X

3.5a

X

4.0

X

4.0a

X

4.1

X

4.2

X

4.2a

X

X

4.3

X

X

4.3a

X

4.4

X

5.0

X

5.1

X

Usage

Access

COMSOL is for academic use only. To use COMSOL you will have to be added to the license server first. Please contact OSC Help to be added.

Setup

Use module avail to view available modules for a given machine. To select a particular software version, type: module load software-name.
For example: To select COMSOL version 4.2a on Oakley, type: module load comsol/42a

Batch Usage

Sample Batch Script (single processor analysis)

When you log into oakley.osc.edu or glenn.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the 4000+ processors in the computing environment, you must submit your COMSOL analysis to the batch system for execution. For example, assume that you have had a comsol script file mycomsol.m in the direcory $PBS_O_WORKDIR. In this directory you should create a batch script that would contain the following information:

See Also

CFX

CFX is a computational fluid dynamics (CFD) program for modeling fluid flow and heat transfer in a variety of applications.

Availability & Restrictions

CFX is available on both the Glenn and Oakley clusters. The following versions are available:

VERSION

GLENN

OAKLEY

14.5

X

15.0

X

Academic License Limitations

Currently, there are in total 25 ANSYS CFD base tokens and 68 HPC tokens for academic users. These base tokens are shared between FLUENT and CFX. These HPC tokens are shared with all ANSYS products we have at OSC.

A base license token will allow to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. A job using a base license token can be submitted to either Glenn or Oakley clusters. A parallel job using HPC tokens (with "ansyspar" flag) however can only be submitted to Glenn clusters due to scheduler issue. For instance, a serial CFX job with 1 core will need 1 base license token while a parallel CFX job with 8 cores will need 1 base license token and 4 HPC tokens.

Commercial License Limitations

For commercial users, there are in total 20 base license tokens and 512 HPC tokens. The base license tokens are shared betweeen FLUENT and CFX. The HPC tokens are shared among available ANSYS products (FLUENT, CFX, ICEMCFD, ANSYS Mechanical, etc.)

Usage

Access

Use of CFX requires validation. Please contact OSC Help for more information.

Set-up

CFX can only be run on the compute nodes of the Oakley and Glenn clusters. Therefore, all CFX jobs are run via the batch scheduling system, either as interactive or unattended jobs. In either case, only once a batch job has been started can the CFX module be loaded. For example, if you'd like to load CFX version 14.5 on Glenn, type:

module load fluent
cfx5

Batch Usage

Sample Usage (interactive execution)

Using the CFX GUI interactivly can be done with the following steps:

Ensure that your SSH client software has X11 forwarding enabled

Connect to either the Oakley or Glenn system

Request an interactive job. The command below will request a one-core, one-hour job. Modify as per your own needs:

qsub -I -X -l walltime=1:00:00 -l software=fluent+1

Once the interactive job has started, run the following commands to setup and start the CFX GUI:

module load fluent
cfx5

Sample Batch Script (serial execution using 1 base token)

An example of running CFX job for one-hour with an input file named "test.def" on Glenn is provided below:

Sample Batch Script (parallel execution using HPC token)

CFX can be run in parallel, but it is very important that you read the documentation in the CFX Manual on the details of how this works. You can find the CFX manuals on-line by following the "Further Reading" link at the bottom of this page.

In addition to requesting the CFX base license token (-l software=fluent+1), you need to request copies of the ansyspar license, i.e., HPC tokens. However the scheduler cannot handle two "software" flags simultaneously, so the syntax changes. The new option is -W x=GRES:fluent+1%ansyspar+[n], where [n] is equal to the number of cores you requested minus 4.

Further Reading

CDO

CDO (Climate Data Operators) is a collection of command line operator to manipulate and analyse climate and NWP model data. It is open source and released under the terms of the GNU General Public License v2 (GPL).

Availability and Restrictions

Versions

CDO is available on Oakley Cluster. The versions currently available at OSC are:

version

oakley

1.5.4

X

1.6.8

X*

* : Default Version

You can use module avail cdo to view available modules on Oakley. Feel free to contact OSC Help if you need other versions for your work.

Access

CDO is available for use by all OSC users.

Usage

Usage on Oakley

Set-up on Oakley

To load the default version of CDO module, use module load cdo. To select a particular software version, use module load cdo/version. For example, use module load cdo/1.5.4 to load CDO version 1.5.4 on Oakley.

Further Reading

CUDA

CUDA™ (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by Nvidia that enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU).

Availability and Compatability

CUDA is available on Ruby, Oakley, and Glenn Clusters. The versions currently available at OSC are

Version

Glenn

Oakley

Ruby

2.3

X

3.0

X

3.1

X

4.0

X

4.1.28

X

4.2.9

X

5.0.35

X

X

5.5

X

X

X

6.0.37

X

6.5.14

X

Usage

Access

CUDA is available for use by all OSC users.

Setup

Use module avail to view available modules for a given machine. To load the appropriate CUDA module, type: module load software-name.
For example: To select CUDA version 4.1.28 on Oakley, type: module load cuda/4.1.28

GPU Computing SDK

The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. On Oakley, the SDK binaries are located in $CUDA_HOME/bin/linux/release ($CUDA_HOME is an environment variable set when you load the module).

Compiling CUDA Code

One can type module show cuda/version-number to view the list of environment variables.
To compile a cuda code contained in a file, let say mycudaApp.cu, the following could be done after loading the appropriate CUDA module:nvcc -o mycudaApp mycudaApp.cu
This will create an executable by name mycudaApp

Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

Debugging CUDA code

cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

Detecting memory access errors

CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

Batch Usage

Following are the sample batch scripts for requesting GPU nodes on Glenn and Oakley. Notice that only the second line is different in the two batch scripts. In case of Oakley one can specify the number of GPUs required.

See Also

NCL

NCL (The NCAR Command Language), is a free interpreted language designed specifically for scientific data processing and visualization. It is a product of the Computational & Information Systems Laboratory at the National Center for Atmospheric Research (NCAR) and sponsored by the National Science Foundation. NCL has robust file input and output: it can read and write netCDF-3, netCDF-4 classic, HDF4, binary, and ASCII data, and read HDF-EOS2, GRIB1, and GRIB2. The graphics are based on NCAR Graphics.

Availability and Restrictions

Versions

NCL/NCAR Graphics is available on Glenn and Oakley Cluster. The versions currently available at OSC are:

version

glenn

oakley

ruby

5.0.0

X*

6.0.0

X

6.2.1

X*

* : Default Version

You can use module avail ncl or module avail ncarg to view available NCL/NCAR Graphics modules. Feel free to contact OSC Help if you need other versions for your work.

Access

NCL/NCAR Graphics is available for use by all OSC users.

Usage

Usage on Glenn

Set-up on Glenn

To load the default version, use module load ncarg. To select a particular version, use module load ncarg-version. For example, use module load ncarg-5.0.0 to load NCAR Graphics 5.0.0 on Glenn Cluster.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named agex06.f .

In order to run it via the batch system, submit the job.txt file with the following command:

qsub job.txt

Usage on Oakley

Set-up on Oakley

To load the default version, use module load ncl or use module load ncarg. To select a particular version, use module load ncl/version or module load ncarg/version. For example, use module load ncl/6.2.1 to load NCL version 6.2.1 on Oakley.

Interactive Batch Session

For an interactive batch session on Oakley, one can run the following command:

qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00

which gives you 12 cores ( -l nodes=1:ppn=12) with 1 hour ( -l walltime=1:00:00). You may adjust the numbers per your need.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named omega.ncl

Further Reading

Clustal W

Clustal W is a general purpose multiple sequence alignment program for DNA or proteins.It produces biologically meaningful multiple sequence alignments of divergent sequences. It calculates the best match for the selected sequences, and lines them up so that the identities, similarities and differences can be seen.

Availability & Restrictions

Clustal W is available without restriction to all OSC users.

The following versions of Clustal W are available on OSC systems:

Version

Glenn

Oakley

1.8.3

X

2.1

X

Usage

Set-up

Setup prior to use of Clustal W is dependent on the system you are using. On the Glenn system, first load the biosoftware module, then load the clustalw module:

module load biosoftw
module load clustalw

On the Oakley system, just load the clustalw module directly:

module load clustalw

Using Clustal W

Once the clustalw module has been loaded, the commands are available for your use. On the Glenn system, the command is

clustalw

On the Oakley system, the command is

clustalw2

The options can be listed interactively by typing clustalw -help or clustalw -check on the command-line.

Further Reading

See Also

EMBOSS

EMBOSS is "The European Molecular Biology Open Software Suite". EMBOSS is a free Open Source software analysis package specially developed for the needs of the molecular biology (e.g. EMBnet) user community. The software automatically copes with data in a variety of formats and even allows transparent retrieval of sequence data from the web. Also, as extensive libraries are provided with the package, it is a platform to allow other scientists to develop and release software in true open source spirit. EMBOSS also integrates a range of currently available packages and tools for sequence analysis into a seamless whole.

Within EMBOSS you will find around hundreds of programs (applications) covering areas such as:

Batch Usage

Further Reading

See Also

Boost

Boost is a set of C++ libraries that provide helpful data structures and numerous support functions in a wide range of aspects of programming, such as, image processing, gpu programming, concurrent programming, along with many algorithms. Boost is portable and performs well on a wide variety of platforms.

Availability & Restrictions

Boost is available without restriction to all OSC Users

The following version of Boost are available on OSC systems:

Version

Glenn

Oakley

RUBY

1.56

X*

X*

Note: This installation is compatible with intel and gnu compilers, but not pgi. This installation contains all of the boost libraries except mpi.

Usage

Set-up

Initalizing the system for use of the Boost library is independent of the compiler you are using. To load the boost module run the following command:

module load boost

Building With Boost

The following environment variables are setup when the Boost library is loaded:

Variable

Use

$BOOST_CFLAGS

Use during your compilation step for C++ programs on Oakley.

$BOOST_LIBS

Use during your link step on Oakley.

Below is a set of example commands used to build and run a file called example2.cpp on Oakley. First copy the example2.cpp and jayne.txt with the following commands.

Further Reading

FFTW

FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data. It is portable and performs well on a wide variety of platforms.

Availability & Restrictions

FFTW is available without restriction to all OSC Users

The following versions of FFTW are available on OSC systems:

Version

Glenn

Oakley

RUBY

2.1

X**

2.1.5

X

X

3.1.2

X

3.3

X*

X**

3.3.1

X*

3.3.2

X**

3.3.4

X

*gnu versions only, **intel version only

Usage

Set-up

Initalizing the system for use of the FFTW library is dependent on both the system you are using and the compiler you are using. Some of the installations on Glenn also depend on the version of MPI you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load the default FFTW3 library, run the following command:

module load fftw3

Building With FFTW

The following environment variables are setup when the FFTW library is loaded:

Variable

Use

$FFTW3_CFLAGS

Use during your compilation step for C programs on Oakley.

$FFTW3_FFLAGS

Use during your compilation step for Fortran programson Oakley.

$FFTW3_LIBS

Use during your link step on Oakley for the sequential version of the library.

$FFTW3_LIBS_OMP

Use during your link step on Oakley for the OpenMP version of the library.

$FFTW3_LIBS_MPI

Use during your link step on Oakleyfor the MPI version of the library.

$FFTW_CFLAGS

Use during your compilation step for C programs on Glenn.

$FFTW_FFLAGS

Use during your compilation step for Fortran programs on Glenn.

$FFTW_LIBS

Use during your link step on Glenn for the sequential version of the library.

$FFTW_MPI_LIBS

Use during your link step on Glenn for the MPI version of the library.

below is a set of example commands used to build a file called my-fftw.c on Oakley.

Further Reading

See Also

FLUENT

FLUENT is a state-of-the-art computer program for modeling fluid flow and heat transfer in complex geometries.

Availability & Restrictions

FLUENT is available on both the Glenn and Oakley clusters. The following versions are available:

Version

OAKLEY

GLENN

13

X

X

14

X

14.5

X

A base license token will allow Fluent to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial FLUENT job with 1 core will need 1 base license token while a parallel FLUENT job with 8 cores will need 1 base license token and 4 HPC tokens.

Academic License Limitations

Currently, there are in total 25 base license tokens and 68 HPC tokens for academic users. These HPC tokens are shared between FLUENT and ANSYS. A job using a base license token can be submitted to either Glenn or Oakley clusters. A parallel job using HPC tokens (with "ansyspar" flag) however can only be submitted to Glenn clusters due to scheduler issue.

NDEMC License Limitations

For NDEMC users, there are in total 10 base license tokens and 512 HPC tokens. These HPC tokens are shared among available ANSYS products (FLUENT, CFX, ICEMCFD, ANSYS Mechanical, etc.) All jobs should be submitted to Oakley cluster.

Usage

Access

Use of FLUENT for academic purposes requires validation. To obtain validation please complete and return the "Academic Agreement to Use FLUENT." This can be obtained from your site consultant or from the files fluent.pdf located in the Academic Agreement Forms.

Set-up

FLUENT can only be run on the compute nodes of the Oakley and Glenn clusters. Therefore, all FLUENT jobs are run via the batch scheduling system, either as interactive or unattended jobs. In either case, only once a batch job has been started can the FLUENT module be loaded with the following command:

module load fluent

Additionally, you can use the following command to see the other FLUENT versions you can load on the cluster you are using:

module avail

Batch Usage

Sample Usage (interactive execution)

Using the FLUENT GUI interactivly can be done with the following steps:

Ensure that your SSH client software has X11 forwarding enabled

Connect to either the Oakley or Glenn system

Request an interactive job. The command below will request a single-core, one hour job. Modify as per your own needs:

qsub -I -X -l walltime=1:00:00 -l software=fluent+1

Once the interactive job has started, run the following commands to setup and start the FLUENT GUI:

module load fluent
fluent

Sample Batch Script (serial execution using 1 base token)

Running FLUENT for five-hours on an input file "run.input" on either Glenn or Oakley:

Sample Batch Script (parallel execution using HPC token)

FLUENT can be run in parallel, but it is very important that you read the documentation in the FLUENT Manual on the details of how this works. You can find the Fluent manuals on-line by following the "Further Reading" link at the bottom of this page, or clicking the "Manuals" link in the left panel of any of the software pages.

In addition to requesting the FLUENT base license token (-l software=fluent+1), you need to request copies of the ansyspar license, i.e., HPC tokens. However the scheduler cannot handle two "software" flags simultaneously, so the syntax changes. The new option is -W x=GRES:fluent+1%ansyspar+[n], where [n] is equal to the number of cores you requested minus 4.

Parallel jobs have to be submitted on Glenn via the batch system. An example of the batch script follows:

Further Reading

See Also

Fitmodel

"'Fitmodel' estimates the parameters of various codon-based models of substitution, including those described in Guindon, Rodrigo, Dyer and Huelsenbeck (2004). These models are especially useful as they accommodate site-specific switches between selection regimes without a priori knowledge of the positions in the tree where changes of selection regimes occurred.

The program will ask for two input files: a tree file and a sequence file. The tree should be unrooted and in NEWICK format. The sequences should be in PHYLIP interleaved or sequential format. If you are planning to use codon-based models, the sequence length should be a multiple of 3. The program provides four types of codon models: M1, M2, M2a, and M3 (see PAML manual). Moreover, M2, M2a and M3 can be combined with 'switching' models (option 'M'). Two switching models are implemented: S1 and S2. S1 constraints the rates of changes between dN/dS values to be uniform (e.g., the rates of changes between negative and positive selection is constrained to be the same as the rate of change between neutrality and positive selection) while S2 allows for differents rates of change between the different classes of dN/dS values.

If you are using a 'switching' model, 'fitmodel' will output file with the following names: your_sequence_file_trees_w1, your_sequence_file_trees_w2, your_sequence_file_trees_w3 and your_sequence_file_trees_wbest. The w1, w2 and w3 files give the estimated tree with probabilities of w1, w2, and w3 (three maximum likelihood dN/dS ratio estimates) calculated on each edge of the tree and for each site. Hence, the first tree in one of these files reports the probabilities calculated at the first site of the alignment. Instead of probabilities, the wbest file allows you to identify which of the tree dN/dS is the most probable on any give edge, at any given site. A branch with label 0.0 means that w1 is the most probable class, 0.5 indicates the w2 is the most probable and 1.0 means that w3 has the highest posterior probability." (README.txt)

Availability & Restrictions

Fitmodel is available to all OSC users without restriction.

The following versions of fitmodel are available on OSC systems:

Version

Glenn

Oakley

0.5.3

X

Usage

Set-up

On the Glenn Cluster fitmodel is accessed by executing the following commands:

module load biosoftw
module load fitmodel

Using fitmodel

fitmodel will be added to the users PATH and can then be run with the following command:

Further Reading

GAMESS

Introduction

The General Atomic and Molecular Electronic Structure System (GAMESS) is a flexible ab initio electronic structure program. Its latest version can perform general valence bond, multiconfiguration self-consistent field, Möller-Plesset, coupled-cluster, and configuration interaction calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs well on open shell and excited state systems and can model relativistic effects. The GAMESS Home Page has additional information.

Availability

GAMESS is available on the Glenn and Oakley Clusters.

Version

Different versions of GAMESS are available on the Glenn and Oakley clusters.

VERSION

GLENN

OAKLEY

24 MAR 2007 (R3)

X

12 JAN 2009 (R3)

X*

1 MAY 2013 (R1)

X*

* - Default Version

Usage

Users are not required to complete a license agreement.

Location

The root of the GAMESS directory tree is/usr/local/gamess/

Execution

GAMESS usage is controlled via modules. Load one of the GAMESS modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

Further Reading

Q-Chem

Introduction

Q-Chem is a general purpose ab initio electronic structure program. Its latest version emphasizes Self-Consistent Field, especially Density Functional Theory, post Hartree-Fock, and innovative algorithms for fast performance and reduced scaling calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs reasonably well within its single reference paradigm on open shell and excited state systems. The Q-Chem Home Page has additional information.

Availability

Q-Chem is available on the Oakley Cluster.

Version

These versions of Q-Chem are available on the Glenn and Oakley clusters.

VERSION

GLENN

OAKLEY

4.0.1

X*

4.2.2

X

4.3

X

* - Default Version

Usage

Users are not required to complete a license agreement.

Location

The root of the Q-Chem directory tree is/usr/local/qchem/

Execution

Q-Chem usage is controlled via modules. Load one of the Q-Chem modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

Further Reading

Gaussian

Gaussian is the most popular general purpose electronic structure program. Its latest version, g09, can perform density functional theory, Hartree-Fock, Möller-Plesset, coupled-cluster, and configuration interaction calculations among others. Geometry optimizations, vibrational frequencies, magnetic properties, and solution modeling are available. It performs well as black-box software on closed-shell ground state systems.

Availability & Restrictions

Gaussian is available to all users that sign a software license agreement, found here. Return the signed form to OSC by email, mail, or fax for access.

The following versions of Gaussian are available on OSC systems:

Version

Glenn

Oakley

g03d01

X*

g03e01

X*

g09a01

X

g09b01

X

X

g09c01

X

X

g09d01

X

* Users are highly encouraged to switch to the g09 version as g03 is no longer supported by Gaussian and has significant limitations on Glenn.

Usage

Set-up

To initalize your environement for use of Gaussian 09, run the following command on Glenn:

module load g09

On Oakley, the command is

module load gaussian

To see other available versions, run the following command:

module avail

Using Gaussian

To execute Gaussian, simply run the Gaussian binary with the input file on the command line:

g09 < input.com

When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen via 'qpeek jobid' when the job is running in a batch queue. Alternatively, Gaussian can be invoked without file redirection:

g09 input.com

in which case the output file will be named 'input.log'; in this form the output cannot be seen via 'qpeek jobid' when the job is running in a batch queue.

Note: OSC does not have a functional distributed parallel version (LINDA) of Gaussian. Parallelism of Gaussian at OSC is only via shared memory. Consequently, do not request more than one node for Gaussian jobs on OSC's clusters.

Further Reading

GLPK

GLPK (GNU Linear Programming Kit) is a set of open source LP (linear programming) and MIP (mixed integer problem) routines written in ANSI C, which can be called from within C programs.

Availability & Restrictions

GLPK is available to all OSC users without restriction.

The following versions are available on OSC systems:

version

glenn

oakley

4.48

X

Usage

Setup

To set up your environment for using GLPK on Oakley, run the following command:

module load glpk

Compiling and Linking

To compile your C code using GLPK API routines, use the environment variable $GLPK_CFLAGS provided by the module:

gcc $GLPK_CFLAGS -c my_prog.c

To link your code, use the variable $GLPK_LIBS:

gcc my_prog.o $GLPK_LIBS -o my_prog

glpsol

Additionally, the GLPK module contains a stand-alone LP/MIP solver, which can be used to process files written in the GNU MathProg modeling language. The solver can be invoked using the following command syntax:

Further Reading

Gnuplot

Introduction

Gnuplot is a portable command-line driven data and function plotting utility. It was originally intended to allow scientists and students to visualize mathematical functions and data.

Gnuplot supports many types of plots in two or three dimensions. It can draw using points, lines, boxes, contours, vector fields surfaces and various associated text. It also supports various specialized plot types.

Further Reading

GPU-Blast

GPU-BLAST is an accelerated version of the popular NCBI-BLAST (www.ncbi.nlm.nih.gov) that uses general-purpose graphics processing unit (GPU). In comparison to the sequential NCBI-BLAST, GPU-BLAST is nearly four times faster, while producing identical results.

Availability & Restrictions

GPU-BLAST is available without restriction to all OSC users.

The following versions of GPU-BLAST are available on OSC systems:

VERSION

NCBI-Blast Version

GLENN

OAKLEY

1.0

2.2.24+

X

1.1

2.2.25+

X

1.1

2.2.26+

X

Usage

Set-up

To set up the environment for gpu-blast on Oakley cluster, do the following:

Further Reading

See Also

GROMACS

GROMACS is a versatile package of molecular dynamics simulation programs. It is primarily designed for biochemical molecules, but it has also been used on non-biological systems. GROMACS generally scales well on OSC platforms. Versions after 4.6 include GPU acceleration.

Availability & Restrictions

GROMACS is available to all OSC users without restriction.

The following versions of GROMACS are available on OSC systems:

Version

Glenn

Oakley

3.3.1

X

3.3.3

X

4.0.3

X

4.5.4

X

4.5.5

X

X

4.6.3

X

Usage

Set-up

Initalizing GROMACS on both Glenn and Oakley is done by loading a gromacs module:

module load gromacs

To see other available versions, run the following command:

module avail

Using GROMACS

To execute a serial GROMACS program interactively, simply run it on the command line, e.g.:

pdb2gmx

Parallel GROMACS programs should be run in a batch environment with mpiexec, e.g.:

mpiexec mdrun_mpi_d

Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision.

Batch Usage

Sample batch scripts and GROMACS input files are available here:

/nfs/10/srb/workshops/compchem/gromacs/

This simple batch script for Oakley demonstrates some important points:

Further Reading

HDF5

HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

Availability & Restrictions

HDF5 is available without restriction to all OSC users.

Different versions of HDF5 are available on the various OSC clusters.

Version

Glenn

Oakley

Ruby

1.6.4

X

1.6.5

X*

1.8.7

X

1.8.8

X

1.8.12

X*

1.8.13

X*

X

* - Default version

Usage

Set-up

To use the HDF5 library, first run the following command:

module load hdf5

or

module load hdf5-serial

Building With HDF5

The HDF5 library provides the following variables for use at build time:

Variable

Use

$HDF5_C_INCLUDE

Use during your compilation step for C programs

$HDF5_CPP_INCLUDE

Use during your compilation step for C++ programs (serial version only)

$HDF5_F90_INCLUDE

Use during your compilation step for FORTRAN programs

$HDF5_C_LIBS

Use during your linking step programs

$HDF5_F90_LIBS

Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

Further Reading

See Also

HMMER

Profile hidden Markov models (profile HMMs) can be used to do sensitive database searching using statistical descriptions of a sequence family's consensus. HMMER uses profile HMMs, and can be useful in situations like:

if you are working with an evolutionarily diverse protein family, a BLAST search with any individual sequence may not find the rest of the sequences in the family.

the top hits in a BLAST search are hypothetical sequences from genome projects.

your protein consists of several domains which are of different types.

HMMER (pronounced 'hammer', as in a more precise mining tool than BLAST) was developed by Sean Eddy at Washington University in St. Louis.

HMMER is a very cpu-intensive program and is parallelized using threads, so that each instance of hmmsearch or the other search programs can use all the cpus available on a node. HMMER on OSC clusters are intended for those who need to run HMMER searches on large numbers of query sequences.

Availability & Restrictions

HMMER is available to all OSC users without restriction.

The following versions of HMMER are available on OSC systems:

Version

Glenn

Oakley

2.3.2

X

3.0

X

Usage

Set-up

To use HMMER on Glenn, first run the following commands:

module load biosoftw
module load hmmer

To use HMMER on Oakley, first run the following command:

module load hmmer

Using HMMER

Once the hmmer module is loaded, the following commands will be available for your use:

Single sequence queries: new to HMMER3
phmmer Search a sequence against a sequence database. (BLASTP-like)
jackhmmer Iteratively search a sequence against a sequence database. (PSIBLAST-like)
Replacements for HMMER2’s functionality
hmmbuild Build a profile HMM from an input multiple alignment.
hmmsearch Search a profile HMM against a sequence database.
hmmscan Search a sequence against a profile HMM database.
hmmalign Make a multiple alignment of many sequences to a common profile HMM.
Other utilities
hmmconvert Convert profile formats to/from HMMER3 format.
hmmemit Generate (sample) sequences from a profile HMM.
hmmfetch Get a profile HMM by name or accession from an HMM database.
hmmpress Format an HMM database into a binary format for hmmscan.
hmmstat Show summary statistics for each profile in an HMM database.

If you need to know options for a command, type the command name followed by "-h", for example:

Access

The Intel Compilers areavailable to all OSC users without restriction.

Usage

Usage on Glenn

Set-up on Glenn

After you ssh to Glenn, the default version of Intel compilers will be loaded for you automatically. To configure your environment for a particular Intel compilers version, use module swap intel-compilers-10.0.023 intel-compilers-version . For example, use module swap intel-compilers-10.0.023 intel-compilers-11.1to load Intel compilers 11.1 on Glenn.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results .
Below is the example batch script ( job.txt ) for a serial run:

Usage on Oakley

Set-up on Oakley

After you ssh to Oakley, the default version of Intel compilers will be loaded for you automatically. To configure your environment for a particular Intel compilers version, usemodule swap intel/12.1.4.39 intel/version . For example, use module swap intel/12.1.4.39 intel/14.0.0.080to load Intel compilers 14.0.0.080 on Oakley.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results .
Below is the example batch script ( job.txt ) for a serial run:

Usage on Ruby

Set-up on Ruby

After you ssh to Ruby, the default version of Intel compilers will be loaded for you automatically. To configure your environment for a particular Intel compilers version, usemodule swap intel/15.0.0 intel/version . For example, use module swap intel/15.0.0 intel/15.0.1to load Intel compilers 15.0.1 on Ruby.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results .
Below is the example batch script ( job.txt ) for a serial run:

Further Reading

See Also

JasPer

Introduction

JasPer is an extensible open source utility designed for the manipulation, conversion, compression and decompression of digital images. Currently supported image formats include bmp, jp2, jpc, jpg, pgx, mif and ras.

Availability

Version 1.900.1 is available on the Oakley cluster.

Usage

In order to use the JasPer library, you must load the JasPer module into your environment. To load the JasPer module, type the following command:

module load jasper

Useful commands

In addition to the JasPer libraries, a few default applications are also provided when the JasPer module is loaded.

To convert an image from one format to another, use the jasper command.

jasper [options]

For a comprehensive list of options available for this command, type

jasper --help

To compare particular qualities of two images, use the imgcmp command.

imgcmp -f [file 1] -F [file 2] -m [metric]

For a full list of acceptable metric arguments, type

imgcmp --help

To view a sequence of one or more images, use the jiv command.

jiv image

Alternatively, if you would like to view multiple images in sequence you can use the --wait option to specify the amount of time between the display of each image, creating a slideshow. The command below will display each image for 5 seconds before displaying the next image.

jiv --wait 5 [image1 image2 image3 ...]

For a full list of options, type

jiv --help

Further Reading

Jmol

Jmol is a simple molecular visualization program. It can read many file formats and can write various formats. Animations of normal modes and simulations are implemented. Jmol is an OpenScience Java application.

GPU Usage

On the Glenn and Oakley OSC clusters, LAMMPS can run on GPU's. See the sample scripts for details. The following text is specific to the Glenn cluster and demonstrates using GPU's to speed up certain pair_style's. This example shows how to load and run such a GPU-enabled version. Here is a sample PBS script to run. It uses one node and two GPU's for the computation.

Please note that the you cannot run more than two threads per node. More than two threads will cause the application to hang, since there are only two GPU's per node. The LAMMPS input and the pbs script must match up with the number of GPU's being used.

Please refer to the documentation for more pair_style's that can be used in such simulations.

Further Reading

LS-DYNA

LS-DYNA is a general purpose finite element code for simulating complex structural problems, specializing in nonlinear, transient dynamic problems using explicit integration. LS-DYNA is one of the codes developed at Livermore Software Technology Corporation (LSTC).

Availability & Restrictions

LS-DYNA is available to OSC users that have filled out the appropriate academic license agreement form found here.

Serial (smp -for single node jobs) and parallel (mpp -for multipe node jobs) versions of LS-DYNA solvers are installed at OSC. You may check the available versions on Oakley using module avail dyna. (On Glenn, use module avail lsdyna for smp solvers, and use module avail mpp for mpp solvers.) In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp971_d_R5 is the mpp solver with double precision on Glenn. The following versions are currently available on OSC cluster systems (including smp/mpp, single/double precision):

Version

Glenn

Oakley

971-R4.2.1

X

971-R5

X

X

971-R5.1.1

X

971-R7.0.0

X

X

Using LS-DYNA

LS-DYNA can be used via the batch system, by requesting the computing resources and launching solvers (smp/mpp) in a script file. See the following example.

Batch Usage

To use LS-DYNA via the batch system, here's what you need to do:

1) copy your input files (explorer.k in the example below) to your work directory at OSC
2) create a batch script, similar to the following file, saved as 'test.job':

when the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR).

Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

See Also

LS-PrePost

Introduction

LS-PrePost is an advanced pre and post-processor that is delivered free with LS-DYNA. The user interface is designed to be both efficient and intuitive. LS-PrePost runs on Windows, Linux, and Unix utilizing OpenGL graphics to achieve fast rendering and XY plotting. The latest builds can be downloaded from LSTC's FTP Site.

The prefered way of accessing LS-Prepost is through OnDemand's Glenn desktop application. This gives you a preconfigured enviornment with GPU acceleration enabled.

Availability

LS-PrePost is available on both Glenn and Oakley clusters. A module was created for LS-Prepost 4.0 on Oakley, and can be loaded with 'module load lspp'. All other versions can be loaded with the corresponding dyna modules.

Version

Glenn

Oakley

v2.3

X

v3.0

X

v3.2

X

v4.0

X

Usage

Running LS-PrePost on Oakley through OnDemand's Glenn Destkop

Below are instructions on running LS-PrePost on Oakley through Glenn's OnDemand desktop interface with GPU acceleration enabled. To run LS-PrePost with a slower X-tunneling procedure, see the specified instructions below.

1) Log in to OnDemand with your HPC username and password.

2) Launch the Glenn Desktop from "Apps" menu.

3) Open a Terminal window (Applications > Accessories > Terminal)

4) Type the following command to connect to oakley:

ssh -X username@oakley.osc.edu

* Where "username" is your username.

5) Once logged in to Oakley, submit an interactive job with the following command:

qsub -X -I -l nodes=1:ppn=12:gpus=2:vis -l walltime=hh:mm:ss

* pick a walltime that is close to the amount of time you will need to for working in the GUI application.

6) Once your job starts, make a note of the hostname for the compute node your job is running on. You can find this information by typing the following command:

hostname

7) Open another Terminal window, and type the following commands:

module load VirtualGL
vglconnect username@job-hostname

* job-hostname is the information you found in step 6; your command might look something like this, for example:

vglconnect ahahn@n0656.ten.osc.edu

You'll be asked a password to connect, which should be your HPC password.

8) Now, you should be connected to your running job's GPU node. Run the following commands to launch LS-PrePost version 4.0:

At startup LS-PrePost displays a graphical interface for model generation and results post-processing.

Running LS-PrePost on Oakley or Glenn with X-11 forwarding

The following procedure will result in a much slower and GUI interface, but may be useful if the above instructions are not working. It can be done completely from the command line, with no need for logging into OnDemand. You may need to edit your terminal settings to enabled x11 forwarding.

Documentation

User-Defined Material for LS-DYNA

This page describes how to specify user defined material to use within LS-DYNA. The user-defined subroutines in LS-DYNA allow the program to be customized for particular applications. In order to define user material, LS-DYNA must be recompiled.

Availability

LS-DYNA with user defined material models are only available on the Glenn Cluster.

Version

The following versions on Glenn are available for use with user defined material models:

smp version:

lsdyna971_d_R4.2.1_umat

mpp versions:

mpp971_s_7600.2.398

mpp971_d_7600.2.398

mpp971_s_R3.1

mpp971_d_R3.1

mpp971_d_R4.2

Usage

The first step to running a simulation with user defined material is to build a new executable. The following is an example done with solver version mpp971_s_7600.2.398.

When you log into the Glenn system, load mpp971_s_7600.2.398 with the command:

module load mpp971_s_7600

Next, copy the mpp971_s_7600 object files and Makefile to your current directory:

cp /usr/local/lstc/mpp971_s_7600.2.398/usermat/* $PWD

Next, update the dyn21.f file with your user defined material model subroutine. Please see the LS-DYNA User's Manual (Keyword version) for details regarding the format and structure of this file.

Once your user defined model is setup correctly in dyn21.f, build the new mpp971 executable with the command:

make

To execute a multi processor (ppn > 1) run with your new executable, execute the following steps:

1) move your input file to a directory on Glenn (pipe.k in the example below)

2) copy your newly created mpp971 executable to this directory as well

See Also

MATLAB

Introduction

MATLAB is a technical computing environment for high-performance numeric computation and visualization. MATLAB integrates numerical analysis, matrix computation, signal processing, and graphics in an easy-to-use envionment where problems and solutions are expressed just as they are written mathematically--without traditional programming.

Versions

Different versions of MATLAB are available on the various OSC clusters.

Version

Glenn

Oakley

Ruby

2011a

X

2011b

X

X

2012a

X*

X

2012b

X

2013a

X*

2013b

X

2014a

X

X*

* - Default version

Availability

Ohio State University students/faculty

MATLAB is available on the Glenn Cluster and the Oakley Cluster. All Ohio State users must be added to the license server before using MATLAB. Please contact OSC Help to be granted access.

Non Ohio State University students/faculty

OSC cannot provide MATLAB licenses for academic use to students and faculty outside of Ohio State at this time. However, if you have your own MATLAB license, including the Parallel Computing Toolbox, you will be able to connect to the MATLAB Distributed Computing Server on Oakley via your MATLAB interface. See HOW-TO: Configure the MATLAB Parallel Computing Toolbox.

Use of MATLAB requires validation. See the Software Form page for the software agreement form for MATLAB for non-OSC users.

Usage

Initialization

To load the default version of MATLAB into your environment, you can use the following command.

module load matlab

To load a specific version of MATLAB on your current system, you can use variations of the previous command. The command format expected by each system differs slightly, thus the invocation of a particular version of MATLAB depends on the system you wish to run on.

For example, if you’d like to load MATLAB 2012a on Glenn, type:

module load matlabR2012a

If you’d like to load MATLAB 2012a on Oakley, type:

module load matlab/R2012a

For a list of available MATLAB versions and the format expected on Glenn, type:

module avail matlab

For a list of all available MATLAB versions and the format expected on Oakley, type:

module spider matlab

Running MATLAB

The following command will start an interactive, command line version of MATLAB on either cluster system:

matlab -nodisplay

If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command “matlab”.

For more information about the matlab command usage, type “matlab –h” for a complete list of command line options.

The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

Running in Batch

MATLAB should be run via the batch system for any scripts that are computationally intensive, long running, or memory intensive.

Interactive Batch Jobs

To run an interactive batch job using the command line version of MATLAB, you can use the following command:

qsub –I –X –l nodes=n:ppn=p –l walltime=hh:mm:ss

After submitting your job, you will be logged on to one of the compute nodes. Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above. The –X flag enables X-11 forwarding on the compute node, so you can use the MATLAB GUI if you choose.

Non-interactive Batch Jobs

In order to run MATLAB non-interactively via the batch system, you will require a batch submission script and a MATLAB M-file containing the script that will be run via the batch system. You can create both of these files using any text editor you like in a working directory on the system of your choice.

Below is an example batch submission script and a simple M-file. The batch script runs the M-file via the batch system.

Parallel Processing in MATLAB

Multithreading

Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled.

The use of multithreading differs slightly depending on which cluster you'd like to run your jobs on. On Oakley, the system will use a number of threads equal to the number of cores you request. Therefore, if you request nodes=1:ppn=4, your job will only spawn four threads. However, on the Glenn cluster the system will use a number of threads equal to the number of cores on one node. You must, therefore, request nodes=1:ppn=8 otherwise your jobs may be killed.

Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may request nodes=1:ppn=1 and include the option "-singleCompThread" when running MATLAB.

Further Reading

See Also

Maven (Apache)

Apache Maven is a plugin-based build automation tool, similar in purpose to GNU Make or Apache Ant. It is most commonly used with Java projects, but also supports other languages such as .NET and C/C++.

Availability & Compatibility

Version

Glenn

Oakley

3.0.4

X

Restrictions

There are no restrictions for this software; any OSC user may make use of the Apache Maven.

Usage

Set-up

To use Maven, load the the "maven" module with the following command.

module load maven

To test that the install worked correctly, run "mvn --version". You should see output similar to that shown below:

Cairo

Cairo is a 2D graphics library with support for multiple output devices. It is designed to produce consistent output on all output media while taking advantage of display hardware acceleration when available.

Availability and Restrictions

Versions

The following versions of Cairo are available on OSC clusters:

VERSION

OAKLEY

1.12.18

X*

*: Default Version

You can use module avail cairo to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Cairo is available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of Cairo on Oakley, run the following command: module load cairo . The default version will be loaded. To select a particular Cairo version, use module load cairo/version . For example, use module load cairo/1.12.18 to load Cairo 1.12.18 on Oakley.

Further Reading

FreeType

FreeType is a freely available software library to render fonts. It is written in C, designed to be small, efficient, highly customizable, and portable while capable of producing high-quality output (glyph images) of most vector and bitmap font formats.

Availability and Restrictions

Versions

The following versions of FreeType are available on OSC clusters:

VERSION

OAKLEY

2.5.5

X*

*: Default Version

You can use
module avail freetype
to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

FreeType is available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of FreeType on Oakley, run the following command:
module load freetype
. The default version will be loaded. To select a particular FreeType version, use
module load freetype/version
. For example, use
module load freetype/2.5.5
to load FreeType 2.5.5 on Oakley.

MKL - Intel Math Kernel Library

Availability & Compatibility

OSC supports single-process use of MKL for LAPACK and BLAS levels one through three. For multi-process applications, we also support the ScaLAPACK, FFTW2, and FFTW3 MKL wrappers. MKL modules are available for the Intel, gnu, and PGI compilers.

Version

Glenn

Oakley

Ruby

Statewide Software

10.0.3

X

10.3.0

X

X

X

11.2.0

X

Restrictions

All OSC users may use the Intel MKL for academic purposes.

Usage

Set-up

To configure your environment for the Intel MKL, load the “mkl” module:

module load mkl

This step is required for both building and running MKL applications. Exception: The "mkl" module is usually not needed when using the Intel compilers; just use the "-mkl" flag on the compile and link steps.

Building With MKL (Oakley)

The Oakley MKL module will automatically configure your environment to locate the appropriate include files. When linking, however, you have a variety of options to choose from depending on your needs. These options come in the form of environment variables defined by the module. Which variable you include when linking will determine what particular configuration of MKL you get.

Environment Variable

Function

$MKL_CFLAGS

Include flags for C

$MKL_FFLAGS

Include flags for Fortran

$MKL_LIBS

Use multithreaded MKL with 32-bit integers. This is the standard configuration.

$MKL_LIBS_INT64

Use multithreaded MKL with 64-bit (“ILP64”) integers.

$MKL_LIBS_SEQ

Use single-threaded (“SEQuential”) MKL with with 32-bit integers.

$MKL_LIBS_SEQ_INT64

Use single-threaded MKL with 64-bit integers.

Notes:

64-bit integers are not supported by the FFTW2 wrappers. (See below.)

The Intel compilers are specially configured for Intel MKL. If you are using this toolchain, you can use the “-mkl” option when linking in place of $MKL_LIBS.

The default, multithreaded MKL libraries will automatically run in a single thread if they are called from an OpenMP parallel region. If you want to force single-threaded behavior throughout your program, choose one of the “_SEQ” variables from the list above.

Fortran 95 BLAS/LAPACK Wrappers

To compile Fortran 95 programs with modules, add the $MKL_F95FLAGS variable to your compilation step. If you need 64-bit integers, use the $MKL_F95_FLAGS_INT64 instead. When linking, you will also need to use $MKL_F95LIBS (or $MKL_F95LIBS_INT64 if using 64-bit integers). These variables will allow your programs to use the BLAS and LAPACK wrappers for MKL.

FFTW Wrappers

A number of “wrappers” are provided in the form of environment variables that allow you to use FFTW APIs with MKL. Variables ending in FLAGS should be included with your compilation step, while variables ending in LIBS should be included in your linking step.

Environment Variable

Function

$MKL_FFTW_CFLAGS

Compile variable for C programs.

$MKL_FFTW_FFLAGS

Compile variable for Fortran programs.

$MKL_FFTW2_D_CLIBS

Linking variable for double-precision FFTW2 C programs.

$MKL_FFTW2_D_FLIBS

Linking variable for double-precision FFTW2 Fortran programs.

$MKL_FFTW2_S_CLIBS

Linking variable for single-precision FFTW2 C programs.

$MKL_FFTW2_S_FLIBS

Linking variable for single-precision FFTW2 Fortran programs.

$MKL_FFTW3_CLIBS

Linking variable for FFTW3 C programs.

$MKL_FFTW3_FLIBS

Linking variable for FFTW3 Fortran programs.

Notes:

FFTW3 is the default wrapper and should be picked up by your linker automatically. If you would prefer to use FFTW2, use the appropriate linking variable.

The FFTW2 wrappers do not support 64-bit (“ILP64”) integers.

Examples

The following examples use the Intel compilers, though they should work using the GNU and Portland Group compilers as well.

Advanced Usage

If you are already familiar with building MKL applications with your chosen build tool (GCC, Intel, or PGI) and you do not wish to use the convenience variables discussed above, you may wish to use the $MKLROOT variable instead. This variable points to the installation directory for Intel MKL. All include files can be found in $MKLROOT/include, for example, and the libraries are in $MKLROOT/lib/intel64.

Running MKL Programs

When running an MKL program, you need to be sure to take the following steps.

Load the mkl module:

module load mkl

If running with parallel MKL, set OMP_NUM_THREADS to match the number of cores per node in your process. In the bash shell, you can accomplish this with:

Further Reading

See Also

PNG

PNG (Portable Network Graphics) is an extensible file format for the lossless, portable, well-compressed storage of raster images. PNG provides a patent-free replacement for GIF and can also replace many common uses of TIFF. Indexed-color, grayscale, and truecolor images are supported, plus an optional alpha channel. Sample depths range from 1 to 16 bits.

Availability and Restrictions

Versions

The following versions of PNG are available on OSC clusters:

VERSION

OAKLEY

1.2.52

X

1.5.9

X*

*: Default Version

You can use module avail png to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

PNG is available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of PNG on Oakley, run the following command: module load png . The default version will be loaded. To select a particular PNG version, use module load png/version . For example, use module load png/1.2.52 to load PNG 1.2.52 on Oakley.

Further Reading

Pixman

Pixman is a low-level software library for pixel manipulation, providing features such as image compositing and trapezoid rasterization. Important users of pixman are the cairo graphics library and the X server. Pixman is implemented as a library in the C programming language. It runs on many platforms, including Linux, BSD Derivatives, MacOS X, and Windows.

Availability and Restrictions

Versions

The following versions of Pixman are available on OSC clusters:

VERSION

OAKLEY

0.32.6

X*

*: Default Version

You can use module avail pixman to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Pixman is available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of Pixman on Oakley, run the following command: module load pixman . The default version will be loaded. To select a particular Pixman version, use module load pixman/version . For example, use module load pixman/0.32.6 to load Pixman 0.32.6 on Oakley.

Further Reading

See Also

parallel-command-processor

Introduction

There are many instances where it is necessary to run the same serial program many times with slightly different input. Parametric runs such as these either end up running in a sequential fashion in a single batch job, or a batch job is submitted for each parameter that is varied (or somewhere in between.) One alternative to this is to allocate a number of nodes/processors to running a large number of serial processes for some period of time. The command parallel-command-processor allows the execution of large number of independent serial processes in parallel. parallel-command-processor works as follows: In a parallel job with N processors allocated, the PCP manager process will read the first N-1 commands in the command stream and distribute them to the other N-1 processors. As processes complete, the PCP manager will read the next one in the stream and send it out to an idle processor core. Once the PCP manager runs out of commands to run, it will wait on the remaining running processes to complete before shutting itself down.

Availability

GLENN

OAKLEY

RUBY

X

X

X

Usage

Here is an interactive batch session that demonstrate the use of parallel-command-processor with a config file, pconf. pconf contains several lines of simple commands, one per line. The output of the commands were redirected to individual files.

As the command "all -j ls -l $TMPDIR" shows, the output files are distributed on the two nodes. In a batch file, pbsdcp can be used to distribute copy the files to $TMPDIR on different nodes and gather output files once execution has completed. This step is important due to the load that executing many processes in parallel can place on the user home directories.

Here is a slightly more complex example showing the usage of parallel-command-processor and pbsdcp:

Documentation

pkg-config

pkg-config is a helper tool used when compiling applications and libraries. It helps you insert the correct compiler options on the command line. This metadata is stored in pkg-config files. These files have the suffix .pc and reside in specific locations known to the pkg-config tool.

Availability and Restrictions

Versions

The following versions of pkg-config are available on OSC clusters:

VERSION

OAKLEY

0.28

X*

*: Default Version

You can use module avail pkg-config to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

pkg-config is available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of pkg-config on Oakley, run the following command: module load pkg-config . The default version will be loaded. To select a particular pkg-config version, use module load pkg-config/version . For example, use module load pkg-config/0.28 to load pkg-config 0.28 on Oakley.

Further Reading

Molpro

Introduction

Molpro is a complete system of ab initio programs for molecular electronic structure calculations. The emphasis is on highly accurate computations, with computationally efficient and extensive treatments of electron correlation. The Molpro program offers multiconfiguration SCF, multireference CI, coupled-cluster routines, local electron correlation methods, and a wide range of supporting features. Additional information can be found at the Molpro website.

Availability & Restrictions

Access requires being added to the molpro group.
Contact oschelp@osc.edu to request to be added.

Molpro is available on the Oakley cluster only.

Version

VERSION

GLENN

OAKLEY

2012.1.13

X*

* - Default Version

Usage

Location

The root of the Molpro directory tree is/usr/local/molpro/

Execution

Molpro usage is controlled via modules. Load one of the Molpro modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

module load molpro

Batch Script Example

After loading the module, Molpro can be used interactively or non-interactively. However, Molpro is most commonly used by submitting a non-interactive batch script, such as the one below. This script and its output can be found in the directory /nfs/10/srb/workshops/compchem/molpro/

The above command would query the sequences in blast_query.fas against the nt database and write out results to the blast_results.txt file in the current working directory. By default, mpiBLAST reads configuration information from ~/.ncbirc. Furthermore, mpiBLAST needs at least 3 processes to perform a search: two processes dedicated for scheduling tasks and coordinating file output, while any additional processes actually perform search tasks.

Extra options to mpiblast

--partition-size=[integer]
Enable hierarchical scheduling with multiple masters. The partition size equals the number of workers in a partition plus 1 (the master process). For example, a partition size of 17 creates partitions consisting of 16 workers and 1 master. An individual output file will be generated for each partition. By default, mpiBLAST uses one partition. This option is only available for version 1.6 or above.

--replica-group-size=[integer]
Specify how database fragments are replicated within a partition. Suppose the total number of database fragments is F, the number of MPI processes in a partition is N, and the replica-group-size is G, then in total (N-1)/G database replicas will be distributed in the partition (the master process does not host any database fragments), and each worker process will host F/G fragments. In other words, a database replica will be distributed to every G MPI processes.

--query-segment-size=[integer]
The default value is 5. Specify the number of query sequences that will be fetched from the supermaster to the master at a time. This parameter controls the granularity of load balancing between different partitions. This option is only available for version 1.6 or above.

--use-parallel-write
Enable the high-performance parallel output solution. Note the current implementation of parallel-write does not require a parallel file system.

--use-virtual-frags
Enable workers to cache database fragments in memory instead of local storage. This is recommended on diskless platforms where there is no local storage attaching to each processor. Default to be enabled on Blue Gene systems.

--predistribute-db
Distribute database fragments to workers before the search begins. Especially useful in reducing data input time when multiple database replicas need to be distributed to workers.

--output-search-stats
Enable output of the search statistics in the pairwise and XML output format. This could cause performance degradation on some diskless systems such as Blue Gene.

--removedb
Removes the local copy of the database from each node before terminating execution.

--copy-via=[cp|rcp|scp|mpi|none]
Sets the method of copying files that each worker will use. Default = "cp"

none : do not copy files, instead use shared storage as local storage.

--debug[=filename]
Produces verbose debugging output for each node, optionally logs the output to a file.

--time-profile=[filename]
Reports execution time profile.

--version
Print the mpiBLAST version.

Please refer to the README file in the mpiBLAST package for performance tuning guide.

Removing a database

The --removedb command line option will cause mpiBLAST to do all work in a temporary directory that will get removed from each node's local storage directory upon successful termination. For example:mpiexec -n 18 mpiblast -p blastx -d yeast.aa -i ech_10k.fas -o results.txt --removedb

The above command would perform a 18 node (16 worker) search of the yeast.aa database, writing the output to results.txt. Upon completion, worker nodes would delete the yeast.aa database fragments from their local storage.

Databases can also be removed without performing a search in the following manner:mpiexec -n 18 mpiblast_cleanup

Batch Usage

Below is a sample batch script for running mpiBLAST job. It asks for 24 processors and 4 hours of walltime.

See Also

MPI Library

MPI is a standard library for performing parallel processing using a distributed-memory model. The Glenn and Oakley clusters at OSC use the MVAPICH implementation of the Message Passing Interface (MPI), which is based on MPICH and optimized for the high-speed Infiniband interconnects.

MVAPICH2, based on the MPI-2 standard, is installed on both Oakley and Glenn. The older MVAPICH, based on the MPI-1 standard, is installed on Glenn only.

Availability & Restrictions

MPI is available without restriction to all OSC users.

Installations are available for the Intel, PGI, and gnu compilers.

The following versions of MVAPICH2 are available on OSC systems:

Version

Glenn

Oakley

1.5

X*

1.6

X

1.7

X*

1.8

X

1.9

X

The following versions of MVAPICH (MPI-1) are available on OSC systems:

Version

Glenn

Oakley

1.1

X*

Some older versions are also available.

*Default version. The default on Oakley is the build corresponding to the currently loaded compiler. The default on Glenn is the MPI-1 build for the PGI compiler.

Usage

Set-up

To set up your environment for using the MPI libraries, you must load the appropriate module. On Oakley this is pretty straightforward:

module load mvapich2

You will get the default version for the compiler you have loaded. (Be sure to swap the intel compiler module for the gnu module if you're using gnu.)

On Glenn you should load the module shown in the table below for the compiler you're using. Be sure to unload the default module before loading a new one.

PGI

Intel

GNU

MPI-1

mpi

mvapich-1.1-fixes-intel

mvapich-1.1-fixes-gnu

MPI-2

mpi2

mvapich2-1.5-intel

mvapich2-1.5-gnu

To see all available mvapich modules on Glenn, use this command: module avail mvapich

For example, to use the MPI-2 library with the gnu compilers:

module unload mpi
module load mvapich2-1.5-gnu

Building With MPI

To build a program that uses MPI, you should use the compiler wrappers provided on the system. They accept the same options as the underlying compiler. The commands are shown in the following table.

C

mpicc

C++

mpicxx

FORTRAN 77

mpif77

Fortran 90

mpif90

For example, to build the code my_prog.c using the -O2 option, you would use:

mpicc -o my_prog -O2 my_prog.c

In rare cases you may be unable to use the wrappers. In that case you should use the environment variables set by the module.

Variable

Use

$MPI_CFLAGS

Use during your compilation step for C programs.

$MPI_CXXFLAGS

Use during your compilation step for C++ programs.

$MPI_FFLAGS

Use during your compilation step for Fortran 77 programs.

$MPI_F90FLAGS

Use during your compilation step for Fortran 90 programs.

$MPI_LIBS

Use when linking your program to the MPI libraries.

For example, to build the code my_prog.c without using the wrappers you would use:

mpicc -c $MPI_CFLAGS my_prog.cmpicc -o my_prog my_prog.o $MPI_LIBS

Batch Usage

Programs built with MPI can only be run in the batch environment at OSC. For information on starting MPI programs using the mpiexec command, see Batch Processing at OSC.

Be sure to load the same compiler and mvapich modules at execution time as at build time.

See Also

MPJ-Express

MPJ-Express is a Java library that provides message passing capabilities for parallel computing in Java applications.

MPJ-Express is available on Oakley.

Availability & Restrictions

MPJ-Express is available without restriction to all OSC users.

The following versions of MPJ-Express are available on OSC systems:

Version

Glenn

Oakley

0.38

X

Note: We have modified MPJ-Express slightly from the default distribution to work nicely with our infrastructure. Installing your own copy of a newer version will not work. Please contact OSC Help if you need a newer version of this library.

Usage

Set-up

To set up your environment for using the MPJ-Express libraries, you must load the appropriate module. On Oakley this is pretty straightforward:

module load mpj-express

Building With MPJ-Express

When you load the MPJ-Express module, we will modify your CLASSPATH to ensure the JAR files are located.

Batch Usage

Programs built with MPJ-Express can only be run in the batch environment at OSC. For information on writing batch scripts, see Batch Processing at OSC.

If you have loaded a Java module when compiling, be sure to load that same module in your batch script to avoid version mismatches.

Here is a basic "Hello World" program, which needs to be in a file called "HelloWorld.java", which we use in our example batch script:

Further Reading

See Also

MrBayes

MrBayes is a program for the Bayesian estimation of phylogeny. Bayesian inference of phylogeny is based upon a quantity called the posterior probability distribution of trees, which is the probability of a tree conditioned on the observations. The conditioning is accomplished using Bayes's theorem. The posterior probability distribution of trees is impossible to calculate analytically; instead, MrBayes uses a simulation technique called Markov chain Monte Carlo (or MCMC) to approximate the posterior probabilities of trees.

Availability & Restrictions

MrBayes is available without restriction to all OSC users.

The following versions of MrBayes are available:

Version

Glenn

Oakley

3.1.2

X

3.2.1

X

Usage

Set-up

To load the MrBayes software on the Glenn system, use the following commands:

module load biosoftw
module load mrbayes

On the Oakley system, use the following command:

module load mrbayes

Using MrBayes

Once mrbayes module is loaded, the commands are available for your use. The command for serial version is

mb

Parallel Mrbayes can only be used via batch system. The command is

mb-parallel

Batch Usage

Below is a sample batch script for running serial mrbayes job. It asks for 1 processor and 30 minutes of walltime.

Further Reading

See Also

NAMD

NAMD is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems. NAMD generally scales well on OSC platforms and offers a variety of modelling techniques. NAMD is file-compatible with AMBER, CHARMM, and X-PLOR.

Availability & Restrictions

NAMD is available without restriction to all OSC users.

The following versions of NAMD are available:

Version

Glenn

Oakley

2.6

X

2.7

X

2.9

X

Usage

Set-up

To load the NAMD software on the Glenn system, use the following command:

module load namd-2.6-mpi

Using NAMD

NAMD is rarely executed interactively because preparation for simulations is typically performed with extraneous tools, such as, VMD.

Batch Usage

Sample batch scripts and input files are available here:

/nfs/10/srb/workshops/compchem/namd/

The simple batch script below demonstrates the important points. It requests 16 processors and 2 hours of walltime. If the job goes beyond 2 hours, the job would be terminated.

Further Reading

NetCDF

NetCDF (Network Common Data Form) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netCDF library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data.

Availability & Restrictions

NetCDF is available without restriction to all OSC users.

The following versions of NetCDF are available at OSC:

Version

Glenn

Oakley

ruby

3.6.2

X

4.1.3

X

4.3.2

X

Fortran 4.4.1

X

C++ 4.2.1

X

Usage

Set-up

To initalize either system prior to using NetCDF, run the following comand:

module load netcdf

To use the parallel implementation of netcdf, run the following command instead:

module load pnetcdf

Building With NetDCF

With the netcdf library loaded, the following environment variables will be available for use:

Variable

Use

$NETCDF_CFLAGS

Use during your compilation step for C or C++ programs.

$NETCDF_FFLAGS

Use during your compilation step for Fortran programs.

$NETCDF_LIBS

Use when linking your program to NetCDF.

Similarly, when the pnetcdf module is loaded, the following environment variables will be available:

VARIABLE

USE

$PNETCDF_CFLAGS

Use during your compilation step for C programs.

$PNETCDF_FFLAGS

Use during your compilation step for Fortran programs.

$PNETCDF_LIBS

Use when linking your program to NetCDF.

For example, to build the code myprog.c with the netcdf library you would use:

icc -c $NETCDF_CFLAGS myprog.c
icc -o myprog myprog.o $NETCDF_LIBS

Batch Usage

You must load the netcdf or pnetcdf module in your batch script before executing a program which is built with the netcdf library

Further Reading

See Also

Parallel netCDF

PnetCDF is a library providing high-performance parallel I/O while still maintaining file-format compatibility with Unidata's NetCDF, specifically the formats of CDF-1 and CDF-2. Although NetCDF supports parallel I/O starting from version 4, the files must be in HDF5 format. PnetCDF is currently the only choice for carrying out parallel I/O on files that are in classic formats (CDF-1 and 2). In addition, PnetCDF supports the CDF-5 file format, an extension of CDF-2, that supports more data types and allows users to define large dimensions, attributes, and variables (>2B elements).

Availability & Restrictions

NetCDF is available without restriction to all OSC users.

The following versions of NetCDF are available at OSC:

Version

Glenn

Oakley

ruby

1.0.1

X

1.5.0

X

Usage

Set-up

To initalize the system prior to using PnetCDF, run the following comand:

module load pnetcdf

Building With PnetDCF

With the pnetcdf module loaded, the following environment variables will be available for use:

Variable

Use

$PNETCDF_CFLAGS

Use during your compilation step for C or C++ programs.

$PNETCDF_FFLAGS

Use during your compilation step for Fortran programs.

$PNETCDF_LIBS

Use when linking your program to PnetCDF.

$PNETCDF

Path to the PnetCDF installation directory

For example, to build the code myprog.c with the pnetcdf library you would use:

Batch Usage (Ruby)

Further Reading

See Also

Octave

Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Octave has extensive tools for solving common numerical linear algebra problems, finding the roots of nonlinear equations, integrating ordinary functions, manipulating polynomials, and integrating ordinary differential and differential-algebraic equations. It is easily extensible and customizable via user-defined functions written in Octave's own language, or using dynamically loaded modules written in C++, C, Fortran, or other languages.

Availability & Restrictions

Octave is available to all OSC users without restriction.

The following versions of Octave are available on OSC clusters:

Version

Glenn

Oakley

2.1.73

X

2.9.12

X

3.6.1

X

3.6.4

X

Usage

Set-up

To initialize either cluster for the use of Octave, run the following command:

module load octave

Using Octave

To run Octave, simply run the following command:

octave

Batch Usage

The following example batch script will an octave code file, mycode.o, via the batch processing system. The script requests one full node of cores on Oakley and 1 hour of walltime.

Further Reading

See Also

OpenACC

OpenACC is a standard for parallel programming on accelerators, such as Nvidia GPUs and Intel Phi. It consists primarily of a set of compiler directives for executing code on the accelerator, in C and Fortran. OpenACC is currently only supported by the PGI compilers installed on OSC systems.

Availability & Restrictions

OpenACC is available without restriction to all OSC users. It is supported by the PGI compilers.

Note: Some of the older compiler versions available on Glenn may not support OpenACC.

Usage

Set-up

OpenACC support is built into the compilers. There is no separate module to load.

Building With OpenACC

To build a program with OpenACC, use the compiler flag appropriate to your compiler. The correct libraries are included implicitly.

Compiler Family

Flag

PGI

-acc -ta=nvidia -Minfo=accel

Batch Usage

An OpenACC program will not run without an acelerator present. You need to ensure that your PBS resource request includes GPUs. For example, to run an OpenACC program on Oakley, your resource request should look something like this: #PBS -l nodes=1:ppn=12:gpus=2.

Once your job is ready and your environment has been configured (refer to Set-up On Glenn), you can then start any of the OpenFOAM utilities by the associated command:

blockmesh

or

icoFoam

or

sonicFoam

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script ( job.txt ) for a serial run:

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script ( job.txt ) for a serial run:

Further Reading

OpenMP

OpenMP is a standard for parallel programming on shared-memory systems, including multicore systems. It consists primarily of a set of compiler directives for sharing work among multiple threads. OpenMP is supported by all the Fortran, C, and C++ compilers installed on OSC systems.

Availability & Restrictions

OpenMP is available without restriction to all OSC users. It is supported by the Intel, PGI, and gnu compilers.

Note: Some of the older compiler versions available on Glenn may not support OpenMP.

Usage

Set-up

OpenMP support is built into the compilers. There is no separate module to load.

Building With OpenMP

To build a program with OpenMP, use the compiler flag appropriate to your compiler. The correct libraries are included implicitly.

Compiler Family

Flag

Intel

-openmp

gnu

-fopenmp

PGI

-mp

Batch Usage

An OpenMP program by default will use a number of threads equal to the number of processor cores available. To use a different number of threads, set the environment variable OMP_NUM_THREADS.

Availability & Restrictions

Usage

Set-up

On the Glenn Cluster paml is accessed by executing the following commands:

module load biosoftw
module load paml

Using PAML

PAML is a collection of several programs that will be added to the users PATH: baseml, basemlg, chi2, codeml, ds, evolver, mcmctree, pamp, and yn00. Each of the programs has separate, but typically similar usage and options.

mcmctree Implements the Bayesian MCMC algorithm of Yang and Rannala for estimating species divergence times

mcmctree requires mcmctree.ctl to be located in the current directory with the following variables set: seqfile, treefile, outfile, RootAge, usedata
The following are optional variables to set in mcmctree.ctl: seed, ndata, clock, model, alpha, ncatG, cleandata, BDparas, kappa_gamma, alpha_gamma, rgene_gamma, sigma2_gamma, finetune, print, burnin, sampfreq, nsample

pamp Implements the parsimony-based analysis of Yang and Kumar

pamp requires pamp.ctl to be located in the current directory with the following variables set: seqfile, treefile, outfile
The following are optional variables to set in pamp.ctl: seqtype, ncatG, nhomo

yn00Implements the method of Yang and Nielson for estimating synonymous and nonsynonymous substitution rates in pairwise comparisons of protein-coding DNA sequences

yn00 requiresyn00.ctl to be located in the current directory with the following variables set: seqfile, outfile
The following are optional variables to set in yn00.ctl: verbose, icode, weighting, commonf3x4, ndata

Control Files

All .ctl files (baseml.ctl, codeml.ctl, mcmctree.ctl, pamp.ctl, and yn00.ctl) have comment line starting with '*'.

ParaView

ParaView is an open-source, multi-platform application designed to visualize data sets of size varying from small to very large. ParaView was developed to support distributed computational models for processing large data sets and to create an open, flexible user interface.

Availability & Restrictions

ParaView is available on Glenn cluster only. The following versions are available:

VERSION

GLENN

OAKLEY

3.3.0

X

3.8.0

X

3.14.1

X

Usage

use module avail paraview to view available ParaView modules. ParaView is normally started with a module load command along with the specific version. For example, to load ParaView version 3.8.0, type:

module load paraview-3.8.0

Following a successful loading of the ParaView module, you can access the ParaView program:

See Also

GNU Compilers

Availability and Restrictions

Versions

The following versions of the GNU compilers are available on OSC systems:

Version

Glenn

Oakley

Ruby

4.4.7

X*

X*

4.8.4

X

4.9.1

X

X

* : Default Version

You can use module avail gnu to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

The GNU compilers are available to all OSC users without restriction.

Usage

Usage on Oakley

Set-up on Oakley

To configure your environment for use of the GNU compilers on Oakley, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions): module load gnu. The default version will be loaded. To select a particular GNU version, use module load gnu/version . For example, use module load gnu/4.8.4 to load GNU 4.8.4 on Oakley.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results .
Below is the example batch script ( job.txt ) for a serial run:

Usage on Ruby

Set-up on Ruby

To configure your environment for use of the GNU compilers on Ruby, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions): module load gnu. The default version will be loaded. To select a particular GNU version, use module load gnu/version . For example, use module load gnu/4.9.1 to load GNU 4.9.1 on Ruby.

Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c and the output file named hello_results .
Below is the example batch script ( job.txt ) for a serial run:

Further Reading

See Also

PGI Compilers

Fortran, C and C++ compilers from the Portland Group. PGI compilers are the default, recommended compilers on Glenn.

Availability & Restrictions

The PGI compilers are available to all OSC users without restriction.

The following versions of the PGI compilers are available on OSC systems:

Version

Glenn

Oakley

7.0

X

7.1

X

8.0

X

9.0

X*

10.0

X

10.5

X

11.6

X

11.8

X

12.5

X

X

12.6

X

X

12.9

X

12.10

X

X*

* - default version

Usage

Set-up

To configure your environment for use of the PGI compilers, follow the system-specific instructions below:

For the Oakley system, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions):

module load pgi

For the Glenn system, run the following commands:

module load pgi

Using the PGI Compilers

Once the module is loaded, compiling with the PGI compilers requires understanding which binary should be used for which type of code. Specifically, use the pgcc binary for C codes, the pgCC binary for C++ codes, the pgf77 for FORTRAN-77 codes and the pgf90 for FORTRAN-90 codes.

Further Reading

See Also

Python

Python is a high-level, multi-paradigm programming language that is both easy to learn and useful in a wide variety of applications. Python has a large standard library as well as a large number of third-party extensions, most of which are completely free and open source. We highly recommend using Python 2.7.1, as we have added a lot of Python packages and tuned them to perform well on our systems.

Usage

To run the Python interpreter using the default system-level Python installation, simply type the command “python”. On Glenn, the default version used is 2.4.3. On Oakley, the default version used is 2.6.6.

Each system also has the Python 2.7.1 module. To load this module, type one of the following commands depending on which system you are using.

On Glenn, type:

module load python-2.7.1

On Oakley, type:

module load python/2.7.1

After the module is loaded, you can run the interpreter by using the command

python

To unload the Python 2.7.1 module, use the command

module unload python

Installed modules

We have installed a number of Python packages and tuned them for optimal performance on our systems. When Python 2.7.1 is loaded, executing module help python will show you the current list of packages available.

Of special note is the fact that numpy and scipy have been compiled to use the optimized math libraries for each system (ACML on Glenn, MKL on Oakley) for best performance.

Due to architecture differences between our supercomputers, we recommend not installing your own packages in ~/.local. Instead, you should install them in some other directory and set $PYTHONPATH in your default environment. For more information about installing your own Python modules, please see our HOWTO.

Running Python Scripts in Batch

Python can technically be used in both interactive and non-interactive batch jobs, but you will most likely only want to run a Python script non-interactively. To do this, you will need to write a batch submission script and a Python script.

Below is a sample batch submission script (using the Oakley syntax) and a simple “Hello World” program written in Python.

Further reading

Quantum Espresso

Quantum ESPRESSO is a program package for ab-initio molecular dynamics (MD) simulations and electronic structure calculations. It is based on density-functional theory, plane waves, and pseudopotentials.

Availability & Restrictions.

Quantum ESPRESSO is open source and available without restriction. We recommend that Oakley be used.

The following versions are available on OSC systems:

Version

Glenn

Oakley

5.0.3

X

Usage

Set-up

You can configure your environment for the usage of Quantum ESPRESSO by running the following command:

module load espresso

Batch Usage

Sample batch scripts and input files are available here:

/nfs/10/srb/workshops/compchem/espresso/

Further Reading

See Also

R

R is a language and environment for statistical computing and graphics. It is similar to the S language and environment developed at Bell Laboratories (formerly AT&T, now Lucent Technologies). R provides a wide variety of statistical and graphical techniques, and is highly extensible.

Availability & Restrictions

R is available to all OSC users without restriction.

The following versions of R are available on OSC systems:

Version

Glenn

Oakley

2.5.1

X

2.8.0

X

2.11.1

X

2.14.1

X

2.15.0

X

X

2.15.2

X

X

3.0.1

X

X

Usage

Set-up

In order to configure your environment for the usage of R, run the following command:

module load R

Using R

Once your environment is configured, R can be started simply by entering the following (rediculously short) command:

R

For a listing of command line options, run:

R --help

Batch Usage

Running R interactivly is not recommended and may violate OSC usage policy. In order to run R in batch, refrence the example batch script below. This script requests one full node on the Oakley cluster for 1 hour of walltime.

Options

-a Specify a column weight file name to assign individual weights to each column of the alignment. Those weights must be integers separated by any type and number of whitespaces within a separate file, see file "example_weights" for an example.
-b Specify an integer number (random seed) and turn on bootstrapping. DEFAULT: OFF
-c Specify number of distinct rate catgories for RAxML when modelOfEvolution is set to GTRCAT or GTRMIX. Individual per-site rates are categorized into numberOfCategories rate categories to accelerate computations. DEFAULT: 25
-d start ML optimization from random starting tree. DEFAULT: OFF
-e set model optimization precision in log likelihood units for final optimization of tree topology under MIX/MIXI or GAMMA/GAMMAI. DEFAULT: 0.1 for models not using proportion of invariant sites estimate 0.001 for models using proportion of invariant sites estimate
-E specify an exclude file name, that contains a specification of alignment positions you wish to exclude. Format is similar to Nexus, the file shall contain entries like "100-200 300-400", to exclude a single column write, e.g., "100-100", if you use a mixed model, an appropriately adapted model file will be written.
-f select algorithm:
"-f a": rapid Bootstrap analysis and search for best-scoring ML tree in one program run
"-f b": draw bipartition information on a tree provided with "-t" based on multiple trees (e.g. form a bootstrap) in a file specifed by "-z"
"-f c": check if the alignment can be properly read by RAxML
"-f d": new rapid hill-climbing
"-f e": optimize model+branch lengths for given input tree under GAMMA/GAMMAI only
"-f g": compute per site log Likelihoods for one ore more trees passed via "-z" and write them to a file that can be read by CONSEL
"-f h": compute log likelihood test (SH-test) between best tree passed via "-t" and a bunch of other trees passed via "-z"
"-f i": perform a really thorough bootstrap, refinement of final BS tree under GAMMA and a more exhaustive algorithm
"-f j": generate a bunch of bootstrapped alignment files from an original alignment file
"-f m": Compare bipartitions between two bunches of trees passed via "-t" and "-z" respectively. This will return the Pearson correlation between all bipartitions found in the two tree files. A file called RAxML_bipartitionFrequencies.outpuFileName will be printed that contains the pair-wise bipartition frequencies of the two sets
"-f n": Compute the log likelihood score of all trees contained in a tree file provided by "-z" under GAMMA or GAMMA+P-Invar
"-f o": old and slower rapid hill-climbing
"-f p": perform pure stepwise MP addition of new sequences to an incomplete starting tree
"-f s": split up a multi-gene partitioned alignment into the respective subalignments
"-f t": do randomized tree searches on one fixed starting tree
"-f w": compute ELW test on a bunch of trees passed via "-z"
"-f x": compute pair-wise ML distances, ML model parameters will be estimated on an MP starting tree or a user-defined tree passed via "-t", only allowed for GAMMA-based models of rate heterogeneity. DEFAULT: new rapid hill climbing
-g specify the file name of a multifurcating constraint tree this tree does not need to be comprehensive, i.e. must not contain all taxa
-h Display this help message.
-iInitial rearrangement setting for the subsequent application of topological changes phase. DEFAULT: determined by program
-jSpecifies if checkpoints will be written by the program. If checkpoints (intermediate tree topologies) shall be written by the program specify "-j" DEFAULT: OFF
-kSpecifies that bootstrapped trees should be printed with branch lengths. The bootstraps will run a bit longer, because model parameters will be optimized at the end of each run. Use with CATMIX/PROTMIX or GAMMA/GAMMAI. DEFAULT: OFF
-l Specify a threshold for sequence similarity clustering. RAxML will then print out an alignment to a file called sequenceFileName.reducedBy.threshold that only contains sequences <= the specified thresold that must be between 0.0 and 1.0. RAxML uses the QT-clustering algorithm to perform this task. In addition, a file called RAxML_reducedList.outputFileName will be written that contains clustering information. DEFAULT: OFF
-L Same functionality as "-l" above, but uses a less exhasutive and thus faster clustering algorithm. This is intended for very large datasets with more than 20,000-30,000 sequences. DEFAULT: OFF
-mModel of Nucleotide or Amino Acid Substitution:
NUCLEOTIDES:
"-m GTRCAT": GTR + Optimization of substitution rates + Optimization of site-specific evolutionary rates which are categorized into numberOfCategories distinct rate categories for greater computational efficiency if you do a multiple analysis with "-#" or "-N" but without bootstrapping the program will use GTRMIX instead
"-m GTRGAMMA": GTR + Optimization of substitution rates + GAMMA model of rate heterogeneity (alpha parameter will be estimated)
"-m GTRMIX": Inference of the tree under GTRCAT and thereafter evaluation of the final tree topology under GTRGAMMA
"-m GTRCAT_GAMMA": Inference of the tree with site-specific evolutionary rates. However, here rates are categorized using the 4 discrete GAMMA rates. Evaluation of the final tree topology under GTRGAMMA
"-m GTRGAMMAI": Same as GTRGAMMA, but with estimate of proportion of invariable sites
"-m GTRMIXI": Same as GTRMIX, but with estimate of proportion of invariable sites
"-m GTRCAT_GAMMAI": Same as GTRCAT_GAMMA, but with estimate of proportion of invariable sites
AMINO ACIDS:
"-m PROTCATmatrixName[F]": specified AA matrix + Optimization of substitution rates + Optimization of site-specific evolutionary rates which are categorized into numberOfCategories distinct rate categories for greater computational efficiency if you do a multiple analysis with "-#" or "-N" but without bootstrapping the program will use PROTMIX... instead
"-m PROTGAMMAmatrixName[F]": specified AA matrix + Optimization of substitution rates + GAMMA model of rate heterogeneity (alpha parameter will be estimated)
"-m PROTMIXmatrixName[F]": Inference of the tree under specified AA matrix + CAT and thereafter evaluation of the final tree topology under specified AA matrix + GAMMA
"-m PROTCAT_GAMMAmatrixName[F]" : Inference of the tree under specified AA matrix and site-specific evolutionary rates. However, here rates are categorized using the 4 discrete GAMMA rates. Evaluation of the final tree topology under specified AA matrix + GAMMA
"-m PROTGAMMAImatrixName[F]": Same as PROTGAMMAmatrixName[F], but with estimate of proportion of invariable sites
"-m PROTMIXImatrixName[F]" : Same as PROTMIXmatrixName[F], but with estimate of proportion of invariable sites
"-m PROTCAT_GAMMAImatrixName[F]": Same as PROTCAT_GAMMAmatrixName[F], but with estimate of proportion of invariable sites
Available AA substitution models: DAYHOFF, DCMUT, JTT, MTREV, WAG, RTREV, CPREV, VT, BLOSUM62, MTMAM, GTR
With the optional "F" appendix you can specify if you want to use empirical base frequencies. Please not that for mixed models you can in addition specify the per-gene AA model in the mixed model file (see manual for details)
-M Switch on estimation of individual per-partition branch lengths. Only has effect when used in combination with "-q". Branch lengths for individual partitions will be printed to separate files. A weighted average of the branch lengths is computed by using the respective partition lengths. DEFAULT: OFF
-nSpecifies the name of the output file.
-oSpecify the name of a single outgrpoup or a comma-separated list of outgroups, eg "-o Rat" or "-o Rat,Mouse", in case that multiple outgroups are not monophyletic the first name in the list will be selected as outgroup, don't leave spaces between taxon names!
-qSpecify the file name which contains the assignment of models to alignment partitions for multiple models of substitution. For the syntax of this file please consult the manual.
-pSpecify a random number seed for the parsimony inferences. This allows you to reproduce your results and will help me debug the program. This option HAS NO EFFECT in the parallel MPI version
-PSpecify the file name of a user-defined AA (Protein) substitution model. This file must contain 420 entries, the first 400 being the AA substitution rates (this must be a symmetric matrix) and the last 20 are the empirical base frequencies
-r Specify the file name of a binary constraint tree. This tree does not need to be comprehensive, i.e. must not contain all taxa
-sSpecify the name of the alignment data file in PHYLIP format
-t Specify a user starting tree file name in Newick format
-T PTHREADS VERSION ONLY! Specify the number of threads you want to run. Make sure to set "-T" to at most the number of CPUs you have on your machine, otherwise, there will be a huge performance decrease!
-uSpecify the number of multiple BS searches per replicate to obtain better ML trees for each replicate. DEFAULT: One ML search per BS replicate
-v Display version information
-wName of the working directory where RAxML will write its output files. DEFAULT: current directory
-xSpecify an integer number (random seed) and turn on rapid bootstrapping
-yIf you want to only compute a parsimony starting tree with RAxML specify "-y", the program will exit after computation of the starting tree. DEFAULT: OFF
-z Specify the file name of a file containing multiple trees e.g. from a bootstrap that shall be used to draw bipartition values onto a tree provided with "-t", It can also be used to compute per site log likelihoods in combination with "-f g" and to read a bunch of trees for a couple of other options ("-f h", "-f m", "-f n").
-#|-N Specify the number of alternative runs on distinct starting trees. In combination with the "-b" option, this will invoke a multiple bootstrap analysis. Note that "-N" has been added as an alternative since "-#" sometimes caused problems with certain MPI job submission systems, since "-#" is often used to start comments. DEFAULT: 1 single analysis

Example

The file brown.philip is from the PAML example and modified to be in PHYLIP format.

RECON

“Proper identification of repetitive sequences is an essential step in genome analysis. The RECON package performs de novo identification and classification of repeat sequence families from genomic sequences. The underlying algorithm is based on extensions to the usual approach of single linkage clustering of local pairwise alignments between genomic sequences. Specifically, our extensions use multiple alignment information to define the boundaries of individual copies of the repeats and to distinguish homologous but distinct repeat element families. RECON should be useful for first-pass automatic classification of repeats in newly sequenced genomes.” (http://selab.janelia.org/recon.html)

Availability & Restrictions

RECON is available to all OSC users without restriction.

The following versions of RECON are available on OSC systems:

Version

Glenn

Oakley

1.0.5

X

Usage

Set-up

On the Glenn Cluster, RECON is accessed by executing the following commands:

module load biosoftw
module load RECON

RECON is a collection of several programs that will be added to the users PATH:imagespread, eledef, eleredef, edgeredef, and famdef. Also added to the users PATH isrecon.pl, which executes each of the aforementioned programs.

Example

Further Reading

RepeatMasker

"RepeatMasker is a program that screens DNA sequences for interspersed repeats and low complexity DNA sequences. The output of the program is a detailed annotation of the repeats that are present in the query sequence as well as a modified version of the query sequence in which all the annotated repeats have been masked (default: replaced by Ns). On average, almost 50% of a human genomic DNA sequence currently will be masked by the program." (http://www.repeatmasker.org/)

Availability & Restrictions

RepeatMasker is available to all OSC users without restriction.

The following versions of RepeatMasker are available on OSC systems:

Version

Glenn

Oakley

2.1

X

Usage

Set-up

On the Glenn Cluster RepeatMasker is accessed by executing the following commands:

module load biosoftw
module load RepeatMasker

RepeatMasker will be added to the users PATH and can be run with the command:

RepeatMasker [-options] <seqfiles(s) in fasta format>

Options

-h(elp)
Detailed help
Default settings are for masking all type of repeats in a primate sequence.
-pa(rallel) [number]
The number of processors to use in parallel (only works for batch files or sequences over 50 kb)
-sSlow search; 0-5% more sensitive, 2-3 times slower than default
-qQuick search; 5-10% less sensitive, 2-5 times faster than default
-qqRush job; about 10% less sensitive, 4->10 times faster than default (quick searches are fine under most circumstances) repeat options
-nolow /-lowDoes not mask low_complexity DNA or simple repeats
-noint /-intOnly masks low complex/simple repeats (no interspersed repeats)
-nornaDoes not mask small RNA (pseudo) genes
-aluOnly masks Alus (and 7SLRNA, SVA and LTR5)(only for primate DNA)
-div [number]Masks only those repeats < x percent diverged from consensus seq
-lib [filename]Allows use of a custom library (e.g. from another species)
-cutoff [number]Sets cutoff score for masking repeats when using -lib (default 225)
-species <query species>Specify the species or clade of the input sequence. The species name must be a valid NCBI Taxonomy Database species name and be contained in the RepeatMasker repeat database. Some examples are:
-species human
-species mouse
-species rattus
-species "ciona savignyi"
-species arabidopsis
Other commonly used species: mammal, carnivore, rodentia, rat, cow, pig, cat, dog, chicken, fugu, danio, "ciona intestinalis" drosophila, anopheles, elegans, diatoaea, artiodactyl, arabidopsis, rice, wheat, and maize

Running options

-gc [number]Use matrices calculated for 'number' percentage background GC level
-gccalcRepeatMasker calculates the GC content even for batch files/small seqs
-frag [number]Maximum sequence length masked without fragmenting (default 40000, 300000 for DeCypher)
-maxsize [nr]Maximum length for which IS- or repeat clipped sequences can be produced (default 4000000). Memory requirements go up with higher maxsize.
-nocutSkips the steps in which repeats are excised
-noisyPrints search engine progress report to screen (defaults to .stderr file)
-nopostDo not postprocess the results of the run ( i.e. call ProcessRepeats).
NOTE: This options should only be used when ProcessRepeats will be run manually on the results.

Output options

-dir [directory name]Writes output to this directory (default is query file directory, "-dir ." will write to current directory).
-a(lignments)
Writes alignments in .align output file; (not working with -wublast)
-invAlignments are presented in the orientation of the repeat (with option -a)
-lcambigOutputs ambiguous DNA transposon fragments using a lower case name. All other repeats are listed in upper case. Ambiguous fragments match multiple repeat elements and can only be called based on flanking repeat information.
-smallReturns complete .masked sequence in lower case
-xsmallReturns repetitive regions in lowercase (rest capitals) rather than masked
-xReturns repetitive regions masked with Xs rather than Ns
-polyReports simple repeats that may be polymorphic (in file.poly)
-sourceIncludes for each annotation the HSP "evidence". Currently this option is only available with the "-html" output format listed below.
-htmlCreates an additional output file in xhtml format.
-aceCreates an additional output file in ACeDB format
-gffCreates an additional Gene Feature Finding format output
-uCreates an additional annotation file not processed by ProcessRepeats
-xmCreates an additional output file in cross_match format (for parsing)
-fixedCreates an (old style) annotation file with fixed width columns
-no_idLeaves out final column with unique ID for each element (was default)
-e(xcln)
Calculates repeat densities (in .tbl) excluding runs of >=20 N/Xs in the query

This page documents usage of the ScaLAPACK library installed by OSC from source. An optimized implementation of ScaLAPACK is included in MKL; see the software documentation page for Intel Math Kernel Library for usage information.

Availability & Restrictions

ScaLAPACK is available to all OSC users without restriction. If you need high performance, we recommend using MKL instead of the standalone scalapack module.

The following versions of ScaLAPACK are available on OSC systems:

Version

Glenn

Oakley

Ruby

1.7

X

2.0.1

X

2.0.2

X

Usage

Set-up

To use the ScaLAPACK libraries in your compilation on Oakley, first load the scalapack module:

module load scalapack

To use the ScaLAPACK libraries in your compilation on Glenn, you must load the acml module then the scalapack module. Both modules are compiler-dependent.

Compiler

Commands

PGI

module load acml
module load scalapack

gnu

module load acml-gfortran
module load scalapack-gnu

Intel

module load acml-intel
module load scalapack-intel

Building With ScaLAPACK

Once loaded, the ScaLAPACK libraries can be linked in with your compilation. To do this, use the following environment variables:

On Oakley:

Variable

Use

$SCALAPACK_LIBS

Used to link ScaLAPACK into either Fortran or C

On Glenn:

Variable

Use

$SCALAPACK_C_LIBS

Used to link ScaLAPACK into a C program

$SCALAPACK_F77_LIBS

Used to link ScaLAPACK into a Fortran program

Further Reading

See Also

SIESTA

SIESTA is both a method and its computer program implementation, to perform efficient electronic structure calculations and ab initio molecular dynamics simulations of molecules and solids. More information can be found from here.

Availability & Restrictions

SIESTA is available on the Oakley cluster. A serial and a parallel build were created in order to meet users' computational needs.

VERSION

GLENN

OAKLEY

3.2-pl-4

X

The SIESTA program is distributed freely to academics. Use of SIESTA for academic purposes requires validation. To obtain validation, please complete and return the "Academic Agreement to Use SIESTA", located on the Academic Agreement Forms Page. Commercial users need to show proof of licence in order to use SIESTA at OSC.

Batch Usage

When you log into oakley.osc.edu, you are actually logged into a linux box referred to as the login node. To gain access to the 4000+ processors in the computing environment, you must submit your SIESTA job to the batch system for execution.

Assume that you have a test case in your work directory (where you submit your job, represented by $PBS_O_WORKDIR), with the input file 32_h2o.fdf. A batch script can be created and submitted for a serial or parallel run. The following are the sample batch scripts for running serial and parallel SIESTA jobs.

SPRNG

Computational stochastic approaches (Monte Carlo methods) based on random sampling are becoming extremely important research tools not only in their "traditional" fields such as physics, chemistry or applied mathematics but also in social sciences and, recently, in various branches of industry. An indication of importance is, for example, the fact that Monte Carlo calculations consume about one half of the supercomputer cycles. One of the indispensable and important ingredients for reliable and statistically sound calculations is the source of pseudo random numbers. SPRNG provides a scalable package for parallel pseudo random number generation which will be easy to use on a variety of architectures, especially in large-scale parallel Monte Carlo applications.

SPRNG 1.0 provides the user the various SPRNG random number generators each in its own library. For most users this is acceptable, as one rarely uses more than one type of generator in a single program. However, if the user desires this added flexibility, SPRNG 2.0 provides it. In all other respects, SPRNG 1.0 and SPRNG 2.0 are identical.

Availability & Restrictions

SPRNG is available to all OSC users without restriction.

The following versions of sprng are available on OSC systems:

Version

Glenn

Oakley

1.0

X

2.0

X

X

Usage

Set-up

To configure your environment for use of SPRNG 1.0 on Glenn, use the following command:

module load sprng

To configure your environment for use of SPRNG 2.0 on Glenn, use the following command:

module load sprng2

To configure your environment for use of SPRNG 2.0 on Oakley, use the following command:

module load sprng/2.0

Note: SPRNG is dependent on the compiler and mpi module and may not be supported for all combinations.

Building With SPRNG 1.0 and 2.0

Once SPRNG 1.0 is loaded, the following environment variables will be defined for use with compilation tools:

Variable

Function

$SPRNG_INCLUDE

Include path for SPRNG header files

$SPRNG_CMRG

Library path and library for CMRG generator

$SPRNG_LCG

Library path and library for LCG generator

$SPRNG_LCG64

Library path and library for 64-bit LCG generator

$SPRNG_LFG

Library path and library for LFG generator

$SPRNG_MLFG

Library path and library for MLFG generator

Alternativly, when SPRNG 2.0 is loaded, the following variables will be defined:

VARIABLE

Function

$SPRNG_INCLUDE

Include path for SPRNG 2.0 header files

$SPRNG_LIBS

Library path and library for SPRNG 2.0 generators

Further Reading

See Also

STAR-CCM+

STAR-CCM+ provides the world’s most comprehensive engineering physics simulation inside a single integrated package. Much more than a CFD code, STAR‑CCM+ provides an engineering process for solving problems involving flow (of fluids and solids), heat transfer and stress. STAR‑CCM+ is unrivalled in its ability to tackle problems involving multi‑physics and complex geometries. Support is provided by CD-adapco.

Availability & Restrictions

CD-adapco releases new version of STAR-CCM+ every four months. The following versions of STAR-CCM+ are available at OSC:

VERSION

GLENN

OAKLEY

6.04.016

X

6.06.017

X

7.02.011

X

7.04.011

X

X

7.06.012

X

7.06.012r8

X

8.02.011

X

8.04.009

X

X

8.06.007

X

9.04.011r8

X

9.04.012

X

10.04.009

X

Feel free to contact OSC Help if you need other versions for your work.

Use module avail starccm to view available STAR-CCM+ modules for a given machine. STAR-CCM+ is normally started with a module load command along with the specific version. For example, to load STAR-CCM+ version 7.06.012 on Oakley, type:

module load starccm/7.06.012

Following a successful loading of the STAR-CCM+ module, you can access the STAR-CCM+ program:

starccm+

OSC Batch Usage

STAR-CCM+ can be run on OSC clusters in either interactive mode or in batch mode. Interactive mode is similar to running STAR-CCM+ on a desktop machine in that the graphical user interface (GUI) will be sent from OSC and displayed on the local machine. Batch mode means that you run the STAR-CCM+ job by submitting a batch script with providing all the STAR-CCM+ input file(s) needed to run the simulation when resources become available.

Interactive Example

To run interactive STAR-CCM+, it is suggested to request necessary compute resources from the login node, with X11 forwarding. The intention is that users can run STAR-CCM+ interactively for the purpose of building their model, preparing input file (.sim file), and checking results. For example, the following line requests one node, one core, for a walltime of one hour:

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=1

This job will queue until resources become available. Once the job is started, you're automatically logged in on the compute node; and you can launch STAR-CCM+ GUI with the following commands:

module load starccm/7.06.012
starccm+

Batch Mode

Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. For a given problem, prepare the input file with STAR-CCM+ (named starccm.sim for example) for the batch run. Feel free to contact OSC Help if you need more information.

Serial execution

The following batch script would be needed for the serial applicaiton (assume the solution will be completed within 30 hours and 1 processor):

To run this job on OSC batch system, the above script (named submit_starccm.job) is to be submitted with the command:

qsub submit_starccm.job

Parallel execution

To take advantage of the powerful compute resources at OSC, you may choose to run distributed STAR-CCM+ for large problems. Multiple nodes and cores can be requested to accelerate the solution time. The following shows an example script if you need 2 nodes with 12 cores per node on Oakley using the inputfile named starccm.sim:

Stata

Stata is a complete, integrated statistical package that provides everything needed for data analysis, data management, and graphics. Release 13 32-processor SMP is currently available at OSC.

Availability & Restrictions

Only academic use is allowed. Please contact oschelp@osc.edu to get validated for using Stata.

The following versions of Stata are available on OSC systems:

Version

Glenn

Oakley

13

X

Stata is no longer available on Glenn. We have upgraded to Stata 13, and migrated to Oakley.

Usage

Set-up

To configure your environment on Oakley for the usage of Stata, run the following command:

module load stata

Using Stata

Due to licensing restrictions, Stata may ONLY be used via the batch system on Oakley. See below for information on how this is done.

Batch Usage

OSC has a 2-user license. However, there is no enforcement mechanism built into Stata. In order for us to stay within the 2-user limit, we require you to run in the context of PBS and to include this option when starting your batch job (the PBS system will enforce the 2 user limit):

Further Reading

See Also

SuperLU

SuperLU is a library for the direct solution of large, sparse, nonsymmetric systems of linear equations on high performance machines. It comes in two different flavors: SuperLU_MT (multithreaded) for shared memory parallel machines and SuperLU_DIST for distributed memory parallel machines.

Availability & Restrictions

SuperLU is available to all OSC users without restriction.

The following versions of SuperLU are available on OSC systems:

Library

Version

Glenn

Oakley

SuperLU_MT

2.0

X

SuperLU_MT

2.3

X

SuperLU_DIST

2.3

X

SuperLU_DIST

3.0

X

SuperLU_DIST

3.1

X

SuperLU is available for all compilers on Oakley but only for Intel compilers on Glenn.

Usage

Set-up

To use the SuperLU libraries in your compilation, first load the appropriate superlu module:

On Oakley, use one of these:

module load superlu_mt

module load superlu_dist

On Glenn, you must first load the acml module:

module load acml-intel

then use one of these:

module load superlu-MT-intel

module load superlu-DIST-intel

Building With SuperLU

Once loaded, the SuperLU libraries can be linked in with your compilation. To do this, use the following environment variables:

On Oakley:

Variable

Use

$SUPERLU_MT_CFLAGS

Include path for multithreaded libraries

$SUPERLU_MT_LIBS

Link flags for multithreaded libraries

$SUPERLU_DIST_CFLAGS

Include path for distributed-memory libraries

$SUPERLU_DIST_LIBS

Link flags for distributed-memory libraries

On Glenn:

Variable

Use

$SUPERLU_INC

Include path

$SUPERLU_LIBS

Link flags for either library, depending on loaded module

Further Reading

See Also

TotalView Debugger

Introduction

TotalView is a symbolic debugger which supports threads, MPI, OpenMP, C/C++ and Fortran, plus mixed-language codes. Advanced features include on-demand memory leak detection, heap allocation debugging and the Standard Template Library Viewer (STLView). Other features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis and control at the thread level give you the power you need to solve tough problems.

Versions

Version

Glenn

Oakley

RUBY

8.3.0

X

8.5.0-1

X

8.8.0-2

X

8.9.2-1

X*

8.14.1-8

X**

X**

* Both standard and CUDA versions are available.

** CUDA debugging included in standard module.

Usage

Building your code for debugging with TotalView

You should build your code with the -g option for symbolic debugging. For example:

icc -g -c mycode.c

or

mpicc -g -c mympicode.c

To debug an MPI program on Ruby, you must build with a debug version of mvapich2. Swap modules before building code, for example (Ruby only):

module swap mvapich2/1.9 mvapich2/1.9-debug

Launching TotalView

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands:

module load totalviewtotalview

Example code to debug can be found on Glenn, Oakley and Ruby in $TOTALVIEW_HOME/linux-x86-64/examples. Refer to the file README.TXT in this directory for instructions for using these examples.

To debug an MPI program on Oakley or Glenn, use the following commands in an interactive job:

TreeBeST

"TreeBeST is an original tree builder for constrained neighbour-joining and tree merge, an efficient tool capable of duplication/loss/ortholog inference, and a versatile program facili- tating many tree-building routines, such as tree rooting, alignment filtering and tree plot- ting. TreeBeST stands for ‚Äò(gene) Tree Building guided by Species Tree‚Äô. It is previously known as NJTREE as the first piece of codes of this project aimed to build a eighbour-joining tree.

TreeBeST is the core engine of TreeFam (Tree Families Database) project initiated by Richard Durbin. The basic idea of this project is to build a full tree constrained by a manually verified seed tree. The tree builder must know how to utilize the prior knowledge provided by human experts. This demand disqualifies any existing softwares. Given this fact, we devised a new algorithm to control the joining step of traditional neighbour-joining. This is origin the constrained neighbour-joining.

When trees are built, they are only meaningful to biologists. Computers generate trees, but they do not understand them. To understand gene trees, a computer must be equipped with some biological knowledges, the species tree. It will teach a computer how to discriminate a speciation from a duplication event and how to find orthologs, provided a correct gene tree.

Unfortunately, gene trees are not always correct. Since the advent of UPGMA algorithm in 1958, we have tried to find a ideal model for nearly half a century. But we failed. Evolution is so complex a thing. A model best fits in one lineage might mean a disaster in another. A unified model is far from being discovered. TreeBeST aims at improving the accuracy of tree building, but it does not try to set up a new model in a traditional way. Instead, it integrates two existing models with the help of species tree, finding the subtree that best fits the models and merging them together to build a new tree incorporating the advantages of the both. This is the tree algorithm." (treebest.pdf)

Availability & Restrictions

TreeBeST is available to all OSC users without restriction.

The following versions of TreeBeST are available on OSC systems:

Version

Glenn

Oakley

1.9.2

X

Usage

Set up

On the Glenn Cluster TreeBeST is accessed by executing the following commands:

module load biosoftw
module load treebest

TreeBeST will be added to the users PATH and can then be run with the command:

treebest <command> [options]

Below lists the commands and their summaries for the treebest program.

Turbomole

TURBOMOLE is an ab initio computational chemistry program that implements various quantum chemistry algorithms. It is developed at the group of Prof. Reinhart Ahlrichs at the University of Karlsruhe.

Availability and Compatability

Turbomole is available on both Oakley and Glenn Clusters. The versions currently available at OSC are

VERSION

GLENN

OAKLEY

5.9.0

X

5.9.1

X

5.10.0

X

6.0.0

X

6.2

X

6.3.1

X

X

To use Turbomole's parallel computing capabilities on Glenn, load the parallel module with the "turbomole-parallel" prefix.

Usage

Access

Turbomole is available for use by all OSC users.

Setup

Use module avail to view available modules for a given machine. To load the appropriate Turbomole module, type:module load software-name.
For example: To select Turbomole version 6.3.1 on Oakley, type:module loadturbomole/6.3.1

TurboVNC

Introduction

TurboVNC is an implementation of VNC optimized for 3D graphics rendering. Like other VNC software, TurboVNC can be used to create a virtual desktop on a remote machine, which can be useful for visualizing CPU-intensive graphics produced remotely.

Availability

VERSION

GLENN

OAKLEY

1.0

X

1.1

X

X [1]

1.2

X

X [2]

[1] -- TurboVNC 1.1's version of vncviewer does not work on Oakley. Use the 1.2 module's version.

[2] -- TurboVNC 1.2's version of vncserver does not work on Oakley. Use the 1.1 module's version.

Usage

Please do not SSH directly to compute nodes and start VNC sessions! This will negatively impact other users (even if you have been assigned a node via the batch scheduler), and we will consider repeated occurances an abuse of the resources. If you need to use VNC on a compute node, please see our HOWTO for instructions.

To use the applications provided in the TurboVNC module, you must first load the module using the following command:

module load turbovnc

To start a VNC server on your current host, use the following command:

vncserver

After starting the VNC server you should see output similar to the following:

New 'X' desktop is hostname:displayStarting applications specified in /nfs/nn/yourusername/.vnc/xstartup.turbovncLog file is /nfs/nn/yourusername/.vnc/hotsname:display.log

Make a note of the hostname and display number ("hostname:display"), because you will need this information later in order to connect to the running VNC server.

To establish a standard unencrypted connection to an already running VNC server, X11 forwarding must first be enabled in your SSH connection. This can usually either be done by changing the preferences or settings in your SSH client software application, or by using the -X or -Y option on your ssh command.

Once you are certain that X11 forwarding is enabled, create your VNC desktop using the vncviewer command in a new shell.

vncviewer

You will be prompted by a dialogue box asking for the VNC server you wish to connect to. Enter "hostname:display".

You may then be prompted for your HPC password. Once the password has been entered your VNC desktop should appear, where you should see all of your home directory contents.

When you are finished with your work on the VNC desktop, you should make sure to close the desktop and kill the VNC server that was originally started. The VNC server can be killed using the following command in the shell where the VNC server was originally started:

vncserver -kill :[display]

For a full explanation of each of the previous commands, type man vncserver or man vncviewer at the command line to view the online manual.

Further Reading

VASP

The Vienna Ab initio Simulation Package, VASP, is a program package for ab-initio quantum-mechanical molecular dynamics (MD) simulations and electronic structure calculations, from first principles.

Availability & Restrictions

Due to licensing considerations, OSC does not provide general access to this software. However, we are available to assist with the configuration of individual research-group installations on both the Oakley and the Glenn Cluster. The existing VASP module on Glenn was setup for a specific research group and is not available for general use. See the The VASP FAQ page for information regrading licensing.

The following versions of VASP are available with restrictions as listed above on OSC systems:

Version

Glenn

Oakley

4.6

X

Usage

Set-up

For those with access you can configure your environment for the usage of VASP on Glenn by running the following command:

module load vasp

Using VASP

Further Reading

See Also

VTK

Introduction

The Visualization ToolKit (VTK) is an open source, freely available software system for 3D computer graphics, image processing and visualization. VTK consists of a C++ class library and several interpreted interface layers including Tcl/Tk, Java and Python.

Version

Version

Glenn

Oakley

5.0.4

X

Usage

You can use the following command to load the VTK module:

module load vtk

VTK example code can be found on the Glenn cluster in $VTK_DIR/Examples. Refer to the file README.txt in this directory for information about the content of the examples.