The Basic Local Alignment Search Tool (BLAST) finds regions of local similarity between sequences. The program compares nucleotide or protein sequences to sequence databases and calculates the statistical significance of matches. BLAST can be used to infer functional and evolutionary relationships between sequences as well as help identify members of gene families.

Below we detail how to run both implementations of BLAST, ncbi-blast and mpiBLAST. While mpiBLAST is the recommended method for large scale simulations, it does not provide so many options in terms of input as BLAST. This imposes a limitation for many users that need more flexibility. These users will need to use BLAST at large scale and should run ncbi-blast.

The first approach consists of running BLAST with threads. Several different nodes can be used for this. The input file will have to split into the same number of nodes used for the job. Since having all the nodes accessing the shared filesystem would create a very high IO load, the database needs to be moved to the local disk of each node (/tmp).

Load the BLAST module to set up the environment:

login1$ module load blast/2.2.29

Lets say we want to use 10 nodes (10 nodes, 16 cores per node, equals 160 cores). After splitting the input file, we have 10 files with names: input.0, input.1, input.2,…, input.9. Now, the rank 0 will work on input.0, rank 1 will work on input.1, and so on. We are going to need an auxiliary executable file, "task.sh" that will do this

First, each task (remember, in this example we have 10 of these) gets its rank. Next, it creates a folder in its own local disk and copies the original database to the local disk. Once the file has been copied, it runs blastp using as input the file input.$rank. Since rank starts at 0 and ends at 9 (in this example), we need the input files to be name exactly like that: input.0,…, input.9. Be careful not to use: input.00, input.01,…, input.09, as that will fail. It is important to notice how we are indicating blastp to use 16 threads.

Now, all we need is to run this script using ibrun. Since BLAST will run using 16 threads per execution, we only need one instance of the previous script ("task.sh") running on each node. Therefore we have to use ibrun with task.sh, and we need to put in the job submission scripts that the number of parallel tasks (those started by ibrun) will be one per node. We achieve that by specifying the same value for -n and -N in the submission script (10 in our example). Our batch script for this example will be:

See how we are requesting 10 nodes ("-N") and 10 processes ("-n"). Then, each of these 10 processes will use 16 threads as previously stated. It is also important to pay attention to how the environment variables are now defined, pointing to "/tmp/DATABASE", which is the local disk of each node. Once all the 10 tasks have finished, the individual output files of each execution are put together into "blast_results_out_10.txt".

Once the mpiBLAST module has been loaded, reformat the database as follows:

login1$ mpiformatdb -N 200 -i DATABASE/uniref90.fasta -o T

This command formats the database into 200 fragments. It will put those fragments in the same folder as the FASTA database. To specify a different location, use the "-n" option.

Typically, both BLAST and mpiBLAST require a set of variables to be defined in the ".ncbirc" file. However, using this file with a large number of cores leads to timeouts since all those cores try to access the file at the same time. That timeout leads to MPI aborts and the computation will fail. In order to avoid that, the variables must be defined as environment variables. Since we are going to submit the job to the queues, it's a good idea to include those variables in the batch script. The following variables must be defined: