Compiling NWChem from source

On this page, a step-by-step description of the build process and necessary and optional environment variables is outlined. In addition, based on the experiences of developers and users how-to's for various platforms have been created. These how-to's will be updated with additional platforms and better environment variables over time.

Setting up the proper environment variables

NWCHEM_TOP defines the top directory of the NWChem source tree, e.g.

When dealing with source from a NWChem release (6.6 in this example)

% setenv NWCHEM_TOP <your path>/nwchem-6.6

when using the NWChem development source

% setenv NWCHEM_TOP <your path>/nwchem

NWCHEM_TARGET defines your target platform, e.g.

% setenv NWCHEM_TARGET LINUX64

The platforms that available are:

NWCHEM_TARGET

Platform

OS/Version

Compilers

LINUX

x86 ppc

RedHat, Suse Suse

GNU, Intel, PGI GNU, xlf

LINUX64

ia64 x86_64 ppc64, ppc64le

RedHat SLES, RedHat SLES, RedHat

Intel GNU, PGI, PathScale, Intel xlf

BGL BGP BGQ

Blue Gene/L Blue Gene/P Blue Gene/Q

SLES (login) CNK (compute)

blrts_xlf bgxlf_r bgxlf_r

LAPI LAPI64

IBM SP

AIX/LAPI

xlf

MACX MACX64

Apple MacOSX

OSX

GNU, xlf, Intel

CYGWIN WIN32

Intel x86

Windows with Cygwin Windows

GNU Compaq

ARMCI_NETWORK must be defined in order to achieve best performance on high performance networks, e.g.

% setenv ARMCI_NETWORK OPENIB

For a single processor system, this environment variable does not have to be defined.

MPI variables

Set to "y" for the NWPW module to use fortran-bindings of MPI (Generally set when USE_MPI is set)

USE_MPIF4

Set to "y" for the NWPW module to use Integer*4 fortran-bindings of MPI. (Generally set when USE_MPI is set on most platforms)

LIBMPI

Name of the MPI library that should be linked with -l (eg. -lmpich)

MPI_LIB

Directory where the MPI library resides

MPI_INCLUDE

Directory where the MPI include files reside

New in NWChem 6.6: If the location of the mpif90 command is part of your PATH env. variable, NWChem will figure out the values of LIBMPI, MPI_LIB and MPI_INCLUDE (if they are not set). Therefore, we recommend not to set LIBMPI, MPI_LIB and MPI_INCLUDE and add the location of mpif90 to the PATH variable, instead.

When MPI is used, the appropriate MPI run command should be used to start an NWChem calculation, e.g.

% mpirun -np 8 $NWCHEM_TOP/bin/${NWCHEM_TARGET}}/nwchem h2o.nw

When all nodes are connected via shared memory and the ch_shmem version of MPICH is installed and used, NWChem can be called directly, e.g.

% $NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem -np 8 h2o.nw

NWCHEM_MODULES defines the modules to be compiled, e.g.

% setenv NWCHEM_MODULES "all python"

The following modules are available:

Module

Description

all

Everything useful

all python

Everything useful plus python

qm

All quantum mechanics modules

md

MD only build

Note that additional environment variables need to be defined to specify the location of the Python libraries, when the python module is compiled. See the optional environmental variables section for specifics.

Adding optional environmental variables

USE_NOFSCHECK can be set to avoid NWChem creating files for each process when testing the size of the scratch directory (a.k.a. creation of junk files), e.g.

% setenv USE_NOFSCHECK TRUE

USE_NOIO can be set to avoid NWChem 6.5 doing I/O for the ddscf, mp2 and ccsd modules (it automatically sets USE_NOFSCHECK, too).
It is strongly recommended on large clusters or supercomputers or any computer lacking any fast and large local filesystem.

% setenv USE_NOIO TRUE

LIB_DEFINES can be set to pass additional defines to the C preprocessor (for both Fortran and C), e.g.

% setenv LIB_DEFINES -DDFLT_TOT_MEM=16777216

Note: -DDFLT_TOT_MEM sets the default dynamic memory available for NWChem to run, where the units are in doubles. Instead of manually defining these one can optionally use the "getmem.nwchem" script in the $NWCHEM_TOP/contrib directory. This script should be run after an initial build of the binary has been completed. The script will assess memory availability and make an educated guess, recompile the appropriate files and relink.

MRCC_THEORY can be set to request the multireference coupled cluster capability to be included in the code, e.g.

% setenv MRCC_THEORY TRUE

Setting Python environment variables

Python programs may be embedded into the NWChem input and used to control the execution of NWChem. To build with Python, Python needs to be available on your machine. The software can be download from http://www.python.org . Follow the Python instructions for installation and testing. NWChem has been tested with Python versions 1.x and 2.x.

The following environment variables need to be set when compiling with Python:

USE_PYTHON64 is needed on platforms were the Python library is found under the /usr/lib64 tree instead of /usr/lib (e.g. 64-bit RHEL6: /usr/lib64/python2.6/config/libpython2.6.so).

PYTHONLIBTYPE=so needs to be set when static libraries are not present, and shared libraries needs to be used, instead (/usr/lib64/python2.6/config/libpython2.6.a vs /usr/lib64/python2.6/config/libpython2.6.so).

PYTHONCONFIGDIR needs to be set when the name of the subdirectory containing the Python library is not config (e.g. config-x86_64-linux-gnu instead of config).

To run with Python, make sure that PYTHONHOME is set as mentioned above. You will also need to set PYTHONPATH to include any modules that you are using in your input. Examples of Python within NWChem are in the $NWCHEM_TOP/QA/tests/pyqa and $NWCHEM_TOP/contrib/python directories.

Optimized math libraries

By default NWChem uses its own basic linear algebra subroutines (BLAS). To include faster BLAS routines, the environment variable BLASOPT needs to be set before building the code. For example, with ATLAS

WARNING: In the case of 64-bit platforms, most vendors optimized BLAS libraries cannot be used. This is due to the fact that while NWChem uses 64-bit integers (i.e. integer*8) on 64-bit platforms, most of the vendors optimized BLAS libraries used 32-bit integers. The same holds for the ScaLAPACK libraries, which internally use 32-bit integers.

A method is now available to link against the libraries mentioned above, using the following procedure:

GotoBLAS2 can be installed with 64bit integers. This is accomplished by compiling the GotoBLAS2 library after having by edited the GotoBLAS2 Makefile.rule file and un-commenting the line containing the INTERFACE64 definition. In other words, the line

#INTERFACE64 = 1
needs to be changed to
INTERFACE64 = 1

ACML and MKL can support 64-bit integers if the appropriate library is chosen. For MKL, one can choose the ILP64 Version of Intel® MKL and the correct recipe can be extracted from the website https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor. For ACML the int64 libraries should be chosen, e.g. in the case of ACML 4.4.0 using a PGI compiler /opt/acml/4.4.0/pgi64_int64/lib/libacml.a

Linking in NBO

The current versions of NBO provide a utility to generate source code that can be linked into computational chemistry packages such as NWChem. To utilize this functionality, follow the instructions in the NBO package to generate an nwnbo.f file. Linking NBO into NWChem can be done using the following procedure:

One can now use "task nbo" and incorporate NBO input into the NWChem input file directly:

nbo
$NBO NRT $END
...
end
task nbo

Building the NWChem binary

Once all required and optional environment variables have been set, NWChem can be compiled:

% cd $NWCHEM_TOP/src
% make nwchem_config
% make >& make.log

The make above will use the standard compilers available on your system. To use compilers different from the default one can either set environment variables:

% setenv FC <fortran compiler>
% setenv CC <c compiler>

Or one can supply the compiler options to the make command, e.g:

% make FC=ifort CC=icc

For example, on Linux FC could be set either equal to ifort, gfortran or pgf90

Note 1: If in a Linux environment, FC is set equal to anything other than the tested compilers, there is no guarantee of a successful installation, since the makefile structure has not been tested to process other settings. In other words, please avoid make FC="ifort -O3 -xhost" and stick to make FC="ifort", instead

Note 2: It's better to avoid redefining CC, since a) NWChem does not have C source that is a computational bottleneck and b) we typically test just the default C compiler.
In other words, the recommendation is to compile with make FC=ifort

Note 3: It's better to avoid modifying the values of the FOPTIMIZE and COPTIMIZE variables.
The reason is that the default values for FOPTIMIZE and COPTIMIZE have been tested by the NWChem developers (using the internal QA suites, among others), while any modification might produce incorrect results.

How-to: Build_nwchem script

The build_nwchem script is an auto-compile tool. It guesses the configuration of your machine based on
the results of a number of tests and compiles a corresponding binary. It is of course possible that the
script guesses something wrong in which case that setting can be corrected by manually specifying the
corresponding environment variable. See the other platform how-to's for details on those variables.
The only requirement to use the script is that the MPI compiler wrappers (e.g. mpif90, mpicc and mpiCC)
should be in your path, and that if your machine uses the "module" software you should have all
modules you want to use loaded. If these requirements are met then simply run

New in NWChem 6.6: If the location of the mpif90 command is part of your PATH env. variable, NWChem will figure out the values of LIBMPI, MPI_LIB and MPI_INCLUDE (if they are not set). Therefore, we recommend not to set LIBMPI, MPI_LIB and MPI_INCLUDE and add the location of mpif90 to the PATH variable, instead.

Example: NERSC Edison

These are variables used for compilation on NERSC Edison, a Cray XC30, as of October 23rd 2015, when using Intel compilers (i.e. after issuing the commands module swap PrgEnv-gnu PrgEnv-intel). Very similar settings can be applied to other Cray XC30 computers, such as the UK ARCHER computer

From our experience using the CCSD(T) TCE module, we have determined that the optimal configuration is to use a single Global Arrays ranks for offloading work to each Xeon Phi card.

On the EMSL cascade system, each node is equipped with two coprocessors, and NWChem can allocate one GA ranks per coprocessor.
In the job scripts, we recommend spawning just 6 GA ranks for each node, instead of 16 (number that would match the number of physical cores).
Therefore, 2 out 6 GA ranks assigned to a particular compute node will offload to the coprocessors, while the remaining 6 cores while be used for traditional CPU processing duties. Since during offload the host core is idle, we can double the number of OpenMP threads for the host (OMP_NUM_THREADS=4) in order to fill the idle core with work from another GA rank (4 process with 4 threads each will total 16 threads on each node).

NWChem itself automatically detects the available coprocessors in the system and properly partitions them for optimal use, therefore no action is required other than specifying the number of processes on each node (using the appropriate mpirun/mpiexec options) and setting the value of OMP_NUM_THREADS as in the example above.

Environmental variables useful at run-time:

OMP_NUM_THREADS is needed for the thread-level parallelization on the Xeon CPU hosts

WARNING: This is just a baseline port that we have tested and validated against our QA suite. There is large room for improvement both for serial performance (compiler options) and parallel performance (use of alternative ARMCI_NETWORKs other than MPI-TS)

As an unsupported alternative, Cygwin might be used with make, perl, and gcc/gfortran version 4 installed (however several Cygwin versions are not a good match for NWChem installation, for example cygwin version 1.7.32 -- current version as of September 2014 -- fails to compile NWChem)

To start the compilation, start the Microsoft makefile utility from the top level source directory by typing "nmake". Reminder: For Compaq visual fortran don't forget to execute the "dfvars" script.

General site installation

The build procedures outlined above will allow use of NWChem within the NWChem directory structure. The code will look for the basis set library file in a default place within that directory structure. To install the code in a general, public place (e.g., /usr/local/NWChem) the following procedure can be applied:

Determine the local storage path for the install files. (e.g., /usr/local/NWChem).

Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/NWChem/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/NWChem/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

Of course users can copy this file instead of making the symbolic link described above and change these defaults at their discretion.

It is can also be useful to use the NWCHEM_BASIS_LIBRARY environment variable when testing a new installation when an old one exists. This will allow you to
overwrite the value of nwchem_basis_library in your .nwchemrc file and point to the new basis library. For example: