README: Parallel Q-Chem


Parallel Q-Chem is capable of carrying out energy (HF & DFT), gradient (HF & DFT), and second derivative (HF & DFT) calculations over multiple shared and distributed processors. Parallel second derivative calculations at MP2 and OD levels with finite difference method are also possible. Other types of calculations can be done only on a single processor. Using Parallel Q-Chem on a single processor runs exactly same as the serial version.

Please note that the instructions in this file are for-your- -information only. You may need to adopt them to the specific setup of your system. We strongly recommend your system administrator read this documentation.

Parallel Environment:

Parallel Q-Chem is implemented with the Message Passing Interface (MPI). Different implementations of the interface are used depending on the platform for which Q-Chem is compiled.

In general, the parallel environment must be properly installed for Parallel Q-Chem to work for each platform. But since Q-Chem, we have included mpich running scripts with Q-Chem distribution such that you can run parallel Q-Chem without installing any parallel packages on all the Linux and MAC platforms. You still need to set up parallel environment for other platforms.

Here are list of the various MPI implementations used by Q-Chem and the websites in which more information can be obtained for each.

MPICH (default) -- http://www-unix.mcs.anl.gov/mpi/mpich/
LAM-MPI (optional) -- http://www.lam-mpi.org/

AIX Parallel Operating Environment -- Available from IBM

SGI Message Passing Toolkit -- http://www.sgi.com/software/mpt/

HP High-Performance MPI;

Compaq MPI -- http://www.hp.com/techservers/software/cmpi.html

Global Arrays:

In order to implement a more efficient dynamic load-balancing model in Parallel Q-Chem, we have utilized the Global Arrays toolkit. Global Arrays works as another layer in conjunction with the above MPI implementations. No installation is necessary for Global Arrays to work with Q-Chem, it is already in our executables. If you however want to learn more about Global Arrays, please refer to the website: http://www.emsl.pnl.gov/docs/global/ga.html

Technical Details:

(1) Q-Chem uses two different scratch directories QCSCRATCH and QCLOCALSCR Please read README.Scratch for details.

(2) Please make sure you have set up $HOME/.rhosts file correctly. Otherwise some of the systems cannot execute parallel jobs. .rhosts file contains the names of all computing nodes.

(3) Running parallel Q-Chem interactively:

qchem [-save] [-np #processors] <input> <output> [<savename>]

The -np option specifies the number of processors. QCHEM will run on 1 processor if -np is not specified. If this command fails, please look at $QC/bin/parallel.csh file and modify mpirun, poe or prun command accordingly. Please read Chapter 2 of Q-Chem User's Guide for the meanings of -save and savename options.

(4) Running Q-Chem in a batch queue:

In order to run Q-Chem using a batch job scheduler like PBS you may have to modify the mpirun command line in $QC/bin/parallel.csh file

mpirun -np #processors -machinefile $PBS_NODEFILE $QC/exe/qcprog.exe input.in $4 >> $QCOUTFILE
mpirun -np #processors $QC/exe/qcprog.exe input.in $4 >> $QCOUTFILE

The reason is that some sites require -machinefile $PBS_NODEFILE in the mpirun command line. Failure to do so will result in running all the requested jobs on the same node. If your site does not have that problem then ignore the modification to $QC/bin/parallel.csh file. It is a good idea to run 'top' command and make sure your jobs are queued on different nodes.

(5) DEC_ALPHA users with Compaq MPI implementation will have to modify the "prun" command in $QC/bin/parallel.csh file to reflect their partition.

(6) Users on most parallel computers are required to submit their jobs through a job scheduler. Please read README.Jobscripts for sample job scripts for some common job schedulers.