3.4.3 Building with MPI

Building MITgcm to use MPI libraries can be complicated due to the variety of different MPI implementations available, their dependencies or interactions with different compilers, and their often ad-hoc locations within file systems. For these reasons, its generally a good idea to start by finding and reading the documentation for your machine(s) and, if necessary, seeking help from your local systems administrator.

The steps for building MITgcm with MPI support are:

1. Determine the locations of your MPI-enabled compiler and/or MPI libraries and put them into an options file as described in Section 3.4.2. One can start with one of the examples in:
   MITgcm/tools/build_options/
   such as linux_ia32_g77+mpi_cg01 or linux_ia64_efc+mpi and then edit it to suit the machine at hand. You may need help from your user guide or local systems administrator to determine the exact location of the MPI libraries. If libraries are not installed, MPI implementations and related tools are available including:
   • MPICH

   • LAM/MPI

   • MPIexec

2. Build the code with the genmake2 -mpi option (see Section 3.4.2) using commands such as:

     %  ../../../tools/genmake2 -mods=../code -mpi -of=YOUR_OPTFILE
     %  make depend
     %  make
   

3. Run the code with the appropriate MPI ``run'' or ``exec'' program provided with your particular implementation of MPI. Typical MPI packages such as MPICH will use something like:

     %  mpirun -np 4 -machinefile mf ./mitgcmuv
   

   Sightly more complicated scripts may be needed for many machines since execution of the code may be controlled by both the MPI library and a job scheduling and queueing system such as PBS, LoadLeveller, Condor, or any of a number of similar tools. A few example scripts (those used for our regular verification runs ) are available at: http://mitgcm.org/viewvc/MITgcm/MITgcm/tools/example_scripts/ or at: http://mitgcm.org/viewvc/MITgcm/MITgcm_contrib/test_scripts/

An example of the above process on the MITgcm cluster (``cg01'') using the GNU g77 compiler and the mpich MPI library is:

  %  cd MITgcm/verification/exp5
  %  mkdir build
  %  cd build
  %  ../../../tools/genmake2 -mpi -mods=../code \
       -of=../../../tools/build_options/linux_ia32_g77+mpi_cg01
  %  make depend
  %  make
  %  cd ../input
  %  /usr/local/pkg/mpi/mpi-1.2.4..8a-gm-1.5/g77/bin/mpirun.ch_gm \
       -machinefile mf --gm-kill 5 -v -np 2  ../build/mitgcmuv

Copyright © 2006 Massachusetts Institute of Technology

Last update 2011-01-09