Software Installation and Compilation

---

System Requirement

This 2DECOMP&FFT library is written in Fortran. A Fortran 95 compatible compiler is the minimum requirement. In addition, the library relies on allocatable enhancement features (defined in ISO TR 15581, and officially part of Fortran 2003) and Cray pointers (for the shared-memory implementation only). These features are almost universally supported by modern Fortran compilers. The portability of 2DECOMP&FFT has been shown to be very good.

For FFT computations, the library can interface with all popular FFT libraries which, of course, need to be supplied. Some advanced software options may rely on other 3rd-party libraries.

Using Binary Package

A binary package may be available on your system (for example as part of the installation) for testing purpose. Suppose a binary package is installed at $DECOMP2D_DIR and a Fortran compiler is defined by $F90, to compile an application test.f90 using the library, do:

      $F90 -I$DECOMP2D_DIR/include test.f90 -L$DECOMP2D_DIR/lib -l2decomp_fft -o test.exe

Please note that when creating real-world applications using 2DECOMP&FFT, a pre-built binary release is unlikely to suit your needs. This is because the library has many software options that are either hardware or application dependent. Users are always recommended to build the library from source.

Building the Library from Source Using GNU Make

To build the library from source, unpack the source package and follow the instructions here. There are Makefiles located in the $DECOMP2D_DIR/src directory. Normally there is no need to modify the main Makefile. Create a Makefile.inc instead from which software options and platform dependent information can be set. A couple of sample Makefile.inc files prepared for popular hardware are provided as templates.

- Software Options

One may choose to turn on/off certain software features in $DECOMP2D_DIR/src/Makefile.inc by appending flags to the OPTIONS variable. All optional features are implemented as code branches using pre-processing flags within source code. Note that some options are not available on all systems and some are dependent on external libraries which need to be installed separately. The available options are:

• -DDOUBLE_PREC - Use this flag to build the library in double-precision mode. Single precision is the default. Click here for further instructions on switching between single precision and double precision in applications
• -DEVEN - All communication code in this library is written to support general uneven data distributions. For this reason by default MPI_ALLTOALLV is used for communication. Turn on this flag to use MPI_ALLTOALL routines instead if:
  • The global data can be evenly distributed across processors in all three pencil orientations, meaning that for a (nx, ny, nz) global data set on a (p_row, p_col) processor grid, nx & ny must be multiples of p_row and ny & nz must be multiples of p_col.
  • The global data cannot be evenly distributed but the smaller messages are padded with extra bytes to match the largest messages. This is known as padded-alltoall optimisation.
• -DGLOBAL_ARRAYS - Use this flag to enable Global Arrays integration. See more details.
• -DOCC - OCC stands for Overlap of Communication and Computation. A set of API to facilitate this in applications may be switch on by setting the -DOCC flag. An external library is required for this to function and instructions are available here.
• -DOVERWRITE - This flag indicates that it is safe to overwrite the input data of the FFTs. Turning on this flag will reduce the memory footprint of several FFT implementations and may improve the overall performance. More details about memory usage can be found here.
• -DSHM - Use this flag to turn on the System V shared-memory version of communication code. An optional package has to be downloaded and installed first. Check the Shared-memory Programming page for technical details.
• -DT3PIO - On supercomputers with LUSTRE file system, turning on this flag to use an external library T3PIO may improve application's I/O performance.

If the FFT part of the library is going to be used, users can also specify a suitable FFT engine by setting the FFT variable. 2DECOMP&FFT can interface with the following FFT libraries: FFTW (version 3.x), ACML, ESSL, MKL, FFTPACK, FFTE and even an experimental GPU engine using CUFFT. Please note that not all options are available on all systems. For example you won't find an MKL installation on a AMD Opteron-based supercomputer. For details, check the FFT page.

- System Dependent Information

Users are also responsible for providing a set of system dependent information for their specific hardware platforms, including compiler flags and external library locations. The following variables need to be set:

• For a compiler suite
  • F90 - Fortran compiler command (this often needs to be the name of the MPI wrapper - e.g. mpif90 rather than gfortran).
  • CPPFLAG - The Fortran compiler flag to turn on preprocessing.
  • OPTIM - The optimisation or debugging flags for the Fortran compiler.
  • CRAYPTR - The flag to turn on Cray pointer support, if required.
  • CC - C compiler command (this often needs to be the name of the MPI wrapper - e.g. mpicc rather than gcc).
  • CFLAGS - C compiler optimisation or debugging flags.
  The last three variables are optional - they are only required if the shared-memory code is used.
• For an external FFT library
  • INC - The include path pointing to the header files of the external library, if applicable.
  • LIBFFT - Extra parameters, such as the library path, used by compilers at link stage.
  Both may be left blank on certain platforms.

Some of the software features above require the use of external libraries. Their paths may need to be supplied at either compile or link time.

Several sample Makefile.inc are distributed with the package, supporting major platforms such as Cray XT/XE, IBM BlueGene, etc.

Once all set-up is done, type make from $DECOMP2D_DIR/src to build 2DECOMP&FFT. If successful, a library file will be created within $DECOMP2D_DIR/lib and a number of Fortran module files will be created within $DECOMP2D_DIR/include to be used by applications. To validate the library, try to build and run a few sample applications distributed with the library. Finally, please consider sending in any successful configurations to help support users with similar hardware.

- (Optionally) FreeIPC for Shared-memory Code

First of all, read this page for technical background before trying this software feature.

To use the shared-memory implementation, download the FreeIPC package, extract it into the $DECOMP2D_DIR/src directory, turn on the -DSHM flag in $DECOMP2D_DIR/src/Makefile.inc and build 2DECOMP&FFT as normal. FreeIPC, provided by Dr. Ian Bush of NAG, allows easy access to System V Inter-Process Communication API from Fortran applications. This package has to be downloaded separately because it uses a different software license.

Building Sample Applications

A number of sample applications are distributed to help validate and demonstrate the proper use of 2DECOMP&FFT (here is a list of sample applications). Each sample code is located in a separate sub-directory under $DECOMP2D_DIR/examples and is designed to test one aspect of the library. Each sample code has its own Makefile, which includes $DECOMP2D_DIR/src/Makefile.inc so options used to compile the base library are inherited to avoid potential conflicting settings. Several sample applications contain code branches defined by pre-processing directives to test those optional library features. If a sample application relies on external libraries, paths to such libraries may need to be set in the Makefile. A README file is available for each sample application. When everything is set up properly, simply type make to build.