Memory Usage

Memory consumption is normally not an issue for applications built on 2DECOMP&FFT. Because the scalability of the library is extremely good, simply increasing the core count would solve most memory-related problems.

For distributed FFTs with 2D decomposition, assuming the size of the input is 1X, the memory footprint of the algorithms is 5X, including:

• 1X for the input data.
• 1X for the output data.
• 2X for the send buffer and receive buffer used by the MPI_ALLTOALL(V) routine.
• 1X for the storage of the intermediate Y-pencil data.

Some of the temporary space may be allocated as needed. But for performance reason (Fortran ALLOCATE can be quite slow for large memory space), it is preferable to have them allocated and stored globally.

The following optimisations are possible and may be introduced in a future version of the library:

• In-place FFT - this allows the FFT output to be stored in the same memory address as the input. This is fairly easy to implement in C2C transforms. But for R2C transforms, because the input and the output are of different data type¹ and may be of different size due to parallel partitioning, the memory may require extra padding (as done in P3DFFT).
• In-place ALLTOALL - this allows the send buffer and receive buffer of the MPI_ALLTOALL routine to be shared. One such implementation can be found in FFTW's transpose routines, available in version 3.3.

In Practice

• When building the library, the -DOVERWRITE flag should be used whenever possible. This allows the FFT input data to be overwritten, which can save memory and may improve performance.

---

Footnotes

1. This can be achieved in Fortran very efficiently using the EQUIVALENCE statement if the input and output are both static arrays. However, this technique does not apply to dynamically allocated arrays and is strongly discouraged in modern Fortran. More discussions can be found here.