Category:Parallelization

VASP makes use of parallel machines splitting the calculation into many tasks, that communicate with each other using MPI. Since a single core cannot perform enough operations, for many complex problems, this parallelization is necessary to finish the calculation in a reasonable time.

Theory

Terminology in high-performance computing (HPC)

CPU

The Central Processing Unit of a computer. A CPU may consist of multiple cores. One or two CPUs can be combined with accelerators like GPUs to form a node.

Core

Node

Socket

Memory

Dynamic random access memory (DRAM), random access memory (RAM)

Process

Patches

Rank

Cache

OpenMP thread

Message Passing Interface (MPI)

-->

Basic parallelization

By default, VASP distributes the number of bands (NBANDS) over the available MPI ranks. But it is often beneficial to add parallelization of the FFTs (NCORE), parallelization over k points (KPAR), and parallelization over separate calculations (IMAGES). All these tags default to 1 and divide the number of MPI ranks among the parallelization options. Additionally, there are some parallelization options for specific algorithms in VASP, e.g., NOMEGAPAR. In summary, VASP parallelizes with

${\displaystyle {\text{total ranks}}={\text{ranks parallelizing bands}}\times {\text{NCORE}}\times {\text{KPAR}}\times {\text{IMAGES}}\times {\text{other algorithm-dependent tags}}.}$

In addition to the parallelization using MPI, VASP can make use of OpenMP threading and/or OpenACC (for the GPU port). Note that running on multiple OpenMP threads and/or GPUs switches off the NCORE parallelization.

MPI setup

The MPI setup determines the placement of the ranks onto the nodes. VASP assumes the ranks first fill up a node before the next node is occupied. As an example when running with 8 ranks on two nodes, VASP expects rank 1–4 on node 1 and rank 5–8 on node 2. If the ranks are placed differently, communication between the nodes occurs for every parallel FFT. Because FFTs are essential to VASP's speed this inhibits the performance of the calculation. A manifestation is an increase in computing time when the number of nodes is increased from 1 to 2. If NCORE is not used this issue is less severe but will still reduce the performance.

To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

How to

Optimizing the parallelization

The performance of a specific parallelization depends on the system, i.e., the number of ions, the elements, the size of the cell, etc, as well as the algorithms, e.g., whether it is a calculation, a many-body–perturbation–theory calculation or a molecular-dynamics simulation using machine-learned force fields. To obtain trustworthy and publishable results, many projects require performing many similar calculations, i.e., calculations with similar input and using the same algorithms. Therefore, we recommend optimizing the parallelization to make the most of the available compute time.

Tip: Run a few test calculations varying the parallel setup, and use the optimal choice of parameters for the rest of the calculations.

For more detailed advice, check the following:

• How to optimize the parallelization in a nutshell

OpenMP/OpenACC

Both OpenMP and OpenACC parallelize the FFTs and therefore disregard any conflicting specification of NCORE.When combining these methods OpenACC takes precedence but any code not ported to OpenACC benefits from the additional OpenMP treads.This approach is relevant because the recommended NVIDIA Collective Communications Library requires a single MPI rank per GPU.

• How to parallelize with multiple OpenMP threads per MPI rank
• How to run on GPUs

Additional parallelization options

KPAR

For Laplace transformed MP2 this tag has a different meaning.

NCORE_IN_IMAGE1

Defines how many ranks work on the first image in the thermodynamic coupling constant integration (VCAIMAGES).

NOMEGAPAR

Parallelize over imaginary frequency points in ${\displaystyle GW}$ and RPA calculations.

NTAUPAR

Parallelize over imaginary time points in ${\displaystyle GW}$ and RPA calculations.