View source for Optimizing the parallelization

To find the optimal [[:Category:Parallelization|parallelization]] setup of a VASP calculation, it is necessary to run tests for each system, algorithm and computer architecture. Below, we offer general advice on how to optimize the parallelization.

__TOC__


==Optimizing the parallelization==

For repetitive tasks, '''a few iterations estimate the performance''' of the full calculation very well.
For example, run only a few electronic or ionic self-consistency steps (without reaching full convergence).
Compare the time various parallelization setups need to perform these few iterations. 

Try to '''get as close as possible to the actual system'''.
Specifically, use the same or a very similar physical system (atoms, cell size, cutoff, ...);
run on the target computational hardware (CPUs, interconnect, number of nodes, ...).
If too many parameters are different, the parallel configuration may not be transferable to the production calculation.

In our experience, VASP yields the best performance by '''combining multiple parallelization options''' because the parallel efficiency of each level drops near its limit.
By default, VASP distributes the number of bands ({{TAG|NBANDS}}) over the available MPI ranks.
But it is often beneficial to add parallelization of the FFTs ({{TAG|NCORE}}), parallelization over '''k''' points ({{TAG|KPAR}}), and parallelization over separate calculations ({{TAG|IMAGES}}).
Additionally, there are some parallelization options for specific algorithms in VASP, e.g., {{TAG|NOMEGAPAR}} for parallelization over imaginary frequency points in <math>GW</math> and RPA calculations. In summary, '''VASP parallelizes with'''

::<math>
\text{total ranks} = \text{ranks parallelizing bands} \times \text{NCORE} \times \text{KPAR} \times \text{IMAGES} \times \text{other algorithm-dependent tags}.
</math>

To optimize the parallelization, follow this recipe:

# '''Create a list of the relevant [[:Category:Parallelization|parallelization INCAR tags]]''' for the specific calculation. '''Read the documentation for each of the relevant tags''' to understand the limits and reasonable choices.
# For any calculation involving [[:Category:Electronic minimization|electronic minimization]], it can be useful to first '''run VASP with {{TAG|ALGO}}=''None''''' set in the {{FILE|INCAR}} file. With {{TAG|ALGO}}=''None'' the [[:Category:Calculation setup|computational setup]] for the [[:Category:Electronic minimization|electronic minimization]] is done without actually performing the minimization. For instance, the FFTs are planned, and the irreducible '''k''' points of the first Brillouin zone are constructed. Therefore, some parameters, e.g., the default number of Kohn-Sham orbitals ({{TAG|NBANDS}}) and the total number of plane waves, are written to the {{FILE|OUTCAR}} file while using barely any computational time.
# Combine the information from the documentation and the [[Command-line arguments#--dry-run / -n|dry run]] into a few possible candidates for a reasonable setup. '''Run test calculations''' on a subset of the production run e.g. by reducing the number of steps.
# Run the production calculation with the best performing setup.

For the common case of [[:Category: Electronic minimization|electronic minimization]] calculations, the following rules of thumb apply:
* Aim to set the number of ranks to the default value of {{TAG|NBANDS}} divided by a small integer. Note that VASP will increase {{TAG|NBANDS}} to accommodate the number of ranks.
* Choose {{TAG|NCORE}} as a factor of the cores per node to avoid communicating between nodes for the FFTs. Mind that {{TAG|NCORE}} cannot be set with [[Combining MPI and OpenMP|OpenMP]] threading and/or the [[OpenACC GPU port of VASP|OpenACC GPU port]]. Use the number of OpenMP threads in this case to for fine grained control over parallelization.
* The '''k'''-point parallelization is efficient but requires additional [[:Category:Memory|memory]]. Given sufficient [[:Category:Memory|memory]], increase {{TAG|KPAR}} up to the number of irreducible '''k''' points. Keep in mind that {{TAG|KPAR}} should factorize the number of '''k''' points. This is especially important to reduce MPI communication between [[Optimizing_the_parallelization#Understanding_the_hardware|NUMA domains]] and compute nodes. Read also {{TAG|KPAR}} for more information.
* For bulk systems with small unit cells ({{TAG|NBANDS}} is small, NKPTS=no of k points is large), {{TAGO|NCORE|1}} and {{TAGO|KPAR|NKPTS}} is optimal.
* For running on GPUs: 
** MPI ranks = no of GPUs
** {{TAGO|KPAR|no of GPUs}} if memory allows
** [[Combining_MPI_and_OpenMP|OpenMP threading]] can be quite important for the parts that still run on the host, because usually no of GPUs is rather small.
* Finally, use the {{TAG|IMAGES}} tag to split several VASP runs into separate calculations. The limit is dictated by the number of desired calculations.

==Understanding the hardware==

Optimizing the parallel execution of VASP depends greatly on the hardware architecture, network topology for MPI communication, and [[:Toolchains|compiler / library toolchain]]. Hence, it is a good start to first get familiar with the computer system you are working with i.e. look out for specific documentation of your HPC center for submitting jobs. Keywords to look out for might be: ''submitting jobs'', ''distribution and binding'', ''MPI rank and thread binding'', ''rank / thread affinity'', ''hybrid MPI and OpenMP'' etc. Usually, HPC facilities put great work in optimizing the computer system in a very special way and one has to follow their recommendations to get the most out of the compute resources.

Each hardware setup is unique since nowadays CPUs and GPUs alike consists not of a single monolithic core anymore. Each processor might consist of multiple tiles or domains that each are connected to their own memory. To leverage the full potential of the hardware one has to reduce communication between cores that are "far away" from each other. To understand this better the concept of NUMA (Non-Uniform Memory Access) domains is useful. In NUMA architecture, memory access times vary depending on the proximity of the memory to the CPU core requesting it — local memory access is quicker, while remote access across domains incurs latency. Modern multi-socket and multi-core systems use NUMA to improve scalability and performance by keeping data close to the cores that use it most often. Understanding and managing NUMA domains is essential for optimizing memory placement and parallel performance in high-performance computing. 

[[File:Numa example.png|alt=NUMA example Epyc 7543|none|thumb|950x950px|Example output of <code>lstopo</code> command for an AMD Epyc 7543 processor. The processor consists of multiple hardware tiles that display as 4 distinct NUMA domains.]]

The picture above displays the NUMA architecture of an AMD Epyc 7543 processor. In total this processor has 512 GB of system memory available. However, this memory is spread across 4 NUMA nodes (pink bars). Each NUMA domain consists of 8 physical CPU cores (each able to work on 2 threads, i.e. P#0/P#32 belong to the same physical core). These domains share not only a chunk of the system memory, but also their own L3 cache. Hence, it is important for this specific processor that not more than 8 cores work on a specific task. If more cores work on the same memory they have to communicate to the other NUMA domains with some latency. This is important for choosing parallelization flags. Hence, we know now that 8 cores is a good work group size for VASP on this processor, and that up to 32 cores can work on the same memory with a small penalty. If we use more than 32 cores we have to communicate between two different processors (nodes / sockets) via a different protocol (MPI); much higher latency. If the tool <code>lstopo</code> is not available on your system you can use the command <code>numactl --hardware</code> to get text output of the same information:

 # numactl --hardware
 available: 4 nodes (0-3)
 node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
 node 0 size: 128262 MB
 node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
 node 1 size: 129018 MB
 node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
 node 2 size: 129018 MB
 node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
 node 3 size: 129003 MB
 node distances:
 node   0   1   2   3 
  0:  10  12  12  12 
  1:  12  10  12  12 
  2:  12  12  10  12 
  3:  12  12  12  10 


Another point that immediately follows from the concept of NUMA domains is that MPI ranks should not jump between NUMA domains; they would loose access to their previous cache. To avoid this one uses ''pinning'' or ''binding'' of MPI ranks. This becomes also important for combining MPI with OpenMP threading. Threads should live close to their parent rank and also not move to share the same caches, i.e. live in the same NUMA domain. Binding ranks and threads to specific cores or regions depends on the software setup. For example in the popular SLURM jobscript submission system this is done via the flag <code>--cpu-bind=cores</code> , for openmpi this is done via <code>--bind-to=core</code>, and for intel MPI via <code>-genv I_MPI_PIN=ON</code>. To bind threads use the environment variable <code>OMP_PROC_BIND</code> and set it to <code>true</code> or <code>close</code>. For threads it is also important to tell them where they should run via <code>OMP_PLACES</code>, which best is set to <code>cores</code>.  


==Related tags an articles==
[[:Category:Parallelization|Parallelization]], {{TAG|KPAR}}, {{TAG| NCORE}}, {{TAG|KPAR}}, {{TAG|IMAGES}}

==References==

[[Category:Parallelization]][[Category:Howto]]