vasp.6.5.0 installation issue with OpenMpi

[bhagyashridevaru_bhat]

Hello,

I have recently got VASP license and installed vasp.6.5.0 with following
dependencies.
1. Binutils
2. GCC 11.3.0
3. OpenMPI 4.1.6
4. FFTW 3.3.10
5. OpenBLAS 0.3.21
6. Scalapack 2.2.0
STEP-1: I compiled and Installed all these dependencies (make -j4 && make
install)
STEP-2: Then chose the makefile.include.gnu_omp and edited
###########makefile.include.gnu_omp (edited)##################
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxGNU\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Dtbdyn \
-Dfock_dblbuf \
-D_OPENMP

CPP = gcc -E -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS)

FC =
/users/bhagyashribhat/Software/dependencies/openmpi/bin/mpif90 -fopenmp
FCL =
/users/bhagyashribhat/Software/dependencies/openmpi/bin/mpif90 -fopenmp

FREE = -ffree-form -ffree-line-length-none

FFLAGS = -w -ffpe-summary=none

OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0

# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = /users/bhagyashribhat/Software/dependencies/gcc-11.3/bin/gcc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS = /users/bhagyashribhat/Software/dependencies/gcc-11.3/bin/g++
LLIBS = -lstdc++

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -march=native
FFLAGS += $(VASP_TARGET_CPU)

# For gcc-10 and higher (comment out for older versions)
FFLAGS += -fallow-argument-mismatch

# BLAS and LAPACK (mandatory)
OPENBLAS_ROOT = /users/bhagyashribhat/Software/dependencies/openblas
BLASPACK = -L$(OPENBLAS_ROOT)/lib -lopenblas

# scaLAPACK (mandatory)
SCALAPACK_ROOT =
/users/bhagyashribhat/Software/dependencies/scalapack-2.2.0/build
SCALAPACK = -L$(SCALAPACK_ROOT)/lib -lscalapack

LLIBS += $(SCALAPACK) $(BLASPACK)

# FFTW (mandatory)
FFTW_ROOT = /users/bhagyashribhat/Software/dependencies/fftw
LLIBS += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp
INCS += -I$(FFTW_ROOT)/include

# HDF5-support (optional but strongly recommended, and mandatory for some
features)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT ?= /path/to/your/hdf5/installation
#LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (recommended)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = g++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(FFTW_ROOT)/include
#LIBS += fftlib
#LLIBS += -ldl

# For machine learning library vaspml (experimental)
#CPP_OPTIONS += -Dlibvaspml
#CPP_OPTIONS += -DVASPML_USE_CBLAS
#CPP_OPTIONS += -DVASPML_DEBUG_LEVEL=3
#CXX_ML = mpic++ -fopenmp
#CXXFLAGS_ML = -O3 -std=c++17 -pedantic-errors -Wall -Wextra
#INCLUDE_ML = -I$(OPENBLAS_ROOT)/include
#######################################################
########################################################

STEP3: cp arch/makefile.include.your_choice ./makefile.include
make DEPS=1 -j32 all
The compilation and installation ran withount errors.

STEP4: Now I am trying to run a basic calculation with the job submitting
script:

#!/bin/bash
#SBATCH --job-name=test3-sns2
#SBATCH --partition=workq
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=128
#SBATCH --cpus-per-task=1
#SBATCH --time=96:00:00
#SBATCH --output=vasp_%j.out
#SBATCH --error=vasp_%j.err

# Load environment
export HOME_DIR=/users/bhagyashribhat/Software/dependencies
export PATH=$HOME_DIR/openmpi/bin:$HOME_DIR/gcc-11.3/bin:$PATH
export
LD_LIBRARY_PATH=$HOME_DIR/gcc-11.3/lib64:$HOME_DIR/openmpi/lib:$HOME_DIR/openblas/lib:$HOME_DIR/scalapack-2.2.0/build/lib:$HOME_DIR/fftw/lib:$HOME_DIR/binutils/lib:$LD_LIBRARY_PATH

# Verify mpirun location
which mpirun

# Move to submission directory
cd $SLURM_SUBMIT_DIR

# Run VASP with TCP/IP to bypass InfiniBand issue
mpirun --mca btl ^openib -np 128
/users/bhagyashribhat/Software/vasp.6.5.0/bin/vasp_std

echo "Job completed successfully."
##############################################
I found that it's updating the OSZICAR and other files very very slowly.
It's taking hours for one iteration for three atom system. I have NCORE = 4
in INCAR.
Please let me know the solution as quick as possible. I am eagerly waiting
to play with vasp.

Thank you
Sincerely
Bhagyashri

[martin.schlipf]

Did you try to run without OpenMP? You can set export OMP_NUM_THREADS=1 to limit the number of threads used. From the slurm file it looks like that should already be used but it is good to double check that. You can look at the beginning of the OUTCAR file where it reports the parallel setup used.

I would also recommend starting with the simplest possible system (e.g. https://vasp.at/tutorials/latest/bulk/part1/). If your VASP installation is somehow not working probably that will already show up there. These systems should complete in a few minutes even on a single core.

Please let me know if you find out anything else.

[bhagyashridevaru_bhat]

Hi Martin,
I also tried with OMP_NUM_THREADS=1. It works.

Thank you very much.

Regards
Bhagyashri

[hszhao.cn@gmail.com]

Following up on this discussion about the VASP performance issue with OpenMP parallelization:

1. Why is it necessary to set OMP_NUM_THREADS=1 in this case? Is this related to the specific toolchain used to compile VASP?

2. Since setting OMP_NUM_THREADS=1 effectively disables OpenMP parallelization, doesn't this make the OpenMP-enabled VASP compilation unnecessary or redundant?

Regards,
Zhao

[martin.schlipf]

Setting OMP_NUM_THREADS=1 is only used for testing purposes. You can check whether the issue results from the multithreading. For production you should of course test which setting of OMP_NUM_THREADS yields the best performance.