VRAM Limitation Handling for Large Systems

[Zhiyuan Yin]

Hi,

I am running VASP 6.4.2 with the Gamma-only GPU version on a system containing NIONS = 577 and NELEC = 6347, and NBANDS from OUTCAR is 3808. The GPU nodes I am using have 2 V100s with 32 GB of VRAM each. Job crashes due to apparent CUDA out of memory. I am doing structure relaxation of the metal substrate for molecule adsorption with:
LREAL = AUTO
PREC = Accurate
ADDGRID = .TRUE.
EDIFF = 1E-6
IBRION = 2
ISIF = 0
EDIFFG = -0.02
ISYM = 0
ENCUT = 500

Is there a best practice for balancing CPU–GPU load in large-scale systems where 32 GB VRAM is not enough?
More specifically, how can I offload more of the FFT or charge density operations to CPU memory to reduce GPU VRAM usage?
Additionally, are there environment variables or INCAR flags that can reduce memory usage on the GPU for production calculations?

Thanks.

[merzuk.kaltak]

Hello Zhiyuan Yin,

We are currently prioritizing optimization of GPU memory utilization.
However, no dedicated control tag exists to regulate CPU/GPU workload distribution or memory allocation.

Given your use of the gamma-only version of VASP, the sole remaining strategy to minimize memory consumption is reduction of the basis set.
The following tags directly influence wavefunction basis set dimensions:

• PREC

• ENCUT

• LREAL

• ROPT