OOM with supercell, vasp_gam

[patrizio_graziosi]

Dears,

I need of performing forces calculation ona supercell with 1536 atoms. It is a Gamma point calculation so I use vasp_gam on a node with 48 cpu.
The convergence is reached smoothly with ENCUT = 1000, but then it goes OOM. This continues even if I reduce ENCUT to 600. Find beolw the INCAR file.

Is reducing ENCUT the only one strategy? Can you support me, please?
Thanks,
Patrizio

ENCUT = 600
LREAL = .FALSE.
PREC = Accurate
IALGO = 38

EDIFF = 1E-6

IBRION=-1;
ISMEAR = 0 ; SIGMA = 0.05

LPLANE = .TRUE.
NSIM = 1
NCORE = 48
KPAR = 1

IVDW=12

LCHARG = .FALSE.
LWAVE = .FALSE.

[michael_wolloch]

Dear Patrizio Graziosi,

Please attach all your input files and the relevant output files (at least OUTCAR) as a .zip file, according to our posting guidelines. Also tell us what material property you are interested in calculating, what VASP version you are using, and which toolchain you used to compile the code (if you compiled it yourself).

Please also give any information you have available on the architecture and memory layout of the machine you are running on.

Please understand that there are Easter holidays right now in Austria, and I will not be able to get back to you before Tuesday, April 22nd.

Thanks, Michael

[patrizio_graziosi]

Dear Michael,

thanks for your quick reply.

I attach a zipped folder with all the files and the error slurm file. I am using 6.4.0 vasp_gam (vasp_std returns the same error) on the italian HPC cluster G100 at CINECA, equipped with 2 x CPU Intel CascadeLake 8260, each with 24 cores, 2.4 GHz, 384GB RAM DDR4. In the sbatch file I ask 370 as this is the maximum available. VASP has been compiled by them and I never encountered this type of problem before - eventual oom issues appeared at the beginning of the self-consistent loop, not after the convergence is reached, and could be sorted out acting on ENCUT.
I am working on a supercell of a molecular organic semiconductor, C12-BTBT-C12, to perform phonon calculation with phonopy. The included POSCAR corresponds to the displacement made by phonopy on the first atom.

I'm looking forward to hear from you,
Patrizio

[michael_wolloch]

Dear Patrizio,

I am a bit confused. In your OUTCAR, 20 electronic steps are completed, and you have set ENCUT to 500 eV. However, the OUTCAR is not fully complete. This would point to an out-of-memory error at the end of the run.
On the other hand, your log file also shows no output but the error. Also in the text, you write that the out-of-memory error happens at the beginning of the run, which does not fit the OUTCAR file.
Is it possible that you mixed the files from different runs?

In any case, I reproduced the OOM event from your calculation on an Intel node that has 2 24-core CPUs (Xeon(R) Gold 6336Y) and 256 GB of memory. I used ENCUT values of 500 and 600 eV, and the error occurred in both cases at the end of the run when the vdW contributions are computed. During the main loop of the calculation, the memory demand is much lower, below 40 GB, and if I remove IVDW = 12 from the INCAR file, the calculation completes without issues.

There are a few options to get your calculation to run:

• Use more nodes, or nodes with higher memory. I also ran the calculation with vdW and 600eV cutoff on a 512 GB node, and it completed. (Note that I still used 48 cores only on the larger machine, although I had more available. I don't know how the data is distributed between cores in this vdW method without checking the implementation in more detail, which I have not done.)

• If you have access to a newer VASP version (>=6.4.3), you could try to use libMBD, which should be faster and may use less memory. For more details, visit the LIBMBD_METHOD page on the wiki.

• Deactivate vdW or choose another method with less memory usage.

I would think that the first option is the most reasonable.

On a side note: I am also not sure why you would increase the ENCUT to 1000 eV if the highest ENMAX value in your POTCAR is 400 eV. This is neither necessary nor recommended.

I hope I understood your problem correctly. Let me know if you have further questions or if this resolves the issue, so I know when to lock the topic.
Cheers, Michael

[patrizio_graziosi]

Dear Michael,

thanks for reaching me. The files are correct, it was my text which was confusing. I meant that this OOM is different from the OOM I received in the past. When, in my previous calculations, I used to encounter OOM issues, it was at the beginning of the scf run, but now, I got it at the end of the scf.
The error for which I opened this ticket happens at the end of the scf, exactly as you reproduced.

About your three options, I get the first twos in once by changing cluster; unfortunately, I have very limited access to this other cluster and I am submitting a proposal (the third option is not practicable for molecular systems).

It is very interesting your side question: I use 1000 eV because I found this is a good value to reach convergence in the unit cell (not supercell) total energy, with ENCUT = 400 I get a much higher total energy.
Also, I did previous calculation with a similar molecule (same elements, same POTCAR) and I found that the phonons frequencies you get with phonopy depend greatley on the ENCUT: with ENCUT = 700 I get many negative frequencies at the Gamma point while increasing at ENCUT = 1000 only one (optical phonon) negative frequency remains. Thus, the highest ENMAX in the POTCAR was 400 eV but the computed phonon frequencies varied significantly from ENCUT to 700 to 1000.

Since in the VASP wiki, wiki/index.php/ENCUT, I read
"The convergence of the quantity of interest with respect to the energy cutoff ENCUT should always be checked."
I expected that ENCUT might need to be set at larger value than highest ENMAX in the POTCAR.
May I ask you if you think that the behavior I observe about the importance to have large ENCUT is strange?

[michael_wolloch]

Dear Patrizio,

I am glad that I understood the problem and that you were able to solve it with some of my suggestions. Note that my third option also included trying a different vdW correction. I was thinking especially about the family of nonlocal vdW-DF functionals. For those, the dispersion correction is baked into the functional, so it is calculated at every electronic step and not at the end of the calculation, so if it runs out of memory, you would at least notice this much earlier. However, I completely understand that you might need to stick to the DFT-D3 correction method for many reasons.

Now to the ENCUT question:
First of all, you are completely right that the quantity of interest (in your case, the phonon frequencies) needs to be converged with respect to ENCUT. However, 1000 eV seems very high for the chosen pseudopotentials. While it is correct that convergence studies might result in a necessary ENCUT > ENMAX, a > 100% increase might lead to problems since the pseudopotentials were not designed and tested for such a large increase. Results are probably fine, but you might run into many different issues, including convergence problems.

For phonon calculations, a lot of different parameters need to be converged, other than ENCUT. Especially convergence of the supercell size is important. You can see some examples in our phonon tutorials. If I read your POSCAR correctly, this is a 2x2x2 supercell? That might be a bigger problem regarding imaginary modes than the lower ENCUT value. But you are right in testing both. Since your unit cell is quite large, a further increase might be computationally challenging, but if you are still getting imaginary frequencies, it could be necessary. (Note that imaginary frequencies could also point to a dynamically unstable crystal structure and a possible phase transition. I have no experience with molecular systems, however, so I am hesitant to give advice here.)

Where you are a bit mistaken, if I read your last post correctly, is converging the total energy with respect to ENCUT for the primitive cell. Total energies are meaningless in DFT; only energy differences are important. For example, the energy difference between two volumes or between the unperturbed system and the one with a phonon displacement. Trying to converge the total energy for a single system is thus generally a bad idea, regardless of what parameter you are trying to converge.

If your convergence studies have indicated that a large ENCUT is necessary for this system, I would recommend using different potentials, especially the hard h variants, which require a higher plane wave cutoff energy, might be suitable. For all elements, you are using h variants are available.

Cheers, Michael

[patrizio_graziosi]

Dear Michael,

thank you so much for the explanation about both VdW (indeed changing it requires more tests) and ENCUT. I'll test the h- variant pseudopotentials.
So, if I understand, when looking for the convergence k-grid and energy cutoff, it is recommended to look at the differences in the total energies, right?
About the supercell, you are 100% correct, but a 3x2x2 grid runs out of memory. At other hand, those molecular systems have a liquid crystal transition and it is possible that the structure determined by XRD is somewhat metastable. Unfortunately a larger supercell, which might address this point, is not yet feasible.

Thanks again for your support,
Cheers,
Patrizio

[michael_wolloch]

Dear Patrizio,

So, if I understand, when looking for the convergence k-grid and energy cutoff, it is recommended to look at the differences in the total energies, right?

Well, if you are interested in total energy differences, then yes. Otherwise, not really.

For example, if you want to compare the stability of two phases of the same material, it is OK to converge an equation of state for the smaller cell, because energy differences with respect to volume will translate well to energy differences between two phases.
It is generally advised to converge with respect to the quantity you want to compute. If you want to converge phonon frequencies, you should converge those. If you want to compute a bandgap, converge with respect to the bandgap. Etc. It is just never correct to converge total energies.

If you cannot converge your phonon frequencies with respect to supercell size, I am doubtful that a higher cutoff will help much. I would rather try to use a larger cell with a lower cutoff (I am thinking of using soft potentials now), which will reduce memory requirements and speed up the calculation.
In the end, more compute nodes with more memory might be the only way to be sure...

Is there anything else you need on this topic, or can I lock it?

Cheers, Michael

[patrizio_graziosi]

Dear Michael,
that's everything, you can lock the topic, thank you so much!

Cheers
Patrizio