ASE itself comes with a NEB implementation which is capable of running NEB with climbing images, but in the case of VASP only in serial, meaning calculating the energy and forces of one image after another. The here described method advances the ASE implementation in two ways:

1. It allows to calculate images in parallel (so to have VASP running in parallel)
2. Can restart the image calculation from its previous one by initialising the WAVECAR which speeds up the calculation compared to using random wave functions as initial guess.

That makes it comparable to the VASP VTST implementation with the advantage of:

• Checking during the simulation your NEB-path == not such a black blackbox
• For large nodes (e.g. 32 cores) two images can be run on one node by defining an appropriate hostfile

With the newest versions of ASE this problems are solved and the NEBs can be run in parallel directly with one image per node by adding the following two keywords to the vasp_calculator.VASP object:

    for i in range(1,N_images+1,1):
        image = initial.copy()
        calc = vasp_calculator.Vasp(
                     directory = '{:02}'.format(i), # Directory name of image i
                     command = 'srun -N 1 -n 32 --exclusive vasp',

The command is here given for Beskow and uses per image one node exclusively. Generally one can replace with the command line the run-command as given in the vasp_run.py file for ASE. The --exclusive flag ensures that each image is run on different nodes.

If the newest version of ASE is not available one can install it using a virtual environment (venv) or makes adjustments as described below. To create the virtual environment and install ASE properly on Hebbe the following commands can be used. On other clusters you have to make sure that the here mentioned modules are all loaded. The exact version is not entirely important, beside for the iccifort and impi where at least version 2019.5 and 2018.5 have to be used respectively. Older versions have a bug which leads to problems in the mpi4py.

Command line:

    >> module purge 
    >> module load GCC/8.3.0
    >> module load Tk/8.6.9
    >> module load Tkinter/3.7.4
    >> module load Python/3.7.4
    >> module load iccifort/2019.5.281
    >> module load impi/2018.5.288

Create the virtual environment and install the needed packages:

    >> python -m venv custom-ase
    >> source custom-ase/bin/activate
    >> pip install --upgrade pip
    >> pip install --upgrade scipy numpy matplotlib mpi4py
    >> pip install --upgrade ase 

To test, if the ase version works, one can for e.g. try and open a traj file. To customise ASE it is necessary to replace two files, neb.py ( Download ) in the main ase folder and vasp.py ( Download ) in calculators/vasp/. They can be found in the virtual environment folder under lib/python3.7/site-packages/ase. The main changes are that in the vasp.py file a new key-word is introduced called '"folder"'. It is optional and if not given its value is none. In the neb.py file upon using parallel=True while submitting the run-file on one core, the calculations of the energies and forces of the images will be spread out over different processes using multiprocesses. That is done for the 'normal' NEB in ASE, which means that the initial and final picture energies are not calculated automatically.

Next, the run_vasp.py has to be adjusted to be able to find the hostfiles for the different NEB-images for submitting the VASP calculations. The path of the run_vasp.py ( Download ) is commonly given in the .bashrc-file in a form like this:

    export VASP_SCRIPT="$HOME/vasp_stuff/run_vasp.py"

Newly introduced is a routine which goes and looks for a file called hostlist_ followed by the folder name in which the VASP calculations shall be submitted.

    #! /usr/bin/env python 
    import os
    import sys

    found_hostfile=False
    submitdir=sys.argv[1]
    if not os.path.exists('run.py'):
        dirpath = os.getcwd()
        foldername = os.path.basename(dirpath)
        print (dirpath)
        print (foldername)

        if os.path.exists(submitdir+'/hostlist_'+foldername):
           hfile =  'hostlist_'+foldername
           found_hostfile = True
        else:
           print ('I have no hostfile which belongs to this folder')
           sys.exit()

    if found_hostfile:
        print ('Found the hostfile:',hfile)
        os.system('cp submitdir/hfile .')
        exitcode = os.system('mpirun -n 20 -f hfile vasp_std')

In this case all the vasp-calculations are executed on 16 cores using the hosts given in the hostfile. The number of cores is clearly adjustable to what one needs, but it should match the length of the hostfile. To run other vasp calculations one can use either a second run_vasp.py and changes with them up-on submission, or add something like:

    else:
        print ('running without hostfile!')
        exitcode = os.system('mpirun vasp_std')

Nowadays the intelmpi is good enough to detect the number of cores, etc. by itself so it does not have to be specified anymore -- at least if a bash submitting script is used. Coming to that, the submission script ( Download ) has to be also edited to create the hostfiles and activate the venv: To run on multiple Nodes with common storage:

   #SBATCH --gres=ptmpdir:1

Get the hostfiles:

    listn=$(scontrol show hostnames $SLURM_NODELIST)
    echo $listn
    a=1
    for node in $listn
    do
        echo $node':20' >> hostlist_0$a
        a=$(( $a + 1 ))
    done

This example assumes using as many nodes as images to be calculated, which is the total number of images-2. The format $node':16' is one way to specify the hosts in a file, but different forms can be used. After copying all the files to the $TMPDIR:

    cp run.py $TMPDIR 
    cp *.traj $TMPDIR
    cp hostlist_* $TMPDIR
    cd $TMPDIR

all the module have to be loaded which are needed and the venv activated:

   module purge
   deactivate
   module load GCC/8.3.0 
   module load Python/3.7.4
   module load Tk/8.6.9
   module load Tkinter/3.7.4
   module load iccifort/2019.5.281
   module load impi/2018.5.288
   module load VASP/5.4.4.16052018
   source $HOME/custom-ase/bin/activate
   echo "Do I run in the virtual env?"
   echo $VIRTUAL_ENV

afterwards the simulation can be started normally with e.g.:

    time python run.py $SLURM_SUBMIT_DIR > $SLURM_SUBMIT_DIR/result.txt

In the control (run) file ( Download ) of the ASE simulation only the folder name needs to be added in the definition of the calculator:

    calc = vasp_calculator.Vasp(
             folder = '{:02}'.format(i),

and the NEB run in parallel:

   neb = NEB(images,parallel=True)

For the example given on the ASE-webpage of an Au atom diffusion over an Aluminium surface ASE-example the following calculation times were obtained comparing different NEB set-ups:

• Nnode = 1, Ncore =16, parallel=False: 7m17.090s
• Nnode = 3, Ncore =16, parallel=False: 6m20.918s
• Nnode = 3, Ncore =16, parallel=True, wavecar=False: 5m57.191s
• Nnode = 3, Ncore =16, parallel=False, wavecar=True: 3m56.060s

The NEB can be evaluated using for example this script (Download) which gives a plot of the energy of the last 4 paths. It is just an example and can be modified in any way.