chengis.me

SIESTA (Spanish Initiative for Electronic Simulations with Thousands of Atoms) and VASP are two commonly-used ab initio packages, for mad physicists to calculate the interesting stuffs of the atoms. But compiling the codes is a very tricky job, especially you want to compile it flawless. Both the packages supports parallel, which means you can use all your available CPU to speed up your work. Our lab got a 8-core Mac Pro, it looks awesome. I compiled one-CPU versions of both packages, and they already runs faster than I used 2-3 cpus on the university’s cluster. But I still want the challenge for a true parallel version on this sexy workstation.

So what’s the problems took me so long to complete?

• A Case-sensitive Volume! The default and trouble-free format is case-insensitive on Mac. But if you wanna build these things, at least make a .dmg and mount it. (of course you can make another partition on disk)
• The Fortran/C Compiler.
• Compile MPI support: The packages need this framework to communicate among CPUs, like network support or something.
• Compile Math Libraries: i.e. BLAS, LAPACK, BLACS, SCALAPACK. Looks geeky? Think them to be your math symbols like cos, sin, lg, etc; the packages simply use them instead of code them one by one. Of course the libraries are in fact for matrix operations instead of simple math functions, but you get it.
• Linking them. We may got many specialists as a team, but you still need to make sure they could talk to each other.

The Fortran/C Compiler

On a mac (it should be intel now) the best available compiler would be the ifort, icc provided by intel themselves. Believe me, they are fast. But they need tricker configurations when you compile. Also, they need to pay for licenses. Of course you can apply a trial license and compile your codes. If you really care about that, you may think about the open source gfortran/gcc. And some kind guys(http://hpc.sf.net) provides usable version for Mac OSX. All you need is just un-archive and use. The intel compilers are easy to install also, they use the standard .dmg and .pkg install routines.

Mission 1 Checked.

The MPI Support

In Leopard (OSX 10.5), apple have included OpenMPI framework. But very sadly, it comes with no fortran support. So we need to re-compile OpenMPI, or you can choose LAMMPI, MPICH 2 etc. They work basically the same, but you can still choose your favorite. I’d say I used OpenMPI because they had a FAQ for OSX. If you use gcc/gfortran, just follow the steps, except you need to make it 64bit! This is a lesson, so beware it: the default setting of gcc/gfortran compile 32bit files.

My options for gcc/gfortran:

./configure --prefix=/opt/openmpi.gcc \
             --with-wrapper-ldflags="-Wl,-search_paths_first" \
             CFLAGS=-m64 CXXFLAGS=-m64

The intel compilers seems to compile 64bit files by default, but you can’t be too careful. My options:

./configure --prefix=/opt/openmpi.intel\
             --with-wrapper-ldflags="-Wl,-search_paths_first" \
             CFLAGS=-m64 CXXFLAGS=-m64 \
             CC=icc CXX=icpc F77=ifort FC=ifort

And then enjoy the sexy scrolling matrix Terminal window, and then make; sudo make install to enjoy for another chance. This step is easy enough too.

Checked Mission 2!

The Math Libraries

Also I would like to mention the built-in LAPACK and ATLAS packages in Leopard, but they don’t quite seem to work neither. SIESTA needs BLAS, LAPACK, BLACS and SCALAPACK (yes, all of them) to build a parallel version. VASP seems a little more merciful, you just need BLAS and LAPACK.

As always, there is a simple way and a tricky way to do this. The simple way: you can use the SCALAPACK installer. And ask it to download and compile the libs (the coolest thing is, it builds all of them) for you. Remind you, the -m64 flag for gcc/gfortran!

The tricky way, you can use Intel’s Math Kernel Library (MKL). Don’t worry, you should already installed them with your compilers on Mac. But the worst thing is, it provides BLAS, LAPACK support except SCALAPACK. They have it on Linux/Win, but not Mac. So you should use the installer mentioned above, just download and compile. But you don’t need to use all the libs, only the SCALAPACK and BLACS.

OK, Mission 3 done.

Linking and Compiling

We finally get to the final part. But before SIESTA/VASP really work, your efforts still mean nothing. Generally speaking, you need to take care of the configuration files in the following ways:

• Compiler: mpif77/mpif90 depends on the specific package. But you should make sure it if you have several versions and do not use gfortran/ifort because they won’t compile parallel apps.
• Flags: What did I say time after time? 64bit for gfortran! And for intel compilers, you surely want a best option to work well and fast. I’m not sure about everything, but this works for me:-O3 -m64 -axSSE4.1,SSSE3 -xSSSE3 -mp1 -prec-div -pc80 -pad -ip
• Linking Libraries: for gcc/gfortran, I simply use the libraries installed by the scalapack installer. and for intel, I use the following settings: (again, it works for me and don’t make me to explain )
  
  

  MKLPATH=/opt/intel/Compiler/11.0/059/Frameworks/mkl/lib/em64t/
  INTEL_MKL=-L$(MKLPATH)\
        -I/opt/intel/Compiler/11.0/059/Frameworks/mkl/include/  \
        -lmkl_lapack\
        $(MKLPATH)/libmkl_intel_lp64.a\
        $(MKLPATH)/libmkl_intel_thread.a \
        $(MKLPATH)/libmkl_core.a\
        $(MKLPATH)/libmkl_intel_thread.a \
        $(MKLPATH)/libmkl_core.a\
        $(MKLPATH)/libmkl_intel_thread.a \
        $(MKLPATH)/libmkl_core.a\
        -liomp5 -lpthread
  LIBS=  $(SCALAPACK_LIBS) $(BLACS_LIBS)  \
         $(INTEL_MKL)

You should know the linking LIBS are just making everything together, so it doesn’t matter what intermediate variables in the makefile(or arch.make for SIESTA). Just make sure the compiler link the libraries in this way.

Mission Complete!

Ta-da

The apps should be working for now. If not, let me provide these tricky makefiles for ya. They may not work for you immediately, but I think they could be a better start than those in the package.

Updates

Peter Klaver suggested the heavy optimization -O3 could be problematic for some of the libs in VASP. If the executable is not working, please check the optimization options. Thanks Peter!

I did have to reduce the optimization level on one file, fftmpi.F90. With -O3 I get an executable that gives me errors ‘Input and output electron density differ’. But reducing optimization for fftmpi.F90 to -O1 fixes it. I had emailed someone who was using linux/ifort earlier. He too found that the mpi version produces those errors unless optimization was reduced.