|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix |
Intel has the most idiotic Web site I ever encounter among software vendors. It is very difficult to get to the download even if you have the license. You need specific version of compiler and libraries:
ifort
(version 12.1.3) You need at least Intel-Composer-XE (which is essentially Fortran compiler, C++ compiler + libraries bundle). See comparison table at https://software.intel.com/en-us/intel-xe-product-comparison.
You can also use gcc and use C preprocessor from gcc to compile VASP but this does not makes much sense as performance of OpenMPI might well be critical.
Purchase Options: Intel® Cluster Studio XE Intel® Parallel Studio XE Intel® C++ Studio XE Intel® Fortran Studio XE Intel® Composer XE Intel® C++ Composer XE Intel® Fortran Composer XE Intel® C / C++ Compiler Y Y Y Y Y Intel® Fortran Compiler Y Y Y Y Y Intel® Math Kernel Library3 Y Y Y Y Y Y Y
If you bought single user license, you generally are better off installing compiler under the user you plan to use vasp, for example vasp.
I think two best recommendation for vasp compilation are:
For the parallel compilation of VASP the following software is mandatory:
- Fortran Compiler
- An implementation of MPI (Message Passing Interface)
- Numerical libraries like BLAS, LAPACK, FFTW, ScaLAPACK
Intel Composer XE (12.1.3)
When installing the Intel Composer XE software package, both the Intel Fortran compiler
ifort
(version 12.1.3) as well as the Intel Math Kernel Library (MKL) in version 10.3 is installed.The Intel MKL includes libraries for BLAS, LAPACK ScaLAPACK and their own implementation of FFT. No additional numerical libraries have to be compiled and installed. Make sure that the following file is sourced in your
~/.bashrc
:source /opt/intel/composerxe/bin/compilervars.sh [ia32|intel64]where you should use
ia32
for 32bit andintel64
for 64bit operating systems.Intel MKL FFTW wrappers
VASP internally uses the calling statements of the FFTW3 which differ to the implementation of the FFT in the MKL. Therefore we have to build wrapper routines which are shipped with the MKL but not compiled:
cd $MKLROOT/interfaces/fftw3xf/ makeAfter a successful compilation
libfftw3xf_intel.a
was created in this directory.openMPI
Download a version of openMPI (we use version 1.4.5) and untar the tarball.
tar -xjvf openmpi-1.4.5.tar.bz2 cd openmpi-1.4.5Use
./configure --helpto get help for all the compilation options available. We used the following configure line, ensuring the Intel Fortran and C compilers are used
./configure CC=icc CXX=icpc F77=ifort FC=ifort --prefix=PREFIXand where
PREFIX
is the absolute path to the directory where it should be installed on your system. Use the following naming for the final directory to know which Fortran compiler was usedopempi-1.4.5-INTEL-12.1.3
. After configuration has finished execute:make make installProbably you need root privileges to execute the last command. Finally we have to make the openMPI directories available for your bash. Therefore we include the following statement in the
~/.bashrc
if [ -f /opt/intel/composerxe/bin/compilervars.sh ] then source /opt/intel/composerxe/bin/compilervars.sh intel64 2> /dev/null export MPI=[PREFIX] export PATH=$MPI/bin:$PATH export LD_LIBRARY_PATH=$MPI/lib:$LD_LIBRARY_PATH export MANPATH=$MANPATH:$MPI/share/man export INFOPATH=$INFOPATH:$MPI/share/man fiIn the first line we test whether the Intel composer is installed by checking whether the
compilervars.sh
file exists. Then we source the appropriate compiler and the openMPI version compiled with this compiler.PREFIX
is the directory you installed openMPI.Compilation
The following flags should be changed in the makefile, so that the most recent version of VASP compiles.
FCWe are using openMPI for the parallel MPI environment and therefore we have to compile with the MPI version of the Fortran compiler
mpif90
. Mind that the used Fortran compiler has to be identical to the compiler used to compile openMPI.FC=mpif90FCLThe
-mkl
option tells the ifort compiler to link all necessary MKL libraries. Mind that this uses the ordinary LAPACK routines and not ScaLAPACK. Therefore you can comment allBLAS
,LAPACK
,BLACS
,SCA
statements in the makefile.FCL=mpif90 -mklFFLAGS
Some namings of compiler flags changed in the most recent version of the Intel Fortran compiler. This is optional, but will prevent warnings and in the future it has to be changed since the old flags are deprecated.
FFLAGS = -free -names lowercase -assume bytereclFFTW
Include the Intel MKL FFT wrapper library which we compiled at the beginning of this section.
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o $(MKLROOT)/interfaces/fftw3xf/libfftw3xf_intel.aSCA
If one wants to use ScaLAPACK routines instead of LAPACK the following line will activate ScaLAPACK support in VASP and includes the necessary libraries. Mind that you have to recompile certain files to add support for ScaLAPACK.
SCA= -lmkl_scalapack_lp64.a -lmkl_blacs_openmpi_lp64Intel© Math Kernel Library Link Line Advisor
A very handy site is the Intel© Math Kernel Library Link Line Advisor. With its help you get a recommendation for the best libraries which you should use on your particular system.
Add profiling support
To add profiling support in the most recent version of VASP you have to add
-DPROFILING
to the preprocessor flags underCPP
CPP = $(CPP_) -DMPI ...... -DPROFILINGAfterwards execute
make clean make vaspto recompile all source files of VASP. For further reading see Profiling
Hybrid openMPI/openMP parallelization
To enable explicit openMP support in VASP add
-openmp
to theFC
line in the makefileFC=mpif90 -openmpSince this subject is more complex see Hybrid openMPI/openMP parallelization fur further information
Here is a copy of Peter Larsson notes which looks like the best and the most current from many similar notes on the WEB, see original at How to compile VASP on NSC's Triolith Jan 9th, 2013. the site contains additional interesting material and is highly recommended. You can compare Peter Larsson's recommendations with Cluster installation of VASP with Intel Compilers by Lev Lafayette
... You need both the regular VASP source code, and the supporting “vasp 5” library:
vasp.5.3.3.tar.gz vasp.5.lib.tar.gz
I suggest to make a new directory called e.g. vasp.5.3.3, where you download and expand them. You would type commands approximately like this:
mkdir 5.3.3 cd 5.3.3 (download) tar zxvf vasp.5.3.3.tar.gz tar zxvf vasp.5.lib.tar.gz
Currently, you want to load these modules:
intel/12.1.4 impi/4.0.3.008 mkl/10.3.10.319
Which you can get bundled in the following module:
VASP 5 libmodule load build-environment/nsc-recommended
Compiling the VASP 5 library is straightforward. It contains some timing and IO routines, necessary for VASP, and LINPACK. My heavy edited makefile looks like this:
.SUFFIXES: .inc .f .F #----------------------------------------------------------------------- # Makefile for VASP 5 library on Triolith #----------------------------------------------------------------------- # C-preprocessor CPP = gcc -E -P -C -DLONGCHAR $*.F >$*.f FC= ifort CFLAGS = -O FFLAGS = -Os -FI FREE = -FR DOBJ = preclib.o timing_.o derrf_.o dclock_.o diolib.o dlexlib.o drdatab.o #----------------------------------------------------------------------- # general rules #----------------------------------------------------------------------- libdmy.a: $(DOBJ) linpack_double.o -rm libdmy.a ar vq libdmy.a $(DOBJ) linpack_double.o: linpack_double.f $(FC) $(FFLAGS) $(NOFREE) -c linpack_double.f .c.o: $(CC) $(CFLAGS) -c $*.c .F.o: $(CPP) $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f .F.f: $(CPP) .f.o: $(FC) $(FFLAGS) $(FREE) $(INCS) -c $*.f
Note the addition of the “-DLONGCHAR” flag on the CPP line. It activates the longer input format for INCAR files, e.g. you can have MAGMOM lines with more than 256 characters. Now compile the library with the “make” command and check that you have the “libdmy.a” output file. Leave the file here, as the main VASP makefile will include it directly from here.
VASP 5 binaryPreparations
I only show how to build the parallel version with MPI and SCALAPACK here, as that is what you should run on Triolith. Navigate to the “vasp.5.3” library where the main source code is:
cd .. cd vasp.5.3
Before we start, we want to think about how to find the external libraries that we need. These are:
- BLAS/LAPACK (for basic linear algebra)
- FFT library (for fast Fourier transform from reciprocal to real space)
- MPI (for parallel communication)
- SCALAPACK (for parallel linear algebra, e.g. orthogonalization of states)
For BLAS/LAPACK, we are going to use Intel’s Math Kernel Library (“MKL” henceforth). The easiest way to link to MKL at NSC is by adding the two following flags to the compiler command:
ifort -Nmkl -mkl=sequential ...
For fast Fourier transforms, we could use the common FFTW library with VASP, but MKL actually contains its own optimized FFTs together with an FFTW interface, so we can use these instead. Provided that we link with MKL, which we are already doing in order to get BLAS/LAPACK, we do not need to do anything more. The linker should pick up the FFTW subroutines automatically.
For MPI, we are going to use Intel’s MPI library. We have already loaded the “impi/4.0.3.008” module, so all we have to do is to add the “-Nmpi” flag to compiler command:
ifort -Nmpi ...
We don’t need to add explicit paths to any MPI libraries, or use the special “mpif90” compiler wrapper.
Editing the makefileI suggest that you start from the Linux/Intel Fortran makefile:
cp makefile.linux_ifc_P4 makefile
It is important to realize that the makefile is split in two parts, and is intended to be used in an overriding fashion. If you don’t want to compile the serial version, you should enable the definitions of FC, CPP etc in the second half of the makefile to enable parallel compilation. These will then override the settings for the serial version.
Start by editing the Fortran compiler and its flags:
FC=ifort -I$(MKL_ROOT)/include/fftw FFLAGS = -FR -lowercase -assume byterecl -Nmpi
We need to add “-Nmpi” to get proper linking with Intel MPI at NSC. Then, we change the optimization flags:
OFLAG=-O2 -ip -xavx
This is to be on the safe side, so that we get AVX optimizations. Include MKL with FFTW like this:
BLAS = -mkl=sequential LAPACK =
We use the serial version of MKL, without any multithreading, as VASP runs MPI on all cores with great success. Set the NSC specific linking options for MKL and MPI:
LINK = -Nmkl -Nmpi
Uncomment the CPP section for the MPI parallel VASP:
CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \ -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \ -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK -DRPROMU_DGEMV -DRACCMU_DGEMV
Change it to something like this:
CPP = $(CPP_) -DMPI -DHOST=\"TRIOLITH-BUILD01\" -DIFC \ -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \ -DMPI_BLOCK=262144 -Duse_collective -DscaLAPACK \ -DRPROMU_DGEMV -DRACCMU_DGEMV -DnoSTOPCAR
CACHE_SIZE is only relevant for Furth FFTs, which we do not use. The HOST variable is written out in the top of the OUTCAR file. It can be anything which helps you identify this compilation of VASP. The MPI_BLOCK variable needs to be set higher for best performance on Triolith. And finally, “noSTOPCAR” will disable the ability to stop a calculation by using the STOPCAR file. We do this to improve file I/O against the global file systems. (Otherwise, each VASP process will have to check this file for every SCF iteration.)
Finally, we enable SCALAPACK from MKL:
SCA= -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
And the parallelized version of the fast Fourier transforms with FFTW bindings:
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
Note that we do not need to link to FFTW explicitly, since it is included in MKL. Finally, we uncomment the last library section:
LIB = -L../vasp.5.lib -ldmy \ ../vasp.5.lib/linpack_double.o \ $(SCA) $(LAPACK) $(BLAS)
We have to do this to include the “$(SCA)” variable. The full makefile can be found here on Triolith:
/software/apps/vasp/5.3.3-18Dec12/build01/makefileCompilingVASP does not have a makefile that supports parallel compilation. So in order to compile we just do:
make... ... ...
When finished, you should find a binary called “vasp”.
RunningWhen you compile according to these instructions, there is no need to set LD_LIBRARY_PATHs and such. Instead, the ifort compiler will hard-code all library paths by using the RPATH mechanism and write information into the binary file about which MPI version you used. This means that you can launch VASP directly like this in a job shell:
mpprun /path/to/vasp
Mpprun will automatically pick up the correct number of processor cores from the queue system and launch your vasp binary using Intel’s MPI launcher.
Posted by Peter Larsson Jan 9th, 2013
|
||||
Bulletin | Latest | Past week | Past month |
|
Submitted by lev_lafayette on Thu, 02/13/2014 - 05:25Previous comments concerning VASP installs still largely apply.
- It still performs ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set.
- It still has an weird and frustrating license which is open source (if you pay them) but not free.
- It still is released without standard autoconfiguration tools and with make files for a variety of system architectures which rarely match was diverse users might wish to use, thus requiring posts like this every time a new version is released.
In this instance we start of with a copy of VASP in the user's home directory. Make a directory called vasp (mkdir vasp) and and extract the tarball of the two VASP files there (tar xvf vasp.5.lib, tar xvf vasp.5.3).
# tar xvf vasp.5.lib.tar.gz
# tar xvf vasp.5.3.3.tar.gz
# module load openmpi-intel
# module load fftw/3.3.2-intel
# cd vasp.5.lib
# cp makefile.linux_ifc_P4 makefile
# vim makefileModify to the following (seriously guys, ifc?):
# diff makefile makefile.linux_ifc_P4
19c19
< FC=ifort
---
> FC=ifcAnd install..
# make
Change to VASP
# cd ../vasp.5.3
# cp makefile.linux_ifc_P4 makefile
# vim makefile
Modify to the following:
diff makefile makefile.linux_ifc_P4
62c62
< FC=mpif90
---
> FC=ifort
149,151c149
< #BLAS= -lguide -mkl
<
< BLAS= -mkl
---
> BLAS= -lguide -mkl
160,162c158
< #LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
<
< LAPACK= /usr/local/intel/composerxe/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_intel_lp64.a
---
> LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
209,210c205,206
< FC=mpif90
< FCL=$(FC)
---
> #FC=mpif90
> #FCL=$(FC)
227,230c223,226
< CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
< -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
< -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK
< # -DRPROMU_DGEMV -DRACCMU_DGEMV
---
> #CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
> # -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
> # -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK
> ## -DRPROMU_DGEMV -DRACCMU_DGEMV
245,247c241,243
< LIB = -L../vasp.5.lib -ldmy \
< ../vasp.5.lib/linpack_double.o \
< $(SCA) $(LAPACK) $(BLAS)
---
> #LIB = -L../vasp.5.lib -ldmy \
> # ../vasp.5.lib/linpack_double.o \
> # $(SCA) $(LAPACK) $(BLAS)
257,258c253
< FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /usr/local/fftw/3.3.2-intel/lib/libfftw3.a
< # /opt/libs/fftw-3.1.2/lib/libfftw3.a
---
> #FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.aDamned if I could find the guide libraries. Note the hardcoded path (ugh). I could recompile Intel...
.. and install
# makeThe code should now compile and generate a 'vasp' executable, which the user should then be able to test, once changed to their user and group. Don't forget the OMP_NUM_THREADS issue previously cited.
Initial executable worked only in serial. Still fighting this and will update as progress continues. "A luta continua", as the Portuguese say.
Intel® Math Kernel Library (Intel® MKL) 11.0 includes a wealth of routines to accelerate application performance and reduce development time. Today's processors have increasing core counts, wider vector units and more varied architectures. The easiest way to take advantage of all of that processing power is to use a carefully optimized computing math library designed to harness that potential. Even the best compiler can't compete with the level of performance possible from a hand-optimized library.
The following table compares the time to run the same benchmark case as above using both executables. Our executable shows speed up with multiple CPUs.
Number of CPUs vasp compiled here /share/apps/vasp.4.6/vasp vasp another build (below) 1 76 (seconds) 75 (seconds) 48 (seconds) 2 59 (seconds) 72 (seconds) 34 (seconds) 4 35 (seconds) 64 (seconds) 29 (seconds) 8 37 (seconds) 65 (seconds) 31 (seconds) ... ... ...
This will create executable vasp in this directory. This time the executable vasp can not run interactively, but can only run in the queue through a PBS script (e.g. mpiexec --comm=ib -np 4 vasp). Notice that here we need to specify the communication channel. (For mvapich1 we need to use --comm=ib and for mvapich2 use --comm=pmi.)
The following table provides the timing information for the same benchmark case studied above.
Number of CPUs vasp compiled here 1 40 (seconds) 2 28 (seconds) 4 27 (seconds) 6 29 (seconds) 8 31 (seconds) 16 215 (seconds) A Word of Caution: Make sure to run a few test cases to confirm your executable not only runs but produces the correct numerical results. For example, we have found that on su-ahpcrc, function BRMIX (broyden.f) was giving serious errors. This was solved by changing the compilation options to
"OFLAG=-O1 -mtune core2 -axW -unroll",
and by changing "ICHARG = 0" to "ICHARG = 2" in INCAR. ("ICHARG=2" is the default when "ISTART=0" or if there are no CHG, CHGCAR, WAVECAR files in the folder.)
... ... ...
Intel MKL instructions
There are "official" instructions to compile VASP with the Intel Compiler family, named Using Intel MKL in VASP. Those instructions are unrelated with the present instructions but can be a good reference for future builds.
I used the ifort 10.1.018 version to compile the VASP(a chemical soft).All the parameters were set up as following(in the makefile):
Makefile:
#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)OFLAG=-O1 -xW -tpp7
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
............
Output:
............
............
ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signalsfortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3
========================================
I have no idea whats wrong with my compilation..
if anyone can help me figure it out.. My email is [email protected]
Thanks.
For more complete information about compiler optimizations, see our Optimization Notice.
Kevin Davis (Intel)Wed, 10/22/2008 - 05:21
The backend component of the compiler suffered an internal compilation error. Since you are already using the latest 10.1.018 update but only a few optimization switches, you could try the first work-around of dropping the optimization from O1 to O0 in OFLAG. If that avoids the error, then use a trial and error search to see if you can identify which source file corresponding to the objects linked triggers the error with compiled at O1. It is also possible the error is triggered by xW so you could do a similar search involving that option. You probably noted the tpp7 is an invalid option, so you can remove it.xqzhang
If the source for VASP can be provided to us then please report the error to Intel Premier here and indicate that you reported this error in the User forum.
You could also consider trying the 11.0 Beta compiler. See the permanent announcement regarding that at the top of this Forum.
Shane Story (Intel)Wed, 10/22/2008 - 13:22
Quoting - Kevin Davis (Intel)The backend component of the compiler suffered an internal compilation error. Since you are already using the latest 10.1.018 update but only a few optimization switches, you could try the first work-around of dropping the optimization from O1 to O0 in OFLAG. If that avoids the error, then use a trial and error search to see if you can identify which source file corresponding to the objects linked triggers the error with compiled at O1. It is also possible the error is triggered by xW so you could do a similar search involving that option. You probably noted the tpp7 is an invalid option, so you can remove it.
If the source for VASP can be provided to us then please report the error to Intel Premier here and indicate that you reported this error in the User forum.
You could also consider trying the 11.0 Beta compiler. See the permanent announcement regarding that at the top of this Forum.
Hi, thanks for your reply.. i tried your way, but still get the same error.
Then i switched to another pc, and changed alittle when linking the blas library by MKL as following:
ifort flag:
OFLAG=-O3 -xW -tpp7
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl_P4 -lguide -lpthrea
=============================================================
for this time, i got a different error:
===========================error=============================
ipo: error #11021: OBJREAD Error: Could not create mapping for /opt/intel/mkl/10.0.5.025/lib/32/
ifort: error #10014: problem during multi-file optimization compilation (code 1)make: *** [vasp] Error 1============================================================
so,what should i do now?
thanks for your help. :)
Shane Story (Intel)Wed, 10/22/2008 - 17:15
Quoting - xqzhangHi, all
I used the ifort 10.1.018 version to compile the VASP(a chemical soft).
All the parameters were set up as following(in the makefile):
Makefile:
#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)OFLAG=-O1 -xW -tpp7
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
............
Output:
............
............
ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signalsfortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3
========================================
I have no idea whats wrong with my compilation..
if anyone can help me figure it out.. My email is [email protected]
Thanks.
Zach LukaszekWed, 10/22/2008 - 17:19
Quoting - xqzhangHi, all
I used the ifort 10.1.018 version to compile the VASP(a chemical soft).
All the parameters were set up as following(in the makefile):
Makefile:
#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)OFLAG=-O1 -xW -tpp7
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# on P4, VASP works fastest with the libgoto library
# so that's what I recommend
#-----------------------------------------------------------------------# Atlas based libraries
#ATLASHOME=/opt/Linux_P4SSE2/lib
#ATLASHOME= $(HOME)/archives/BLAS_OPT/ATLAS/lib/Linux_P4SSE2/
#BLAS= /opt/Linux_P4SSE2/lib -lf77blas -latlas -lmkl
#BLAS= -L$(ATLASHOME) -lf77blas -latlas# use specific libraries (default library path might point to other libraries)
#BLAS= $(ATLASHOME)/libf77blas.a $(ATLASHOME)/libatlas.a# use the mkl Intel libraries for p4 (www.intel.com)
# mkl.5.1
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl_p4 -lpthread# mkl.5.2 requires also to -lguide library
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
BLAS= /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
#BLAS=-L/opt/intel/mkl/lib/32 -lmkl -lguide -lpthread# even faster Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#BLAS= /opt/libs/libgoto/libgoto_p4_512-r0.6.so# LAPACK, simplest use vasp.4.lib/lapack_double
LAPACK= ../vasp.4.lib/lapack_double.o# use atlas optimized part of lapack
#LAPACK= ../vasp.4.lib/lapack_atlas.o -llapack -lcblas# use the mkl Intel lapack
#LAPACK= -lmkl_lapack
............
Output:
............
............
ifort -FR -lowercase -O1 -tpp7 -xW -prefetch- -prev_div -unroll0 -vec_report3 -c fft3dlib.f90
ifort: command line remark #10148: option '-tp' not supported
ifort: command line warning #10156: ignoring option '-p'; no argument required
ifort -FR -lowercase -FR -O0 -c main.f90
rm -f vasp
ifort -o vasp main.o base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o chain.o dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o fft3dfurth.o fft3dlib.o -L../vasp.4.lib -ldmy ../vasp.4.lib/linpack_double.o ../vasp.4.lib/lapack_double.o /opt/intel/mkl/10.0.5.025/lib/32 -lmkl -lguide -lpthread
backend signalsfortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.
ifort: error #10014: problem during multi-file optimization compilation (code 3)make: *** [vasp] Error 3
========================================
I have no idea whats wrong with my compilation..
if anyone can help me figure it out.. My email is [email protected]
Thanks.
For instructions on how to edit the VASP makefile to use MKL, see this application note:
http://www.intel.com/support/performancetools/libraries/mkl/sb/CS-028850.htm
Thanks, Shane
Tue, 05/29/2012 - 10:59
Dear future people: I actually just ran into this issue minutes ago while compiling Q-Chem 4.0 with parallel flags at the ohio supercomputing center. I was using intel-compilers-10.0.023 and updating to 11.1.056 fixed the problem immediately for me. I don't know if this is useful to anyone, but I figured there should be a record of it for future googlers.
| software.intel.com
This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI.
Building VASP* with Intel® MKL and Intel® CompilersStep 1 - Overview
This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI.
VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudo potentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate. [Ref: VASP]
Version Information
This application note was created to help users who benchmark clusters using VASP to also incorporate the latest version of Intel® MKL on Linux platforms on Xeon systems. Specifically, we'll address Intel® MKL version 11.1
More information on VASP can be found from http://cms.mpi.univie.ac.at/vasp/
Step 2 - Downloading VASP Source Code
VASP is not public-domain or share-ware, and will be distributed only after a license contract has been signed. Please visit VASP homepage to know more details on obtaining the license.
Prerequisites
- Intel® MKL contains highly optimized FFT and also the wrappers for FFTW, which can be obtained from the following options:
- Download a FREE evaluation version of the Intel® MKL product.
- Download the FREE non-commercial* version of the Intel® MKL product.
- Purchase a copy of Intel® MKL
All of these can be obtained at: http://www.intel.com/software/products/mkl.
- Intel® C++ and FORTRAN compilers can be obtained from http://www.intel.com/softwre/products/compilers.
- Intel MPI can be obtained from http://www.intel.com/software/products/cluster.
Open source MPI (MPICH2) can be obtained from http://www-unix.mcs.anl.gov/mpi/mpich/ .
Step 3 - Configuration
Use the following commands to extract the VASP files:
view sourceprint?
1
$tar -xvzf vasp.tgz vasp.lib.tgz
This will create vasp and vasp.lib directories
Make sure that MPI, C++ and FORTRAN compilers are installed and in the PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and Intel® MKL libraries.
Note: This application note is written specifically for use with the Intel compilers and MPI.
Step 4 - Building VASP
- Build FFTW wrapper library for Intel® MKL
Change directory to <your MKL installation>/interfaces/fftw3xf
Assuming default installation
view sourceprint?
1
$cd /opt/intel/mkl/interfaces/fftw3xf
Build the fftw 3.x FORTRAN wrapper library for Intel® 64 (previously known as em64t) platform using the Intel compilers.
view sourceprint?
1
$make libintel64 compiler=intel
This will create the fftw wrapper library "libfftw3xf_intel.a" in lib/em64t directory of your Intel® MKL installation.
- Build libdmy.a
Change directory to vasp.x.x.lib
Modify the makefile.linux_ifc_P4 Makefile to point to the correct Intel FORTRAN compiler.
view sourceprint?
1
FC=ifort
Run the following command from vasp.x.x.lib, using the Makefile for linux using the Intel compiler.
view sourceprint?
1
$make -f makefile.linux_ifc_P4
After a successful compilation, libdmy.a will be built in the same directory.
- Build VASP
Change directory to vasp.x.x
Export the environment variable MPIFC to point to the Intel MPI Fortran compiler mpiifort.
view sourceprint?
1
$export MPIFC=mpiifort
Edit the makefile.linux_ifc_P4 to link with Intel® MKL libraries and change the Fotran compiler
Under FORTRAN compiler and linker part of the makefile
view sourceprint?view sourceprint?
1
FC=$(MPIFC)
1
CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
2
-DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
3
-DMPI_BLOCK=64000 -DscaLAPACK -Duse_collective -Ddo_loops -DMKL_ILP64
Change the FORTRAN flags section as shown here.
view sourceprint?view sourceprint?
1
FFFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw -xAVX
01
MKLROOT=/opt/intel/composer_xe_2013_sp1/mkl
02
MKL_PATH=$(MKLROOT)/lib/intel64
03
MKL_FFTW_PATH=$(MKL_PATH)/interfaces/fftw3xf
04
05
BLAS= -mkl=cluster
06
07
LAPACK= -mkl=cluster
08
09
…
10
.……
11
…………
Comment out the existing FFT3D line
view sourceprint?view sourceprint?
1
#FFT3D = fftdfurth.o fftdlib.o
1
FFT3D= fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_intel.a
2
3
INCS = -I$(MKLROOT)/include/fftw
To use Intel® MKL ScaLAPACK, modify additional options for CPP for parallel version adding
-DscaLAPACK to the CPP line as mentioned above and specify Intel® MKL ScaLAPACK as below.
view sourceprint?
1
SCA=$(MKL_PATH)/libmkl_scalapack_ilp64.a $(MKL_PATH)/libmkl_intelmpi_ilp64.a
Run the following command to build vasp
view sourceprint?
1
$make -f makefile.linux_ifc_P4
This will create the VASP executable in the current directory.
Step 5 - Running VASP
Run vasp by executing mpiexec command with your required parameters. Below, for e.g., to run 32 processes use as shown with your workloads.
view sourceprint?
1
$mpiexec -n 32 ./vasp
Appendix B - Known Issues and Limitations
There could be a compilation error on nonlr.F file as reported below.
view sourceprint?
01
mpif90 -fc=ifort -FR -names lowercase -assume byterecl -O2 -ip -I/opt/intel/compiler/2013_sp1.1.106/composer_xe_2013_sp1.1.106/mkl/include/fftw -c nonlr.f90
02
03
nonlr.F(3069): error #6404: This name does not have a type, and must have an
explicit
type. [LM]
04
DO LM=1,LMMAXC
05
nonlr.F(3069): error #6404: This name does not have a type, and must have an
explicit
type. [LM]
06
DO LM=1,LMMAXC
07
----------------^
08
nonlr.F(3069): error #6063: An INTEGER or REAL data type is required in
this
context. [LM]
09
DO LM=1,LMMAXC
10
----------------^
11
nonlr.F(3206): error #6404: This name does not have a type, and must have an
explicit
type. [LM]
12
DO LM=1,LMMAXC
13
----------------^
14
nonlr.F(3206): error #6063: An INTEGER or REAL data type is required in
this
context. [LM]
15
DO LM=1,LMMAXC
16
----------------^
By applying the patch mentioned below to the nonlr.F, this compilation error can be fixed.
view sourceprint?
1
3002c3002
2
< INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
3
---
4
> INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND
5
3144c3144
6
< INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
7
---
8
> INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND
Appendix C - References
VASP (Vienna Ab-Initio Package Simulation)
How to compile VASP on NSC's Triolith - Peter Larsson
This is the best guide for compilation of VASP that I have found on Internet. Partially reproduced above. Peter Larson uses intel MPI with VASP.
Recommended Links
Google matched content
Softpanorama Recommended
Top articles
Sites
Compilation
- VASP 5.2 With Intel Compilers on 64-bit AMD Opteron Cluster CentOS Linux Lev Lafayette
- How to compile VASP on NSC's Triolith - Peter Larsson
- Compiling VASP.4.X, f90 compilers
- How to compile VASP - Peter Larsson
- UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F
- vasp - Computational Materials Physics - Universität Wien
- How to build VASP-4.6.36 and VASP-5.2.8 on Intel Westmere with Infiniband network
Forums
- UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F Texas university
- UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F
Etc
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
You can use PayPal to to buy a cup of coffee for authors of this site Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March 12, 2019