From NWChem
Viewed 5455 times, With a total of 6 Posts
|
|
|
Clicked A Few Times
Threads 11
Posts 32
|
|
| 7:39:34 PM PST - Wed, Jan 2nd 2013 |
|
i could be wrong on this but it seems that NWChem is not using the MKL it was linked against. I say this for two reasons.
1) The CPU time of all the test jobs seem for the NWChem built with the standard libraries and the one built with MKL are identical (+- 2%)
2) I have noticed that in the final linking step to generate NWChem binary "-lblas and -llapack" are there, which i suspect are the default libraries for NWChem
gfortran -L/home/joe/nwchem-6.1.1-src/lib/LINUX64 -L/home/joe/nwchem-6.1.1-src/src/tools/install/lib -o /home/joe/nwchem-6.1.1-src/bin/LINUX64/nwchem nwchem.o stubs.o -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ldangchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldplot -ldrdy -lvscf -lqmmm -lqmd -letrans -lpspw -ltce -lbq -lcons -lperfm -ldntmc -lccca -lnwcutil -lga -lpeigs -lperfm -lcons -lbq -lnwcutil -L/home/joe/mkl/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm -llapack -lblas -L/home/joe/mpich-install/lib -lmpich -lopa -lmpl -lrt -lpthread
The following is the script I use for environment variables
#!/bin/bash
export NWCHEM_TOP=$HOME/nwchem-6.1.1-src
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=$HOME/mpich-install
export LIBMPI="-lmpich -lopa -lmpl -lrt -lpthread"
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LARGE_FILES=TRUE
export NWCHEM_MODULES=all
export MKLROOT=$HOME/mkl/mkl/lib/intel64
export BLASOPT="-L$MKLROOT -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm"
#make clean
make FC=gfortran FFLAGS="-fdefault-integer-8 -m64" nwchem_config
make FC=gfortran FFLAGS="-fdefault-integer-8 -m64" HAS_BLAS=yes
Any ideas? Thanks in advance
|
|
Edited On 7:42:02 PM PST - Wed, Jan 2nd 2013 by DouDou
|
|
|
-
Huub Forum:Admin, Forum:Mod, NWChemDeveloper, bureaucrat, sysop
|
|
|
Forum Regular
Threads 1
Posts 185
|
|
| 11:15:09 AM PST - Thu, Jan 3rd 2013 |
|
Hi DouDou,
I think that actually MKL is probably being used. However, you might not be able to see that easily for two reasons:
1. In the usual test jobs the BLAS calls take very little time. So using a much more efficient BLAS library will still not change the CPU time much, probably too little to see.
2. The presence of -llapack and -lblas is correct. Those libraries are included as MKL does not necessarily provide all LAPACK and BLAS routines. The routine that are not found in MKL are then picked up from the LAPACK and BLAS implementations in the source code. For this to work properly the LAPACK and BLAS libraries have to be listed after the MKL libraries, which you can easily see is correct on your link line.
My recommendation is to use a tool like ldd on the binary and check that the binary was actually linked against the libraries you want. Alternatively you try and use the debugger on the binary and have it stop the program in for example dgemm (a BLAS routine) and see what the debugger says about where that routine comes from. Alternatively (but more work) you can put some print statements in the source code of the BLAS routines in NWChem. If the MKL versions are used you should not see anything from those print statements.
I am sorry that it is not so easy to see what libraries the code uses, but if you really want to know there are ways to find out but it takes a little work.
Best wishes,
Huub
|
|
|
-
Edoapra Forum:Admin, Forum:Mod, bureaucrat, sysop
|
|
|
Forum Vet
Threads 9
Posts 1520
|
|
| 11:52:19 AM PST - Thu, Jan 3rd 2013 |
|
How to check if mkl was linked
|
Doudou,
Here is the command for checking if your binary contains any mkl blas. If the command doesnot return anything,
the you do not have mkl blas objects in your nwchem binary.
nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm
|
|
|
|
|
|
Clicked A Few Times
Threads 11
Posts 32
|
|
| 2:31:37 PM PST - Thu, Jan 3rd 2013 |
|
Thanks Huub and Edopra for your responses,
'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again
|
|
|
-
Edoapra Forum:Admin, Forum:Mod, bureaucrat, sysop
|
|
|
Forum Vet
Threads 9
Posts 1520
|
|
| 10:20:41 AM PST - Fri, Jan 4th 2013 |
|
What about the ldd command, i.e.
ldd $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem
Does this show any mkl library?
Quote:DouDou Jan 3rd 1:31 pmThanks Huub and Edopra for your responses,
'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again |
|
|
|
|
|
Clicked A Few Times
Threads 11
Posts 32
|
|
| 10:42:47 AM PST - Fri, Jan 4th 2013 |
|
Quote:Edoapra Jan 4th 9:20 amWhat about the ldd command, i.e.
ldd $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem
Does this show any mkl library?
Quote:DouDou Jan 3rd 1:31 pmThanks Huub and Edopra for your responses,
'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again
Yes, they are there
libmkl_scalapack_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_scalapack_ilp64.so (0x00002aaaaaaad000)
libmkl_cdft_core.so => /home/joe/mkl/mkl/lib/intel64/libmkl_cdft_core.so (0x00002aaaab2b5000)
libmkl_gf_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_gf_ilp64.so (0x00002aaaab4d1000)
libmkl_sequential.so => /home/joe/mkl/mkl/lib/intel64/libmkl_sequential.so (0x00002aaaabbd3000)
libmkl_core.so => /home/joe/mkl/mkl/lib/intel64/libmkl_core.so (0x00002aaaac271000)
libmkl_blacs_intelmpi_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.so (0x00002aaaad47f000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003790800000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003790c00000)
libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002aaaad708000)
libm.so.6 => /lib64/libm.so.6 (0x0000003790400000)
librt.so.1 => /lib64/librt.so.1 (0x0000003791400000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003669600000)
libc.so.6 => /lib64/libc.so.6 (0x0000003790000000)
/lib64/ld-linux-x86-64.so.2 (0x000000378f000000)
|
|
Edited On 10:49:27 AM PST - Fri, Jan 4th 2013 by DouDou
|
|
|
|
|
|
Gets Around
Threads 3
Posts 85
|
|
| 9:11:08 AM PST - Wed, Dec 10th 2014 |
|
Quote:DouDou Jan 3rd 2:39 am
-L/home/joe/mkl/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm -llapack -lblas -L/home/joe/mpich-install/lib -lmpich -lopa -lmpl -lrt -lpthread
Any ideas? Thanks in advance
You can read up on the left-to-right convention of the GNU linker (and possibly others) to understand why the presence of the built-in Netlib libraries are not used if the symbols are found in MKL first.
Jeff
|
|
|
AWC's:
2.5.10 MediaWiki - Stand Alone Forum Extension
Forum theme style by: AWC
