[molpro-user] Some progress on running MOLPRO using Infiniband/Topspin
Shenggang Li
shenggangli at gmail.com
Fri May 26 00:47:55 BST 2006
Hi, after some trial and error, I was able to make some progress in running
MOLPRO with Infiniband/Topspin libraries. Because the special requirement
of this MPI library, which accepts the following format of arguments:
mpirun [-paramfile=parameterfile] -np n -hostfile hostfile executable
arguments
I have to use a wrapper (mpirun_molpro) of the following format:
#!/bin/csh -f
echo MOLPRO_KEY=\"$MOLPRO_KEY\" > /home/sli/parameterfile
#echo $MOLPRO_GA_DEBUG
echo MOLPRO_MEMMAX=$MOLPRO_MEMMAX >> /home/sli/parameterfile
echo MOLPRO_MEMORY=$MOLPRO_MEMORY >> /home/sli/parameterfile
echo MOLPRO_MXMBLK=$MOLPRO_MXMBLK >> /home/sli/parameterfile
#echo $MOLPRO_MXMBLN
#echo $MOLPRO_UNROLL
#echo $MOLPRO_MINDGM
#echo $MOLPRO_MINDGL
#echo $MOLPRO_MINDGC
#echo $MOLPRO_MINDGR
#echo $MOLPRO_MINDGV
#echo $MOLPRO_FLOPDGM
#echo $MOLPRO_FLOPMXV
#echo $MOLPRO_FLOPDGV
#echo $MOLPRO_FLOPMXM
#echo $MOLPRO_MPPLAT
#echo $MOLPRO_MPPSPEED
#echo $MOLPRO_NOBLAS
echo MOLPRO_CACHE=$MOLPRO_CACHE >> /home/sli/parameterfile
#echo $MOLPRO_IOS_IMPLEMENTATION
echo LIBMOL=$LIBMOL >> /home/sli/parameterfile
#echo $MOLPRO_GA_STACK
#echo $MOLPRO_GA_HEAP
echo MOLPRO_GA_MEMORY=$MOLPRO_GA_MEMORY >> /home/sli/parameterfile
echo MOLPRO_MAX_MEMORY=$MOLPRO_MAX_MEMORY >> /home/sli/parameterfile
echo MOLPRO_LOG=$MOLPRO_LOG >> /home/sli/parameterfile
#echo $MOLPRO_DEF
echo MOLPRO_PWD=$MOLPRO_PWD >> /home/sli/parameterfile
echo MOLPRO_SCR=$MOLPRO_SCR >> /home/sli/parameterfile
#echo $MOLPRO_NOHUP
#echo $MOLPRO_SCR_11
#echo $MOLPRO_SCR_12
#echo $MOLPRO_SCR_13
#echo $MOLPRO_SCR_14
#echo $MOLPRO_SCR_15
#echo $MOLPRO_SCR_16
#echo $MOLPRO_SCR_17
#echo $MOLPRO_SCR_18
#echo $MOLPRO_SCR_19
echo MOLPRO_OUTPUT=$MOLPRO_OUTPUT >> /home/sli/parameterfile
echo MOLPRO_EXECUTABLE=$MOLPRO_EXECUTABLE >> /home/sli/parameterfile
#echo $MOLPRO_SCR_
#echo $MOLPRO_SCR_1
echo MOLPRO_BLASLIB=$MOLPRO_BLASLIB >> /home/sli/parameterfile
#echo $MOLPRO_TRAP
#echo $MOLPRO_IFC
#echo $PRINTER
#echo $XWDOPT
#echo $XPRCMD
echo DISPLAY=$DISPLAY >> /home/sli/parameterfile
#echo $DEBUG_ENV_NAME
#echo $INDEX_DIRECTORY_ENVIRONMENT
echo TMPDIR=$TMPDIR >> /home/sli/parameterfile
#echo $TRAP
#echo $NOTRAP
/usr/local/topspin/mpi/mpich/bin/mpirun_ssh
-paramfile=/home/sli/parameterfile $1 $2 -hostfile $4 $5
Commented lines are the undefined environmental variables, which might be
necessary once I have done the tuning. The following are the actual
commands initiated by mpirun_ssh captured by the -show option
command: /usr/bin/ssh compute-11-1.local cd /home/sli/scratch/molpro;
/usr/bin/env MPIRUN_HOST=compute-11-1.local MPIRUN_PORT=38590 MPIRUN_RANK=0
MPIRUN_NPROCS=2 MPIRUN_ID=29194
MPIRUN_NODENAME=compute-11-1.localMOLPRO_KEY="xxxxReplacedxxxLegalxxxReasonxxx"
MOLPRO_MEMMAX=2147483647
MOLPRO_MEMORY=7272727 MOLPRO_MXMBLK=32 MOLPRO_CACHE=262144
LIBMOL=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/lib/
MOLPRO_GA_MEMORY=8000000 MOLPRO_MAX_MEMORY=7445729
MOLPRO_LOG=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test/h2o.log
MOLPRO_PWD=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test
MOLPRO_SCR=/home/sli/scratch/molpro MOLPRO_OUTPUT=/home/sli/apps/molpro-
2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test/h2o.out
MOLPRO_EXECUTABLE=molprop_2002_6_i8_p4_mpi MOLPRO_BLASLIB=pthread
DISPLAY=localhost:10.0 TMPDIR=/home/sli/scratch/molpro
/home/sli/apps/molpro-2002.6
/pgi-i8-mpich-mkl-mellanox-ga4/lib/molprop_2002_6_i8_p4_mpi.exe
command: /usr/bin/ssh compute-11-2.local cd /home/sli/scratch/molpro;
/usr/bin/env MPIRUN_HOST=compute-11-1.local MPIRUN_PORT=38590 MPIRUN_RANK=1
MPIRUN_NPROCS=2 MPIRUN_ID=29194
MPIRUN_NODENAME=compute-11-2.localMOLPRO_KEY="xxxxReplacedxxxLegalxxxReasonxxx"
MOLPRO_MEMMAX=2147483647
MOLPRO_MEMORY=7272727 MOLPRO_MXMBLK=32 MOLPRO_CACHE=262144
LIBMOL=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/lib/
MOLPRO_GA_MEMORY=8000000 MOLPRO_MAX_MEMORY=7445729
MOLPRO_LOG=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test/h2o.log
MOLPRO_PWD=/home/sli/apps/molpro-2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test
MOLPRO_SCR=/home/sli/scratch/molpro MOLPRO_OUTPUT=/home/sli/apps/molpro-
2002.6/pgi-i8-mpich-mkl-mellanox-ga4/test/h2o.out
MOLPRO_EXECUTABLE=molprop_2002_6_i8_p4_mpi MOLPRO_BLASLIB=pthread
DISPLAY=localhost:10.0 TMPDIR=/home/sli/scratch/molpro
/home/sli/apps/molpro-2002.6
/pgi-i8-mpich-mkl-mellanox-ga4/lib/molprop_2002_6_i8_p4_mpi.exe
When I run molpro by
molpro -n 2 --mpirun-machinefile /home/sli/hostfile h2o.in
on the following input (h2o.in)
***, H2O
mem,100,m
geomtyp=xyz
geometry={
3
H2O
O -1.11111111 2.46666663 0.00000000
H -0.15111111 2.46666663 0.00000000
H -1.43156570 3.37160246 0.00000000
}
basis=avtz;
int;
hf;
ccsd(t);
I got the following output (h2o.out)
ARMCI configured for 2 cluster nodes. Network protocol is 'Mellanox VAPI'.
MPP nodes nproc
compute-11-1.loc 1
compute-11-2.loc 1
ga_uses_ma=true, calling ma_init with heap size 4010000 determined
by size of -G / nprocs
Primary working directories: /home/sli/scratch/molpro
Secondary working directories: /home/sli/scratch/molpro
blaslib=pthread
MPP tuning parameters: Latency= 1000 Microseconds, Broadcast speed= 20
MB/sec
default implementation of scratch files=ga
Variables initialized (306), CPU time= 0.00 sec
Default parameters read. Elapsed time= 0.02 sec
Checking input...
Passed
Variable memory set to 7272727 words, buffer space 230000 words
**********************************************************************************************************************************
Variable memory released
Basically the program didn't do anything. My guess is that it didn't
receive the input file, but I can't find any environmental variable pointing
to the input file (MOLPRO_INPUT?). I think I am very close to get MOLPRO to
run with Infiniband. Can anyone help me?
Thank you very much!
--
Shenggang Li
Shelby Hall L103A
Department of Chemistry
University of Alabama
Tuscaloosa, AL 35487-0336
Office: 205-348-4906
http://bama.ua.edu/~sli/fcf (FCF User Support Website)
http://bama.ua.edu/~sli/pople (Pople User Support Website)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.molpro.net/pipermail/molpro-user/attachments/20060525/5ad50e1a/attachment.html>
More information about the Molpro-user
mailing list