Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
ga_installation [2021/06/02 08:27]
qianli
ga_installation [2021/11/16 10:10] (current)
qianli Warn against openmpi file problem
Line 3: Line 3:
 We recommend building Molpro with the [[https://github.com/GlobalArrays/ga|Global Arrays toolkit]], using the latest stable version. We recommend building Molpro with the [[https://github.com/GlobalArrays/ga|Global Arrays toolkit]], using the latest stable version.
  
-There are different ways (GA "runtimes") to configure the GA library which significantly affect the excution and options, and they are therefore important from a users point of view.+There are different ways (GA "runtimes") to configure the GA library which significantly affect the execution and options, and they are therefore important from a users point of view.
 The current default of the GA installation is to use ''%%--with-mpi-ts%%'', which is **very slow** and should be avoided. The current default of the GA installation is to use ''%%--with-mpi-ts%%'', which is **very slow** and should be avoided.
 Our current recommendations are: Our current recommendations are:
  
-  * Use ''%%--with-sockets%%'' for single workstations and servers connected by ethernet. +  * Use ''%%--with-sockets%%'' for single workstations and servers connected by ethernet. We recommend using [[https://www.mpich.org/ | mpich]]
-  * Use ''%%--with-openib%%'' for nodes connected by InfiniBand, also preferably on top of the latest openmpi with UCX. +  * Use ''%%--with-openib%%'' for nodes connected by InfiniBand, preferably on top of the latest version of [[https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX|openmpi with UCX]]
-  * When a helper process on each node can be tolerated, use ''%%--with-mpi-pr%%'', preferably on top of the latest version of [[https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX|openmpi with UCX]].+  * When a helper process on each node can be tolerated, use ''%%--with-mpi-pr%%'', also preferably using the latest version of openmpi with UCX.
  
-The first two GA runtimes do not need any helper processes, but has a serious disadvantage that with ''--ga-impl ga'' (the default for multi-node calculations since Molpro 2020.2 and for all calculations in earlier versions), the GA software frequently crashes or gives wrong numbers if several GAs are allocated whose total size exceeds 16 GB (2 MW). This can be avoided by using ''--ga-impl disk'' (default for single-node calculations since Molpro 2020.2), or allocating in the beginning of the job a very large GA with at least the size of the total GA space needed in the calculation (which is not known in advance). The latter can be done with the ''-G'' or ''-M'' molpro options.+Note that openmpi may not work with ''--ga-impl disk'' when multiple parallel Molpro calculations are executed simultaneously on a node. 
 + 
 +The first two GA runtimes do not need any helper processes, but has a serious disadvantage that with ''--ga-impl ga'' (the default for multi-node calculations since Molpro 2021.2 and for all calculations in earlier versions), the GA software frequently crashes or gives wrong numbers if several GAs are allocated whose total size exceeds 16 GB (2 MW). This can be avoided by using ''--ga-impl disk'' (default for single-node calculations since Molpro 2021.2), or allocating in the beginning of the job a very large GA with at least the size of the total GA space needed in the calculation (which is not known in advance). The latter can be done with the ''-G'' or ''-M'' molpro options.
 See [[Running Molpro on parallel computers]] for more details. See [[Running Molpro on parallel computers]] for more details.
  
Line 17: Line 19:
  
 The ''socket'' and ''openib'' GA runtimes uses System V shared memory while the ''mpi-pr'' runtime uses POSIX shared memory. In the former case you may need to raise the value of ''SHMMAX'' and ''SHMALL'' kernel parameters, while in the latter case to raise the size-limit of ''/dev/shm''. Also note that even in successful calculations shared memory segments are sometimes left in ''/dev/shm''. These can accumulate to make a machine unusable, and should therefore be deleted from time to time. The ''socket'' and ''openib'' GA runtimes uses System V shared memory while the ''mpi-pr'' runtime uses POSIX shared memory. In the former case you may need to raise the value of ''SHMMAX'' and ''SHMALL'' kernel parameters, while in the latter case to raise the size-limit of ''/dev/shm''. Also note that even in successful calculations shared memory segments are sometimes left in ''/dev/shm''. These can accumulate to make a machine unusable, and should therefore be deleted from time to time.
 +
 +
  
 Queries regarding Global Arrays installations should be sent directly to the Global Arrays team, any Molpro related queries will assume a fully functional Global Arrays suite with all internal tests run successfully. Queries regarding Global Arrays installations should be sent directly to the Global Arrays team, any Molpro related queries will assume a fully functional Global Arrays suite with all internal tests run successfully.