forked from cea-hpc/wi4mpi
-
Notifications
You must be signed in to change notification settings - Fork 0
/
doc_users.en
221 lines (163 loc) · 10.8 KB
/
doc_users.en
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
############################# Wi4MPI License ###########################
# `04/04/2016` #
# Copyright or (C) or Copr. Commissariat a l'Energie Atomique #
# #
# IDDN.FR.001.210022.000.S.P.2014.000.10800 #
# This file is part of the Wi4MPI library. #
# #
# This software is governed by the CeCILL-C license under French law #
# and abiding by the rules of distribution of free software. You can #
# use, modify and/ or redistribute the software under the terms of #
# the CeCILL-C license as circulated by CEA, CNRS and INRIA at the #
# following URL http://www.cecill.info. #
# #
# The fact that you are presently reading this means that you have #
# had knowledge of the CeCILL-C license and that you accept its #
# terms. #
# #
# Authors: #
# - Delforge Tony <tony.delforge.tgcc@cea.fr> #
# - Ducrot Vincent <vincent.ducrot.tgcc@cea.fr> #
# #
########################################################################
:Authors:
Tony Delforge (tony.delforge.tgcc@cea.fr)
Vincent Ducrot (vincent.ducrot.tgcc@cea.fr)
Developer Documentation
=======================
Contents
---------
1. Introduction
2. Install
1. Preload version
2. Interface version
3. How to use wi4mpi
4. Developers words
------------
INTRODUCTION
============
MPI is a standard in HPC community which allows a simple use of clusters. Nowoaday, there are several implementation (OpenMPI, BullxMPI, MPT, IntelMPI,MPC, ...) each of which involves a specific ABI (Application Binary Interface) for an application compiled with a specific MPI implementation.
With wi4mpi the ABI doesn't matter anymore. In fact, wi4mpi is a software allowing an ABI translation from an MPI implementation MPI1 to a MPI implementation MPI2, meaning that an application compiled with an MPI1 MPI implementation can now be run with a MPI2 implementation without re-compliation.
INSTALL
=======
The installation procedureis simply consists of a make clean all install sequence.
There are two different installations for wi4mpi encompassing the two different ways to use the software. Both are discussed in the next part (How to use wi4mpi):
- The preload version
- The interface version
Preload version:
----------------
First make sure you got an OpenMPI loaded in your environment, otherwise the IntelMPI --> OpenMPI conversion will not be performed correctly.
Explanation
For such a conversion wi4mpi need to export the ompi_mpi_... symbols for a perfect execution. Otherwise some undefined symbol errors will be triggered.
End explanation
To compil the source code go to scr directory then use the following comands:
- make clean
- make preload
- make install (be careful, by default the program is installed in /usr/local)
In order to change the install directory please use the prefix variable as follow:
- make prefix=... install
The install directory should now contain the following file:
- bin/wi4mpi
- etc/wi4mpi.cfg
- libexec/wi4mpi/ **Directory which contain all the library allowing the translation**
- libexec/wi4mpi/fakelibINTEL/ **Directory which contain all the empty library called for an INTEL_OMPI conversion**
- libexec/wi4mpi/fakelibOMPI/ **Directory which contain all the empty library called for an OMPI_INTEL conversion**
- libexec/wi4mpi/fakelibCXX/ **Directory which contain the empty c++ library in order to avoid some issues when link is done via mpicxx**
Interface version:
------------------
To compile this version go to the src/ and use the following commands:
- make clean
- make interface
- make install (be careful, by default the program is installed in /usr/local)
In order to change the install directory please use the prefix variable as follow:
- make prefix=... install
The install directory should now contain the following file:
- bin : contain all the wrapper mpi
- include : contain all the necessary include to compile an mpi application
- lib : contain the library which an MPI application is link with
- lib_IMPI : contain the library which perform the ABI translation between our ABI and the IntelMPI ABI
- lib_OMPI : contain the library which perform the ABI translation between our ABI and the OpenMPI ABI
"""Both version can be compiled at the same time, and installed in the same directory : make clean all && make prefix=... install"""
How to use wi4mpi
=================
As mentionned in the previous section, there are two different ways to use our library.
The first way is to use the preload version which allowed any MPI application to be run with a different MPI implementation than the one used for the compilation without a re-compilation process.
We currently support 4 conversions :
- OpenMPI --> OpenMPI
- OpenMPI --> IntelMPI
- IntelMPI --> IntelMPI
- IntelMPI --> OpenMPI
For each conversion, the first MPI implementation points the implementation used for the compilation, and the second one indicate the implementation selected for execution.
To do so, some environment variables have to be set first.
Key:
- MPI_ROOT : environment variable which refers to the MPI install directory
Common:
- export WI4MPI_INTERNAL_INCLUDES="path_to_install/INTERNAL/include"
- OpenMPI --> OpenMPI :
export TRUE_MPI_LIB="$env(MPI_ROOT)/lib/libmpi.so"
export TRUE_MPI_F_LIB="$env(MPI_ROOT)/lib/libmpi_mpifh.so"
export LD_PRELOAD="path_to_wi4mpi_install/libexec/libwi4mpi_OMPI_OMPI.so $TRUE_MPI_F_LIB $TRUE_MPI_LIB"
export LD_LIBRARY_PATH="path_to_wi4mpi_install/libexec/fakelibCXX:$LD_LIBRARY_PATH"
export WI4MPI_APP_INCLUDES="path_to_wi4mpi_install/INTERNAL/preload/include/OMPI_OMPI"
export WI4MPI_COMPILE_OPT "-Dompi_mpich -DOMPI_OMPI"
- OpenMPI --> IntelMPI
export TRUE_MPI_LIB="$env(MPI_ROOT)/lib64/libmpi.so"
export TRUE_MPI_F_LIB="$env(MPI_ROOT)/lib64/libmpifort.so"
export LD_PRELOAD="path_to_wi4mpi_install/libexec/libwi4mpi_OMPI_INTEL.so $TRUE_MPI_F_LIB $TRUE_MPI_LIB"
export LD_LIBRARY_PATH="path_to_wi4mpi_install/libexec/fakelibCXX:path_to_wi4mpi_install/libexec/fakelibOMPI:$LD_LIBRARY_PATH"
export WI4MPI_APP_INCLUDES "path_to_wi4mpi_install/preload/include/OMPI_INTEL"
export WI4MPI_COMPILE_OPT "-Dompi_mpich -DOMPI_INTEL"
- IntelMPI --> IntelMPI
export TRUE_MPI_LIB="$env(MPI_ROOT)/lib64/libmpi.so"
export TRUE_MPI_F_LIB="$env(MPI_ROOT)/lib64/libmpifort.so"
export LD_PRELOAD="path_to_wi4mpi_install/libexec/libwi4mpi_INTEL_INTEL.so $TRUE_MPI_F_LIB $TRUE_MPI_LIB"
export LD_LIBRARY_PATH="path_to_wi4mpi_install/libexec/fakelibCXX:$LD_LIBRARY_PATH"
export WI4MPI_APP_INCLUDES "path_to_wi4mpi_install/preload/include/INTEL_INTEL"
export WI4MPI_COMPILE_OPT "-Dmpich_mpich -DINTEL_INTEL"
- IntelMPI --> OpenMPI
export TRUE_MPI_LIB="$env(MPI_ROOT)/lib/libmpi.so"
export TRUE_MPI_F_LIB="$env(MPI_ROOT)/lib/libmpi_mpifh.so"
export LD_PRELOAD="path_to_wi4mpi_install/libexec/libwi4mpi_OMPI_INTEL.so $TRUE_MPI_F_LIB $TRUE_MPI_LIB"
export LD_LIBRARY_PATH="path_to_wi4mpi_install/libexec/fakelibCXX:path_to_wi4mpi_install/libexec/fakelibINTEL:$LD_LIBRARY_PATH"
export WI4MPI_APP_INCLUDES "path_to_wi4mpi_install/preload/include/INTEL_OMPI"
export WI4MPI_COMPILE_OPT "-Dmpich_ompi -DINTEL_OMPI"
At the TGCC ( Très Grand Centre de Calcul du CEA) we used a module environment with a flavor system allowing a simple way to switch between all the available conversion.
The protocol works as follow:
- module load wi4mpi: The default wi4mpi module is loaded with his two flavors 'from' and 'to' which set the desired conversion:
- flavor/wi4mpi/from/openmpi
- flavor/wi4mpi/to/intelmpi
In this specific case, the 'from' flavor indicate the MPI implementation used to compiled the application, while the 'to' flavor is used to tell to wi4mpi which MPI implementation is designed by the user as the runtime of the application.
The second way we mentionned is to use the interface. The purpose of this interface is to compile any application with it, and then change the MPI implementation used for the execution without any process of compilation. So wi4mpi translate the interface ABI (named CCC) to the MPI implementation ABI used to be run.
To do so, some environment variables have to be set as follow:
- export WI4MPI_INTERNAL_INCLUDES="path_to_install/INTERNAL/include"
- export CCC_LIBMPI_ROOT="/path_to_wi4mpi_install_root"
- export CCC_LIBMPICC=icc
- export CCC_MPIFC=ifort
- export CCC_MPICXX=icpc
- CCC --> IntelMPI
setenv TRUE_MPI_LIB $env(MPI_ROOT)/lib64/libmpi.so
setenv TRUE_MPI_F_LIB $env(MPI_ROOT)/lib64/libmpifort.so
setenv WRAPPER_WI4MPI $env(CCC_LIBMPI_ROOT)/lib_IMPI/libwi4mpi_CCC_INTEL.so
setenv WI4MPI_APP_INCLUDES="path_to_install/interface/include/OMPI_OMPI"
setenv WI4MPI_COMPILE_OPT="-Dompi_ompi -DOMPI_OMPI"
- CCC --> OpenMPI
setenv TRUE_MPI_LIB $env(MPI_ROOT)/lib/libmpi.so
setenv TRUE_MPI_F_LIB $env(MPI_ROOT)/lib/libmpi_mpifh.so
setenv WRAPPER_WI4MPI $env(CCC_LIBMPI_ROOT)/lib_OMPI/libwi4mpi_CCC_OMPI.so
setenv WI4MPI_APP_INCLUDES="path_to_install/interface/include/OMPI_INTEL"
setenv WI4MPI_COMPILE_OPT="-Dompi_ompi -DOMPI_INTEL"
Once again this solution has been set up at the TGCC with the module environment.
module load wi4mpi: The default wi4mpi module is loaded with it two default falvors 'from' and 'to' (we consider the interface version as the default version) :
- flavor/wi4mpi/from/interface (default)
- flavor/wi4mpi/to/openmpi (default)
Here is the following flavors available:
- flavor/wi4mpi/from/interface
- flavor/wi4mpi/from/openmpi
- flavor/wi4mpi/from/intelmpi
- flavor/wi4mpi/to/openmpi
- flavor/wi4mpi/to/intelmpi
This flavor system allow an automatic and dynamic management of the MPI environment based on the underlying MPI modules. For example, the command "module switch flavor/wi4mpi/from/interface flavor/wi4mpi/from/openmpi" allow user to switch the way to use the wrapper from 'interface' to 'preload', but also update the MPI environment.
Developers words
================
First of all thanks a lot for using the wi4mpi software. Nevertheless be aware that this software is still under development and that some bugs may remain. If you encounter one of them, don't hesitate to contact one of the authors and so to participate to the software robustness.
Second, wi4mpi is compatible with the 3.1 MPI norm. Don't hesitate to participate :-)