-
Notifications
You must be signed in to change notification settings - Fork 11
/
Copy pathREADME.developer
166 lines (116 loc) · 7.63 KB
/
README.developer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
# @HEADER
#
########################################################################
#
# Zoltan Toolkit for Load-balancing, Partitioning, Ordering and Coloring
# Copyright 2012 Sandia Corporation
#
# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
# the U.S. Government retains certain rights in this software.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the Corporation nor the names of the
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY SANDIA CORPORATION "AS IS" AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SANDIA CORPORATION OR THE
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Questions? Contact Karen Devine kddevin@sandia.gov
# Erik Boman egboman@sandia.gov
#
########################################################################
#
# @HEADER
lriesen@sandia.gov
November 29, 2010
Notes on typedefs that were just introduced in order to support 64-bit global counts and IDs:
ZOLTAN_ID_TYPE:
==============
A ZOLTAN_ID_TYPE is still the type that we use for global IDs passed in by the user. However now the ZOLTAN_ID_TYPE can be set at configuration time. It can be unsigned int, unsigned long, or unsigned long long. The default is unsigned int.
In CMake you can choose:
-D Zoltan_ENABLE_UINT_IDS:Bool=ON
-D Zoltan_ENABLE_ULONG_IDS:Bool=ON
-D Zoltan_ENABLE_ULLONG_IDS:Bool=ON
Using autoconf it's:
--with-id-type=uint
--with-id-type=ulong
--with-id-type=ullong
To print a ZOLTAN_ID_TYPE use ZOLTAN_ID_SPEC:
printf("GID: " ZOLTAN_ID_SPEC ", LID %d\n", my_gid, my_lid);
To send a ZOLTAN_ID_TYPE in an MPI message, use ZOLTAN_ID_MPI_TYPE:
MPI_Bcast(&gid, 1, ZOLTAN_ID_MPI_TYPE, 0, MPI_COMM_WORLD);
To silence compiler warnings, you can properly specify a constant of type ZOLTAN_ID_TYPE using ZOLTAN_ID_CONSTANT:
a = ZOLTAN_ID_CONSTANT(99)
ZOLTAN_GNO_TYPE:
===============
ZOLTAN_GNO_TYPE is a new typedef which Zoltan uses for global counts and for internal global IDs regardless of the definition of ZOLTAN_ID_TYPE. It is defined to be ssize_t, so it is signed and it will be 64 bits on a 64-bit architecture and 32 bits on a 32-bit architecture.
The MPI_Datatype for ZOLTAN_GNO_TYPE is returned by Zoltan_mpi_gno_type().
Any time that Zoltan creates a new problem from the application supplied problem it uses ZOLTAN_GNO_TYPE for the object IDs.
indextype and weighttype:
========================
At compile time, the Zoltan library defines indextype and weighttype to coincide with the third party graph libraries, if any, that it has been configured to work with.
The "indextype" will be the data type used by the third party library (TPL) for global IDs.
The "weighttype" will be the data type used by the TPL for weights.
Based on configure-time parameters, Zoltan will recognize and correcty set types for Metis, ParMetis, 32- or 64-bit Scotch or 32- or 64-bit PTScotch.
As a developer, it is important to know where the boundary is in the code for the use of Zoltan's types (ZOLTAN_ID_TYPE, ZOLTAN_GNO_TYPE, float) versus the types used by the TPL (indextype, weighttype). A simplified explanation is that indextype and weighttype are only used in source code that is in the "tpls" directory.
More specifically, it goes like this:
1. Zoltan calls the application query functions to obtain the graph using Zoltan's data types for IDs and weights.
2. Zoltan builds graph and matrix structures in Zoltan_ZG_Build using Zoltan's data types. (I suspect the "ZG" stands for "Zoltan Graph").
3. The graph is exported to the TPL data structures in Zoltan_ZG_Export. This is the point where arrays are converted if necessary to the data types used by the third party libraries. The C-structures with names like ZOLTAN_Third_* and others found in tpls/third_library.h use the indextype and weighttype data types.
4. Of course the TPL is called with indextype and weighttype objects.
5. Zoltan_Postprocess_Graph is called on the TPL structures and writes the ZOLTAN_Output_Part structure which uses Zoltan's data types.
6. Zoltan_Third_Export_User uses the ZOLTAN_Output_Part structure to write the part assignments to the structures returned to the user.
To print a indextype or weighttype use TPL_IDX_SPEC or TPL_WGT_SPEC respectively.
If the TPL weight type is a floating point type, then TPL_FLOAT_WEIGHT will be defined.
If the TPL weight type is an integral type, then TPL_INTEGRAL_WEIGHT will be defined.
Other useful TPL configuration macro definitions can be found in tpls/third_library_const.h.
Zoltan can be configured to use both Scotch and ParMetis as long as the 32-bit version of Scotch is used.
Assumptions:
===========
sizeof(ZOLTAN_GNO_TYPE) >= sizeof(ZOLTAN_ID_TYPE)
sizeof(ZOLTAN_ID_TYPE) >= sizeof(int)
Some changes to support 64-bit IDs:
==================================
Zoltan_Map_Create() used to assume it was handling keys that were multiples of ZOLTAN_ID_TYPEs. Now you supply the number of bytes in the key, not the number of ZOLTAN_ID_TYPEs.
Because testing of this branch involves running large memory problems, I added the function Zoltan_write_linux_meminfo() which will write out the contents of /proc/meminfo on a Linux machine. The new function Zoltan_Memory_Get_Debug() returns the debug level set in mem.c by Zoltan_Memory_Debug(). zdrive has a new input option
zoltan memory debug level = n
which will set the debug level. Then after partitioning, zdrive checks the debug level and if there was an error and it is running on a linux machine it will dump out /proc/meminfo.
I modified the configure script to define HOST_LINUX on a linux machine.
I wrote three tests in tests/Large_Data that test PHG, RCB and RIB with arbitrarily large numbers of objects. They have signal handlers that call Zoltan_write_linux_meminfo() on a Linux machine. One test can be configured to use 64-bit IDs when it has less than 2*10^9 IDs.
Limitations:
===========
The reftree and oct methods have not been converted to support 64-bit global IDs and global numbers.
INTERESTING CHART:
=================
32 and 64 bit data models (ILP - integer/long/pointer):
type LP32 ILP32 ILP64 LLP64 LP64
char 8 8 8 8 8
short 16 16 16 16 16
_int32 32
int 16 32 64 32 32
long 32 32 64 32 64
long long 64
pointer 32 32 64 64 64
ILP32 is most widely used.
LP64 is most widely used.
LLP64 is ILP32 with new 64 bit int added to it - used for Win64.