forked from globus/GridFTP-DSI-for-HPSS
-
Notifications
You must be signed in to change notification settings - Fork 0
/
INSTALL
291 lines (236 loc) · 13.9 KB
/
INSTALL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
BUILD AND INSTALL HPSS
======================
Building HPSS is beyond the scope of this guide but you should have a working
HPSS installation. This version of the DSI was tested against HPSS 7.4.2. The
DSI requires either a full HPSS build or a 'clnt' HPSS build. Note whether the
HPSS build is 32bit or 64bit; you'll need to build GridFTP and the DSI that same
way.
POINTS OF INTEREST ON THIS VERSION
==================================
* This DSI is likely to have 64bit-ism and so 64bit is recommended.
* This DSI requires UDA support in HPSS for checksums and quotas.
* The format of the stored checksums are compatible with 7.4.2 but
not necessarily with earlier versions.
* BZ2487 is a show stopper; you must have it
* The GridFTP client you choose must use ALLO (always and corretly)
* Checkout the README.xdr if you want to understand why the DSI
loading sequence is setup as it is
* Because of the XDR issue, this DSI most likely only works on Linux
* Restarts are not supported until Globus supplies a way to decouple
restart from performance markers. See http://jira.globus.org/browse/GT-517.
CONFIGURATION OPTIONS
=====================
* --enable-quota_support puts a special UDA mark on files that are renamed. This
funtionality is part of the NCSA HPSS quota system. There are other parts of
the quota system that are not distributed publically that make this funciton.
If you would like to have that code, you can contact me. Otherwise, do not
enable this feature.
REQUIRED HPSS PATCHES
=====================
BZ2487 - PIO returns wrong bytes moved when reading without regard to file
length, also returns incorrect data. The gap handling code may be
erroneously adding the gap offset to the total bytes moved, or at
least adding additional bytes when handling gap information. This
bug may not exist without the patch for BZ2146. The fix is available
for 7.4.1, not 7.3.3.
RECOMMENDED HPSS PATCHES
========================
BZ2146 - PIO with HPSS_PIO_HANDLE_GAP does not properly handle 32 IOD SrcSink
descriptor boundaries. HPSS breaks transfers into 32 segments per
transfer (IOD/IOR combos). If you do not specify HPSS_PIO_HANDLE_GAP,
hpss_PIOExecute() will kickout similar to gap handling; it is up to
the client to reissue the hpss_PIOExecute() call with the necessary
offset and length. When HPSS_PIO_HANDLE_GAP is used, hpss_PIOExecute()
will treat this segment-limit-kickout as a gap and resume the transfer
without the kickout. However, this bug causes hpss_PIOExecute() to get
stuck in an infinite loop.
This bug has been fixed in HPSS 7.4.1 but not in 7.3.3. The portable
solution is to not use HPSS_PIO_HANDLE_GAP and have the DSI handle
the gaps. Note that if a site uses the "Fixed Length, Classic Style"
allocation scheme (which is recommended for GridFTP to get optimal
performance) then this bug is not encountered because the segment
count remains low (4).
This DSI is currently not affected by this bug because it does not use
HPSS_PIO_HANDLE_GAP.
BZ2819 - PIO 60 second delay impacts small file performance. There is a small
percentage chance that, after a transfer completes, HPSS PIO will
wait 60 seconds before informing the client that the transfer has
completed. This fix has been implemented in 7.4.1 and will be ported
to 7.3.3.
HPSS BUGS TO MONITOR
====================
BZ1660 - PIO callback buffer switching does not persist across calls to
hpss_PIOExecute. The buffer passed to hpss_PIORegister() is passed
to it's callback with a pointer-to-pointer variable. It seesm that
the buffers can be swapped from call to call however, each time
hpss_PIOExecute() is called (which happends multiple times per
transfer for files with gaps or large segment counts) the buffer
passed to hpss_PIORegister() is passed to the callback on its first
invocation after each call to hpss_PIOExecute(). If buffers have been
swapped, the callback will receive a buffer that may be currently
used else where. The fix is to avoid buffer swapping.
IBM HPSS has decided that this is intended behavior and so the DSI works
around the issue.
BZ2856 - Enabling HPSS_API_REUSE_CONNECTIONS returns address already in use.
This one sets a limit on how many active connections we can have.
GridFTP and HPSS are both socket hogs. Quick, successive file transfers
can lead the system to run out of available ports. There is no fix for
this bug at this time. The number of ephemeral ports can be increased
and the amount of time a socket spends in timed wait can be decreased
to help avoid this issue.
PERFORMANCE
===========
(Short version) GridFTP installations benefit from and take full advantage of
classes of service that use fixed length classic style allocation. In short,
you'll get the best performance from the GridFTP interface (actually any HPSS
interface) if the segment count is below 32.
(Longer version) HPSS has multple disk/tape allocation algorithms used to
allocate space for incoming data. Fixed length allocation gives you equal size
chunks to store data in. This was deemed wasteful because the last block was
most certainly never filled. Variable length allocation was created to solve this
problem; it will give you increasingly larger segments as data is stored and
truncates the last block. This is a win for most situations when HPSS is unsure
how much data is to be stored for the given file.
Using either of these allocation mechanisms (any variable length allocation or
fixed w/o knowing the file size), HPSS is free to continue to allocate segments
until all the data is stored. This has a definite performance impact because
internally HPSS retrieves data in 32-segment chunks. This means when you request
a file from HPSS, internally it breaks it up into multiple transfers, each of which
is <= 32 segments. Functionally, this is transparent to the client. In terms of
performance, the client will see a high load followed by a pause followed by a
high load, etc.
In order to avoid the performance hit, you can use fixed length allocation with
segment counts < 32 and take advantage of the fact that any WELL-BEHAVED GridFTP
client will inform HPSS of the size of the incoming file before the transfer
begins. In fact, the DSI is designed to require this. If a GridFTP client is
not well behavad, the DSI will act as though a zero length transfer is about to
occur and will handle it as such. So you'll know if the client is not doing the
right thing.
GRIDFTP PATCHES
===============
These are fixed in Globus Toolkit 5.2.4:
GT-297 globus_ftp_control_data_query_channels() SIGSEGV on proxy expiration
Still believed to exist in GT 5.2.5:
GT-296 globus_ftp_control_data_read() race condition
GT-376 globus_ftp_control_data_query_channels() SIGSEGV on ABRT
GT-377 abort() in globus_ftp_control_handle_destroy() -> globus_list_remove()
GT-378 abort() in globus_i_gfs_data_session_stop()
GT-379 Abort() in globus_i_gfs_data_request_recv()
GT-380 Abort in globus_l_gfs_new_server_cb()
INSTALL GRIDFTP
===============
In order to build the module, you'll need to install the globus-gridftp-server-devel
and libgssglue-devel rpms. In order to run the HPSS GridFTP server, you'll need to
install the globus-gridftp-server RPM. This DSI is only tested against 64 bit builds.
See the GT documentation for installing from RPM.
http://toolkit.globus.org/toolkit/docs/5.2/5.2.5/admin/install/#install-bininst
BUILD AND INSTALL THE DSI
=========================
Build the HPSS DSI with the following options. Set the value of '--libdir' to the
directory where you want to place the DSI. For your sanity, I recommend putting it
in a new directory with the version information stored in the name.
Ex. /usr/local/gridftp_hpss_dsi-1.0. You can then set the environment variable
LD_LIBRARY_PATH to this location so that the GridFTP server can find the DSI.
If you use Globus RPMs, you shoudn't need to specify --with-globus because it
defaults to /usr/
./configure --with-hpss=/opt/hpss -libdir=/usr/local/gridftp_hpss_dsi-<version>.
Install the DSI into libdir:
make install
USE USER HPSSFTP FOR GRIDFTP
============================
GridFTP requires a privileged user with control permission on the core server's
client interface in order to log into HPSS and then it changes its credentials
to that of the actual user (you). Think of it as a type of setuid process that
logs in, does what it must, then changes the process owner to you.
In the past, we recommended creating a new HPSS user named 'gridftp' for this
purpose. However, it was discovered that HPSS allows any connection with control
permission on the core server's client interface to bypass Gate Keeper site hooks.
That little tidbit breaks site specific accounting and quota deployments.
HPSS has a special privileged user named 'hpssftp' which has the necessary permissions
and special code within HPSS to not allow it to bypass the Gate Keeper callouts. Thus
we recommend that you use the 'hpssftp' user in gridftp.conf.
You will however want the GridFTP process to run as a non privileged user. For this
reasons, we run the GridFTP server as local UNIX user 'gridftp' but configure it to
use hpssftp's credentials to log into HPSS.
Add the new user 'gridftp' to the system password file. Your entry should look
something like the following:
gridftp:x:316:316:GridFTP Server:/home/gridftp:/bin/bash
Now you'll need to create a keytab file for the gridftp user. This will be used
by the DSI in order to authenticate to HPSS. For sites using unix
authentication with HPSS (rather than kerberos):
[root@hpss gridftp_hpss_dsi_0_1]# /opt/hpss/bin/hpss_unix_keytab -f /var/hpss/etc/gridftp.keytab add hpssftp
For sites using kerberos authentication with HPSS, you'll need to create
and use a kerberos keytab file (rather than a unix keytab). The kerberos
utility 'ktutil' can be used for that purpose.
CONFIGURING XINETD
==================
The globus gridftp server does not need to run as a privileged user. You should
create a system account for this purpose. It is important to make sure that both
the keytab files generated above and the GSI host key are both owned by this
system account. In the following example, I will use the system account
'gridftp'. Since our HPSS installation currently uses the system password file
(instead of the HPSS specific password file) this is the same account created
earlier for authentication to HPSS. However, if your site uses the HPSS specific
password file, you will need to create a separate system account for the GridFTP
server process (they may both have the same name).
Add an entry to /etc/services for the default gsiftp port:
gsiftp 2811/tcp # GSI FTP
Here's an example xinetd entry (/etc/xinetd.d/gridftp). Note that the DSI
requires that the server run with threads so you must use -threads with a value
of 2 or greater.
service gsiftp
{
flags = IPv4
wait = no
user = gridftp
# You may need to set group depending upon the account chosen
group = gridftp
server = /usr/local/globus/sbin/globus-gridftp-server
# auth-level 6 = 2 for frontend default + 4 for no-change-uid/gid
# 0 - Disables all authorization checks
# 1 - Authorize identity
# 2 - Authorize all file/resource accesses (Default for fronend)
# 4 - Disable changing process uid to authenticated user
server_args = -inetd -threads 2 -auth-level 7 -dsi hpss_local -disable-command-list SCKS,APPE,REST
env = GRIDMAP=/etc/grid-security/grid-mapfile
env = LD_LIBRARY_PATH=/opt/hpss/lib:/usr/local/gridftp_hpss_dsi-<version>
socket_type = stream
per_source = 100
}
CONFIGURING WITHOUT XINETD
==========================
If you are launching GridFTP without xinetd (for example, a Globus Connect Server install),
you can configure the server as follows.
Add these lines to /etc/gridftp.d/extra:
auth_level 7
load_dsi_module hpss_local
disable_command_list SCKS,APPE,REST
Also, in order for the GridFTP server to find the HPSS DSI, you must edit your start script
to include: LD_LIBRARY_PATH="<path to DSI install dir>". For example, if you launch GridFTP
from /etc/init.d/globus-gridftp-server and the DSI is installed in /usr/local/hpss_dsi, then
add the following line to /etc/init.d/globus-gridftp-server:
export LD_LIBRARY_PATH=/usr/local/hpss_dsi
SETUP THE DSI CONFIG FILE
=========================
Review 'ConfigFile' in the source directory for any changes you may wish to
make for your site. Then copy ConfigFile into place on the target system. The
DSI will use the following search order for locating the configuration file:
1) $HPSS_PATH_ETC/gridftp.conf
2) /var/hpss/etc/gridftp.conf
Make sure the configuration file's permissions allow for the GridFTP process to
read it. For example, if you chose to run the GridFTP server as user 'gridftp',
make sure the user 'gridftp' has read access to the configuration file.
HPSS Configuration Files
========================
The HPSS DSI needs to run on a system that has sufficient HPSS configuration files installed
to permit it to talk to the appropriate HPSS servers (such as an HPSS mover node), as well as
to perform authentication of users. These files typically are kept under /var/hpss/etc.
The HPSS user authentication files may be separate from the system's authentication files
(i.e. in /var/hpss/etc/{group,passwd}), but can be set in /var/hpss/etc/env.conf to point
to any suitable files. The DSI will also need to store credential files in the /var/hpss/cred
directory.
Kerberos Configuration
======================
Kerberos must be configured for access to the proper Kerberos realm that contains HPSS.
This file is usually kept in /etc/krb5.conf. You may need to enable the allow_weak_crypto
option in the [libdefaults] section if the DSI module can not talk to the HPSS servers.