-------------------ABOUT ZHT-------------------
ZHT is a zero-hop distributed hash table(see also the paper, http://datasys.cs.iit.edu/projects/ZHT/ZHT-CRC-PID2666213-Final.pdf).

-------------------SOFTWARE REQUIREMENTS-------------------
Google protocol buffers c binding, VERSION 0.15 or later
		http://code.google.com/p/protobuf-c/downloads/list


Google protocol buffers c++ binding, VERSION 2.4.1
		http://code.google.com/p/protobuf/downloads/list


-------------------ABOUT ZHT API, SPECIAL ATTENTION-------------------
ZHT supports the API set such as insert/lookup/remove/append/compare_swap.

When you are using append, BE CAREFUL THAT your key and value CAN NOT contain the colon(:), which is used 
internally to delimite key/value pairs.


-------------------COMPILE AND INSTALL-------------------
FIRSTLY install Google protocol buffers c++ binding, 
AND THEN install Google protocol buffers c binding since the latter dependes on the former.

Assuming that you installed them to default directory, possibly that is, /usr/local
1. vim/gedit ~/.bashrc
2. append the following lines to the end,
	export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH
3. source ~/.bashrc
4. cd to ZHT src directory
5. TO COMPILE ZHT, RUN
   make

ORTHERWISE, assuming that you will install them to /home/xiaobingo/install, follow THE ORDER:
1. firstly install Google protocol buffers c++ binding, by running
   1) cd to <Google protocol buffers c++ binding> home directory
   2) ./configure --prefix=/home/xiaobingo/install
   3) make
   4) make install
2. then install Google protocol buffers c binding, in this step, you may get error like: ERROR: protobuf headers are required.,
   follow the instructions, 
   1) cd to <Google protocol buffers c binding> home directory
   2) ./configure --prefix=/home/xiaobingo/install CXXFLAGS=-I/home/xiaobingo/install/include LDFLAGS=-L/home/xiaobingo/install/lib
   3) make
   4) make install
  
   
If you installed Google protocol buffers c and c++ bindings to customized directory, e.g./home/xiaobingo/install, MAKE SURE TO EXPORT them,
1. vim/gedit ~/.bashrc
2. append the following lines to the end,
	export USER_LIB=/home/xiaobingo/install/lib/
	export USER_INCLUDE=/home/xiaobingo/install/include/
	export LD_LIBRARY_PATH=/home/xiaobingo/install/lib/
3. source ~/.bashrc
4. cd to ZHT src directory
5. TO COMPILE ZHT, RUN
   make

, otherwise, you probably fail to compile ZHT or locate dependencies required by ZHT runtime.

   
-------------------RUN THE BENCHMARK-------------------
To run benchmark:
1. ./zhtserver -z zht.conf -n neighbor.conf, OR
   ./zhtserver -z zht.conf -n neighbor.conf -p 40000, to specify your own port that overrides PORT in zht.conf, OR
   ./zhtserver -z zht.conf -n neighbor.conf -p 40000 -f novoht.db, to specify your own disk file to which data is persisted.

2. ./zht_ben -z zht.conf -n neighbor.conf -o 1000

NOTE that:
	The port you specified to listen on at zhtserver(e.g. the PORT in zht.conf, or ./zhtserver -z zht.conf -n neighbor.conf -p 40000)
	MUST BE CONSISTENT WITH the port of that machine in neighbor.conf, otherwise, the zhtclient CANNOT reach the zhtserver by that port.
		
In ./zhtserver -z zht.conf -n neighbor.conf -p 40000 or
   ./zhtserver -z zht.conf -n neighbor.conf -p 40000 -f novoht.db
	zht.conf: is to configure PROTOCOL and PORT, and other parameters,
	neighbor.conf: is to configure neighbored nodes(or ZHT servers) in ZHT overlay network,
	40000: means zhtserver port to listen on, which overrides PORT in zht.conf, if missing, that PORT in zht.conf will be used.
	novoht.db: is the disk file to which the data are persisted, if missing, the data reside in memory. Please specify different
			   files if you are going to run multiple zhtservers in the same node. 
	
In ./zht_ben -z zht.conf -n neighbor.conf -o 1000 
	zht.conf is to configure PROTOCOL and PORT, and other parameters,
	neighbor.conf is to configure neighbored nodes(or ZHT servers) in ZHT overlay network,
	1000 means running 1000 sequential iterations of INSERT, LOOKUP, REMOVE and APPEND.

NOTE that:
The parameter ID and VALUE(e.g. PROTOCOL TCP) for both zht.conf and neighbor.conf is SPACE delimited.
The conf files support #-style comment-out.
The blank lines will be ignored as comment-outs while parsing the conf files.

	
-------------------RUN TEST CASES FOR C++ BINDING-------------------
To run test cases for zht c++ binding:
1. cd to ZHT src directory
2. make
3. ./zhtserver -z zht.conf -n neighbor.conf
4. ./zht_cpptest -z zht.conf -n neighbor.conf


-------------------RUN TEST CASES FOR C BINDING-------------------
To run test cases for zht c binding:
1. cd to ZHT src directory
2. make
3. ./zhtserver -z zht.conf -n neighbor.conf
4. ./zht_ctest -z zht.conf -n neighbor.conf


-------------------ZHT STANDALONE DEPLOYMENT-------------------
To deploy ZHT on a single node and run your own ZHT application:
1. cd to ZHT src directory
2. ./zhtserver -z zht.conf -n neighbor.conf
3. ./your_zht_app -z zht.conf -n neighbor.conf

NOTE that your_zht_app is your binary that invokes ZHT client to INSERT/LOOKUP/REMOVE/APPEND key/value pairs.


-------------------ZHT CLUSTER DEPLOYMENT-------------------
To deploye ZHT on a cluster of nodes and run your own ZHT application:
1. cd to ZHT src directory
2. copy zht binary(e.g. zhtserver) to every node, or using shared NFS directory.
3. edit neighbor.conf to contain IP and PORT pair(e.g. 192.168.1.100 50000) of all ZHT servers you are going to start, DO NOT USE localhost.
4. copy zht.conf and neighbor.conf to every node, or using shared NFS directory.
5. In every node, run ./zhtserver -z zht.conf -n neighbor.conf, assuming <PORT 50000> in zht.conf is used. 
6. run your ZHT applications.


-------------------DEVELOPE YOUR ZHT APPLICATIONS-------------------
To develope your own ZHT applications, please refer to <c_zhtclient_test.c>, which extensively tests ZHT client c binding, internally invoking c++ binding.
OR see also <cpp_zhtclient_test.cpp> on how to use ZHT API in c++ programs.

To compile your own ZHT applications, link to <libzht.a> and include cpp_zhtclient.h and c_zhtclient.h for C++ and C programs, respectively.

See also <doc/ZHT-dev-tutorial-CS554@cs.iit.pptx> for tutorial on how to built ZHT applications step by step.

-------------------HOW TO COMPILE ZHT TO USE MPI PROTOCOL-------------------
1. INSTALL MPI, e.g. MPICH or OpenMPI, assuming that the mpi is installed to /users/xiaobingo/install
2. vim/gedit ~/.bashrc
3. append the following lines to the end,
	export PATH=/users/xiaobingo/install/bin:$PATH
4. source ~/.bashrc
5. cd to ZHT src directory
6. TO COMPILE ZHT, RUN
   make

-------------------ZHT(USING MPI) STANDALONE DEPLOYMENT-------------------
1. TO START zht-mpiserver and zht-mpibroker(e.g. ON SINGLE NODE), RUN:
mpiexec -np 4 ./zht-mpiserver -z zht.conf -n neighbor.mpi.conf : ./zht-mpibroker -z zht.conf -n neighbor.mpi.conf

3. TO RUN THE BENCHMARK:
see also <RUN THE BENCHMARK> section.

4. TO RUN TEST CASES FOR C++ BINDING:
see also <RUN TEST CASES FOR C++ BINDING> section

5. TO RUN TEST CASES FOR C BINDING:
see also <RUN TEST CASES FOR C BINDING> section


NOTE THAT:
mpiexec: mpi command to launch mpi jobs.
-np 4 ./zht-mpiserver: allocate 4 processors to run 4 zht-mpiserver processes. 
-z zht.conf: zht conf file as input argument to zht-mpiserver or zht-mpibroker, the same file mentioned above.
-n neighbor.mpi.conf: negibor conf file as input argument to zht-mpiserver or zht-mpibroker, containing localhost in this case.			  
zht-mpibroker: the process launched on each node to talk to zht-mpiserver on behalf of zhtclient. 

PAY SPECIAL ATTENTIONS:
There is ONLY ONE INSTANCE of zht-mpibroker allowed to be deployed ON EVERY NODE.

zht-mpibroker MUST appear after zht-mpiserver IN ORDER in order to ensure the processes of zht-mpiserver
are assigned to proper mpi-ranks that serve processing incoming requests from zhtclients.
 

-------------------ZHT(USING MPI) CLUSTER DEPLOYMENT-------------------
1. TO START zht-mpiserver and zht-mpibroker(e.g. ON MANY NODEs), RUN:
mpiexec -f neighbor.mpi.conf -np <#processes> ./zht-mpiserver -z zht.conf -n neighbor.mpi.conf : ./zht-mpibroker -z zht.conf -n neighbor.mpi.conf
OR
mpiexec -hosts 192.168.1.100,192.168.1.101 -np <#processes> ./zht-mpiserver -z zht.conf -n neighbor.mpi.conf : ./zht-mpibroker -z zht.conf -n neighbor.mpi.conf

2. TO RUN THE BENCHMARK:
see also <RUN THE BENCHMARK> section.

3. TO RUN TEST CASES FOR C++ BINDING:
see also <RUN TEST CASES FOR C++ BINDING> section

4. TO RUN TEST CASES FOR C BINDING:
see also <RUN TEST CASES FOR C BINDING> section


NOTE THAT:
mpiexec: mpi command to launch mpi jobs.
-f neighbor.mpi.conf: negibor conf file containing nodes on which MPI processes run, in which each line is ip address(e.g. 192.168.1.100), DO NOT USE localhost.
-hosts 192.168.1.100,192.168.1.101: two nodes(comma separated ip addresses) on which MPI processes run.
-np <#processes> ./zht-mpiserver: allocate <#processes> processors to run <#processes> zht-mpiserver processes. 
-z zht.conf: zht conf file as input argument to zht-mpiserver or zht-mpibroker, the same file mentioned above.
-n neighbor.mpi.conf: negibor conf file as input argument to zht-mpiserver or zht-mpibroker, in which each line is ip address(e.g. 192.168.1.100), DO NOT USE localhost.			  
zht-mpibroker: the process launched on each node to talk to zht-mpiserver on behalf of zhtclient. 

PAY SPECIAL ATTENTIONS:
There is ONLY ONE INSTANCE of zht-mpibroker allowed to be deployed ON EVERY NODE.

zht-mpibroker MUST appear after zht-mpiserver IN ORDER in order to ensure the processes of zht-mpiserver
are assigned to proper mpi-ranks which serve processing incoming requests from zhtclients.


-------------------HOW TO CONFIGURE-------------------
There are two conf files needed, one is zht.conf, the other is neighbor.conf or neighbor.mpi.conf.

zht.conf:
PROTOCOL and PORT are the options to configure what protocols and ports(if applicable) 
over which zhtclient and zhtserver talk to each other. 

If you specified your own port by -p option(e.g. ./zhtserver -z zht.conf -n neighbor.conf -p 40000), this will override PORT defined in
zht.conf.

If you specified the disk file by -f option(e.g. -f novoht.db), the data will be persisted to that file, otherwise, they reside in memory.
Please specify different files if you are going to run multiple zhtservers in the same node. 


neighbor.conf:
This file is used for TCP/UDP protocol.

The IP and PORT pairs of neighbored nodes(or ZHT servers) in ZHT overlay network are configured, e.g. 192.168.1.100 50000, 192.168.1.100 50001 

The port you defined to listen on at neighbored node(or ZHT server)(e.g. the PORT in zht.conf, or ./zhtserver -z zht.conf -n neighbor.conf -p 40000)
MUST BE CONSISTENT WITH the port of that machine in neighbor.conf, otherwise, the zhtclient CANNOT reach the zhtserver by that port.


neighbor.mpi.conf:
This file is used for MPI protocol.

The IP(s) of neighbored nodes(or ZHT servers) in ZHT overlay network are configured, e.g. 192.168.1.100, no port is necessary for MPI protocol. 

NOTE that:
The parameter ID and VALUE(e.g. PROTOCOL TCP) for both zht.conf and neighbor.conf is SPACE delimited.
The conf files support #-style comment-out.
The blank lines will be ignored as comment-outs while parsing.