Getting Started

Software Dependencies

• JDK 1.7, Python 2.7, APSW 3.8 (http://rogerbinns.github.io/apsw/).
• Passwordless ssh.

The master branch contains the latest stable version of exareme prebuilt and ready to install, if you have downloaded the tar from the master branch you can skip the next section (Development).

The dev branch contains all the newest features of exareme. If you want to install this version of exareme that dev has you need to build it. You can build it and get the tar by following the instructions of the next section.

Development

• Requires

• maven 3.0+

• Build

• CLONE dev branch.
• RUN "mvn clean install -Pmultiarch -DskipTests=true" create a tarball (exareme/exareme-distribution/target/exareme-*.tar.gz) ready for production intallation.

Installing from tarball

• After installing the software dependencies you can unpack the tarball by executing :
  

$ mkdir ~/exareme
$ tar xf exareme-*tar.gz -C ~/exareme
$ cd ~/exareme

• Installation directory structure (under ~/exareme/ ) :
  

  • /bin/ - binary scripts.
    
  • /etc/ - configuration and properties files.
    
  • /var/ - logs and pids files.
    
  • /lib/ - exareme, madis libraries and dependencies.
    

• Current release can operate in local or distributed mode. You can configure the system accordingly by set up the appropiate files in ~/exareme/etc/exareme/.

  • exareme-env.sh exports the required environment variables like java etc.
  • Local mode requires only the master ip which you can provide by set up the master file.
  • Distributed mode requires to set up the master and workers files. Workers file format requires one worker ip per line.
  • Example:
    

$ cat ~/exareme/etc/exareme/master
192.168.0.1
$ cat ~/exareme/etc/exareme/workers
192.168.0.2
192.168.0.3

Running Exareme

• In ~/exareme/bin directory, you can find the exareme-admin.sh script which is responsible for the administration tasks for both local/distributed mode.
  

$ cd ~/exareme
$ ./bin/exareme-admin.sh --install  #You can skip it in local mode.
$ ./bin/exareme-admin.sh --start
$ ./bin/exareme-admin.sh --status
$ ./bin/exareme-admin.sh --stop
$ ./bin/exareme-admin.sh --kill
$ ./bin/exareme-admin.sh --uninstall #You can skip it in local mode.

Executing Queries

• In order to execute exareme queries you can use our jdbc driver or the console or our [REST] (https://github.com/madgik/exareme/wiki/Rest-api) API. By executing ./bin/exareme-admin.sh --console a madis console will be prompt along with some exa-like operators available.

  • exaquery - submit an EXADFL query.
  • exaqueryscript - submit an EXADFL queryscript.
  • exaresult - returns result table.

• Exa-like operators requires host, port, db property.

  • you can set this global properties by executing.
    mterm> var ‘db’ ‘/data/demo/tpch’;
  • or you can set as operator property
    mterm> exaresult ‘db:/data/demo/tpch’ nation;
  • host and port are optional and can be skipped if the console is running on the same host as master node.

Example usage of exa-like operators:

Sample import

$ ./bin/exareme-admin.sh --console
mterm> exaquery ‘db:/path/to/db/’ distributed create table lineitem as external
select
 cast(c1 as int) as l_orderkey,
 cast(c2 as int) as l_partkey,
 cast(c3 as int) as l_suppkey,
 cast(c4 as int) as l_linenumber,
 cast(c5 as int) as l_quantity,
 cast(c6 as float) as l_extendedprice,
 cast(c7 as float) as l_discount,
 cast(c8 as float) as l_tax,
 cast(c9 as text) as l_returnflag,
 cast(c10 as text) as l_linestatus,
 cast(c11 as text) as l_shipdate,
 cast(c12 as text) as l_commitdate,
 cast(c13 as text) as l_receiptdate,
 cast(c14 as text) as l_shipinstruct,
 cast(c15 as text) as l_shipmode,
 cast(c16 as text) as l_comment
from (file '/path/to/datasets/lineitem.tbl.gz' delimiter:| fast:1);

Sample query

$ cat /path/to/queries/q6.sql
distributed create temporary table lineitem_partial to 1 as
select sum(l_extendedprice * l_discount) as revenue_PARTIAL
from lineitem
where l_shipdate >= '1994-01-01' 
and l_shipdate < '1995-01-01' 
and l_discount > 0.07 
and l_discount < 0.09 
and l_quantity < 24;

distributed create table q6_result as
select sum(revenue_PARTIAL) as revenue
from lineitem_partial;

$ ./bin/exareme-admin.sh --console
mterm> exaqueryscript ‘db:/path/to/db’ file ‘/path/to/queries/q6.sql’;

Sample result

$ ./bin/exareme-admin.sh --console
mterm> select * from (exaresult ‘db:/path/to/db’ q6_result) limit 2;