Skip to content

Latest commit

 

History

History
136 lines (81 loc) · 4.63 KB

README.markdown

File metadata and controls

136 lines (81 loc) · 4.63 KB
Raw

Example Storm Topologies

Learn to use Storm!

Table of Contents

Getting started

Prerequisites

First, you need java and git installed and in your user's PATH. Also, two of the examples in storm-starter require Python and Ruby.

Next, make sure you have the storm-starter code available on your machine. Git/GitHub beginners may want to use the following command to download the latest storm-starter code and change to the new directory that contains the downloaded code.

$ git clone git://github.com/nathanmarz/storm-starter.git && cd storm-starter

storm-starter overview

storm-starter contains a variety of examples of using Storm. If this is your first time working with Storm, check out these topologies first:

  1. ExclamationTopology: Basic topology written in all Java
  2. WordCountTopology: Basic topology that makes use of multilang by implementing one bolt in Python
  3. ReachTopology: Example of complex DRPC on top of Storm

After you have familiarized yourself with these topologies, take a look at the other topopologies in src/jvm/storm/starter/ such as RollingTopWords for more advanced implementations.

If you want to learn more about how Storm works, please head over to the Storm project page.

Using storm-starter with Leiningen

Install Leiningen

The storm-starter build uses Leiningen 2.0. Install Leiningen by following the leiningen installation instructions.

Running topologies with Leiningen

To run a Java topology

$ lein deps
$ lein compile
$ java -cp $(lein classpath) storm.starter.ExclamationTopology

To run a Clojure topology:

$ lein deps
$ lein compile
$ lein run -m storm.starter.clj.word-count

Using storm-starter with Maven

Install Maven

Maven is an alternative to Leiningen. Install Maven (preferably version 3.x) by following the Maven installation instructions.

Running topologies with Maven

storm-starter contains m2-pom.xml which can be used with Maven using the -f option. For example, to compile and run WordCountTopology in local mode, use the command:

$ mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.WordCountTopology

You can also run clojure topologies with Maven:

$ mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.clj.word_count

Packaging storm-starter for use on a Storm cluster

You can package a jar suitable for submitting to a Storm cluster with the command:

$ mvn -f m2-pom.xml package

This will package your code and all the non-Storm dependencies into a single "uberjar" at the path target/storm-starter-{version}-jar-with-dependencies.jar.

Running unit tests

Use the following Maven command to run the unit tests that ship with storm-starter. Unfortunately lein test does not yet run the included unit tests.

$ mvn -f m2-pom.xml test

Using storm-starter with IntelliJ IDEA

Importing storm-starter as a project in IDEA

The following instructions will import storm-starter as a new project in IntelliJ IDEA.

  • Copy m2-pom.xml to pom.xml. This is requried so that IDEA (or Eclipse) can properly detect the maven configuration.
  • Open File > Import Project... and navigate to the top-level directory of your storm-starter clone (e.g. ~/git/storm-starter).
  • Select Import project from external model, select "Maven", and click Next.
  • In the following screen, enable the checkbox Import Maven projects automatically. Leave all other values at their defaults. Click Next.
  • Click Next on the following screen about selecting Maven projects to import.
  • Select the JDK to be used by IDEA for storm-starter, then click Next.
    • At the time of this writing you should use JDK 6.
    • It is strongly recommended to use Sun/Oracle JDK 6 rather than OpenJDK 6.
  • You may now optionally change the name of the project in IDEA. The default name suggested by IDEA is "storm-starter". Click Finish once you are done.