In the following tutorials we will be looking on how to use Apache Flink with Java. Learn more about Flink at http://flink.apache.org/
In the following section we will look at different components which makes part of the Apache Flink ecosystem. The below image and the following explanation was taken from this source, big thanks to the Dataflair Team.
TODO -> Explain the ecosystem
Parts of this section was taken from Apache Flink GitHub page from the "Building Apache Flink from Source" section. As from this date the current stable version of Apache Flink is 1.8.0, so the following tutorials will be based on this version.
- Unix-like environment (we use Linux, Mac OS X, Cygwin)
- git
- Maven (we recommend version 3.2.5)
- Java 8 (Java 9 and 10 are not yet supported)
We are currently using Ubuntu 18.04.1 LTS. I suggest installing Ubuntu or Debian if you want to use a Linux distro. If you are using Windows you can set up a Virtual Machine and install one of the suggested Linux distros. Any one using Mac OS X is good to go, since it is a Unix-like environment.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install git
- Step 1: Install Homebrew package manager.
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew doctor
- Step 2: Install Git.
brew install git
sudo add-apt-repository ppa:webupd8team/java # when prompted press ENTER
sudo apt update
sudo apt install oracle-java8-installer
To install Java 8 on Mac OS X, follow this tutorial.
The following steps were taken from this source, big thanks to Gobinath Loganathan.
- Step 1: Download the recommended version 'apache-maven-3.2.5-bin.tar.gz' from this source.
- Step 2: Move to the /opt directory.
cd /opt
- Step 3: Extract the maven archive into the opt directory.
sudo tar -xvzf ~/Downloads/apache-maven-3.2.5-bin.tar.gz
- Step 4: Edit the /etc/environment file.
# open nano and edit environment variables
sudo nano /etc/environment
# add the following environment variable
M2_HOME="/opt/apache-maven-3.2.5"
# append the bin directory to the PATH variable
/opt/apache-maven-3.2.5/bin
# so the result should be something similar to the below
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/apache-maven-3.2.5/bin"
M2_HOME="/opt/apache-maven-3.2.5"
# once you added the environment variable and appended the PATH
# press "ctrl + o" to save changes and "ctrl + x" to close nano.
- Step 5: Update the mvn command.
sudo update-alternatives --install "/usr/bin/mvn" "mvn" "/opt/apache-maven-3.2.5/bin/mvn" 0
sudo update-alternatives --set mvn /opt/apache-maven-3.2.5/bin/mvn
- Step 6: Check version to confirm installation.
mvn --version
Once all the prerequisites are installed execute the following commands to build from source.
git clone https://github.com/apache/flink.git
cd flink
mvn clean package -DskipTests # this will take up to 10 minutes
Flink is now installed in build-target
NOTE: Maven 3.3.x can build Flink, but will not properly shade away certain dependencies. Maven 3.0.3 creates the libraries properly. To build unit tests with Java 8, use Java 8u51 or above to prevent failures in unit tests that use the PowerMock runner.
We suggest to follow the tutorials in order, since some steps required in one tutorial may be found in a previous tutorial.
The following suggestions were taken from the following sources; TODO -> Write this section https://www.quora.com/How-do-I-learn-Apache-Flink