Merge pull request #36 from unicef/feature/download-jars-in-dockerfile

Docker build improvements: Download jars in Drill dockerfile and signature check in Drill and Zookeeper dockerfiles

Author: merlos

README.md

@@ -13,6 +13,7 @@ This extension has been created to fit the needs of [UNICEF's magasin](https://g
 
 | Chart Version | Drill Version | Zookeeper Version | Release Date       | Release notes          |
 |---------------|---------------|-------------------|--------------------|------------------------|
+| 0.9.0         | 1.21.2        | 3.9.3             | 2024-12-02         | [Release notes v0.9.0](https://github.com/unicef/magasin-drill/releases/tag/releases%2Fv0.9.0) |
 | 0.8.0         | 1.21.2        | 3.9.2             | 2024-09-04         | [Release notes v0.8.0](https://github.com/unicef/magasin-drill/releases/tag/releases%2Fv0.8.0) | 
 | 0.7.2         | 1.21.2        | 3.9.2             | 2024-08-07         | [Release notes v0.7.2](https://github.com/unicef/magasin-drill/releases/tag/releases%2Fv0.7.2) |
 | 0.7.1         | 1.21.1        | 3.9.1             | 2024-02-05         | [Release notes v0.7.1](https://github.com/unicef/magasin-drill/releases/tag/releases%2Fv0.7.1) |

charts/drill/Chart.yaml

@@ -1,8 +1,8 @@
 apiVersion: v2
 name: drill
-version: 0.8.0
-appVersion: 1.21.2-3.9.2
-description: Helm charts for deploying Apache Drill on a Kubernetes cluster
+version: 0.9.0
+appVersion: 1.21.2-3.9.3
+description: Helm charts for deploying Apache Drill and Zookeeper on a Kubernetes cluster
 type: application
 keywords:
   - apache

charts/drill/values.yaml

@@ -19,7 +19,7 @@ drill:
   # Total CPU for each Drill pod
   cpu: 4000m
   # Drill image with tag
-  image: merlos/drill:1.21.2
+  image: merlos/drill:1.21.2-2024-12
   # Drill's HTTP port
   httpPort: 8047
   # User Server port. The Bit Server Port and Data Server ports are +1 and +2
@@ -140,7 +140,7 @@ zookeeper:
   # Total CPU for each ZooKeeper pod
   cpu: 500m
   # ZooKeeper image with tag
-  image: merlos/zookeeper:3.9.2
+  image: merlos/zookeeper:3.9.3
   # set annotations to ZooKeeper pods
   podAnnotations: {}
     # responsible: devops

docker/README.md

@@ -9,39 +9,40 @@ The `build.sh` script in the `docker/zookeeper` folder builds the zookeeper imag
 It takes two arguments:
 
 ```shell
-./build.sh -r <registry> -v <zookeeper-version>
+./build.sh -r <registry> -v <zookeeper-version> [-t <tag>] [-p] 
 ```
-Where `registry` is the URL of the registry (for example, an Azure registry whatever.acr.io). It can also be a Docker Hub username.
+Where `registry` is the URL of the registry (for example, an Azure registry whatever.acr.io). For Docker Hub, just use your username.
 
-The project version is the version of Zookeeper that you want to setup in the image. For instance "3.9.1", The list of versions is here: https://archive.apache.org/dist/zookeeper/
+The project version is the version of Zookeeper that you want to setup in the image. For instance "3.9.3", The list of versions is here: https://archive.apache.org/dist/zookeeper/
 
 Example:
 ```shell
 cd docker/zookeeper
-./build.sh -r merlos -v 3.9.1
+./build.sh -r merlos -v 3.9.3
 ```
 
-Alternatively it accepts `-p` to push the image to the registry `-t`` to create a different tag. 
+By default it does not push the image to the registry. You can add `-p` to push the image to the registry.
+
+By default it creates the same tag as the version. You can use `-t` to create a different tag. For instance: 
 
 ```shell
-./build.sh -r merlos -v 3.9.1 -t 3.9.1-deb -p
+./build.sh -r merlos -v 3.9.3 -t 3.9.3-deb -p
 ```
 
 In this image: 
 * zookeeper is run by the user `zk`
-* zookeper is installed in `/opt/zookeeper`
+* zookeeper is installed in `/opt/zookeeper`
 * Data is kept in `/var/lib/zookeeper`
 * Output logs are kept in `/var/log/zookeeper/`
 
-
-You have som images available in https://hub.docker.com/r/merlos/zookeeper
+You have some images available in https://hub.docker.com/r/merlos/zookeeper
 
 ### Build Drill
 
 Similarly,  the folder `docker/drill` contains the `build.sh` script that  builds, tags and uploads a docker image. It also takes two arguments:
 
 ```shell
-./build.sh <registry> <drill-version>
+./build.sh -r <registry> -v <drill-version> [-t <tag>] [-p]
 ```
 
 The list of drill versions are in this link https://archive.apache.org/dist/drill/.
@@ -50,15 +51,15 @@ Example:
 
 ```shell
 cd docker/drill
-./build.sh -r merlos -v 1.21.1
+./build.sh -r merlos -v 1.21.2
 ```
 
-Where merlos is a user in dockerhub and 1.21.1 is the version of Drill. For versions older than 1.19.0 you may need to tweak the 
+Where merlos is a user in dockerhub and 1.21.2 is the version of Drill. 
 
 Similarly to the zookeper script, it also accepts `-t`  and `-p`.
 
 ```shell
-./build.sh -r merlos -v 1.21.1 -t 1.21.1-deb -p
+./build.sh -r merlos -v 1.21.2 -t 1.21.2-deb -p
 ```
 
 In this image: 
@@ -67,5 +68,4 @@ In this image:
 * Output logs are kept in `/var/log/drill/`
 * Copies the two jars included in the `jar/` folder (azure-storage and handoop-azure)
 
-
-You have some images available on https://hub.docker.com/r/merlos/drill/tags
+You have some images available already on https://hub.docker.com/r/merlos/drill/tags

docker/drill/Dockerfile

@@ -1,17 +1,72 @@
+# Stage 1: DRILL download and verification
+FROM debian:stable-slim AS verifier
+
+ARG VERSION
+ARG MIRROR=https://archive.apache.org
+
+ENV PROJECT=drill
+
+# Install necessary tools
+RUN apt-get update && \
+    apt-get install -y wget gpg gpgv && \
+    rm -rf /var/lib/apt/lists/*
+
+# Download Drill, sha512 checksum, GPG signature and PGP keys
+RUN wget ${MIRROR}/dist/${PROJECT}/$VERSION/apache-${PROJECT}-${VERSION}.tar.gz -P /tmp/
+RUN wget ${MIRROR}/dist/${PROJECT}/$VERSION/apache-${PROJECT}-${VERSION}.tar.gz.sha512 -P /tmp/
+RUN wget ${MIRROR}/dist/${PROJECT}/$VERSION/apache-${PROJECT}-${VERSION}.tar.gz.asc -P /tmp/
+RUN wget ${MIRROR}/dist/${PROJECT}/KEYS -P /tmp/
+
+# Create the drill pgp keyring
+RUN gpg --no-default-keyring --keyring /tmp/${PROJECT}-keyring.gpg --import /tmp/KEYS
+
+# Verify SHA512 checksum
+RUN cd /tmp && echo "$(cat apache-${PROJECT}-${VERSION}.tar.gz.sha512) apache-${PROJECT}-${VERSION}.tar.gz" | sha512sum -c -
+
+# Verify GPG signature
+RUN gpgv --keyring /tmp/${PROJECT}-keyring.gpg /tmp/apache-${PROJECT}-${VERSION}.tar.gz.asc /tmp/apache-${PROJECT}-${VERSION}.tar.gz
+
+# Extract Drill
+RUN mkdir -p /tmp/apache-${PROJECT}
+RUN tar -xzvf /tmp/apache-${PROJECT}-$VERSION.tar.gz -C/tmp/apache-${PROJECT}/ --strip=1
+
+# remove the tar file
+RUN rm -f /tmp/apache-${PROJECT}-$VERSION.tar.gz
+
+# Test Drill
+RUN ls -la /tmp/apache-drill
+
+
+
+#
+# Stage 2: Setup & dependency download
+#
 FROM debian:stable-slim
 
 ARG VERSION
 RUN echo $VERSION
-ARG MIRROR=https://archive.apache.org
-RUN echo $MIRROR
+
+LABEL maintainer="merlos"
+LABEL description="Apache Drill Docker Image (by https://magasin.unicef.io)"
+
+# Set environment variables
+ENV DRILL_VERSION=$VERSION
+ENV AZURE_STORAGE_VERSION=8.6.6
+ENV HADOOP_AZURE_VERSION=3.4.1
 
 # Install Pre-Requisite Packages
-RUN apt-get update
-RUN apt-get upgrade -y
-RUN apt-get install -y openjdk-17-jre jed procps stress
+RUN apt-get update && \ 
+    apt-get upgrade -y && \
+    apt-get install -y  \
+      openjdk-17-jre \
+      jed \
+      procps \
+      stress \
+      wget \
+      figlet && \
+    rm -rf /var/lib/apt/lists/* 
 
 # Display project name in bash
-RUN apt-get install -y figlet
 RUN echo "figlet Drill && echo \"\nDockerized for magasin (https://github.com/unicef/magasin) \n\n\"" > /etc/bash.bashrc
 
 # Clean apt cache
@@ -25,34 +80,54 @@ RUN mkdir /opt/drill && chown drill /opt/drill
 RUN mkdir /var/lib/drill  && chown drill /var/lib/drill && chgrp drill /var/lib/drill && chmod 700 /var/lib/drill
 RUN mkdir /var/log/drill && chown drill /var/log/drill && chgrp drill /var/log/drill && chmod 700 /var/log/drill
 
-# Install Drill
-ADD ${MIRROR}/dist/drill/$VERSION/apache-drill-${VERSION}.tar.gz /tmp/apache-drill-$VERSION.tar.gz
-RUN tar -xzf /tmp/apache-drill-$VERSION.tar.gz -C/opt/drill --strip=1
-RUN rm -f /tmp/apache-drill-$VERSION.tar.gz
+# Copy the verified Drill files from the verifier stage
+COPY --from=verifier /tmp/apache-drill /opt/drill
 
-# Change conf folder owner as some files are required to be created tmporarily.
+# Change conf folder owner as some files are required to be created temporarily.
 RUN chgrp -v -R drill /opt/drill
 RUN chown -R drill /opt/drill/conf 
 
-# Expose ports: Drill web UI, user port, control port and Data port
-# https://drill.apache.org/docs/ports-and-bind-addresses-used-by-drill/
-EXPOSE 8047 31010 31011 31012
+# Set user to test the service
+USER drill
 
-# Test Drill
-#RUN echo "select * from sys.version;" > /tmp/version.sql
-#RUN /opt/drill/bin/sqlline -u jdbc:drill:zk=local --verbose --run=/tmp/version.sql
+# Test Drill before changing the default conf 
+RUN echo "select * from sys.version;" > /tmp/version.sql
+RUN /opt/drill/bin/sqlline -u jdbc:drill:zk=local --verbose --run=/tmp/version.sql
+RUN rm -f /tmp/version.sql
+
+# Set root 
+USER root
 
 # Copy Scripts
 COPY distrib-env.sh drill-distrib.conf /opt/drill/conf/
 COPY isLive.sh isReady.sh preStop.sh /opt/drill/bin/
 COPY start.sh /opt/drill/start.sh
 
-# Copy local jars
-COPY jars/* /opt/drill/jars/3rdparty/
+# Download 3rd party dependencies
+# Available versions in https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/
+RUN wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/${HADOOP_AZURE_VERSION}/hadoop-azure-${HADOOP_AZURE_VERSION}.jar  -P /opt/drill/jars/3rdparty
+# Ensure the integrity of the downloaded file (sha1)
+RUN wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/${HADOOP_AZURE_VERSION}/hadoop-azure-${HADOOP_AZURE_VERSION}.jar.sha1  -P /opt/drill/jars/3rdparty
+RUN cd /opt/drill/jars/3rdparty && echo $(cat hadoop-azure-${HADOOP_AZURE_VERSION}.jar.sha1) hadoop-azure-${HADOOP_AZURE_VERSION}.jar | sha1sum -c -
+
+
+# Available versions in https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/
+RUN wget https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/${AZURE_STORAGE_VERSION}/azure-storage-${AZURE_STORAGE_VERSION}.jar  -P /opt/drill/jars/3rdparty
+# Ensure the integrity of the downloaded file (sha1)
+RUN wget https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/${AZURE_STORAGE_VERSION}/azure-storage-${AZURE_STORAGE_VERSION}.jar.sha1  -P /opt/drill/jars/3rdparty
+RUN cd /opt/drill/jars/3rdparty && echo $(cat azure-storage-${AZURE_STORAGE_VERSION}.jar.sha1) azure-storage-${AZURE_STORAGE_VERSION}.jar | sha1sum -c -
 
-# Set user to run the service
+
+# Expose ports: Drill web UI, user port, control port and Data port
+# https://drill.apache.org/docs/ports-and-bind-addresses-used-by-drill/
+EXPOSE 8047 31010 31011 31012
+
+
+RUN chown -R drill /tmp/drill 
+RUN chgrp -R drill /tmp/drill 
+
+# Set the user to run the container
 USER drill
 
 # Start Drill
-ENTRYPOINT /opt/drill/start.sh
-
+ENTRYPOINT ["/opt/drill/start.sh"]

docker/drill/jars/azure-storage-8.0.0.jar

@@ -1,12 +1,23 @@
 #!/usr/bin/env bash
 
+
+figlet -f slant "Drill"
+
+echo "Docker Image build information"
+echo "---------------------------------"
+echo DRILL_VERSION= $DRILL_VERSION
+echo AZURE_STORAGE_VERSION= $AZURE_STORAGE_VERSION
+echo HADOOP_AZURE_VERSION= $HADOOP_AZURE_VERSION
+echo "---------------------------------"
+
 # Be verbose 
-set -x
+#set -x
 DRILL_HOME="/opt/drill"
 DRILL_ENV_FILE="${DRILL_HOME}/conf/distrib-env.sh"
 DRILL_CONF_FILE="${DRILL_HOME}/conf/drill-distrib.conf"
 DRILL_CONF_OVERRIDE_FILE="${DRILL_HOME}/conf/drill-override.conf"
 
+echo "Setting up memory..."
 # Configuring Drill memory
 DRILL_MAX_MEMORY_VALUE=`echo ${DRILL_MAX_MEMORY} | tr -dc '0-9'`
 sed -i "s/__DRILL_MAX_MEMORY_VALUE__/${DRILL_MAX_MEMORY_VALUE}/g" ${DRILL_ENV_FILE}
@@ -43,7 +54,7 @@ fi
 ${DRILL_HOME}/bin/drillbit.sh status
 
 # Stop being verbose
-set +x
+#set +x
 
 # Keep container alive
 while true; do sleep 5; done