Support multi-arch image building for couchdb docker image #60

sudswas · 2018-01-22T10:50:44Z

This commit introduces a way by which we can now build multi-arch
docker images for couchdb. Some of this work is inspired from the
https://github.com/kubernetes/kubernetes/tree/master/build/debian-base

This is the first cut and it can be enhanced based on the couchdb
community's preference. This image is required atleast for the
openwhisk project where we are driving the multi-arch support.

Usage:

     `make build` ==> Retains the old behavior of builds
     `make build ARCH=ppc64le` ==> Builds it for a specific arch

Overview

This PR tries to aid multi-arch builds for couchdb docker image. A makefile is introduce to allow the users to choose an architecture to build the docker image on. No changes are done for the default case that exists today. This uses the qemu static emulator to do the required work.

The README changes are not updated as a part of this PR.

Testing recommendations

Build it for all different architectures by specifying the ARCH parameter during make build.

Checklist

Code is written and works correctly;
Changes are covered by tests;
Documentation reflects the changes;

This commit introduces a way by which we can now build multi-arch docker images for couchdb. Some of this work is inspired from the https://github.com/kubernetes/kubernetes/tree/master/build/debian-base This is the first cut and it can be enhanced based on the couchdb community's preference. This image is required atleast for the openwhisk project where we are driving the multi-arch support. Usage: `make build` ==> Retains the old behavior of builds `make build ARCH=ppc64le` ==> Builds it for a specific arch Travis file is updated to use `make` instead of docker commandline to do the builds.

sudswas · 2018-01-22T15:04:18Z

@wohali Hi, it would be great to have multi-arch manifest images of couchdb on dockerhub. This PR helps in building the multi-arch couchdb image for ppc64le and amd64 using the same Makefile. We could enhance the makefile to include the manifest tool information as well but i am not sure how you would like to proceed. Let me know your thoughts!

wohali · 2018-01-22T16:10:07Z

Hi @sudswas , thanks for the submission and the pull request! I appreciate the enthusiasm and interest in our project very much. Welcome, from a sister Apache project!

I think this work is premature for CouchDB until these platforms are added to our regular CouchDB CI workflow in Travis and/or Jenkins.

If you look at these links (1, 2, 3) you'll see that there have been multiple issues on some of the non-Intel platforms. ppc64le is closest, but without regular checks on each desired platform, it would be irresponsible for us to start advertising that support through Docker, wouldn't you agree?

FYI, we don't and shouldn't use this repository for our CI work; building the Docker images shouldn't always include running the full test suite as well. If you are interested in helping us get regular CI work going on additional platforms, and helping fix the bugs that are found, we'd very much welcome the support! 💃

Travis for the main repo is here: https://github.com/apache/couchdb/blob/master/.travis.yml
Jenkins for the main platform is here: https://github.com/apache/couchdb/blob/master/Jenkinsfile (looks like it started having issues while I was on break, I'll have to look when I have time, if someone else doesn't)

We'd like to solve the HW problem as much as possible on real hardware, as preferentially suggested in the official-images ticket by Docker project leadership. Apache Jenkins appears to now have an arm64 machine available for us, but the 3 ppc64 nodes are reserved for hadoop use only. If you know of any (IBM?) resources to help donate ppc64/s390 hardware for Apache CI use, Apache Infra would be most appreciative.

As I have no objections to the approach you've taken, I'm going to leave the PR open until we get these other issues resolved.

Thanks again for your help!

wohali · 2018-01-22T16:12:37Z

2.1.1/Dockerfile.build

+FROM BASEIMAGE
+MAINTAINER CouchDB Developers dev@couchdb.apache.org
+
+CROSS_BUILD_COPY qemu-ARCH-static /usr/bin/


Since this is a cross build file, perhaps a more informative filename than Dockerfile.build should be used. Perhaps Dockerfile.crossbuild?

Yeah makes sense. Done.

wohali · 2018-01-22T16:13:31Z

2.1.1/Dockerfile.build

@@ -0,0 +1,128 @@
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not


I am concerned about this file falling out of sync with the main Dockerfile. Is it possible to generate all images from the same file?

I took care of it. Basically now everything should get done via one file only.

wohali · 2018-01-22T16:16:18Z

2.1.1/Makefile

+                > $(TEMP_DIR)/Dockerfile.build
+	docker run --rm --privileged multiarch/qemu-user-static:register --reset
+	curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-$(QEMUARCH)-static.tar.gz | tar -xz -C $(TEMP_DIR)
+	# This gosu command is not necessary to run when we are building the image. Hence drop it.


Why is this so? How does this build process differ from the amd64 one in which we presumably do need to run that command?

Well so - the qemu emulator isn't quite perfect w.r.t gosu. While it's able to build gosu, running a gosu command does not work. I thought since this command is not doing anything much, we might as well drop it.

wohali · 2018-01-22T16:18:54Z

I realized I didn't reply to the manifest tool question. I'd like to wait for docker manifest to show up in the CLI before we do that - looks like the PR has landed and support should be available Real Soon Now.

sudswas · 2018-01-23T16:27:29Z

Thanks a lot @wohali for the quick review.
Besides fixing the code review comments...I have worked with an engineer to raise a JIRA request here: https://issues.apache.org/jira/browse/INFRA-15887 . It appears that we have atleast a couple of nodes that's not really getting used at the moment. Interim, i will also look at the generic couchdb CI and see what's going on over there.

wohali · 2018-01-24T06:01:47Z

HI there, I saw the JIRA request has been closed and the ppc64le build is suitable as a target, running Ubuntu 16/xenial.

I'll work on adding ppc64le builds to our Jenkinsfile this week (it should be quick, assuming the tests all pass consistently) and will then revisit here.

Update: Jenkins is blocked for other reasons (tests failing) so this work is blocked until Jenkins is all green again. @jiangphcn has said that he may be able to assist, but holidays start soon for him. If you are able to help diagnose and resolve these failures at all, that'd be most useful.

abdasgupta · 2018-06-06T06:51:17Z

Raised a new PR as the continuation of this work because the author of this issue is unable to complete this work.. Please follow this PR: #84 for the continuation of this work.

abdasgupta · 2018-06-06T06:53:30Z

It would be great if someone from maintainers can close this PR.

wohali · 2018-06-06T16:27:06Z

Closed in favour of #84 .

sudswas force-pushed the multi-arch-builds branch 3 times, most recently from bea995f to 066b0b7 Compare January 22, 2018 14:19

sudswas force-pushed the multi-arch-builds branch from 066b0b7 to 000f83c Compare January 22, 2018 14:56

wohali reviewed Jan 22, 2018

View reviewed changes

wohali mentioned this pull request Jan 22, 2018

Add multi-platform support to CI configuration apache/couchdb#1121

Closed

Fixing code review comments

f3d0bec

This was referenced Feb 1, 2018

Support for ppc64le and s390x? #66

Closed

Provide and publish s390x images #67

Open

Provide and publish ppc64le images #68

Closed

abdasgupta mentioned this pull request Jun 6, 2018

Support multi-arch image building for couchdb docker image #84

Closed

3 tasks

wohali closed this Jun 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multi-arch image building for couchdb docker image #60

Support multi-arch image building for couchdb docker image #60

sudswas commented Jan 22, 2018 •

edited

Loading

sudswas commented Jan 22, 2018 •

edited

Loading

wohali commented Jan 22, 2018

wohali Jan 22, 2018

sudswas Jan 23, 2018

wohali Jan 22, 2018

sudswas Jan 23, 2018

wohali Jan 22, 2018

sudswas Jan 23, 2018

wohali commented Jan 22, 2018

sudswas commented Jan 23, 2018

wohali commented Jan 24, 2018 •

edited

Loading

abdasgupta commented Jun 6, 2018 •

edited

Loading

abdasgupta commented Jun 6, 2018

wohali commented Jun 6, 2018

		@@ -0,0 +1,128 @@
		# Licensed under the Apache License, Version 2.0 (the "License"); you may not

Support multi-arch image building for couchdb docker image #60

Support multi-arch image building for couchdb docker image #60

Conversation

sudswas commented Jan 22, 2018 • edited Loading

Overview

Testing recommendations

Checklist

sudswas commented Jan 22, 2018 • edited Loading

wohali commented Jan 22, 2018

wohali Jan 22, 2018

Choose a reason for hiding this comment

sudswas Jan 23, 2018

Choose a reason for hiding this comment

wohali Jan 22, 2018

Choose a reason for hiding this comment

sudswas Jan 23, 2018

Choose a reason for hiding this comment

wohali Jan 22, 2018

Choose a reason for hiding this comment

sudswas Jan 23, 2018

Choose a reason for hiding this comment

wohali commented Jan 22, 2018

sudswas commented Jan 23, 2018

wohali commented Jan 24, 2018 • edited Loading

abdasgupta commented Jun 6, 2018 • edited Loading

abdasgupta commented Jun 6, 2018

wohali commented Jun 6, 2018

sudswas commented Jan 22, 2018 •

edited

Loading

sudswas commented Jan 22, 2018 •

edited

Loading

wohali commented Jan 24, 2018 •

edited

Loading

abdasgupta commented Jun 6, 2018 •

edited

Loading