-
Notifications
You must be signed in to change notification settings - Fork 37
How we set up continuous integration on Jenkins
Adam Hooper edited this page Feb 8, 2021
·
23 revisions
-
At https://cloud-images.ubuntu.com/locator/ec2/, pick the latest
hvm:ebs-ssd
instance for the region (us-east-1
in our case)- Choose the dedicated
CI
VPC, with IAM roleJenkins-CI
. - Tag it
Environment
:ci
- Give it the
jenkins-ci
security group (inbound port 443) - Launch it!
- Choose the dedicated
-
Name it
Jenkins-CI
-
Give yourself temporary SSH access (via the security group) and SSH in.
-
Install Jenkins:
sudo apt-get update && sudo apt-get dist-upgrade
- Follow the instructions at https://pkg.jenkins.io/debian-stable/ to install:
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add - sudo apt-add-repository 'deb https://pkg.jenkins.io/debian-stable binary/' sudo apt-get update sudo apt-get install openjdk-11-jre-headless jenkins
-
Set up GitHub for authentication: at https://jenkins-ci.overviewdocs.com/securityRealm/finishLogin set up
overview-jenkins-ci
with a URL ofhttps://jenkins-ci.overviewdocs.com
. -
Setup Jenkins:
- From your computer,
ssh -L 8080:localhost:8080 ubuntu@[JenkinsIP]
and browse to http://localhost:8080 - Copy/paste the administrator password in (as prompted by Jenkins)
- "Select plugins to install" => choose defaults, ...
- plus: embeddable-build-status, JUnit Plugin, GitHub Plugin, GitHub Authentication Plugin, Amazon EC2 Plugin, Kubernetes CLI
- minus: Ant Plugin, Gradle Plugin
- Skip creating the administrator user. Click "Start Using Jenkins"
- "Manage Jenkins" => "Configure System":
-
# of executors
:0
-
Jenkins URL
:https://jenkins-ci.overviewdocs.com
-
System Admin e-mail address
:admin@overviewdocs.com
-
SMTP Server
:email-smtp.us-east-1.amazonaws.com
- Check
User SMTP Authentication
and enter Amazon's SMTP settings
-
- "Manage Jenkins" => "Configure Global Security"
-
Security realm
=>Github Authentication Plugin
- Enter
Client ID
andClient Secret
from theoverview-jenkins-ci
app page on GitHub -
Authorization
=>GitHub Committer Authorization Strategy
- Enter comma-separated
Admin User Names
- Check
Use GitHub repository permissions
, and don't fill inParticipant in Organization
- Check
Grant READ permissions for Anonymous Users
- Check
Grant ViewStatus permissions for Anonymous Users
- Save. You'll be locked out.
-
- From your computer,
-
Adjust DNS to point to your new server.
-
Set up the HTTPS proxy:
- Install programs
sudo apt-get install haproxy sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot sudo certbot certonly --standalone -d jenkins-ci.overviewdocs.com -m admin@overviewdocs.com --agree-tos -n
- Fill in
/etc/haproxy/haproxy.cfg
:global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy user haproxy group haproxy daemon # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy-1.5.14&openssl=1.0.1e&hsts=yes&profile=modern ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets defaults log global mode http option httplog option dontlognull option forwardfor option http-server-close timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 502 /etc/haproxy/errors/502.http frontend jenkins-ci.overviewdocs.com bind :80 bind :443 ssl crt /etc/haproxy/ssl.pem redirect scheme https if !{ ssl_fc } reqadd X-Forwarded-Proto:\ https use_backend jenkins backend jenkins server jenkins01 127.0.0.1:8080
- Fill in
/etc/letsencrypt/post-renew.sh
:#!/bin/sh cat \ /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/privkey.pem \ /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/fullchain.pem \ > /etc/haproxy/ssl.pem chown haproxy:haproxy /etc/haproxy/ssl.pem chmod 0600 /etc/haproxy/ssl.pem systemctl restart haproxy
- Fill in
/etc/letsencrypt/pre-renew.sh
:#!/bin/sh systemctl stop haproxy
chmod +x /etc/letsencrypt/post-renew.sh /etc/letsencrypt/pre-renew.sh
-
/etc/letsencrypt/post-renew.sh
(will actually start haproxy) - Fill in
/etc/cron.daily/letsencrypt
:#!/bin/sh exec /usr/bin/certbot renew \ --pre-hook /etc/letsencrypt/pre-renew.sh \ --post-hook /etc/letsencrypt/post-renew.sh \ --quiet
chmod +x /etc/cron.daily/letsencrypt
- Install programs
-
Set up EC2 spot-instance nodes
- Browse to Jenkins, "Manage Jenkins" -> "Configure System"
- "Add New Cloud" => "Amazon EC2"
- Check
Use EC2 instance profile to obtain credentials
- Choose region
us-east-1
- Create a private key
jenkins-ci
in AWS (EC2 => Key Pairs => Create key pair calledjenkins-ci
. This will download ajenkins-ci.pem
in your browser) - Enter the private key in Jenkins by copy/pasting from
jenkins-ci.pem
. - Advanced -> "Add" an AMI....
-
Description
:Jenkins-CI
-
AMI ID
:ami-026c8acd92718196b
(from https://cloud-images.ubuntu.com/locator/ec2/ amd64 ebs-ssd us-east-1 bionic) -
Instance Type
: something big,Availability Zone
: whatever's best. At time of writing,C48xlarge
andus-east-1d
have a good price. (Use the AWS console to find a combination that seems reliable.) -
Spot Max Bid Price
: whatever --0.5
means 50 cents per hour. (Expect 3 builds per hour. Jenkins won't build every commit, if you commit lots of commits at the same time.) -
Choose Bid Type
:persistent
-
Security group names
:test-slave
(which grants SSH access toJenkins-CI
security group: Jenkins will SSH into it) -
Remote FS root
:/home/ubuntu
-
Remote user
:ubuntu
-
Root command prefix
:sudo
-
Slave command prefix
:sudo
-
Labels
:test-slave
-
Idle termination time
:30
-
Advanced
=>Number of Executors
:1
- Check
Stop/Disconnect on Idle Timeout
-
Tags
:role:jenkins-test-slave
,Name:jenkins-test-slave
,Environment:ci
-
IAM Profile
: The description of your IAM "instance-profile" Role. This starts witharn:
and ends withinstance-profile/jenkins-test-slave
-- a role with permission to write tos3://overview-builds
. - Add this Init script:
#!/bin/bash set -ex DEBIAN_FRONTEND=noninteractive apt-get -y -qq update DEBIAN_FRONTEND=noninteractive apt-get -y -qq dist-upgrade DEBIAN_FRONTEND=noninteractive apt-get -y -qq update DEBIAN_FRONTEND=noninteractive apt-get -y -qq install apt-transport-https awscli ca-certificates curl openjdk-11-jre-headless software-properties-common # https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" DEBIAN_FRONTEND=noninteractive apt-get -y -qq update DEBIAN_FRONTEND=noninteractive apt-get -y -qq install docker-ce systemctl start docker usermod -G docker ubuntu curl -L https://github.com/kubernetes/kops/releases/download/1.12.2/kops-linux-amd64 > /usr/local/bin/kops curl -L https://storage.googleapis.com/kubernetes-release/release/v1.15.1/bin/linux/amd64/kubectl > /usr/local/bin/kubectl curl -L "https://github.com/docker/compose/releases/download/1.28.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/{docker-compose,kops,kubectl}
- Set
Block device mapping
to/dev/sda1=:24
(because Docker images often grow to greater than 8GB) - Check 'Associate Public IP' (or it won't be able to download things)
-
Add Docker Hub credentials
- Make sure you have access to https://hub.docker.com/r/overview/
- Browse to Jenkins and click
Credentials
(left sidebar),(global)
, and thenAdd Credentials
(left sidebar) - Add a
Username with Password
with ScopeSystem
, ID and Description bothdocker-hub
, and the username and password that you used to log in to https://hub.docker.com.
-
Add
overview-server
project- Browse to Jenkins and click
Create new jobs
- Name
overview-server
,Pipeline
,OK
- Check
GitHub project
and enter URLhttps://github.com/overview/overview-server
- Check
GitHub hook trigger for GITScm polling
- Check
Quiet period
and enter120
seconds (builds cost time and money; this can save us a few) - Make the Pipeline script
Pipeline script from SCM
. SelectGit
as a source, and enterhttps://github.com/overview/overview-server
- Make
*/master
and*/feature-*
are the branch specifiers
- Browse to Jenkins and click
-
Add Kubernetes credentials
- Browse to Jenkins and click
Credentials
(left sidebar),(global)
, and thenAdd Credentials
(left sidebar) - Add a
Secret text
with ScopeGlobal
, ID and Description bothoverview-production-kubernetes
, and the secret text you get following the instructions inoverview-server/kubernetes/README.md
- Browse to Jenkins and click
-
Hook up to GitHub
- Add
[![Build Status](http://jenkins-ci.overviewdocs.com/job/overview-server/badge/icon)](http://jenkins-ci.overviewdocs.com/job/overview-server/)
toREADME.md
- In
overview-server
project settings on GitHub, add "Integrations and Services" -> "Jenkins (GitHub plugin)" ->https://jenkins-ci.overviewdocs.com/github-webhook/
- Add
-
Hook up to Slack
- On Jenkins, add "Slack" plugin in
Manage Plugins
- On Slack, add "Jenkins" plugin
- Copy/paste the
Team Domain
andIntegration Token
from Slack into Jenkins - (That's it -- the
Jenkinsfile
will handle the rest.)
- On Jenkins, add "Slack" plugin in