Docker in Jenkins, in Docker
Jenkins is still a viable and popular CI platform. I still use it at work where it does the job, and I don't feel that there is a compelling technological reason to abandon it yet. Well, there may be one - Jenkins has no built-in support for nested Docker, ie, running Docker containers whilst itself running in Docker. The Jenkins team maintains an official Docker images so you can easily run Jenkins dockerized, and Jenkins has plenty of Docker plugins, but you can't do both at once. At least not out of the box. It is possible, with a few tweaks.
Custom container
To get this to work, we're going to modify the official Jenkins Docker image. This is pretty simple, and if you use a lot of containers it worth getting comfortable overriding existing images this way.
This is the Dockerfile :
# base this container on the official Jenkins image at a fixed version
FROM jenkins/jenkins:2.213
# the jenkins image we're extending will already be set to user:jenkins, switch to root so we can install stuff
USER root
RUN apt-get update \
# we need some utils to fetch files
&& apt-get -y install wget \
&& apt-get -y install curl \
# libs required by docker
&& apt-get -y install libseccomp2 \
&& apt-get -y install iptables \
&& apt-get -y install libdevmapper-dev \
# install docker @ 19.03 explicitly, version locking is essential
&& wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.2.6-3_amd64.deb \
&& wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce-cli_19.03.5~3-0~ubuntu-bionic_amd64.deb \
&& wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.5~3-0~ubuntu-bionic_amd64.deb \
&& dpkg -i docker-ce-cli_19.03.5~3-0~ubuntu-bionic_amd64.deb \
&& dpkg -i containerd.io_1.2.6-3_amd64.deb \
&& dpkg -i docker-ce_19.03.5~3-0~ubuntu-bionic_amd64.deb \
&& curl -L "https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose \
# clean up docker install files
&& rm docker-ce-cli_19.03.5~3-0~ubuntu-bionic_amd64.deb \
&& rm containerd.io_1.2.6-3_amd64.deb \
&& rm docker-ce_19.03.5~3-0~ubuntu-bionic_amd64.deb \
# docker-compose needs to be explicitly set to executable as we didn't use an installer for it
&& chmod +x /usr/local/bin/docker-compose \
# add jenkins user to docker group so it can access docker daemon
&& usermod -aG docker jenkins \
# remove utils. We'll keep curl because it's useful to have lying around
&& apt-get remove wget -y
# revert to user jenkins, the container expects to run as this
USER jenkins
Build your image with
docker build -t yourcontainerimagename .
And put the resulting image wherever is convenient - in my case I store it on Docker hub so it's easily accessible regardless where I need it.
What did we do in this Dockerfile? We installed the Docker package in the official Jenkins container image - simple enough. We also installed a specific version of Docker, more on this in a bit.
Start it
When you run the Docker app inside a Docker container, on the surface it might appear that app is doing things trapped in its container, but that's not what is actually happening, and this should always be kept in mind. The Docker instance inside a container is always running the hosts's Docker context - you simply pass the host context down into each container, regardless of nesting depth. To pass it in we use Docker volume. If you run
docker ps -a
on the host or inside any guest container, you will always get the same output, which is a list of containers running on the host. In addition, relative structures like directories are always relative to the host.
With that said, there are three requirements for running Jenkins in Docker
- Your host and guest containers must have identical versions of Docker installed throughout.
- You have to pass the Docker socket in to each container and thus overwrite the container's native Docker socket. When a given container uses to its own Docker socket, it's actually using the host's.
- If you're going to write to any volume-mounted filesystem items passed in from the host, you have to pass in a host user that has permission to write to those items.
You can start your Jenkins container with this compose script
version: "2"
services:
jenkins:
container_name: jenkins
image: yourcontainerimagename
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock/:rw # the host's docker socket
- /opt/jenkins/data:/var/jenkins_home/:rw # a host folder Jenksin can persist in
user: "root:${GID}" # a host user that can write to the persist folder
We pass in a host folder /opt/jenkins/data for Jenkins to store its workspaces etc in. We also pass in root as the user, which is the user that owns this folder. We can pass in any user as long as it also owns the folder Jenkins will write to, in my case it was root.
Use it
Suppose we wanted to use a container in a Jenkins job to do some work, in this case, the official NodeJS container. Our Jenkins job would have some NodeJS files in the job workspace; we want to pass that workspace into our NodeJS container as a volume and then manipulate those files on the filesystem. This would be our Jenkins shell script
docker run \
-u ${UID}:${GID} \
-v /opt/jenkins/data/workspace/"$JOB_NAME"/build:/tmp/build \
node:14-slim \
bash -c "cd /tmp/build && npm install"
We pass Jenkins' user to the Node container so that container has write permission to the job folder. We also pass our Jenkins job workspace to the container as a volume, but that path is relative to the host machine, not to the Jenkins container from which we're running this command. Inside the Jenkins conatainer, workspaces live in /var/jenkins_home/workspace/. but on the host it's /opt/jenkins/data/workspace/..
Reference
I keep my custom dockerized Jenkins container source code on Github.