Docker containers as virtual machines for build environments
This article describes a way to use Docker containers as lightweight virtual machines suitable for building software. Containerizing build environments makes it possible to build with different Linux distributions and to isolate the build environment from the host system, with only a negligible performance penalty.
Containers vs. VMs
Docker containers are not real virtual machines, they are only virtualized on the OS level. This leads to some interesting advantages and disadvantages compared to building with fully virtual machines: (using a tool like Vagrant)
- A Docker container runs directly on the same Linux kernel and CPU as the host system. This means running a different Linux distribution in the container requires it to be compatible with the host system's kernel and CPU. Docker containers can't be used to cross compile for different operating systems or for different CPU architectures, like it's possible with VM's.
- A container isn't configured with a fixed amount of memory and CPU cores from the start like a VM is, it simply takes resources as needed like any other program. This saves a lot of memory compared to running a real VM.
- Files in the container can be shared with the host system with highly efficient bind mounts or volumes. When using a real VM files can't be shared as efficiently, as some kind of network file system or file sync mechanism must be used.
- Docker containers tend to start and exit faster than VM images boot and shutdown.
Dockerfiles, images and containers
When using Docker it's important to understand what some basic terms mean, so let's start with some basic points here:
Docker images are templates used to create containers. An image can't be run, a container can.
A container is in many ways comparable to a virtual machine instance. It's created from an image with some configuration values and has a life of its own. It has files in a private file system. It can be stopped and started multiple times. It can be attached to with a terminal and operated like a separate system.
A Dockerfile is a recipe for defining a new Docker image. A Dockerfile typically specifies what to add to an existing docker image in order to get the new image.
Why this method isn't using a Dockerfile
Since our goal here it so emulate the experience of a virtual machine that will be manually managed and operated, as opposed to a fully automatic build solution, there is no good reason to define a custom image with a Dockerfile. Basing our container directly on one of the generic Linux distribution images like:
ubuntu:20.04 is perfectly adequate.
Most of the container customization and configuration must be scripted anyways.
A note about users in containers
Docker is usually run as root on the host system in order to accomplish the OS level virtualization, and the user inside the container starts out as UID:0, GID:0 (root) as well. This is not as bad as it sounds, because the container can only access mounted volumes and its own files.
It's however not very convenient to have all the generated build artifacts owned by root. That's why, in this method, a user named
builder is created inside the container with the same UID and GID as the user that ran the script on the host system. Files created by
builder in the container will thus appear to be owned by the operating user in the host system.
First, install Docker on the host system:
sudo apt install docker.io
Change directory to an empty folder and create a couple of scripts:
#!/bin/bash CONT_NAME=build_test THIS_DIR=`readlink -f "./"` USER_UID=`id -u` USER_GID=`id -g` IMAGE=ubuntu:20.04 echo "This will instantiate a Docker container named: $CONT_NAME for building with: $IMAGE A user named: builder will be made in the container that mirrors the current user's UID: $USER_UID and GID:$USER_GID Current dir will be mounted as /build inside the container " read -p "Press [Enter] key to proceed" MOUNT1="$THIS_DIR:/build" ENV_OPTS="-e USER_UID=$USER_UID -e USER_GID=$USER_GID" sudo docker run -it --init -v "$MOUNT1" $ENV_OPTS --name $CONT_NAME --entrypoint "/build/bootstrap" $IMAGE
A couple of things to note here:
- The script learns the current user's UID and GID, and passes them along as environment variables to the container so the bootstrap script can setup the
- The current dir is mounted as
/buildin the container. Extra mounts may be added by adding more
--entrypointoption instructs Docker to run a script when the container starts or is attached to.
--initoptions are necessary to make the container's terminal act reasonably close to a regular Linux terminal.
In the same folder, make a script called:
#!/bin/bash if id "builder" &> /dev/null; then echo "Bootstrap was already done" else echo "" echo "Bootstrapping build container" echo "-----------------------------" echo "" echo "Adding user: builder with UID: $USER_UID, GID: $USER_GID" groupadd -g $USER_GID builder adduser --disabled-password --gecos "" --uid $USER_UID --gid $USER_GID builder echo "" echo "Installing basic packets and setting up locale" apt update apt upgrade -y apt install -y locales dialog command-not-found python3 software-properties-common wget locale-gen en_US.UTF-8 fi export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LANGUAGE=en_US.UTF-8 cd /build bash
The bootstrap script starts by testing if the user "builder" has already been created to determine if bootstrapping has already been done.
Generating and setting up the locale environment is necessary to make the container's terminal support international characters and to make for example Python use UTF-8 decoding by default when opening files. Otherwise ASCII would be used as default leading to all sorts of issues.
In the same folder, make one more script named:
#!/bin/bash CONT_NAME=build_test echo "Starting (if necessary) and attaching to container: $CONT_NAME" sudo docker container start $CONT_NAME sudo docker container attach $CONT_NAME
This script is handy in order to restart or reconnect to the container after detaching or stopping it.
Make the scripts executable:
chmod +x container_make container_attach bootstrap
Building Python in a container
With these 3 scripts ready, let's make the container:
As the container comes up, and the bootstrap script finishes, the terminal prompt is now a shell inside the container running as root. Install the dependencies for building:
apt install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev
Change to the intended build user:
Download and build Python:
tar xf Python-3.9.5.tgz
make -j 4
That should work out of the box. Notice the build artifacts are already available in the mounted dir on the host system.
To exit the container, which will stop it, run:
exit or press
To detach from the container and keep it running, press:
To start again or reattach, run:
The container will persist until it's deleted:
sudo docker container rm build_test
How to list all containers running or stopped:
sudo docker ps -a
I hope you enjoyed this content!
Comments powered by Talkyard