Introduction
I have just replaced my personal laptop and realized that my development environment setup requires some improvements. Arch Linux installation, as base system, was relatively straight forward as I had it scripted before but I found that all my development environments is a mess. My ad-hoc projects often require completely unrelated stacks:
- flutter, dart, fastlane,
- python, fastapi, gcloud, terraform,
- nextjs, vercel,
- C++, llvm and friends.
- …
So I gave it a few minutes of thinking and gathered the requirements.
Goals
- Isolated base environment for each development stack that is quick to setup, reproducible and flexible,
- Based on Arch Linux as preference but easy to switch to different distros if required,
- Each development environment inherits my dotfiles configuration from the host (vim with plugins, bash profile, etc.),
UX requirements in details:
- Create a new project environment for python development:
a create --type=python-dev my-super-project - Enter a project environment and fire up tmux:
a shell my-super-project -c "tmux new"
That is it. I can create a project and enter it. Of course the project should persist accross machine reboots.
Solution
Docker: multi-stage builds, images, containers, volumes. That’s right. Probably nothing new for you but I was surprised that there is not much material covering docker based development. We can find tones of CI/CD articles though that helps. As an example I use flutter and python development environments.
Base image
First I need a base. Something that I would always need regardless of the type of development. It might be different for you but for me it is e.g.:
- tmux
- vim
- git
- ripgrep
- fzf
- …
Again, my distro of choice is Arch but you can swap for whatever suits you best.
FROM archlinux:latest as base-dev
ARG USER=pawel
ARG UID=1000
ARG GID=1000
RUN groupadd -g $GID -o $USER && \
useradd -m -u $UID -g $GID -o -s /bin/bash $USER
RUN pacman -Suy --noconfirm && pacman --noconfirm -S \
base-devel \
fzf \
git \
make \
man-db \
openssh \
python \
ranger \
ripgrep \
sudo \
tldr \
tmux \
unzip \
vim \
wget
RUN echo "$USER ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER $USER
Important point above is that I use the same username, user id and group id as my host system. That allows to mount my $HOME from the host. More on this later.
Python image
Now I can create more pythonic environment based on base-dev from above.. python is already installed in base-dev but we need probably more. Here it comes:
FROM base-dev as python-dev
ARG USER=pawel
ENV WORKSPACE="/workspace"
RUN sudo pacman -Suy --noconfirm && sudo pacman --noconfirm -S \
python-black
USER $USER
WORKDIR $WORKSPACE
You can include whatever you need there.
Flutter image
FROM base-dev as flutter-dev
ARG USER=pawel
ENV WORKSPACE="/workspace"
ENV JAVA_VERSION="8"
ENV ANDROID_TOOLS_URL="https://dl.google.com/android/repository/commandlinetools-linux-8092744_latest.zip"
ENV ANDROID_VERSION="31"
ENV ANDROID_BUILD_TOOLS_VERSION="29.0.3"
ENV ANDROID_ARCHITECTURE="x86_64"
ENV ANDROID_SDK_ROOT="/opt/android-sdk"
ENV ANDROID_PREFS_ROOT="$WORKSPACE"
ENV FLUTTER_VERSION="2.10.1"
ENV FLUTTER_URL="https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_$FLUTTER_VERSION-stable.tar.xz"
ENV FLUTTER_HOME="/opt/flutter"
ENV PATH="$ANDROID_SDK_ROOT/cmdline-tools/tools/bin:$ANDROID_SDK_ROOT/platform-tools:$ANDROID_SDK_ROOT/platforms:$FLUTTER_HOME/bin/cache/dart-sdk/bin:$FLUTTER_HOME/bin:$PATH"
RUN sudo pacman -Suy --noconfirm && sudo pacman --noconfirm -S \
jdk8-openjdk
USER $USER
WORKDIR $WORKSPACE
# android sdk
RUN sudo chown -R $USER $WORKSPACE && \
sudo mkdir -p $ANDROID_SDK_ROOT && sudo chown -R $USER $ANDROID_SDK_ROOT && \
mkdir -p $ANDROID_PREFS_ROOT/.android && \
touch $ANDROID_PREFS_ROOT/.android/repositories.cfg && \
curl -o /tmp/android_tools.zip $ANDROID_TOOLS_URL && \
unzip -qq -d "$ANDROID_SDK_ROOT" /tmp/android_tools.zip && \
rm /tmp/android_tools.zip && \
mkdir -p $ANDROID_SDK_ROOT/cmdline-tools/tools && \
mv $ANDROID_SDK_ROOT/cmdline-tools/bin $ANDROID_SDK_ROOT/cmdline-tools/tools && \
mv $ANDROID_SDK_ROOT/cmdline-tools/lib $ANDROID_SDK_ROOT/cmdline-tools/tools && \
yes "y" | sdkmanager "build-tools;$ANDROID_BUILD_TOOLS_VERSION" && \
yes "y" | sdkmanager "platforms;android-$ANDROID_VERSION" && \
yes "y" | sdkmanager "platform-tools" && \
yes "y" | sdkmanager "cmdline-tools;latest"
#&& yes "y" | sdkmanager "emulator" \
#&& yes "y" | sdkmanager "system-images;android-$ANDROID_VERSION;google_apis_playstore;$ANDROID_ARCHITECTURE" \
# flutter
RUN curl -o /tmp/flutter.tar.xz $FLUTTER_URL && \
sudo mkdir -p $FLUTTER_HOME && sudo chown -R $USER $FLUTTER_HOME && \
tar xf /tmp/flutter.tar.xz --strip-components 1 -C $FLUTTER_HOME && \
rm /tmp/flutter.tar.xz && \
flutter config --no-analytics && \
flutter precache && \
yes "y" | flutter doctor --android-licenses && \
flutter doctor
Full example is here: https://github.com/kurdybacha/dotfiles/blob/master/bin/Dockerfile
Tooling
With a few lines on python we can create a tool that meets UX requirements for the Goals section above.
It mounts $HOME as home dir inside so all host’s dotfiles are inherited. Working directory is set to \workspace and is backed up with docker volume so it can persist container removal e.g. in case I want to upgrade and rebuild it later.
#!/usr/bin/python
import argparse
import subprocess
import sys
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="command")
build = subparsers.add_parser("create", help="Creates a new dev cotainer")
build.add_argument(
"--type",
type=str,
choices=["flutter-dev", "python-dev"],
help="Type of a container",
required=True,
)
build.add_argument("name", help="Name of a container", nargs="?")
shell = subparsers.add_parser("shell", help="Shells into a dev container")
shell.add_argument("name", help="Name of the container")
shell.add_argument("-c", help="Command to run")
args = parser.parse_args()
if args.command == "create":
p = subprocess.run(
f"docker container ls -f name=^{args.name}$ -q", shell=True, capture_output=True
)
if p.stdout:
sys.exit(f'Container "{args.name}" already exists')
p = subprocess.run(
f"docker build --target {args.type} -t {args.type} "
f"--build-arg UID=$(id -u) --build-arg GID=$(id -g) "
f"-f $HOME/.bin/Dockerfile .",
shell=True,
)
if p.returncode == 0:
subprocess.run(f"docker volume create {args.name}", shell=True)
subprocess.run(
f"docker run -id -h {args.name} "
f"-v $HOME:$HOME -v {args.name}:/workspace "
f"--privileged --device=/dev/bus -v /dev/bus/usb:/dev/bus/usb "
f"--name {args.name} {args.type}",
shell=True,
)
elif args.command == "shell":
p = subprocess.run(
f"docker ps -f name=^{args.name}$ -q", shell=True, capture_output=True
)
if not p.stdout:
p = subprocess.run(f"docker start {args.name}", shell=True)
if p.returncode == 0:
subprocess.run(
f"docker exec -e HISTFILE=/workspace/.bash_history "
f"-it {args.name} /bin/bash "
f'-c "{args.c}"' if args.c else "",
shell=True,
)
Full code is here: https://github.com/kurdybacha/dotfiles/blob/master/bin/a
Working example
Let’s create a flutter project:
host$ a create --type=flutter-dev my-app
...
Successfully built 4621b27ed8ef
Successfully tagged flutter-dev:latest
my-app
658c0ef50dbebf378e058bf5cf087b172260918ee925e7e76dd9fd6723f6bb32
host$ a shell my-app -c /bin/bash
my-app:/workspace$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.10.1, on Arch Linux 5.16.9-arch1-1, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[!] Android Studio (not installed)
[!] Connected device
! No devices available
[✓] HTTP Host Availability
! Doctor found issues in 3 categories.
my-app:/workspace$
Here we are inside the docker image with all ready to start hacking.