Golang and Docker Multi-Stage Build

I have noticed a common pattern amonst some new utilities such as kubectl, kops and terraform: There’s only 1 single executable file to install, and by ‘install’ it can be put anywhere as long as it’s in $PATH. This was before I learned some Golang but it’s easy to find out that the reason behind this pattern is that they are all written in Go.

And in the containers’ realm, the new-ish multi-stage build steps of Docker released in 2017 are super beneficial to Golang containers. A TL;DR example looks like:

  1. use a 1GB Debian container with all Golang tools and build dependencies to build the Golang executable( FROM ... AS in the sample ).
  2. put the executable into a tiny run-time container such as Alpine Linux, resulting in a < 20MB container image(depending on the size of the app obviously) ( COPY --FROM in the sample )

A multi-stage ‘hello world’ Dockerfile looks like:

FROM golang:1.12.5-alpine3.9 as builder
ENV GO111MODULE=on
RUN apk update --no-cache && \
apk add git
WORKDIR /app
ADD ./ /app
RUN go build -o golang-test .

FROM alpine:3.9.4
WORKDIR /app
RUN addgroup -g 2000 golang && \
adduser -D -u 2000 -G golang golang
USER golang
COPY --from=builder /app/golang-test .
CMD ["/app/golang-test"]
EXPOSE 8000

Note: To be able to use the multi-stage feature, the Docker version has to be > 17.06.

🙂

Install Nvidia 396 Driver for Ubuntu 18.04

Just some quick notes to upgrade Nvidia driver from 390 to 396 on Ubuntu 18.04. By default Ubuntu 18.04 only has Nvidia driver 390.

# add ubuntu official ppa for nvidia
# all commands below run as root
add-apt-repository ppa:graphics-drivers/ppa
apt update
# install 396 files
apt install nvidia-dkms-396 nvidia-driver-396 nvidia-compute-utils-396 libnvidia-compute-396
# reboot at the end.

After reboot, the new driver can be verified by:

nvidia-smi 
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.54 Driver Version: 396.54 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 Off | N/A |
| 44% 67C P2 150W / 230W | 3211MiB / 8117MiB | 100% Default |
+-------------------------------+----------------------+----------------------+

🙂

Home VPN with OpenVPN

Here are step to run a simple OpenVPN service at home, so that I can access home network easily while not at home.

First, clone the git repo for OpenVPN docker container:

git clone https://github.com/kylemanna/docker-openvpn.git

I can use the pre-built docker image from docker hub but it has just been breached so I’d rather build it myself:

cd docker-openvpn && docker build -t openvpn .

Create a docker volume to persist data if the OpenVPN container to be rebuilt:

export $OVPN_DATA=ovpn_data
docker volume create --name $OVPN_DATA

Generate OpenVPN configurations, if there’s no DNS record for the server, use the public IP of the home broadband alternatively.

docker run -v $OVPN_DATA:/etc/openvpn --log-driver=none --rm openvpn ovpn_genconfig -u udp://VPN.SERVERNAME.COM

Build a new secret key which will be used to generate user keys. I’d advise to use a strong password which can be saved in a password manager or vault. This is needed everytime when I create a new user.

docker run -v $OVPN_DATA:/etc/openvpn --log-driver=none --rm -it openvpn ovpn_initpki

Then the OpenVPN server container can be run as a service:

docker run -v $OVPN_DATA:/etc/openvpn -d -p 1194:1194/udp --cap-add=NET_ADMIN openvpn

Generate the first user profile. The password for secret key will be needed. Then retrieve the OpenVPN configuration with the 2nd command.

docker run -v $OVPN_DATA:/etc/openvpn --log-driver=none --rm -it openvpn easyrsa build-client-full <username> nopass
docker run -v $OVPN_DATA:/etc/openvpn --log-driver=none --rm openvpn ovpn_getclient <username> > <username>.ovpn

This .ovpn file can be used to configure OpenVPN client softwares on laptops or phones.

At last, ensure UDP 1194 port is forwarded to the host of the docker container. This is usually done in the home broadband router.

Ansible, CloudFormation and Jinja2

CloudFormation is the corner stone to provision infrastructure in AWS with code, however it’s not very DRY, ie. poor modularization, almost static variables and templates. So here comes Ansible.

However at the moment Ansible’s CloudFormation module doesn’t support Jinja2 in templates, like other modules do. Luckily there’s a work-around to get the Ansible-CloudFormation-Jinja2 trio working together.

A simple CloudFormation snippet with Jinja2 variable:

# roles/test-stack/templates/mycf.yaml.j2
...
Parameters:
VpcId:
Type: String
Default: '{{ template_params.vpc_id }}'
...

I don’t put Jinja2 variable directly to replace the !Ref intrinsic function, because I think this is a softer approach so if there’s any parameter verification in the template it still works.

Then the template can be loaded to Ansible’s cloudformation module like this:

# roles/test-stack/tasks/main.yaml
- name: create a stack
cloudformation:
stack_name: '{{ stack_names.test_stack }}'
state: present
template_body: "{{ lookup('template', 'templates/mycf.yaml.j2') }}"
tags: '{{ template_tags }}'
...

The inventory may look like this:

# inventory/local
all:
hosts:
dev:
ansible_connection: local
gather_facts: false

All parameters can go into the global variable file group_vars/all or host variable file host_vars/dev so later I can create a set of parameters for production.

# group_vars/all
stack_names:
test_stack: test1
...
# host_vars/dev
template_params:
vpc_id: vpc-xxxxxxxxxxxxxxx

template_tags:
app_env: dev

And the playbook can be simple as:

# deploy.yaml
- name: deploy stacks
hosts: dev
tags: dev
roles:
- test-stack

Finally, this CloudFormation stack can be deployed by running:

$ ansible-playbook -i inventory/local deploy.yaml --tags dev

🙂