Building My Odroid-C2 Docker Cloud Part 2 – Building Docker 1.12.0 (with Swarm Mode)


In my previous article, I documented my search for a stable Linux to run my Docker cloud on. The next thing for me to do is to select the version of Docker to use. I am in a dilemma in which I want to use the latest Docker 1.12 with inbuilt swarm mode instead of having to build my own cluster using additional packages such as Consul, Zookeeper, etc. but the version I got using apt-get is version 1.10. There is an update but it is still only 1.11. Since clustering (swarm mode) is in-built in 1.12, there is no point pursuing a dead end in Docker 1.11. Looking around the Internet, I could not find any pre-built Docker 1.12 package for Odroid-C2 anywhere. Reluctant as I was, I had no choice but to build my own Docker 1.12 for my Odroid-C2 from source. As I was new to this, I expected issues and confirmed that Murphy’s Law still rules. This article documents my attempt to build Docker 1.12.0 on my Odroid-C2.


Getting the Source Code

I cloned the Docker repository. The version I got by default was 1.13. I went into so many problems during the build that I thought maybe I should have used a more stable version in 1.12.0.  However,  I found myself encountering exactly the same problems so 1.13 was not the problem. To clone the repository, do this:

git clone #this gets you v1.13.0
cd docker

git checkout v1.12.0 #this gets you v1.12.0


The Docker build script is clever in that it includes all the dependencies in the build in, you guessed it, a docker container and performs the build there. This means that, in order to use the build script, you need to have the package installed. I had docker 1.11 installed on my Odroid-C2.  The command:

sudo make build

builds a container with all dependencies (almost!) for creating the binaries. It took around 1 hour to complete on my Odroid-C2 after I  implemented all the workaround summarised in the “Making a Successful Build” section. And the command:

sudo make binary

builds the binaries for the client and daemon.

Configuring Multi-architecture Hangs

Running “sudo make build” started the build and I saw the build script build and run many intermediate containers until it reached the step where it tried to install the multi-architecture support for armhf (32 bit). Odroid-C2 is aarch64. The build always got stuck there. I left it running for 40 minutes and monitored it using top. I saw that it used no cpu time at all during that 40 minutes. I terminated and restarted the build and got stuck at the same place. I told myself that I needed no 32 bit support so I commented out the RUN command below in Dockerfile.aarch64:

# Install armhf loader to use armv6 binaries on armv8
#RUN dpkg --add-architecture armhf \
#       && apt-get update \
#       && apt-get install -y libc6:armhf

Restarting the build, I got over that problem and encountered another issue…


Running out of Disk Space

I should have seen this one coming. I am using a 8G SD card for the OS on my Odroid-C2. I ran out of space as the build process created a number of containers which varied from several hundred megabytes to over 1 gigabytes in size. I pulled out a used USB hard drive from a drawer, reformatted it, created an ext4 file system and added it to the /etc/fstab file.

Then, I:

  • sudo service docker stop
  • Copied the whole /var/lib/docker directory to the hard driver’s docker directory
  • Deleted the /var/lib/docker directory
  • Created the symbolic link “docker” in /var/lib that point to the docker directory on the USB drive
  • sudo mount -a
  • sudo service docker start


Usb Drive
Usb Drive

This fixed the disk space problem and the build command succeeded in building the container with dependencies to build the binaries. The next step was to build the binaries by running the command:

sudo make binary

Running  out of Memory

The build failed during the compiling phase complaining “out of memory” error. So I created a 2G swap file to fix this issue:

dd if=/dev/zero of/media/usbdrive/swap/swapfile bs=1M count=2000
chmod 600 /media/usbdrive/swap/swapfile
mkswap /media/usbdrive/swap/swapfile
swapon /media/usbdrive/swap/swapfile

I then encountered yet another issue.

Missing Go

The build was restarted and it came to a place where it executed a docker run command with lots of parameters followed by the warning:

#WARNING: I don't seem to be running inside a container...

This is quite unexpected as I thought the build was run inside the container created earlier!!!

Then it failed with make error: 1 when I reached places where it tried to use “go” to continue the build and resulted in “Signal: Killed”. I could not find any logs that could tell me what went wrong. I retried several times with the same result. I thought to myself that since go was involved, let me see what version of go was installed on my Odroid-C2. Maybe it was the wrong version? It turned out that go was not installed. So I installed it:

sudo apt-get update
sudo apt-get install golang

Then I ran the build again and after a tantalising 20 minutes or so, the build was completed successfully.

The build created 1 directory for the client (~/docker/bundles/1.12.0/binary-client ) and 1 for the daemon (~/docker/bundles/1.12.0/binary-daemon ). To test out the newly built binaries, do the following:

sudo -s
services docker stop
cd pathOfDaemonDirectory

From another virtual terminal, execute from pathOfClientDirectory:

./docker info shows "Server Version: 1.12.0".
Docker Info command
Docker Info command

I also ran the busybox httpd image in a container to make sure that it worked. This is the same test I ran in my previous article.

Summary: Making a Successful Build

Here is a summary of what needed to be done to make a successful docker 1.12.0 build on my Odroid-C2 after cloning the docker repository from github:

(Note: You must have installed the package (I had v1.11 installed on may Odroid-C2) before you start the build.)

  1. Comment out the –add-architecture RUN command in the Dockerfile.aarch64 docker file
  2. Add a hard drive and move the /var/lib/docker directory to your hard drive if you are using a 8G SD card
  3. Create a swap partition or swap file on your hard drive
  4. Install go on your Odroid-C2

The you can proceed to run the following commands from your docker directory:

sudo make build
sudo make binary

Now you know how to build Docker 1.12 on your Odroid-C2 from source. In the next instalment, I shall explore the Docker 1.12 built-in swam mode in forming a cluster using my 5 Odroid-C2s. So, stay tuned.

8 thoughts on “Building My Odroid-C2 Docker Cloud Part 2 – Building Docker 1.12.0 (with Swarm Mode)”

  1. Great post!
    I’ve followed your instructions with an aditional problem:
    swap file was not enough to compile docker 1.12.1 ( no armhf )
    cause of lowmemorykiller error:
    I ve solved that going with:

    sudo -s
    sudo echo '9999' > /sys/module/lowmemorykiller/parameters/adj
    sudo echo '1' > /sys/module/lowmemorykiller/parameters/minfree

    Original values were

    sudo echo '0,1,6,12' > /sys/module/lowmemorykiller/parameters/adj
    sudo echo '1536,2048,4096,16384' > /sys/module/lowmemorykiller/parameters/minfree

    1. Thanks Luis. Could you let me know if V1.12.1 has resolved the load balancing and user-defined network issues I described in Part 4 of this series please?

  2. Got Docker 1.10.2… Then tried to build Docker 1.12.1…

    root@odroid64 [09/26/2016 1:32:27] /home/odroid/dev/docker ((v1.12.1) *) # make deb
    docker build -t “docker-dev:HEAD” -f “Dockerfile.aarch64″ .
    Sending build context to Docker daemon 152.6 MB
    Step 1 : FROM aarch64/ubuntu:wily
    —> 18163c30cc69
    Step 2 : RUN apt-get update && apt-get install -y apparmor aufs-tools automake bash-completion btrfs-tools build-essential createrepo curl dpkg-sig g++ gcc git iptables jq libapparmor-dev libc6-dev libcap-dev libltdl-dev libsqlite3-dev libsystemd-dev mercurial net-tools parallel pkg-config python-dev python-mock python-pip python-websocket gccgo –no-install-recommends
    —> Using cache
    —> dbbaf9d6b6c0
    Step 3 : RUN dpkg –add-architecture armhf && apt-get update && apt-get install -y libc6:armhf
    —> Using cache
    —> 3197bb343173
    Step 4 : ENV LVM2_VERSION 2.02.103
    —> Using cache
    —> 4889c37f4736
    Step 5 : RUN mkdir -p /usr/local/lvm2 && curl -fsSL “${LVM2_VERSION}.tgz” | tar -xzC /usr/local/lvm2 –strip-components=1
    —> Using cache
    —> 24d53ad288ae
    Step 6 : RUN set -e && for f in config.guess config.sub; do curl -fsSL -o “/usr/local/lvm2/autoconf/$f” “;a=blob_plain;f=$f;hb=HEAD”; done
    —> Using cache
    —> 4aaffe94e94c
    Step 7 : RUN cd /usr/local/lvm2 && ./configure –build=”$(gcc -print-multiarch)” –enable-static_link && make device-mapper && make install_device-mapper
    —> Using cache
    —> e1875ad5d060
    Step 8 : ENV SECCOMP_VERSION 2.3.1
    —> Using cache
    —> c3e11a4029e5
    Step 9 : RUN set -x && export SECCOMP_PATH=”$(mktemp -d)” && curl -fsSL “${SECCOMP_VERSION}/libseccomp-${SECCOMP_VERSION}.tar.gz” | tar -xzC “$SECCOMP_PATH” –strip-components=1 && ( cd “$SECCOMP_PATH” && ./configure –prefix=/usr/local && make && make install && ldconfig ) && rm -rf “$SECCOMP_PATH”
    —> Using cache
    —> 5961bf224495
    Step 10 : ENV GO_VERSION 1.6.3
    —> Using cache
    —> 5c1d2f12d2dd
    Step 11 : RUN mkdir /usr/src/go && curl -fsSL${GO_VERSION}.src.tar.gz | tar -v -C /usr/src/go -xz –strip-components=1 && cd /usr/src/go/src && GOOS=linux GOARCH=arm64 GOROOT_BOOTSTRAP=”$(go env GOROOT)” ./make.bash
    —> Using cache
    —> 243119835d5c
    Step 12 : ENV PATH /usr/src/go/bin:$PATH
    —> Using cache
    —> 34cdb2821446
    Step 13 : ENV GOPATH /go:/go/src/
    —> Using cache
    —> 77eca4cb7b56
    Step 14 : ENV REGISTRY_COMMIT 47a064d4195a9b56133891bbb13620c3ac83a827
    —> Using cache
    —> 3a64bb7a9bd9
    Step 15 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone “$GOPATH/src/” && (cd “$GOPATH/src/” && git checkout -q “$REGISTRY_COMMIT”) && GOPATH=”$GOPATH/src/$GOPATH” go build -o /usr/local/bin/registry-v2 && rm -rf “$GOPATH”
    —> Using cache
    —> 6eb9d5525998
    Step 16 : ENV NOTARY_VERSION v0.3.0
    —> Using cache
    —> 97ae7bba5b45
    Step 17 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone “$GOPATH/src/” && (cd “$GOPATH/src/” && git checkout -q “$NOTARY_VERSION”) && GOPATH=”$GOPATH/src/$GOPATH” go build -o /usr/local/bin/notary-server && GOPATH=”$GOPATH/src/$GOPATH” go build -o /usr/local/bin/notary && rm -rf “$GOPATH”
    —> Using cache
    —> 202136e2b286
    Step 18 : ENV DOCKER_PY_COMMIT 7befe694bd21e3c54bb1d7825270ea4bd6864c13
    —> Using cache
    —> e2fdb04542a3
    Step 19 : RUN git clone /docker-py && cd /docker-py && git checkout -q $DOCKER_PY_COMMIT && pip install -r test-requirements.txt
    —> Using cache
    —> 5f9b4850ca11
    Step 20 : RUN git config –global ‘’
    —> Using cache
    —> 7b1b7c6fdddc
    Step 21 : RUN groupadd -r docker
    —> Using cache
    —> 91ffc700ceff
    Step 22 : RUN useradd –create-home –gid docker unprivilegeduser
    —> Using cache
    —> a11de9da0ea0
    Step 23 : VOLUME /var/lib/docker
    —> Using cache
    —> d8d8c34b8a14
    Step 24 : WORKDIR /go/src/
    —> Using cache
    —> 75d06772242f
    Step 25 : ENV DOCKER_BUILDTAGS apparmor pkcs11 seccomp selinux
    —> Using cache
    —> b91ac7d4f4df
    Step 26 : RUN ln -sfv $PWD/.bashrc ~/.bashrc
    —> Using cache
    —> df86d7150c58
    Step 27 : RUN ln -sv $PWD/contrib/completion/bash/docker /etc/bash_completion.d/docker
    —> Using cache
    —> 49568cef39cf
    Step 28 : COPY contrib/ /go/src/
    —> Using cache
    —> 9f835cda676c
    Step 29 : RUN ./contrib/ /docker-frozen-images aarch64/buildpack-deps:jessie@sha256:6aa1d6910791b7ac78265fd0798e5abd6cb3f27ae992f6f960f6c303ec9535f2 aarch64/busybox:latest@sha256:b23a6a37cf269dff6e46d2473b6e227afa42b037e6d23435f1d2bc40fc8c2828 aarch64/debian:jessie@sha256:4be74a41a7c70ebe887b634b11ffe516cf4fcd56864a54941e56bb49883c3170 aarch64/hello-world:latest@sha256:65a4a158587b307bb02db4de41b836addb0c35175bdc801367b1ac1ddeb9afda
    —> Using cache
    —> 434cf2c408da
    Step 30 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone –depth 1 -b v1.0.5 “$GOPATH/src/” && git clone –depth 1 -b v1.4 “$GOPATH/src/” && go get -v -d && go build -v -o /usr/local/bin/go-md2man && rm -rf “$GOPATH”
    —> Using cache
    —> dfb1b6da0c33
    Step 31 : ENV TOMLV_COMMIT 9baf8a8a9f2ed20a8e54160840c492f937eeaf9a
    —> Using cache
    —> 2a2c58b50ce7
    Step 32 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone “$GOPATH/src/” && (cd “$GOPATH/src/” && git checkout -q “$TOMLV_COMMIT”) && go build -v -o /usr/local/bin/tomlv && rm -rf “$GOPATH”
    —> Using cache
    —> 74d4cc7beaad
    Step 33 : ENV RUNC_COMMIT cc29e3dded8e27ba8f65738f40d251c885030a28
    —> Using cache
    —> ed5ae4637b54
    Step 34 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone “$GOPATH/src/” && cd “$GOPATH/src/” && git checkout -q “$RUNC_COMMIT” && make static BUILDTAGS=”seccomp apparmor selinux” && cp runc /usr/local/bin/docker-runc && rm -rf “$GOPATH”
    —> Using cache
    —> fce24ab9d746
    Step 35 : ENV CONTAINERD_COMMIT 0ac3cd1be170d180b2baed755e8f0da547ceb267
    —> Using cache
    —> 90ea07bf47dd
    Step 36 : RUN set -x && export GOPATH=”$(mktemp -d)” && git clone “$GOPATH/src/” && cd “$GOPATH/src/” && git checkout -q “$CONTAINERD_COMMIT” && make static && cp bin/containerd /usr/local/bin/docker-containerd && cp bin/containerd-shim /usr/local/bin/docker-containerd-shim && cp bin/ctr /usr/local/bin/docker-containerd-ctr && rm -rf “$GOPATH”
    —> Using cache
    —> ed2dc6cfb4d2
    Step 37 : ENTRYPOINT hack/dind
    —> Using cache
    —> 3167f309543a
    Step 38 : COPY . /go/src/
    —> 795541028fe8
    Removing intermediate container a409bd0c7574
    Successfully built 795541028fe8
    # WARNING! I don’t seem to be running in a Docker container.
    # The result of this command might be an incorrect build, and will not be
    # officially supported.
    # Try this instead: make all

    # GITCOMMIT = 23cf638-unsupported
    # The version you are building is listed as unsupported because
    # there are some files in the git repository that are in an uncommited state.
    # Commit these changes, or add to .gitignore to remove the -unsupported from the version.
    # Here is the current list:
    M Makefile
    bundles/1.12.1 already exists. Removing.

    —> Making bundle: dynbinary (in bundles/1.12.1/dynbinary)
    Building: bundles/1.12.1/dynbinary-client/docker-1.12.1
    Created binary: bundles/1.12.1/dynbinary-client/docker-1.12.1
    Building: bundles/1.12.1/dynbinary-daemon/dockerd-1.12.1
    Created binary: bundles/1.12.1/dynbinary-daemon/dockerd-1.12.1
    Building: bundles/1.12.1/dynbinary-daemon/docker-proxy-1.12.1
    Created binary: bundles/1.12.1/dynbinary-daemon/docker-proxy-1.12.1

    —> Making bundle: build-deb (in bundles/1.12.1/build-deb)
    —> Making bundle: .integration-daemon-start (in bundles/1.12.1/build-deb)
    +++ /etc/init.d/apparmor start
    * Starting AppArmor profiles
    Warning from stdin (line 1): /sbin/apparmor_parser: cannot use or update cache, disable, or force-complain via stdin
    Warning failed to create cache: (null)
    INFO: Waiting for daemon to start…
    +++ exec dockerd –debug –host unix:///go/src/ –storage-driver aufs –pidfile bundles/1.12.1/build-deb/ –userland-proxy=true
    —> Making bundle: .detect-daemon-osarch (in bundles/1.12.1/build-deb)
    #docker build -t docker-manpage-dev -f “man/Dockerfile.aarch64″ ./man
    #docker run \
    # -v /go/src/ \
    # docker-manpage-dev
    find: `contrib/builder/deb/aarch64/*/': No such file or directory
    —> Making bundle: .integration-daemon-stop (in bundles/1.12.1/build-deb)
    +++++ cat bundles/1.12.1/build-deb/
    ++++ kill 5299
    ++++ /etc/init.d/apparmor stop
    * Clearing AppArmor profiles cache
    All profile caches have been cleared, but no profiles have been unloaded.
    Unloading profiles will leave already running processes permanently
    unconfined, which can lead to unexpected situations.

    To set a process to complain mode, use the command line tool
    ‘aa-complain’. To really tear down all profiles, run the init script
    with the ‘teardown’ option.”
    Makefile:86: recipe for target ‘deb’ failed
    make: *** [deb] Error 1

    Any idea?

    I copied all the binaries to the /usr/bin directory…

    odroid@odroid64 [09/26/2016 1:45:28] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo cp dynbinary-daemon/docker
    dockerd dockerd-1.12.1.md5 docker-proxy docker-proxy-1.12.1.md5
    dockerd-1.12.1 dockerd-1.12.1.sha256 docker-proxy-1.12.1 docker-proxy-1.12.1.sha256
    odroid@odroid64 [09/26/2016 1:45:28] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo cp dynbinary-daemon/docker-proxy-1.12.1 /usr/bin/
    odroid@odroid64 [09/26/2016 1:46:03] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo ln -s /usr/bin/dockerd-1.12.1 /usr/bin/dockerd
    odroid@odroid64 [09/26/2016 1:46:03] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo ln -s /usr/bin/dockerd-1.12.1 /usr/bin/dockerd
    odroid@odroid64 [09/26/2016 1:46:03] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo ln -s /usr/bin/docker- /usr/bin/dockerd
    docker-1.12.1 docker-proxy-1.12.1
    odroid@odroid64 [09/26/2016 1:46:03] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo ln -s /usr/bin/docker-proxy-1.12.1 /usr/bin/docker-proxy
    odroid@odroid64 [09/26/2016 1:46:28] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ ls -la /usr/bin/docker*
    lrwxrwxrwx 1 root root 22 Sep 26 01:43 /usr/bin/docker -> /usr/bin/docker-1.12.1
    -rwxr-xr-x 1 root root 14598008 Sep 26 01:43 /usr/bin/docker-1.12.1
    lrwxrwxrwx 1 root root 23 Sep 26 01:45 /usr/bin/dockerd -> /usr/bin/dockerd-1.12.1
    -rwxr-xr-x 1 root root 43251760 Sep 26 01:45 /usr/bin/dockerd-1.12.1
    lrwxrwxrwx 1 root root 28 Sep 26 01:46 /usr/bin/docker-proxy -> /usr/bin/docker-proxy-1.12.1
    -rwxr-xr-x 1 root root 2927968 Sep 26 01:46 /usr/bin/docker-proxy-1.12.1

    But can’t run it…

    odroid@odroid64 [09/26/2016 1:47:21] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ ps aux | grep docker
    odroid 14616 0.0 0.0 5560 644 pts/3 S+ 01:47 0:00 grep docker
    root 24928 4.7 1.9 578396 33580 ? Sl Sep25 10:10 docker daemon
    odroid@odroid64 [09/26/2016 1:47:25] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ kill -7 24928
    -bash: kill: (24928) – Operation not permitted
    odroid@odroid64 [09/26/2016 1:47:33] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo kill -7 24928
    odroid@odroid64 [09/26/2016 1:47:35] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ sudo kill -7 24928
    odroid@odroid64 [09/26/2016 1:47:36] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ docker ps
    Cannot connect to the Docker daemon. Is the docker daemon running on this host?
    odroid@odroid64 [09/26/2016 1:47:38] ~/dev/docker/bundles/1.12.1 ((v1.12.1) *) $ docker daemon
    dockerd: error while loading shared libraries: cannot open shared object file: No such file or directory

    1. Oops… Forgot to do apt-get install -y btrfs-tools libsqlite3-dev libdevmapper-dev

      All good…

      root@odroid64 [09/26/2016 3:32:04] /home/odroid/dev/docker ((v1.12.1) *) # docker version
      Version: 1.12.1
      API version: 1.24
      Go version: go1.6.3
      Git commit: 23cf638-unsupported
      Built: Mon Sep 26 07:19:30 2016
      OS/Arch: linux/arm64

      root@odroid64 [09/26/2016 3:32:07] /home/odroid/dev/docker ((v1.12.1) *) # docker daemon
      INFO[0000] libcontainerd: new containerd process, pid: 4026
      WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=65536
      INFO[0001] [graphdriver] using prior storage driver "aufs"
      ^CINFO[0001] Processing signal 'interrupt'
      INFO[0002] Graph migration to content-addressability took 0.00 seconds
      WARN[0002] Your kernel does not support cgroup blkio weight
      WARN[0002] Your kernel does not support cgroup blkio weight_device
      WARN[0002] mountpoint for pids not found
      INFO[0002] Loading containers: start.
      INFO[0002] Firewalld running: false
      ^CINFO[0002] Processing signal 'interrupt'
      INFO[0002] Default bridge (docker0) is assigned with an IP address Daemon option --bip can be used to set a preferred IP address

      INFO[0003] Loading container: done.
      INFO[0003] Daemon has completed initialization
      INFO[0003] Docker daemon commit=23cf638-unsupported graphdriver=aufs version=1.12.1
      INFO[0003] API listen on /var/run/docker.sock
      INFO[0003] stopping containerd after receiving terminated

      1. Thanks Marcello. I am finding 1.12.0 having lots of issues. I am going to build 1.12.1 to try it out to see if the issues have been fixed. Thanks for the detailed description/log.

        1. Hi Andy,

          Yeah the full details are described at… Took me the entire Sunday to get it… If you will try to build, it has steps from this blog and from others that I tried to document… Please comment on the Github issue so we know if there are other problems… So far, just the manpage script.

          Hope to help others… Really hard to get a similar experience like RPi with HypriotOS…

  3. Thanks for the info demaniak. You just confirmed what I discovered when I tried to build 1.12.1.

Comments are closed.