Docker Cluster In VuShell-small

Running Docker Swarm Mode on My Facelifted ODROID-C2 Cluster

Recap

Around 6 months ago, I was experimenting with Docker swarm mode on my ODROID-C2 cluster consisting of 5 ODROID-C2s. Docker was running fine on a single machine using ‘docker run’ command but neither overlay network, routing mesh nor load balancing was working in the swarm mode. I tried using different versions of docker (1.12.x and 1.13.x) compiled on my ODROID-C2 to no avail as documented in a previous article. I also tried running Kubernetes on the ODROID-C2 cluster. Again I couldn’t get the networking part work. I suspected that the kernel was missing certain modules needed for Docker/Kubernetes networking. As the OSes at that time I tested were not stable enough, I stopped my experimentation on ARM-based machines.  Instead I played with Openshift running on INTEL 64 bit machines as INTEL always has better support than ARM in any open source project. What rekindled my passion to get Docker swarm mode working is the hardware I have lying around not being used: an ODROID VU7 multi-touch screen and a VuShell for VU7. I wanted to make use of them to do some useful work.

Facelift

I assembled the VU7 screen and an ODROID-C1+ on the VuShell enclosure. Then I thought to myself, why not put my ODROID-C2 cluster there as well? So I did. You can see the screen displaying a soft keyboard in the photo below. All ODROID single board computers together with an 8-port gigabit switch and a SSD are also put inside the VuShell enclosure. The ODROID cardboard box houses the power supply. The tiny router uses WDS (Wireless Distribution System) to connect to my main router to provide ethernet Internet access for all the ODROIDs housed in the VuSHell as they don’t have inbuilt wifi.

Here is the updated bill of materials for my Docker cluster:

  • 5 X ODROID-C2s
  • 1 X ODROID-C1+ for monitoring the cluster eg, running the Docker Swarm Visualizer (more on the Visualizer later)
  • 1 X D-Link DGS-1008A 8 Port Gigabit Desktop Switch
  • 1 X 5V 15A switching power supply
  • 1 X SSD Disk Drive
  • 1 X USB3.0 to SATA Bridge Board from Hardkernel (I wish ODROID-C2 had USB3)
  • 7 X ethernet cable
  • 1 X 1 to 8 power splitter cable
  • 1 X ODROID VU7 multi-touch screen
  • 1 X VuShell for ODROID
  • 1 X XiaoMi Mi WiFi Router Youth Edition
Docker Cluster In VuShell-small
Docker Cluster In VuShell-small

Replacing Armbian Xenial Server OS

Since I have the suspicion that the cause of docker swarm mode not working was due to some missing/incompatible kernel modules in the OS, I decided to switch to another OS. I noticed that Hardkernel recently released Ubuntu 16.04 (v2.3) for the ODROID-C2 so I thought I’d give it a try. The version of Hardkernel’s Ubuntu OS I tried months earlier was unstable but the current version worked without any of those issues I experienced before. I was happy and told myself that this time it might work;-)

To facilitate ease of operation, I installed and configured the following packages:

  • parallel-ssh on the docker manager to allow me to issue commands from the docker manager to all nodes only once
  • generate ssh keys for users: odroid and root on all members of the cluster such that they can ssh into each other without password
  • nfs-kernel-server on the manager and nfs-common on all nodes
  • curl on the manager for testing
  • dnsutils on all nodes

Docker Swarm Mode Reboot

I installed docker.io using apt-get and did a quick ‘docker run’ test using my httpd image. It worked. I wanted to try out the swarm mode next to see if it will work with the new OS. Here is a screenshot of the versions of software being used. It is interesting to note that the Hardkernel Ubuntu came with zram pre-installed for swap.

Software Version
Software Version

Creating a Swarm

Their host names and static IP addresses are as follows:

  • c2-swarm-00 – 192.168.1.100 (manager)
  • c2-swarm-01 – 192.168.1.101 (node 1)
  • c2-swarm-02 – 192.168.1.102 (node 2)
  • c2-swarm-03 – 192.168.1.103 (node 3)
  • c2-swarm-04 – 192.168.1.104 (node 4)

Only c2-swarm-00 has a SSD drive connected.

To bring up swarm mode on my cluster, issue the following command on the manager:

docker swarm init --advertise-addr 192.168.1.100

which returns

Swarm initialized: current node (8jw6y313hmt3vfa1fme1dinro) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-2gvqzfx48uw8zcokwl5033iwdel2rl9n96lc0wj1qso7lrztub-aokks5xcm5v7c4usmeswsgg1k \
192.168.1.100:2377

To make the other nodes join the cluster, issue the ‘docker swarm join’ command above on each node. This can be done using the parallel-ssh to issue the command once from the manager and  executed on each node.

Here is a screenshot of running ‘docker ps’ command using parallel-ssh.

parallel-ssh
parallel-ssh

Now we have a docker swarm running.

One annoyance of the Docker swarm I found is that after you shut all nodes down, when you power them on again, all nodes will be “Active” but “Down” when you use the ‘docker node ls’ to find out the status of your nodes. Since the nodes are down, all services will be running on the manager. The fix is to run ‘docker ps’ on every node. This will change their status from ‘Down’ to ‘Ready’ and everything is fine again. parallel-ssh is a convenient tool to do this since all you have to do is issue the ‘docker ps’ command once from your manager.

Running Docker Swarm Visualizer and HTTPD Services

To help visualize what is going on in the swarm, I built the image based on Docker Samples on Github. I’ve pushed it to docker hub so that anyone can use it. The image’s name is: mrdreambot/arm64-docker-swarm-visualizer. I then deployed it as a service by issuing the following command from the manager:

docker service create \

--name=dsv \

--publish=8080:8080/tcp \

--constraint=node.role==manager \

--mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \

mrdreambot/arm64-docker-swarm-visualizer

I then Pointed the browser at the master:

http://192.168.1.100:8080

and observed the changes reported by the visualizer when deploying the httpd service:

docker service create --replicas 3 --network home-net --name httpd -p 80:80 mrdreambot/arm64-busybox-httpd

The command line output of listing the services is shown below:

Services
Services

Here is a Docker Swarm Visualizer screenshot showing the nodes on which the service replicas are run.

Docker Swarm Visualizer
Docker Swarm Visualizer

Routing Mesh, Load Balancing and Self-healing

The routing mesh in the swarm allows a request to reach a service although the service is not running on the node where the request has been received. This means that although the httpd service is run on c2-swarm-00,  c2-swarm-03 and c2-swarm-04, one can point the browser at any one of the 5 nodes and still gets a response with the ODROID-Docker graphics. And that was the behaviour observed.

In addition to providing a routing mesh, swarm also performs load balancing. To test the load balancing feature, I hit the manager multiple time using my browser:

http://192.168.1.100/cgi-bin/lbtest

Notice the hostnames (container Id) and IP addresses are different in the following screenshots.

Load Balancing - Httpd first call
Load Balancing – Httpd first call
Load Balancing - Httpd second call
Load Balancing – Httpd second call

 

The tests were repeated using the curl command:

curl http://192.168.1.100/cgi-bin/lbtest

Here is a screenshot on the output of curl commands:

Load Balancing - curl output
Load Balancing – curl output

As for a demo on self-healing, I shut down c2-swarm-04 and you can see from the visualizer as well as the command line that another httpd container was spun up on c2-swarm-02 to replace the one on c2-swarm-04. This is because when we started the service, we specified replica=3 and the Docker swarm will maintain the desired number of replicas.

Service Recovery

Service Recovery - Visualizer
Service Recovery – Visualizer

I then shut down the rest of the nodes and left only the manager running and here is what the Visualizer showed:

Service Recovery - all running on manager
Service Recovery – all running on manager

All worked as expected!

Conclusion

The new OS really made a difference. The Docker swarm mode is now working on my ODROID-C2 cluster as expected, at last. In the next instalment, I shall upgrade docker to 1.13.x to experiment with the new docker stack deployment feature. A stack is a collection of services that make up an application. It automatically deploys multiple services that are linked to each other obviating the need to define each one separately. In other words, docker-compose in swarm mode. Stay tuned!