Fish And Chips Club

Building My Odroid-C2 Docker Cloud Part 4 – Application Deployment

Web Application with Database Backend

So far, I’ve only deployed trivial application on my docker cluster. In this article, I want to explore the level of difficulty in deploying a more realistic application. Being a lazy person who does not want to create a web application from scratch, I searched the Internet for a ready-made web application using a database backend as this kind of application is commonly deployed in the production environments. I came across a WEB4J sample application called the “Fish and Chips Club” which should do the job. From now on, I am going to refer to this application as “Fish”. This application includes features to:

  • edit club members
  • edit local restaurants
  • edit ratings of each restaurant
  • add new lunches (a given restaurant on a given day)
  • RSVP for each upcoming lunch
  • interact using a simple discussion board
  • produce simple reports
  • provide a simple search page

And it uses 3 databases running on MySQL. You can find out more about how to configure this application here.

The disadvantage of using ARM64 architecture machines like ODROID-C2 is that you don’t have that many docker images readily available to you on Docker Hub to choose from like INTEL-based machines. I can find only 1 MySQL docker image and no Apache Tomcat images at all! And this application requires Tomcat to run. However, this is not a showstopper as I can always create my own docker image!

In the following sections, I am going to deploy MySQL and Fish in different ways and assess their pros and cons. It will be a great learning experience for me as, like most of you, I am new to this too ;-)

Deploying MySQL Using Docker Run

The only MySQL docker image I found is bobsense/mysql:

Running the image in a container and populating the database is quite straight forward. Follow the procedure below:

docker run -d -p 3306:3306 \
--name mysql \
-e MYSQL_USER=fishuser \
-e MYSQL_PASSWORD=fish456 \
-e MYSQL_DATABASE=fish \
-v /media/usbdrive/fish-mysql2:/u01/my3306/data \
bobsense/mysql

Once the MySQL container is up and running, I use the MySQL client to connect to it and populate the database using the database SQL file provided by Fish in the WEB-INF/datastore/mysql directory. Change to that directory and issue:

mysql -u fishuser -p -P 32768 -h 127.0.0.1 < CreateALL.SQL

Note that you have to use 127.0.0.1 instead of localhost otherwise it will try to use the mysql.sock for communication with the server and not TCP and fail. I can even access the database remotely from another machine, see screenshot below where you can see the 3 databases created for fish (fish, fish_access and fish_translation):

MySQL from another Machine
MySQL from another Machine

Problem occurs only after I have created and populated the database. I stopped and removed the container and re-deployed it the next day. It just exited whenever I started it. It appears that there is an issue with the run.sh script. I replaced it with my own modified version. The following only shows the part I changed and not the complete file.

if [ -d $DATA_DIR ]; then
 /u01/my3306/bin/mysqld_safe --defaults-file=/etc/my.cnf --basedir=/u01/my3306 --datadir=/u01/my3306/data
 exit
fi

changed to

if [ -d $DATA_DIR ]; then
 /u01/my3306/bin/mysqld_safe --defaults-file=/etc/my.cnf --basedir=/u01/my3306 --datadir=/u01/my3306/data
 while [ 1 ]
 do
    sleep 1d
 done
fi

Then I use the “-v option” in the command to make it work:

docker run -d -p 3306:3306 \
--name mysql \
-e MYSQL_USER=fishuser \
-e MYSQL_PASSWORD=fish456 \
-e MYSQL_DATABASE=fish \
-v /media/usbdrive/fish-mysql2:/u01/my3306/data \
-v /media/usbdrive/sw/run.sh:/root/run.sh \
bobsense/mysql

Running Fish in Tomcat (Not Containerized)

Fish requires MySQL database configuration as documented here to make it work. I installed Apache Tomcat version V8.5.4 and Fish V4.10.0.0 on my c2-swarm-00 (Swarm Manager) machine with a hard disk. After a few trials, I was able to get the Fish application to work by setting up the correct parameters in fish.xml, the context element that represents the Fish application. I put this in Tomcat’s conf/Catalina/localhost directory which is one way of deploying the Fish application on Tomcat and this is the method suggested by WEB4J for Fish.

<!-- Example of a Tomcat configuration file for this application. -->

<!-- The 'YOUR_MYSQL_...' items refer to a MySQL account. The given values are phony. Please change them. -->

<!-- Assumes MySQL is running on the default port 3306. Please change if necessary. -->

<!-- Maps a URL context under Tomcat to a location on your file system. Please change 'docBase': -->
<Context docBase="/fish" reloadable="true">

 <!-- How Tomcat should make Datasources, for the exclusive use of the application: -->
 <Resource 
 name="jdbc/fish" 
 auth="Container" 
 type="javax.sql.DataSource" 
 username="fishuser"
 password="fish456"
 driverClassName="com.mysql.jdbc.Driver"
 url="jdbc:mysql://SERVER:PORT/fish?useServerPrepStmts=false"
 maxTotal="10"
 maxIdle="5"
 />
 <Resource 
 name="jdbc/fish_translation" 
 auth="Container" 
 type="javax.sql.DataSource" 
 username="fishuser"
 password="fish456"
 driverClassName="com.mysql.jdbc.Driver"
 url="jdbc:mysql://SERVER:PORT/fish_translation"
 maxTotal="10"
 maxIdle="5"
 />
 <Resource 
 name="jdbc/fish_access" 
 auth="Container" 
 type="javax.sql.DataSource" 
 username="fishuser"
 password="fish456"
 driverClassName="com.mysql.jdbc.Driver"
 url="jdbc:mysql://SERVER:PORT/fish_access"
 maxTotal="10"
 maxIdle="5"
 />


 <!-- How Tomcat should look for user names and passwords, when confirming end user credentials during login: -->
 <Realm 
 className="org.apache.catalina.realm.JDBCRealm" 
 connectionURL="jdbc:mysql://SERVER:PORT/fish_access"
 driverName="com.mysql.jdbc.Driver"
 roleNameCol="Role"
 userCredCol="Password" 
 userNameCol="Name" 
 userRoleTable="UserRole" 
 userTable="Users" 
 connectionName="fishuser"
 connectionPassword="fish456"

 />
</Context>

Replace SERVER:PORT with the server name/IP address and port number of your mysql server.

The Fish application is interacting with the MySQL database running in a docker container exposing the standard MySQL port 3306. And it works. Now that I know what parameters Ito configure to make Fish work, I am ready to create a docker image for it.

Creating A Docker Image for Fish

The next thing I wanted to do was to dockerized Fish ie, create a Fish docker image. The images contains Tomcat and the Fish web application. I built my Fish image based on the docker image “msimoes/arm64-java” and the Tomcat/Fish setup described in the previous section.

Here is the Dockerfile:

FROM msimoes/arm64-java
MAINTAINER Andy Yuen
COPY apache-tomcat /tomcat
COPY fish /fish
COPY run.sh /fish/
EXPOSE 8080
CMD sh /fish/run.sh

And the run.sh script:

#!/bin/bash
MYSQL_SERVER=${MYSQL_SERVER:-"mysql"}
MYSQL_PORT=${MYSQL_PORT:-"3306"}
FISHXML=/tomcat/conf/Catalina/localhost/fish.xml
echo set server
sed -i "s/SERVER/${MYSQL_SERVER}/g" ${FISHXML}
sed -i "s/PORT/${MYSQL_PORT}/g" ${FISHXML}
#Output log
echo "======================================================="
echo "You have the MySQL configuration as follows:"
echo "MYSQL_SERVER : \"${MYSQL_SERVER}\"" 
echo "MYSQL_PORT : \"${MYSQL_PORT}\""
echo "">
/tomcat/bin/catalina.sh run

I built the image using the command:

docker build -t mrdreambot/fish:latest .

You can see the build in progress in the following screen shot:

Docker Build in Progress
Docker Build in Progress

And the image is available after the build:

Listing fish Image
Listing fish Image

There are a couple of environment variables to tell Fish about the MySQL Server (IP address) and the MySQL port to use. These environment variables are:

MYSQL_SERVER – name/IP address of MySQL server

MYSQL_PORT – port to access MySQL

Although it is not that difficult to build a docker image, it poses a maintenance issue for production: this is a manual process. Imagine you have 100’s of applications to maintain! This is where product like Openshift excels. You can think of Openshift as Docker + Kubernetes + additional enterprise grade tools. Openshift provides change triggers that one can set up in its buildconfig to trigger an automatic build if the image it relies on changes or the source code of the application changes (using webhooks). Of course, you can also fix the image version or source code version and vary the other. You can automate your build by writing some automation scripts without using Openshift but that introduces yet more things to maintain.

Deploying the Fish Image using Docker Run

To run the image in a container, I used the following docker command (note the environment variables I passed to the container):

docker run -d -p 8080:8080 \
--name fish \
-e MYSQL_SERVER=192.168.1.100 \
-e MYSQL_PORT=3306 \
mrdreambot/fish

And it works.

Fish Container Running

Here is a screen shot of the Fish application:

Fish And Chips Club
Fish And Chips Club

Scaling is a manual process by running containers using the classical docker run command. Running multiple containers is possible provided each one uses a different port.

Deploying MySQL As A Swarm Mode Service

In previous sections, I deployed both MySQL and Fish on C2-swarm-00 using Docker run ie, not using Swarm Mode. In this section, I am going to deploy them as services in Swarm Mode.

Please note that only 1 instance of MySQL mapped to the database directory on disk can be running at a time. However, I can scale up the number of replicas of Fish to run though. Remember that only c2-swarm-00 has a hard disk? I am going to run MySQL as a service with only 1 instance and set up a constraint that the service can only be run on the host c2-swarm-00 using the following command:

docker service create  \
--name mysql \
-p 3306:3306 \
-e MYSQL_USER=fishuser \
-e MYSQL_PASSWORD=fish456 \
-e MYSQL_DATABASE=fish \
--constraint 'node.hostname == c2-swarm-00' \
--mount type=bind,src=/media/usbdrive/fish-mysql2,dst=/u01/my3306/data \
--mount type=bind,src=/media/usbdrive/sw/mysqlroot,dst=/root \
bobsense/mysql

This is very similar to running MySQL using the classical docker run command. The main difference is that if the MySQL service should fall over, docker swarm will automatically start another instance to replace it.

Deploying Fish As A Swarm Mode Service

The deployment is very similar to that of using docker run earlier but by using a service, I can scale Fish ie, running multiple replicas spreading across the 5 ODROID-C2s in my docker swarm cluster easily. Just issue the command:

docker service create \
-p 8080:8080 \
--name fish \
-e MYSQL_SERVER=192.168.1.100 \
-e MYSQL_PORT=3306 \
mrdreambot/fish

You can see that the services are running using the following command:

docker service ls
Services Running
Services Running

Swarm mode is supposed to perform load balancing meaning that if you invoke a service on the swarm node that is not running the service, it will redirect you to the service. This means that if you are running the service on 3 nodes, you can actually invoke the service in any of the nodes in the swarm cluster and not only on the nodes running the service. That was not the behaviour I witnessed using swarm mode. I could not access the service on the nodes not running it. And after I scaled up and down the number of replicas of the service, sometimes I could not even invoke the service on the node running the service! I searched the Internet and many people reported the same issue. The problem was thought to be swarm mode not updating the IPVS (IP Virtual Server) tables correctly. IPVS is the kernel module responsible for load balancing. Some users even reported that the problem is still not resolved in Docker 1.12.1.

 

Deploying Services in a User-defined Network

The next thing to try is to create a user-defined (overlay) network for the mysql and fish services. Using this approach, the fish application can access the MySQL database by name instead of using a hard-coded IP address. To deploy mysql and fish services, I first removed the existing services created in previous sections and issued the following commands:

docker network create --driver overlay fish-net

docker service create --network fish-net \
--name mysql \
-p 3306:3306 \
-e MYSQL_USER=fishuser \
-e MYSQL_PASSWORD=fish456 \
-e MYSQL_DATABASE=fish \
--constraint 'node.hostname == c2-swarm-00' \
--mount type=bind,src=/media/usbdrive/fish-mysql2,dst=/u01/my3306/data \
--mount type=bind,src=/media/usbdrive/sw/mysqlroot,dst=/root \
bobsense/mysql

docker service create --network fish-net \
-p 8080:8080 \
--name fish \
-e MYSQL_SERVER=mysql \
-e MYSQL_PORT=3306 \
mrdreambot/fish

After creating the network, one can see an overlay network (fish-net) has been setup up for the swarm:

Overlay Network fish-net
Overlay Network fish-net

However, fish did not work. It showed the login screen but was not able to authenticate after I entered the user name and password repeatedly. It did not seem to be able to talk to mysql! I used “docker exec” to get into fish and installed curl. The curl http://fish:8080 command returned the tomcat page so the name fish was resolved. See screen shot below:

curl fish From Fish Container
curl fish From Fish Container

I then used “docker run” to get into mysql and issued “curl http://fish:8080” but it informed me that it could find no route to fish!

curl mysql From Fish Container
curl mysql From Fish Container

 

So I decided to create another service which is just a busybox container in the same user-defined network: fish-net.

docker service create --network fish-net \
--name busybox \
arm64el/busybox-arm64el \
sleep 1d

 

I also identified the IP addresses of the containers running mysql and fish using “docker service inspect” and the docker Ids using “docker ps”. Then I used “docker exec” to get into the busybox container to diagnose the user-defined network problem. Here is what I see:

busybox can ping  its own IP identified in the docker service inspect command but it can’t ping either the mysql or fish IP address::

Pinging Busybox
Pinging Busybox

 

nslookup also failed on the service names:

nslookup
nslookup

Now I am convinced that User-defined Network is not working in the Docker 1.12.0 that I built from source. Does it work on INTEL architecture? I don’t know. Please let me know if you have tried it.

What We Learned

We learned that:

  • There are multiple ways that one can deploy docker containers: docker run, docker service create
  • Using either deployment approach, one can deploy multiple instances of the Fish application although using docker service create is much easier
  • You may not find docker images you are looking for in docker hub if you are using ARM64 architecture machines. There are very few images available for ARM64 on docker hub.
  • building your own docker image using a Dockerfile is not that difficult for the Fish application but on-going maintenance may be an issue.
  • Swarm mode load balancing is not working, at least not working reliably.
  • User-defined overlay network is not working. Containers in the network cannot access each other by name for some reason. It could be just my version of Docker 1.12.0 that I built from source. I don’t know for sure!
  • Based on the findings of my exploration on swarm mode in 1.12.0 that I built from source, I don’t think I would use it in production. The emphasis is the version that I built from source as the problems may be caused by  ARM64 peculiarities and the underlying kernel modules that my ODROID-C2s are running on. If you have used 1.12.0 on INTEL architecture and did not encounter the problems I encountered, please leave a comment to let me know.

What next?

It is a bit disappointing discovering that some of the main features (auto load balancing and user-defined overlay network) in docker swarm mode are not working for me. Other aspect sI’d like to explore are microservices, green/blue deployment and rolling updates. I’d also like to compare pros and cons of using Docker Swarm Mode vs Kubernetes/Openshift for orchestration and running production workloads. I have no idea the effort required to achieve these goals at the moment. But I shall press ahead whenever I find time. Until then, may your journey in exploring docker swarm mode turn out as fruitful as mine! It has been an incredible learning experience and a joy to share my finding with you.