The aim of this article is to introduce our new approach to simplify the on-premise installations with Docker and what it implies.
Toucan solutions are SaaS solutions. We run them, update them, back them up and monitor them with everything hosted on our side. However, with the growth of the business and the sensitive nature of some data, some clients asked us to host our solutions directly on their infrastructure to respect their data security policy.
This can range from “data are too sensitive to be hosted elsewhere” to “nothing gets out from their networks because it is the government”.
A few years ago, we created our first on-premise package for the Toucan backend stack to answer these needs. We already talked about this “DevOps” journey on a previous blog post.
The result of this work was a bench of Ansible scripts, which allows you to auto-magically deploy a full Toucan backend stack. All these scripts are now used by our team to deploy and update all our SaaS customers stacks on our infrastructure, with the latest weekly release.
However, even if our package works and our customers install it without any help from our support team, we knew we had room for improvement. E.g: some mandatory requirements are not always simple to enable for our customers: like having a network access to some famous code repositories like GitHub or the official MongoDB place.
In these cases, we always find a workaround, but it is not optimal:
This is why we tried to figure out a way to install the Toucan stack with fewer requirements.
Obviously it’s a solution, why did we not do it ? 😀
VM is an “old” concept and a lot of our customers already have a VM management system.
So for each release, we would just need to create and send a VM with the whole Toucan stack…
… and that’s not so easy for several reasons:
We felt like it would be tricky to maintain and automate this.
But there may be another way…
We deliberately ignored Docker as an on-premise solution for a long time because it was not used by our customers (Fortune 500 companies).
By pure logic, we would rather focus on projects and features that could be used by all our customers, and not only one or two.
However, a change is going to come (like Sam Cook told us). Docker is becoming a target for almost all our on-premise customers or prospects… Well done, guys !
We had to be on this train and offer something cool with it.
Our backend stack is simple and composed with several services and processes: MongoDB, Redis, gunicorn and celery python workers, nginx, supervisor
One part of the Docker philosophy is to have one container for each service/process and then you orchestrate them with tools like Docker Compose or others.
But there are too many tools with different versions and potential breaking changes… What we want is to be agnostic about them. For example with Docker Compose you have 3 main versions, which require specific Docker versions (cf. the famous compatibility matrix). Moreover, “compose” files are not compatible between major versions.
Remember, we try to reduce the requirements. Therefore, we need to avoid imposing new technologies or versions. The docker philosophy didn’t work with ours.
We wanted to support our application/stack and not the side technologies used by our customers. We had to focus on our product.
This is why we decided to create one container, which holds the full Toucan backend stack as described as below:
Other software editors (like Discourse) chose exactly the same “one container approach” for the same reasons: simple, easy to launch and agnostic.
When you start from scratch, creating the Docker image can take time and be very difficult.
But if you remember, we already got our Ansible scripts, which allow us to deploy and configure automatically the full stack.
Because we use them daily for our SaaS infrastructure, we were really confident about them.
So as a first try, the build of the Toucan backend image could be a basic Ubuntu image where we would launch our shipping scripts with the right settings!
This would be a perfect quick win.
The Dockerfile could be something like this:
FROM ubuntu:16.04
# Backend sources
# ###############
ADD ./ backend_source/
# Start to install the Backend stack
# ##################################
RUN cd backend_source/ && \
make fulldeploy stage=docker # launch our Ansible scripts
# Expose only the HTTP port
# #########################
EXPOSE 80
RUN chmod +x /data/toucan_scripts/entrypoint.sh
ENTRYPOINT [ "bash", "-c", "/data/toucan_scripts/entrypoint.sh" ]
That’s all!
After some minimal changes and trials, it worked like a charm without rewriting everything.
All the “smartness” is in the Ansible scripts, which are robust and easy to maintain.
We absolutely wanted to avoid having some parts of the configuration in the Dockerfile and other parts in the shipping. We try to keep things simple.
Moreover, we are not tied up to Docker, we could have the same approach with VM or a future new technology as long as we are able to use our Ansible scripts. Of course, we could reduce and optimize the size of the image, but it’s only a first version and the beginning of a new “devops” journey ;).
Please note that we also created a “smart” entry point bash script, which lets the users configure the container as they want. Kind of like the security settings or the details about Redis, MongoDB and mail settings.
It’s actually the only part we will need to maintain in parallel with the Toucan backend evolution.
Now you have the same configuration flexibility between the usual on-premise package and the backend Docker image.
Our CI/CD is totally agnostic, it only knows how to check out a project and launch a command according to it.
Each time a new release is created, it triggers the generation of a package via a make package command included on each managed repository. Then the result of the command is automatically uploaded to our download place. So to plug the creation and upload of the Docker image, we only needed to add a new target (package-docker) and configured our CICD to launch it for each release.
That’s all, pretty simple, isn’t it?
Now you can run a full Toucan backend with less than 5 commands:
# download the latest version of the Toucan Toco image
$ curl -u "USER:PASSWORD" https://get-package.toucantoco.com/laputa/docker_backend-latest.tar -o /data/tmp/docker_backend-latest.tar
# load the image in your Docker system
$ docker load < /data/tmp/docker_backend-latest.tar
# check the image is well imported
$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
toucantoco/backend v43.4.4 02d4415908d8 2 days ago 3.2GB
# launch a container with the Toucan Toco backend image
$ docker run -d -p 80:80 toucantoco/backend:v43.4.4
# test your container with the usual monitoring curl
$ curl -A "Plop 4rQD3KzCxWzTYaRyp0NSEfd6" http://127.0.0.1/status
As you can see, you only need Docker and bash commands to run your own Toucan backend stack. There is nothing more to install and it’s absolutely straightforward.
Of course for a production use, we suggest you to check the available configurations, which are described in detail in our online documentation (to persist data, enable HTTPS…).
Because we “capitalize” on our existing robust shipping scripts, we found a quick win to create a real Toucan backend image without changing the tools or the way we work. We are now able to have two ways to install the Toucan backend stack:
This new on-premise approach is also pretty interesting because you’re able to test it without reserving hardware modules or asking your whole IT team to open network flows. You can create your own PoC in only 5 minutes.
If you previously thought our installation was too complicated, let’s talk again about it, we believe we can challenge your view now.
Of course the journey is still long about Docker and the on-premise context, we already have plenty of ideas for the future about optimization, features and so on… But thanks to Docker, we hope to unlock a lot of new business opportunities where the requirements were too difficult to meet.
In the end, we are happy that we managed to simplify the installation and update for all our customers.