In this blog post, we show how we developed a CLI tool to install and manage StreamPipes. In addition, we present a template project that enables you to easily create a CLI tool for your own docker application. All you need is a working docker-compose.yml file.
StreamPipes has a Microservice architecture and we use Docker and Docker Compose for all of our development, testing, and deployment processes. In the beginning, we developed the CLI only to ease the installation process (e.g. managing multiple versions), but over time we realized that the tool is much more powerful. For example, StreamPipes has multiple developer roles: Some developers are working on the frontend, others on the backend, and some are developing new algorithms and data sinks for StreamPipes. In the figure below, you can see different configurations that are used by different developers. All services are running in docker, except for the ones the developer is currently working on. As a result, we had many different docker-compose.yml files and when we made changes, we had to synchronize them all.
Therefore, our goal was to speed up the setup of a development environment, so that developers can focus on their specific programming task. That’s why we developed a command line tool that automates the setup of different environments. All what is needed is a docker-compose.yml file that describes all services of the application and the template CLI project from GitHub. Afterwards, you need to split up your compose file and create an individual file for each service. Put those files in individual folders in the services folder. Each folder should have the same name as the service name in the docker-compose file. That’s it! Now you can use the CLI. With the default commands you are able to start, stop, and configure the application. But how is this better than just using docker itself? The advantage is the flexibility and extensibility of the tool. You can extend the script with tasks you have to perform regularly, like cleaning up volumes, or setting up testing environments.
In the following we will describe the tool in more detail. You can see all files and folders of the CLI Tool in the next listing. The first file “container-cli.m4” contains the script. We use Argbash, which eases the use of bash script parameters and flags. In this file you can easily add new custom commands, flags or other functionalities. The next file is the actual bash script. It is generated by argbash. Since this file is generated automatically, you should not change anything. The docker-compose.yml file is the main file. This should contain all services, networks, or volume information that is relevant for all the other services of your application. All your other services are located in the services folder. For each service, create a folder with the name of the service and place a docker-compose.yml in the folder configuring the service. Therefore, you can copy and paste the configuration from your current docker-compose.yml. The script checks the services folder dynamically, this means you can add services at a later point in time without changing anything else. In our example, we have four services: Backend, UI, database, and a test database containing data for the end2end tests. Which services should be started when running the command "container-cli start" is defined in the active-service file. This file contains the names of all services, each in one line, that should be started. Often it is required to have multiple configurations of your app. You can save them in the templates folder. By running “container-cli set-template ui_developer”, you can select the template ui_developer. With this command, all services defined in the template are written to the active-services file. (e.g. the ui_developer template contains just the database and backend and the e2e-test template contains the backend, ui, and test-database).
The last file is the tmpl_env file. It contains configuration variables for the individual services. Those variables can also be changed by the bash script dynamically, because the tmpl_env file is moved to the .env file in the same folder. We use the configuration for example to dynamically set the IP Address of the host system or set the application version. To use a variable in a docker-compose file just write it in braces, as shown below.
We hope this tool is helpful for your use case. When you have any problems, feedback, or ideas for improvements, contact us or open an issue on GitHub.