Data Service Skeleton ProjectDave Redfern
Published: 21 Feb 02:48 in Standalone
Data Service Starter Project
This is a skeleton project that pre-configures a shared data services project to act as the storage / front-end for micro services. It is intended to be used in conjunction with the Symfony Micro Service and provides the DNS, proxy, rabbit queue and database(s).
- commented docker-compose.yml file
- docker configuration including Postgres, dns, Traefik, and RabbitMQ
- docker is configured to use Docker volumes
Assorted readme files are included for different parts of the service setup:
From GitHub, create a new repository from the data-service-skeleton template and then check it out to your dev machine. Alternatively checkout a detached copy from GitHub and then push to a new repo later.
Customise the base files as you see fit; change names, (especially the service names), config values etc to suite your needs. Then:
docker-compose up -d to start the docker environment in dev mode. Be sure to read Service Discovery to understand some of how the docker environment is setup.
Recommended First Steps
This project uses
example.dev and placeholders though out. Your first step should be to set the project names, and dev domain that you will use.
The domain name is set in several places, it is strongly recommended to change this to something more useful. The following files should be updated:
The following docker services are pre-configured for development:
- PostgreSQL 12
- RabbitMQ 3.7 + management console
- syslog-ng 3
- Traefik 2.2
Docker Service Names
The Docker container names will be prefixed by a project name defined in the
.env file. This is the constant
COMPOSE_PROJECT_NAME. If you remove it, the current folder name will be used instead. For example: you create a new project called "db-service", without setting the COMPOSE constant the containers started via
docker-compose will be prefixed with
db-service_. If you have a lot of docker projects, they may have similar folder names, so using this constant avoids collisions.
DNSmasq is included to provide local DNS resolution to avoid needing entries in your /etc/hosts file. Be sure to check out the instructions in Service Discovery.
If you use a remote docker host, then the scripts will check for the following shell constants:
DNS_HOST_IP is the IP address of the host running DNSmasq i.e. the docker host IP address.
DOCKER_HOST is the host name of the docker host, and must be fully qualified. For example:
DOCKER_HOST="tcp://my-docker-host.example.dev:2375". This is presuming you have a trusted docker host that is not secured. If you do not have DNS translation, then
DOCKER_HOST_ALT can be used with the IP address of the docker host:
_HOST_ constants are used in
docker-compose.yml on the proxy service.
Contributions are welcome! Spot an error, want additional docs or something better explaining? Then create a ticket on the project, or open a PR.
Traefik is being used to act as a load-balancer and proxy for the various micro and data services. Traefik requires access to your Docker Host. On a local dev box this means sharing the
/var/run/docker.sock file with the container, but on remote hosts, exposing the remote docker host port either insecure in a highly trusted environment (port 2375), or over SSL (2376).
The build process will automatically use the local docker socket file unless otherwise configured.
To fully use the service discovery feature, a local DNS service has been added. This allows for local host resolution of the *.example.dev domain name.
Note: it is suggested to move the base data services into a separate project so that they can be shared with other micro services.
- http://dns.example.dev/ - DNSMasq console
- http://proxy.example.dev/ - Traefik Console / Monitoring
- http://rabbit.example.dev/ - RabbitMQ Management Panel
Setup Local DNS Resolution
These instructions are for macOS. For other operating systems, Google it or set custom DNS servers to use your localhost port 1034.
Create a new resolver configuration file:
sudo mkdir /etc/resolver cd /etc/resolver sudo nano -w example.dev
Add the following contents to this file:
domain example.dev nameserver 127.0.0.1 port 1034 search_order 10
Save the changes (Ctrl+O) (oh, not zero) and exit (Ctrl+X).
Check your DNS via
sudo scutil --dns it should have output similar to:
resolver #8 domain : example.dev nameserver : 127.0.0.1 port : 1034 flags : Request A records, Request AAAA records reach : 0x00030002 (Reachable,Local Address,Directly Reachable Address) order : 10 DNS configuration (for scoped queries) resolver #1 nameserver : 22.214.171.124 nameserver : 126.96.36.199 if_index : 8 (en0) flags : Scoped, Request A records reach : 0x00000002 (Reachable)
Finally: make sure any local
/etc/hosts entries are removed otherwise they will interfere with the DNS resolution. Either comment them out or delete the lines entirely.
/etc/resolver only reloads on file changes, not file edits. If you make a mistake and need to reload the file,
sudo touch tmp and then
sudo rm tmp to force a reload.
Note: you may need to clear your dns cache as well:
sudo killall -HUP mDNSResponder
Automatic Service Discovery
Traefik acts as a proxy and load balancer in a similar way to nginx. It listens on port 80 (or any other) and provides a gui (usually on 8080, but proxy.example.dev:80 gives access as well). LetsEncrypt can be setup to provide SSL as well as HTTP auth etc.
To register containers with Traefik (called
proxy in this project), you need to label the container with specific tags. Any web service should be labeled with:
- traefik.http.* with configuration directives
By default, Traefik will not register new services - they must be explicitly configured.
For example: to expose the example App API and have Traefik route it:
services: app: build: context: . dockerfile: src/Resources/docker/dev/app/Dockerfile networks: - backend labels: traefik.enable: true traefik.http.routers.app.rule: "Host(`app.example.dev`)" traefik.http.routers.app.tls: true traefik.http.services.app.loadbalancer.server.port: 8080
By default, the proxy service has automatic SSL forwarding on all hosts, however each app still needs to explicitly set this. If you are using a
.dev domain name, then these must be served over SSL.
Each of the
http config options needs setting to the services name that this config applies to. In this example, the service name is "app" - that is the key under the services. This is then used in each of the router and services options. If your service is named "webserver" then these labels would be written as:
services: webserver: labels: traefik.enable: true traefik.http.routers.webserver.rule: "Host(`app.example.dev`)" traefik.http.routers.webserver.tls: true traefik.http.services.webserver.loadbalancer.server.port: 8080
server.port is the INTERNAL container port that Traefik should forward requests to. Typically this is whatever is exposed in the containers Dockerfile e.g.: 8080, 9000, 5432, etc etc. If there is only one port exposed, this option can be left of, however if there is more than one it must be provided.
All that is left to do is
dc up -d and Traefik will pick up the new container and it will be made available via whatever hostname was set (presuming you are also using the DNS resolver).
dc down services will automatically be removed.
As the Traefik config is done through labels, they can be added safely to docker-compose files without interfering with any other configuration.
Note: you should ensure that any services use the same named network that Traefik is running under to avoid issues with routing. By default the network name is:
mycompany_network_backend. You should change this to reflect your projects name; then in services that need registering ensure that an external network is defined:
networks: mycompany_network_backend: external: true