Explanations¶

A walk around the settings¶

Tamarco is an automation framework for managing the lifecycle and resources of the microservices. The configuration has a critical role in the framework, all the other resources and components of the framework strongly depend on the settings.

When you have thousands of microservices running in production the way to provide the configuration of the system becomes critical. Some desirable characteristics of a microservice settings framework are:

The configuration should be centralized. A microservice compiled in a container should be able to run in different

environments without any change in the code. For example, the network location of a database or its credentials

aren’t going to be the same in a production environment or a staging environment.
The configuration should be able to change in runtime without restarting the microservices. For example, you should

be able to update the configuration of your WebSocket server without close the existing connections.
The configuration should have redundancy. One of the advantages of a microservice architecture is the facility to

obtain redundancy in your services, you should be able to run the microservices in several machines if someone

fails, the others should be able to work correctly. Nothing of this has a sense if your services aren’t able to

read the configuration, so to take the benefits of this architectural advantage, all critical services of the

system must be redundant as well.

The external backend supported by this framework right now is etcd v2, we strongly recommend its use in production with Tamarco.

Other settings backends are available to develop:

Dictionary
File based (YML or JSON)

Settings structure¶

The settings can be configured from a simple YAML in etcd [Link of how to configure an etcd from a file]. A generic setting could be the following:

etcd_ready: true
system:
   deploy_name: tamarco_tutorial
   logging:
       profile: PRODUCTION
       file: false
       stdout: true
   resources:
       amqp:
           host: 172.31.0.102
           port: 5672
           vhost: /
           user: guest
           password: guest
           connection_timeout: 10
           queues_prefix: ""
       kafka:
           bootstrap_servers: 172.31.0.1:9092,172.31.0.2:9092
   microservices:
       http_server:
           application_cache_seconds: 10

The etcd_ready setting is written by the etcd configuration script when it finishes configuring all the other settings. This prevents the microservices from reading the settings before the environment is properly configured.

All the other tamarco settings are inside a root_path named “system”. The settings under the root path are:

Deploy_name. Name that identifies a deploy, used by default by logging and metrics resources with the purpose of

distinct logs and metrics from different deploys. Possible use cases: allow to filter logs of deploys in different

regions or by develop, staging and production with the same monitoring system.
Logging: Configuration of the logging of the system, it is out of resources because this configuration can’t be

avoided since it is a core component, all the microservices and all resources emit logs. More information about the

possible configuration in [TODO link to logging section].
Resources: configurations of the resources of the system, it can be used by one or more microservices. See:

How to setup settings for a resource.
Microservice: configuration of the business logic of each microservice. This section also has a special property,

all the other settings can be configured by in this section for a specific microservice. See:

How to setup settings for a specific microservice.

Microservice lifecycle¶

Start¶

When the microservice is initialized, the following steps are performed, automatically:

Start provisional logging with default parameters. Needed in case of some error before being able to read the final

logging configuration from the settings.
Initialize the settings. All the other resources of the framework depend on being able to read the centralized

configuration.
Initialize the logging with the proper settings. With the settings available, the next step is to be sure that all

the resources can send proper log messages in case of failure before starting them.
Call the pre_start of the microservice, that triggers the pre_start of the microservices. Operations that need to

be performed before starting the microservice. For example, a HTTP server could need to render some templates before

start the server. It is not advisable to perform I/O operations in the pre_start statement.
Call the start of the microservice, they are going to start all the resources. In the start statement the resources

are expected to perform the initial I/O operations, start a server, connect to a database, etc.
Call the post_start of the microservice, it is going to call the post_start of all the resources. In this step all

the resources should be working normally because they should be started in the previous step.

Tamarco builds a dependency graph of the order in that the resources should be initialized.

Status of a resource¶

All the resources should report their state, it can be one of the followings:

NOT_STARTED
CONNECTING
STARTED
STOPPING
STOPPED
FAILED

The status of all the resources are exposed via an HTTP API and used by the default restart policies to detect when a resource is failing.

Resource restart policies¶

The status resources come by default with the microservice and their responsibility is to apply the restart policies of the microservice and report the state of the resources via an HTTP API.

There are two settings to control automatically that a resource should do when it has a FAILED status:

system:
    resources:
        status:
            restart_policy:
                resources:
                    restart_microservice_on_failure: ['redis']
                    restart_resource_on_failure: ['kafka']

Where the microservice is identified by the name of the resource instance in the microservice class.

Keep in mind that the most recommended way is not to use these restart policies and implement a circuit breaker in each resource. But sometimes you could want a simpler solution and in some cases, the default restart policies can be an acceptable way to go.

Stop¶

The shut down of a microservice can be triggered by a restart policy (restart_microservice_on_failure), by a system signal, by a resource (not recommended, a resource shouldn’t have the responsibility of stopping a service) or by business code.

A service only should be stopped calling the method stop_gracefully of the microservice instance.

The shut down is performed doing the following steps:

Call stop() method of the microservice, it is going to call the stop() of all the resources.
Call post_stop() method of the microservice, it is going to call the post_stop() method of all the resources.
The exit is going to be forced after 30 seconds if the microservice didn’t finish the shut down in this time or

some resource raises an exception stopping the service.

Overwrite lifecycle methods¶

The lifecycle methods are designed to be overwritten by the user, allowing to execute code at a certain point of the lifecycle. Just take into account that these methods are asynchronous and that the super() method should be called.

The available methods are:

pre_start
start
post_start
stop
post_stop

from tamarco import Microservice

class LifecycleMicroservice(Microservice):

    async def pre_start(self):
        print("Before pre_start of the service")
        await super().pre_start()
        print("After pre_start of the service")

    async def start(self):
        print("Before start of the service")
        await super().start()
        print("After start of the service")

    async def post_start(self):
        print("Before post_start of the service")
        await super().start()
        print("After post_start of the service")

    async def stop(self):
        print("Before stop of the service")
        await super().stop()
        print("After stop of the service")

    async def post_stop(self):
        print("Before post_stop of the service")
        await super().stop()
        print("After post_stop of the service")


def main():
    microservice = LifecycleMicroservice(Microservice)
    microservice.run()

def __name__ == '__main__':
    main()

Microservice base class¶

All the microservices must inherit from the Tamarco Microservice class. Let’s take a deeper look into this class.

To launch the microservice, we use the run function:

.. code-block:: python

from tamarco.core.microservice import Microservice

class MyMicroservice(Microservice):
[…]

ms = MyMicroservice() ms.run()

When we run the microservice, there is a certain order in the setup of the service and then the event loop is running until an unrecoverable error occurs, or it is stopped.

Setup steps:

1. Configure and load the microservice settings (and of its resources if used). 1. Configure and start the logging service. 1. Pre-start stage: run all the Tamarco resources pre_start methods (only the resources actually used by the microservice). This method can be overriden if we want to do some coding in this step. But don’t forget to call to the Tamarco function too!

1. Start stage: run all the Tamarco resources start methods (only the resources actually used by the microservice). Also collects all the task declared in the microservice (using the @task decorator in a method) and launch them. Generally in this stage is when the database connections, or other services used by the resources are started. This start method can be overriden if we want to do some coding in this step. But don’t forget to call to the Tamarco function too! 1. Post-start stage: run all the Tamarco resources post_start methods (only the resources actually used by the microservice). This method can be overriden if we want to do some coding in this step. But don’t forget to call to the Tamarco function too! 1. Stop stage: run all the Tamarco resources stop methods (only the resources actually used by the microservice). In this stage all resources and tasks are stopped. This method can be overriden if we want to do some coding in this step. But don’t forget to call to the Tamarco function too! 2. Post-stop stage: run all the Tamarco resources post_stop methods (only the resources actually used by the microservice). This step is useful if you want to make some instructions when the microservice stops. This post_stop method can be overriden if we want to do some coding in this step. But don’t forget to call to the Tamarco function too!

Microservice cookicutter template¶

When you install the tamarco python package is available a _tamarco_ command. Calling this command you can create a new microservice skeleton answering before a few questions:

$ tamarco start_project

1. Project name: project name. In the same directory when you execute the tamarco command the script will create a folder with this name and all the initial files insite it. Used also in the docs and README files. 1. Project slug: project short name. Inside of the project name folder, a folder with this name is created and all the microservice logic code should be here. Used also in the docs files. 1. Full name: author’s full name. Used in the docs files. 1. Email: author’s email. Used in the docs files. 1. Version: initial project version. It will be copied to the setup.cfg file. 1. Project short description: this text will be in the initial README file created.

The project skeleton will be:

<project_name>
   |
   |- docs (folder with the files to generate Sphinx documentation)
   |
   |- tests (here will be store the microservice tests)
   |
   |- <project_slug>
   |     |
   |     |- logic (microservice business logic code)
   |     |
   |     |- resources (code related with the microservice resources: databases, ...)
   |     |
   |     |- meters.py (application meters: prometheus, ...)
   |     |
   |     |- microservice.py (microservice class inherited from Tamarco Microservice class)
   |
   |- .coveragerc (coverage configuration file)
   |
   |- .gitignore
   |
   |- app.py (entrypoint file for the microservice)
   |
   |- Dockerfile
   |
   |- HISTORY.md
   |
   |- Makefile (run the tests, generate docs, create virtual environments, install requirements, ...)
   |
   |- README.md
   |
   |- requirements.txt
   |
   |- setup.cfg (several python packages configurations: bumpversion, flake8, pytest, ...)
   |