Rex Server & Agent (Overlord & Drone)

This text is work in progress and will be updated when i have new ideas ;-)

You're all welcome to discuss this feature. Your whishes, your suggestions how to implement it, ...

You can discuss this RFCs here: https://github.com/RexOps/Rex/issues/676

Currently Rex supports only push method because there is no running agent on the remote systems that asks frequently a master server if it should do things.

With Rex-Overlord we want to add the capability to also run rex in such a pull mode. With some other nice enhancements.

Naming

The names are stolen from Starcraft

• Server-Component: Rex-Overlord
• Agent-Component: Rex-Drone

Architecture

 --------      --------       --------    
| drone1 |    | drone2 |     | drone3 |
 --------      --------       --------
    |             |              |
    |_____________|______________|     https/websocket connection
                  |                    from drone to overlord
              ----------
             | overlord |
              ----------

Communication

The communication should work over https and websockets. The authentication should be done by ssl client certificates. With websockets we have the possibility to send commands in both directions.

Features

• Rex-Drone should be pluggable so that it can be extended with custom commands. It should be also possible to monitor and to react on events that are send by the drone. For example inotify events from the filesystem.

• Rex-Drone should send inventory information to the overlord so that it is possible to query overlord for specific things on the drone. For example to check which packages and which versions are installed on a specific host. And also to allow to run commands on hosts that fullfill a special requirement. For example to update apache2 on systems that have an old version of apache2 installed.

• Rex-Overlord should expose a search api to the Rexfile so it is possible to query overlord from the Rexfile. For example if you want to setup a reverse proxy server and want to get all application server that belongs to a special group to write the configuration file.

• Rex-Overlord should also be able to handle states. For example if you want to deploy a database cluster which needs one master server to be setup at first and after this the nodes.

• Inventory information that Rex-Drone is sending to the Overlord should be easily extendable. So that it is possible for users to define custom inventory keys. For example to automatically detect in which environment a system is running or other things.

Workflow

My current idea of the workflow is this:

I'm speaking here of Modules and Services.

• A Module is a Rex Module which is independant of any data. For example an apache module. This module can be used by many services. An example of a module can be seen here: https://github.com/RexOps/rex-keepalived

• A Service is a Rexfile with a meta.yml file which describes the service and holds the dependencies to the Modules. An example of a service can be seen here: https://github.com/RexOps/service-loadbalancer

1. So as soon as someone has written his service he can upload this service to Rex-Overlord with a rex cli command (rex overlord --push)

2. The Overlord will check the version in the meta.yml file and if this version is already uploaded to the overlord it will refuse the upload.

3. (RFC) Now we need to have a way to define which task of this service (and version) should run on which server. We can do this with the already available group() and environment() functions. But it would also be good to have a possibility to define this with other things. For example if someone is using foreman, rex.io or another datacenter-management-tool and want to configure which services should be executed on which server in this management tool.

4. (RFC) After the above definition, we need to be able to upload the service to the rex drone and tell the rex-drone which task it should run locally. The question here is, should we also take care of dependencies of the service? For example if someone is using JSON::XS perl module in his service (or in a module the service is using) should the user be responsible to install all requirements or should this be done by rex-drone automatically? (I would vote for the first, because the installation of such requirements can be done with a second service that just gets executed first)

5. After the Rex-Drone has executed the task, it have to upload the report to the Rex-Overlord.