Make your OpenStacks Collaborative
This PoC aims at making several independent OpenStack clouds
collaborative. The main idea consists in extending the OpenStack CLI
to define the collaboration process. A dedicated option, called
--os-scope, specifies which services (e.g., compute, image,
identity, …) of which OpenStack cloud (e.g., CloudOne, CloudTwo,
…) an OpenStack is made of to perform the CLI request. For instance,
the provisioning of a VM in CloudOne with an image in CloudTwo looks
like as follows:
openstack server create my-vm --flavor m1.tiny --image cirros \
--os-scope '{"compute": "CloudOne", "image": "CloudTwo"}'
This approach enables the segregation of the infrastructure into distinct areas. It removes the relying on a single control plane and thus, resolves network partitioning and scalability challenges in most cases.
To get more insight, read the how it works section or try it by yourself.
Get the code with a git clone.git clone --recurse-submodules [email protected]:BeyondTheClouds/openstackoid.git -b stable/rocky --depth 1
Then starts the two OpenStack clouds (require tmux v2 and Vagrant v2.2 – if you don’t want to use tmux, refer to the setup section).
cd openstackoid; ./setup-env.sh
The previous command starts two tmux windows and launches two
OpenStack clouds, each in a virtual machine, thanks to Vagrant. After
15 to 20 minutes, the time for Devstack to deploy the two OpenStack
clouds, the output may look like the following. The top window
connects to OpenStack CloudOne and the bottom one to CloudTwo.
These two clouds are completely independent and shared no services.
stack@CloudOne:~$ ───────────────────────────────────────────────────────────────────────────────────────────────────────────── stack@CloudTwo:~$
From there you can issue a standard openstack command, such as
openstack image list. Note the difference of ID.
stack@CloudOne:~$ openstack image list +--------------------------------------+--------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------+--------+ | 440263d5-20a7-432b-b0db-693787bd2579 | cirros-0.3.5-x86_64-disk | active | +--------------------------------------+--------------------------+--------+ ───────────────────────────────────────────────────────────────────────────────────────────────────────────── stack@CloudTwo:~$ openstack image list +--------------------------------------+--------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------+--------+ | 45da7b62-163c-4c78-aac7-363cbf4627a4 | cirros-0.3.5-x86_64-disk | active | +--------------------------------------+--------------------------+--------+
Moreover, you can use the --os-scope option that tells OpenStack of
a specific cloud to do something by using a service from another
cloud. For example, you can tell to the CloudOne to start a VM by
using the image service from CloudTwo. In the scope, a service is
implicitly bound to the local cloud, which prevents to explicitly
specify all services. Note the ID of image at the end that comes
from CloudTwo.
stack@CloudOne:~$ openstack server create my-vm \
--os-scope '{"image": "CloudTwo"}' \
--image cirros-0.3.5-x86_64-disk \
--flavor m1.tiny \
--wait
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| ... | ... |
| image | cirros-0.3.5-x86_64-disk (45da7b62-163c-4c78-aac7-363cbf4627a4) |
| name | my-vm |
| ... | ... |
| status | ACTIVE |
| user_id | 2d9440f8a4d546c88d1f5b661dc6e69b |
+-------------------------------------+-----------------------------------------------------------------+
─────────────────────────────────────────────────────────────────────────────────────────────────────────
stack@CloudTwo:~$ openstack image list
+--------------------------------------+--------------------------+--------+
| ID | Name | Status |
+--------------------------------------+--------------------------+--------+
| 45da7b62-163c-4c78-aac7-363cbf4627a4 | cirros-0.3.5-x86_64-disk | active |
+--------------------------------------+--------------------------+--------+
🎉
See misc/examples.sh for other examples.
- Same project, domain id for non-public resources
- Same keystone credential
- Resource of another cloud should be accessible from the first one (e.g., image is OK, flat network is NOK unless the two clouds share the same infra).
In brief, every OpenStack cloud comes with a proxy (here HAProxy)
in front of it. In such deployment, a service (e.g., Glance API of
CloudOne) is available via two addresses:
- The Backend address (i.e.,
10.0.2.15/image) that directly targets Glance API. - The Frontend address (i.e.,
192.168.141.245:8888/image) that targets HAProxy. HAProxy then evaluates the request and, in most cases, forwards it to the Backend.
Here, we add HAProxy the capability to interprets the --os-scope.
Instead of forwarding the request to the local Backend, HAProxy
determines the cloud of the targeted service from the scope and
URL. It then forwards the request to the local Backend only if the
current cloud is equivalent to the determined one. Otherwise, it
forwards the request to the Frontend of the determined cloud.
As an example, here is a sample of the HAProxy configuration on
CloudOne for the image service.
listen http-proxy
bind 192.168.141.245:8888 # (ref:local-front)
http-request del-header X-Forwarded-Proto if { ssl_fc }
use_backend %[lua.interpret_scope] # (ref:lua-scope)
# Target concrete backend
backend CloudOne_image_public
server CloudOne 10.0.2.15:80 check inter 2000 rise 2 fall 5 # (ref:local-back)
# Target HA of OS cloud named CloudTwo
backend CloudTwo_image_public
http-request set-header Host 192.168.141.245:8888
server CloudTwo 192.168.142.245:8888 check inter 2000 rise 2 fall 5 # (ref:remote-front)
# Do the same for compute, identity, ...
The lua.interpret_scope line (lua-scope) is a Lua script that
determines the name of the backend based on the --os-scope '{"image":
"CloudTwo"} and URL of the targeted service. From there, it
forwards the request whether to the local Backend 10.0.2.15 (l.
(local-back)) or Frontend of the remote cloud 192.168.142.245 (l.
(remote-front)).
openstack endpoint list --format json \ -c "Service Type" -c "Interface" -c "URL" -c "Region"
{ "services" :
[
{
"Service Type": "image",
"Interface": "public",
"URL": "192.168.141.245:8888/image",
"Region": "CloudOne",
"Frontend": "192.168.141.245:8888",
"Backend": "10.0.2.15:80"
},
...
{
"Service Type": "image",
"Interface": "public",
"URL": "192.168.142.245:8888/image",
"Region": "CloudTwo",
"Frontend": "192.168.142.245:8888",
"Backend": "10.0.2.15:80"
},
...
]
}HAProxy determines from the --os-scope the address of the targeted
service. Which means, the scope has to be defined for every request
and subsequent requests. For instance, when Alice does an openstack
server create --os-scope ..., the value of the --os-scope should
not only be attached to the initial POST /servers request made by
the CLI. But also, to all subsequent requests of the workflow,
including Nova request to Keystone to check Alice credentials, Nova
request to Glance to check/get the image. Glance request to Keystone
to check Alice credentials … and so on.
A first solution is to modify the OpenStack code of all services to
ensure that, e.g., when Alice contacts Nova with a specific
--os-scope, then Nova propagates that --os-scope in the subsequent
requests. However, in OpenStackoïd, we want to avoid as much as
possible modifications to the vanilla code.
Another naive implementation would try to implement the scope
propagation at HAProxy level – and keep OpenStack code as it is.
Unfortunately, this doesn’t work since HAProxy is unlikely to figure
out that, e.g., the current request from Nova to Glance comes from a
previous request from Alice to Nova with a specific --os-scope.
Luckily, every OpenStack service already propagates information from
one service to another during the entire workflow of a command: the
Keystone X-Auth-Token that contains Alice credentials. Here we reuse
that information to piggyback the --os-scope. Then, HAProxy seeks
for the X-Auth-Token, extracts the scope and finally interprets it
to forwards the request to the good cloud.
--os-scope.
The Vagrantfile contains the description of the two All-in-One
OpenStack at its top (see os_clouds). The :name refers to the name
of the cloud, :ip to the Frontend address (has to be accessible by
other clouds), and :ssh to the port used by Vagrant for SSH
connections. Doing a vagrant up reads that configuration and starts
two Ubuntu/16.04 VMs with these characteristics. Adding a third entry
in os_clouds and running vagrant up again will start a third
All-in-One OpenStack.
os_clouds = [
{
:name => "CloudOne",
:ip => "192.168.141.245",
:ssh => 2141
},
{
:name => "CloudTwo",
:ip => "192.168.142.245",
:ssh => 2142
}
]It is also possible to start only one OpenStack cloud by giving its
name after the vagrant up. For instance, the following command only
starts and configures the CloudOne.
vagrant up CloudOne
A vagrant up <CloudName>, on its first run, automatically deploys
OpenStack with Devstack and then configures it for the --os-scope.
But, it is possible to only run the deployment of Devstack with the
following commands.
vagrant up <CloudName> --no-provision vagrant provision <CloudName> --provision-with devstack
The --provision-with devstack refers to the Ansible
playbooks/devstack.yml playbook. In brief, this playbook:
- Adds a stack user.
- Clones Devstack stable/rocky.
- Generates a local.conf.
- Runs Devstack deployment.
If something goes wrong during the execution of this playbook,
everything is OK. Simply rerun the vagrant provision <CloudName>
--provision-with devstack, since Ansible playbooks are idempotent.
In the same manner of the previous section, it is also possible to
only run the configurations of one OpenStack cloud to interpret the
--os-scope with the next command.
vagrant provision <CloudName> --provision-with os-scope
The --provision-with os-scope refers to the Ansible
playbooks/os-scope.yml playbook. In brief, this playbook:
- Computes the list of services as explained in the “How it works” (see, Generating the HAProxy configuration file).
- Uses that list to generate the HAProxy configuration file, and then deploys HAProxy.
- Installs a new plugin for python-openstackclient that adds the
--os-scopein the CLI. - Workaround the rest client instance variable in Keystonemiddleware (see, Rest client instance variable in Keystonemiddleware).
- Ensures that HTTP requests of OpenStack services go through the proxy (on that particular point, read the next section).
If something goes wrong during the execution of this playbook,
everything is OK. Simply rerun the vagrant provision <CloudName>
--provision-with os-scope, since Ansible playbooks are idempotent.
Devstack doesn’t provide HAProxy deployment by default and we want to
avoid the modification of Devstack – or any other OpenStack services
– as much as possible. Thus, we deployed HAProxy after Devstack and
then ensure each request to OpenStack goes through the proxy thanks to
the HTTP_PROXY environment variable. This is referenced in the
current code with the [HACK] tag. In a real-world deployment (à la
Kolla), services are already hidden behind HAProxy and code marked
with the [HACK] tag should be removed.
. ├── keystonemiddleware@... Fork of k-middleware │ └── ... ├── misc Miscellaneous │ ├── examples.sh - OS CLI examples with the --os-scope │ └── ... ├── playbooks List of provisioning playbooks │ ├── devstack.yml - Devstack provisioning │ ├── os-scope.yml - OpenStackoïd provisioning │ └── haproxy - HAProxy conf files for OpenStackoïd ├── python-openstackoidclient OpenStackoïd CLI plugin │ └── ... ├── setup-env.sh Tmux setup script └── Vagrantfile Vagrant conf that setups the 2 OS
We would like to thanks members of the OpenStack community, and especially members of the OpenStack Berlin Hackathon (team 5) which have laid some of the initial foundation for this work: