object_database

An in-memory distributed objects in python, with transactions, plus a reactive web layer.

Quickstart

Use the included docker-compose file to quickly spin up a container to try out object database.

1. Make sure docker is installed on your system
2. Clone this repository (e.g., git clone [email protected]:APrioriInvestments/object_database)
3. CD into the cloned repo
4. Run docker-compose up
5. Go to localhost:8000 in your browser to see the object database web test application

N.B. if installing docker for the first time (at least on mac) you may need to: a) startup the docker desktop app (it will download some more things and set vars) b) run a container such as the example one docker run -d -p 80:80 docker/getting-started

Installation

The recommended way to install object_database is by cloning the repo and pip install from the repo. The public PyPI release is out of date.

0. Make sure you have python3-dev and openssl and gcc are installed

   1. apt-get install gcc
   2. run apt-get install python3-dev && apt-get install libssl-dev
   3. it is also advisable to run apt-get update && sudo apt-get install --reinstall build-essential (especially if you are getting gcc cc1/execvp related build/install errors)

1. Clone repo git clone [email protected]:APrioriInvestments/object_database

2. Create a fresh virtual environment with python 3.6-3.8

   via venv

   cd object_database
   python3 -m venv .venv
   . .venv/bin/activate
   export PYTHONPATH=`pwd`

   via conda

   cd object_database
   conda create -y -n odb python=3.8
   conda activate odb
   export PYTHONPATH=`pwd`

3. Build and install from source:

   pip install -e .

Major components

This repo has 3 major components and 1 notable major dependency:

• distributed object engine
• service manager ("k8s-lite")
• reactive web layer (Cells)
• typed_python

In-memory distributed object database

The core of this repo is a distributed in-memory object database engine. Using this engine, you can define type-safe collections of objects that can be safely and consistently written/read by multiple clients concurrently. The database engine supports transactions and takes care of consistency and state syncing (via optimistics concurrency control).

For details and examples see object engine docs

Service manager (k8s-lite)

Object database has a service management engine can manage distributed compute, run web services, run jobs, and more. This engine lives in ./object_database/service_manager and uses a running object database to manage state.

For more details see service manager docs

Cells (reactive web layer)

TODO: link to Cells docs from @dkrasner

Typed python

ODB uses typed_python to define object schemas and offer safety/performance. It is also the most frequent source of build/install problems. Most notably, you need to make sure that this repo and typed_python are built on the same version of numpy (see pyproject.toml for dependencies and version constraints for the build, requirements.txt package installation deps). Note that the pyproject.toml is required so the C++ extensions are built on the correct version of numpy.

For more details see typed_python docs.

Support matrix

As of 2022-04-28, only python 3.6-3.8 is supported (due to typed_python not working for 3.9/3.10). Linux: works more or less out of the box MacOS: requires a decent amount of futzing Windows: ahahahahaha good one