Skip to content

Latest commit

 

History

History
420 lines (281 loc) · 18.4 KB

REPORT.md

File metadata and controls

420 lines (281 loc) · 18.4 KB
Raw

Rough Opal Grant Report

This document will give a comprehensive view of the state of the grant work as its duration comes to a close.

Background

Rough-Opal is a grant accepted in October 2022. The duration spanned from September to December 2022.

Deliverables and Development Roadmap

The original proposal containing grant deliverables has not since been edited.

Below the Deliverables and Development Roadmap from that post have been turned into a Table of Contents. This document ends with a Closing included in the ToC.

The links for each item snap to a section that references prs/issues/commits related to them. Items that are not linked have been included in the parent or child or are insignificant. Crossed-out items will not be completed as part of this grant and include a written section for explanation. All other ToC items have been or will be completed (see Still Ongoing for incomplete but still planned items).

Changes Since Then

At the start of the grant, the protocol and implementation shared the name opal. The grant proposal and the implementation's repo mentioned possible name changes. Names have been changed for the protocol and implementation to hldb and welo, respectively. The project now has its own Github org located at github.com/hldb.

Two significant changes involve removing Zzzync replicator and Heavy Testing from this grant's roadmap.

Table of Contents



Deliverables

Opal Base Feature Set

Issue: hldb/welo#8

Locally persisted Database replicas

This required persisting the replicas and indexes of each database. Enables database reads (store index), and writes (replica) immediately upon being opened.

Easy Custom Database States

Creating custom databases is easy now but can still be much easier, so leaving this unfinished. Necessary documentation will be added for this feature when it's ready.

Here is the key-value store model showing what will be needed. It includes a reducer, an init function (which supplies the initial value for the state), action creators, and selectors. These are all similar to Redux's design.

Pubsub Heads Exchange Replicator

This replicator is a reimplementation of the OrbitDB replicator. Welo nodes join a shared pubsub channel unique per database. When nodes see peers join the shared channel, nodes attempt to join a direct channel to advertise database updates.

Unlike OrbitDB, the replicator has a specification, and replication tests are not flaky.

Zzzync Replicator integration

Unfortunately, a Zzzync replicator was not added. Building one will take more time than expected to do right. In this case to do right means the result is: relatively efficient/fast and general purpose (can be used apart from web3storage).

The proposal identified this as the most difficult part of the grant.

It's still possible to have a naive/experimental Zzzync replicator fairly quickly. However, I didn't see this as worth it and am looking at designing a system for use long-term.

I did inquire about this with Irakli from web3storage, who always has some good information.

IPLD Schema Validation (if ready in javascript)

Validating the schema at the IPLD layer will likely be part of hardening the implementation in the nearer future.

Automated Release: ci, change log, and api docs

The release used is similar to js-libp2p's pipeline. It creates a changelog inside a PR; after it's merged, the workflow cuts a new Github release and publishes it to npm. A changelog is generated and the version is bumped by looking at the Conventional Commits in the commit messages.

The API docs have been automated. They look nice but can be improved in the future. JSDoc is parsed by the tool to add useful information. The Typescript types are used to show parameters.

Zzzync Development Plans

See Zzzync Development Plans

Opal-Spec 1.0-beta

Release Draft Spec for Opal and main modules

Release Draft Spec for Live Replicator

Release Draft Spec for Zzzync

See Zzzync Replicator integration

Release completed Specifications for Opal and Zzzync 1.0-betas

Development Roadmap

Opal Repo Init

write unit tests

95% coverage according to codecov.io

There are more than a few testing-related commits; so I won't link them all. Feel free to check out the tests. I plan on organizing them a bit more in the future.

prep for adding features

This work had to do with cleaning up code before adding new stuff.

build interfaces for manifest modules

Created Typescript interfaces for the different manifest modules.

rewrite manifest module registry

Fixed the module registry where modules referenced by the manifest are registered.

rework store module
rework classes to use Libp2p's startable interface

Libp2p's startable interface is a Typescript interface for managing the lifecycle of objects. To make things easier when working with async start/stop I made Playable that classes inside the project use.

make databases locally persistent

Opal Replication and Perf

add live replicator (Libp2p pubsub + IPFS)

test replication and replicated states

write benchmarks

This is not complete yet. It is part of Still Ongoing. Benchmarks will be added for read/write/replication, and metrics will be tracked over time with github-action-benchmark.

automate release with generated API docs and changelog

See Automated Release: ci, change log, and api docs for more detail

release draft spec for Opal and main modules

release alpha with expected public API changes

The live replicator was unfinished at the time; as a result, an alpha was never released.

Zzzync Replicator

See Zzzync Replicator integration

Release

Heavy Testing

I'll be looking at doing the heavier testing with tools like Testground around the same time as performance and hardening. I did not quite reach this step during the grant.

Usage References

Opal and Zzzync automated (and nice-looking) API docs

Decided to use api-documenter instead of Typedoc because it had much nicer-looking generated markdown.

See Automated Release: ci, change log, and api docs for more detail

Write base FAQ.md document for common user questions

hldb/welo#39

Basic Tutorial document added to repo or blogged

hldb/welo#40

NodeJS and Create React App examples

Welo has not been tested with browsers yet. Support is tracked in browser tests issue.

Release Opal and Zzzync 1.0-beta

The 1.0.0 release was prematurely cut while trying out different release pipelines.

Latest release is now v1.0.2

completed protocol specification
typescript implementation (with public API locked until 1.0)

The original plan was to turn the project into Typescript toward the end of the grant. However, Steve, my partner on Cypsela, recommended I use Typescript immediately.

Thanks Steve

It helped a bunch and likely sped up development pretty considerably.

Closing

Accomplishments

Welo, pronounced "way-low", is now a working peer-to-peer database written in Typescript. It's very similar to OrbitDB but does not need to load all entries into memory before opening a database (which can take multiple seconds; IIRC 2 seconds for every 1000 entries). Because Welo write replicas and store indexes to disk, databases do not need to load all entries into memory; exposing read and write immediately regardless of database size. It will be easy, starting from here, to lazily load as much of the database into memory as the user sees fit.

Publishing, testing, API docs, and checking/updating dependencies are now automated. Similar CI/CD actions to the js-libp2p repo test and create releases on push to the master branch.

I created copy-deps, a package that makes updating shared dependencies easier. This package comes in handy when upgrading ipfs-core as it overlaps a handful of dependencies welo uses.

A specification for the Opalescent database is published. This is a living specification for version 1.0-beta and will continue to be updated as things are changed and added.

Still Ongoing

This section will be updated as items are completed

Next Steps

This section outlines ideas to work on following this grant

After completing what is included in the section above, I plan to look at designing an IO-efficient replica structure in IPLD. Intuitively this also seems relevant to Zzzync replicators.

Testing, performance, and hardening the implementation will be the main focuses afterward, in that order. I'd also like to learn Typescript a bit better. I know general use right now but more advanced knowledge will enable me to write even cleaner code that requires less testing.

The protocol is in a good state for now, but look forward to adding a low-latency replication protocol after Zzzync.

Will explore these path options more in a follow-up grant proposal.

Remarks

This was my first solo grant project. The admins managing the grant procedure and communication were great. It's always been a pleasure when interacting with anyone from the Protocol Labs umbrella.

There are some areas I can improve, like tracking progress and creating deliverables/roadmaps. Hopefully, these will get easier to judge.

I would like to see how transparent/auditable I can make grant work progress in the future by linking directly to issues/prs.

Amendment 1

Zzzync is being added back with a demo web app to close out the grant. This list will link directly to the relevant issues for better transparency.

ETA: June 17th

Grant budget is still capped at 30K

This amendment will count as the second two milestones, with the first two marked as complete.