Uploading Files

Users can now upload files into the DeepLynx data warehouse. At time of writing (August 2020) files will either be stored on the local filesystem, or using Azure's Blob Storage service.

Configuration

Users must configure the FILE_STORAGE_METHOD environment variable and set it to be either azure_blob or filesystem. Users must set additional environment variables with each of the methods, those are described inside the .env-sample file at the project's root.

Here is a brief diagram illustrating how file upload and storage works.

How-to

1. User POSTS a multipart form to /containers/:id/datasources/:id/files. That multipart form can have numerous files and normal field/values attached.
2. Deep Lynx separates the files from the metadata and uploads them to the configured file storage.
3. Deep Lynx creates a record in the files table with the files location on the configured storage method and some metadata about the file along with the user provided metadata.
4. The endpoint returns an array of File objects, which contains the newly assigned Deep Lynx ID.
5. Users can query files directly from the files table using the GraphQL query layer.