Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: experimental S3 cache backend #6732

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

lyoung-confluent
Copy link
Contributor

@lyoung-confluent lyoung-confluent commented May 20, 2024

Description

This PR updates the existing fanal S3 cache implementation added many years ago with various fixes/new tests and enables it's usage via the --cache-backend s3://... CLI flag. Specifically, I removed the .index objects which are not necessary because S3 is strongly consistent and took advantage of the HeadObject API for the MissingBlobs function. While this uses the AWS S3 SDK, it can be pointed at any S3 compatible API (ex: GCS, Cloudflare R2, etc) via the endpoint url configuration feature.

Related issues

Related PRs

Checklist

  • I've read the guidelines for contributing to this repository.
  • I've followed the conventions in the PR title.
  • I've added tests that prove my fix is effective or that my feature works.
  • I've updated the documentation with the relevant information (if needed).
  • I've added usage information (if the PR introduces new options)
  • I've included a "before" and "after" example to the description (if the PR is a user interface change).

@lyoung-confluent lyoung-confluent changed the title feat(cache): experimental S3 backend feat: experimental S3 cache backend May 20, 2024
@lyoung-confluent lyoung-confluent mentioned this pull request May 21, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

S3 fanal cache backend
1 participant