Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upsert logic needs to be redesigned #125

Open
mpgreg opened this issue Nov 14, 2023 · 1 comment
Open

Upsert logic needs to be redesigned #125

mpgreg opened this issue Nov 14, 2023 · 1 comment

Comments

@mpgreg
Copy link
Contributor

mpgreg commented Nov 14, 2023

Describe the bug
Current code is not atomic and is not batch safe. Also no roll-back.

If a documents chunks span a batch the upsert will remove docs and reinsert in one batch and then redo it in the next batch. Documents inserted in the first batch will be deleted and no inserted.

Version

To Reproduce
Steps to reproduce the behavior:

Expected behavior
Upsert should be atomic, batch-safe and allow roll-back if any of the batches fail.

Screenshots

Additional context

@mpgreg
Copy link
Contributor Author

mpgreg commented Nov 14, 2023

This is really a bug for the pre-release weaviate provider but documenting here since it impacts ask astro.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant