Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List Splitting/Clipping #565

Open
JMMackenzie opened this issue Jan 15, 2024 · 0 comments
Open

List Splitting/Clipping #565

JMMackenzie opened this issue Jan 15, 2024 · 0 comments
Labels
effort:medium More complex, might take several days to a week enhancement New feature or request performance Efficiency improvements wip

Comments

@JMMackenzie
Copy link
Member

Describe the solution you'd like
There is a bunch of prior work on splitting postings lists; this allows the high impact list to be traversed (more likely) than the low impact list, and since the low impact lists are typically long, accelerates retrieval. Recent work in EMNLP findings showed that this is highly beneficial to learned sparse retrieval mechanisms. I would like to see if this can be pulled into the PISA engine with perhaps a more elegant design (I currently have a private prototype used in the EMNLP experiments but it's a bit rough for release still). This would then supersede the (currently empty) repo here: https://github.com/JMMackenzie/postings-clipping

Paper: https://aclanthology.org/2022.findings-emnlp.205/

Additional context
TBA - This is a draft issue for now.

@JMMackenzie JMMackenzie added enhancement New feature or request performance Efficiency improvements wip effort:medium More complex, might take several days to a week labels Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort:medium More complex, might take several days to a week enhancement New feature or request performance Efficiency improvements wip
Projects
None yet
Development

No branches or pull requests

1 participant