Adding a BM25 baseline?

[malteos]

Hi all,

I made a little script to evaluate the MTEB retrieval tasks with BM25 (especially to have a strong baseline for the multilingual tasks that): https://gist.github.com/malteos/178a1b77ac362cd7857a054e2d9c07cb

Is this something worth adding to this repo or the mteb scripts repo? If so, where do you recommend to add this? I could make PR then.

Best
Malte

[KennethEnevoldsen]

Wouldn't it be possible to write it as an Encoder module to avoid wrapping the MTEB module?

(e.g. see #888)

[KennethEnevoldsen]

from comment #888 I think it is great to add this in as well. I would love to in the future more natively support BM25 models

@malteos will you make the PR? We might want to also discuss potential refactors to allow for native implementation of BM25 (or similar).

Note that the BM25MTEB wrap might be unecc. instead, you can probably do:

tasks = mteb.get_tasks(... task_type == "retrieval")

[malteos]

I will make a PR in the next weeks. I cannot promise any exact date though.