How do I set the distance function the modl should use?

[PhilipMay]

I have models that are trained for cosine sim. and others are trained for dot product.
How do I specify that for evaluation?

[Muennighoff]

For some tasks it is automatically computed with both e.g. here

mteb/mteb/evaluation/evaluators/PairClassificationEvaluator.py

Line 80 in 2341c48

["dot", "Dot-Product", dot_scores, True],

& here

mteb/mteb/evaluation/evaluators/SummarizationEvaluator.py

Line 110 in 2341c48

"dot": {

I think we should do that for all other tasks where it's only cos sim right now (& maybe also make it configurable). It looks like it's Reranking & Retrieval where it's always cos sim right now 🤔 Do you want to open a PR to fix it?

[rasdani]

I found multiple locations, where one can set a similarity function. The one at the very bottom I tried out myself, but could not tell if it had any effect, since I was using normalized embeddings anyway.

You can choose the similarity function when initialising the Evaluator. The two in question would be RetrievalEvaluator.py and RerankingEvaluator.py.

mteb/mteb/evaluation/evaluators/RetrievalEvaluator.py

Line 36 in 2341c48

score_functions: List[Callable[[torch.Tensor, torch.Tensor], torch.Tensor]] = {

mteb/mteb/evaluation/evaluators/RerankingEvaluator.py

Line 31 in 2341c48

similarity_fct=cos_sim,

Or the one I tried was in evaluate() of AbsTaskRetrieval.py

mteb/mteb/abstasks/AbsTaskRetrieval.py

Line 41 in 2341c48

score_function="cos_sim",

For this you would need to pass the kwarg here:

mteb/mteb/evaluation/MTEB.py

Line 271 in 2341c48

results = task.evaluate(model, split, **kwargs)

So basically in your run-Script (see examples), as .run(..., score_function="dot"), if I remember correctly.