Add distinct_1/2 metric #108
Conversation
| import datasets | ||
| from nltk import ngrams | ||
| from rageval.metrics import Metric, add_attribute | ||
|
|
There was a problem hiding this comment.
import/from is not standardized here
| "hypothesis": datasets.Value("string"), | ||
| } | ||
| ), | ||
| codebase_urls=["https://github.com/Hannibal046/SelfMemory/blob/58d8b611ad51605091c7555c0f32dce6702dadbf/src/utils/metrics_utils.py"], |
There was a problem hiding this comment.
reference link is better to be replaced by https://github.com/Hannibal046/SelfMemory/blob/main/src/utils/metrics_utils.py
|
|
||
|
|
||
| @dataclass | ||
| @add_attribute('mtype', 'Diversity') |
There was a problem hiding this comment.
maybe Diversity is not a good metric type
There was a problem hiding this comment.
This metric is lacked of a unit test file.
_answer_precision is not a suitable file name. It is recommended to change it to _answer_perplexity
| from rageval.metrics import Metric, add_attribute | ||
|
|
||
|
|
||
| _CITATION = """\ |
There was a problem hiding this comment.
This citation doesn't seem to be correct.
| longer than the max input length of the model, then it is truncated to the | ||
| max length for the perplexity computation. | ||
|
|
||
| Examples: |
There was a problem hiding this comment.
This makes it difficult to pass CI tests in an environment without a GPU.
| import evaluate | ||
| from evaluate import logging | ||
| from rageval.metrics import Metric, add_attribute | ||
|
|
There was a problem hiding this comment.
import/from is not standardized here
| self, | ||
| predictions: List[str], | ||
| pipeline, | ||
| ) -> Tuple[float, List[float]]: |
There was a problem hiding this comment.
Where is the entry point of the _compute() method?
I add a new evaluation metric, Distinct 1/2, for the generate task evaluation. I have uploaded the new scripts "_answer_distinct12.py" and "test_answer_distinct12.py", and the modified version of "init.py".