Author: QIU Tian
Affiliation: Zhejiang University
English | 简体中文
- Create
your_evaluator.py. - In
your_evaluator.py, define your evaluator. You are free to play, or you can refer to our default evaluator that computes the accuracy, recall, precision, and f1-score indefault.py.
# your_evaluator.py
__all__ = ['YourEvaluator']
...
class YourEvaluator(...):
...-
In
__init__.py,-
Import your evaluator.
-
Register your evaluator in
build_evaluator().
-
# __init__.py
...
from .your_evaluator import YourEvaluator
def build_evaluator(args):
evaluator_name = args.evaluator.lower()
...
if evaluator_name == 'your_evaluator':
return YourEvaluator(...)
...- When using your evaluator, set
--evaluatortoyour_evaluator. Note thatyour_evaluatordoes not have to be consistent with the evaluator class nameYourEvaluator.