- Well tested
- The ROUGE-X scores are tested compare with original Perl script (ROUGE-1.5.5.pl).
- The BLEU score is calculated by SacréBLEU, that produces the same values as official script (
mteval-v13a.pl) used by WMT.
- Multi-language
- Not only English, Japanese are also supported. The other language is extensible easily.
Of course, implementation is Pure Python!
from sumeval.metrics.rouge import RougeCalculator rouge = RougeCalculator(stopwords=True, lang="en") rouge_1 = rouge.rouge_n( summary="I went to the Mars from my living town.", references="I went to Mars", n=1) rouge_2 = rouge.rouge_n( summary="I went to the Mars from my living town.", references=["I went to Mars", "It's my living town"], n=2) rouge_l = rouge.rouge_l( summary="I went to the Mars from my living town.", references=["I went to Mars", "It's my living town"]) # You need spaCy to calculate ROUGE-BE rouge_be = rouge.rouge_be( summary="I went to the Mars from my living town.", references=["I went to Mars", "It's my living town"]) print("ROUGE-1: {}, ROUGE-2: {}, ROUGE-L: {}, ROUGE-BE: {}".format( rouge_1, rouge_2, rouge_l, rouge_be ).replace(", ", "\n"))
from sumeval.metrics.bleu import BLEUCalculator bleu = BLEUCalculator() score = bleu.bleu("I am waiting on the beach", "He is walking on the beach") bleu_ja = BLEUCalculator(lang="ja") score_ja = bleu.bleu("私はビーチで待ってる", "彼がベンチで待ってる")
sumeval r-nlb "I'm living New York its my home town so awesome" "My home town is awesome"
output.
{
"options": {
"stopwords": true,
"stemming": false,
"word_limit": -1,
"length_limit": -1,
"alpha": 0.5,
"input-summary": "I'm living New York its my home town so awesome",
"input-references": [
"My home town is awesome"
]
},
"averages": {
"ROUGE-1": 0.7499999999999999,
"ROUGE-2": 0.6666666666666666,
"ROUGE-L": 0.7499999999999999,
"ROUGE-BE": 0
},
"scores": [
{
"ROUGE-1": 0.7499999999999999,
"ROUGE-2": 0.6666666666666666,
"ROUGE-L": 0.7499999999999999,
"ROUGE-BE": 0
}
]
}
Undoubtedly you can use file input. Please see more detail by sumeval -h.
pip install sumeval
- BLEU is depends on SacréBLEU
- To calculate
ROUGE-BE,spaCyis required. - To use lang
ja,janomeorMeCabis required.- Especially to get score of
ROUGE-BE,CaboChais needed additionally.
- Especially to get score of
sumeval uses two packages to test the score.
- pythonrouge
- It calls original perl script
pip install git+https://github.com/tagucci/pythonrouge.git
- rougescore
- It's simple python implementation for rouge score
pip install git+git://github.com/bdusell/rougescore.git
The tokenization and dependency parse process for each language is located on sumeval/metrics/lang.
You can make language class by inheriting BaseLang.