bleuscore
is a fast BLEU score calculator written in rust. You can check try the web demo here for a quick experience.
The python package has been published to pypi, so we can install it directly:
uv:uv add bleuscorepip:pip install bleuscore
The usage is exactly same with huggingface evaluate:
- import evaluate
+ import bleuscore
predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
["hello there general kenobi", "hello there !"],
["foo bar foobar"]
]
- bleu = evaluate.load("bleu")
- results = bleu.compute(predictions=predictions, references=references)
+ results = bleuscore.compute(predictions=predictions, references=references)
print(results)
# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0,
# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
TLDR: We got more than 10x speedup when the corpus size beyond 100K
We use the demo data shown in quick start to do this simple benchmark. You can check the benchmark/simple for the benchmark source code. Detailed per-PR results are tracked automatically via the Benchmark workflow.
