Hello, After dowloading the contents from git and instantiating the object, i get this error :-

/content/image-captioning-bottom-up-top-down
Traceback (most recent call last):
  File "eval.py", line 11, in <module>
   from jury import Jury 
  File "/usr/local/lib/python3.7/dist-packages/jury/__init__.py", line 1, in <module>
    from jury.core import Jury
  File "/usr/local/lib/python3.7/dist-packages/jury/core.py", line 6, in <module>
    from jury.metrics import EvaluationInstance, Metric, load_metric
  File "/usr/local/lib/python3.7/dist-packages/jury/metrics/__init__.py", line 1, in <module>
    from jury.metrics._core import (
  File "/usr/local/lib/python3.7/dist-packages/jury/metrics/_core/__init__.py", line 1, in <module>
    from jury.metrics._core.auto import AutoMetric, load_metric
  File "/usr/local/lib/python3.7/dist-packages/jury/metrics/_core/auto.py", line 23, in <module>
    from jury.metrics._core.base import Metric
  File "/usr/local/lib/python3.7/dist-packages/jury/metrics/_core/base.py", line 28, in <module>
    from datasets.utils.logging import get_logger
ModuleNotFoundError: No module named 'datasets.utils'; 'datasets' is not a package

Can you please check what could be the issue

CLI implementation for the package the read from txt files.

Draft Usage: jury evaluate --predictions predictions.txt --references references.txt

NLGEval uses single prediction and multiple references in a way that u specify multiple references.txt files for mulitple references, and like this on API.

My idea is to have a single prediction and refenence file including multiple predictions or multiple references. In a single txt file, maybe we can use some sort of special separator like "<sep>" instead of a special char like [",", ";", ":", "\t"] maybe tab seperated would be OK. Wdyt ? @fcakyon @cemilcengiz

Hey, when computing BLEU score (snippet), facing reshape error in _compute_single_pred_single_ref.

Could you assist with the same.

from jury import Jury

scorer = Jury()

# [2, 5/5]
p = [
        ['dummy text', 'dummy text', 'dummy text', 'dummy text', 'dummy text'],
        ['dummy text', 'dummy text', 'dummy text', 'dummy text', 'dummy text']
    ]

# [2, 4/2]
r = [['be looking for a certain office in the building ',
      ' ask the elevator operator for directions ',
      ' be a trained detective ',
      ' be at the scene of a crime'],
     ['leave the room ',
      ' transport the notebook']]

scores = scorer(predictions=p, references=r)

Output:

Traceback (most recent call last):
  File "/home/axe/Projects/VisComSense/del.py", line 22, in <module>
    scores = scorer(predictions=p, references=r)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/core.py", line 78, in __call__
    score = self._compute_single_score(inputs)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/core.py", line 137, in _compute_single_score
    score = metric.compute(predictions=predictions, references=references, reduce_fn=reduce_fn)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/datasets/metric.py", line 404, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/metrics/_core/base.py", line 325, in _compute
    result = self.evaluate(predictions=predictions, references=references, reduce_fn=reduce_fn, **eval_params)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/metrics/bleu/bleu_for_language_generation.py", line 241, in evaluate
    return eval_fn(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/metrics/bleu/bleu_for_language_generation.py", line 195, in _compute_multi_pred_multi_ref
    score = self._compute_single_pred_multi_ref(
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/metrics/bleu/bleu_for_language_generation.py", line 176, in _compute_single_pred_multi_ref
    return self._compute_single_pred_single_ref(
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/metrics/bleu/bleu_for_language_generation.py", line 165, in _compute_single_pred_single_ref
    predictions = predictions.reshape(
  File "/home/axe/VirtualEnvs/pyenv3_8/lib/python3.8/site-packages/jury/collator.py", line 35, in reshape
    return Collator(_seq.reshape(args).tolist(), keep=True)
ValueError: cannot reshape array of size 20 into shape (10,)

Process finished with exit code 1

Hey, how are different bleu scores calculated?

For the give snippet, why are all bleu(n) scores identical? And how does this relate to nltk's sentence_bleu (weights) ?

from jury import Jury

scorer = Jury()
predictions = [
    ["the cat is on the mat", "There is cat playing on the mat"], 
    ["Look!    a wonderful day."]
]
references = [
    ["the cat is playing on the mat.", "The cat plays on the mat."], 
    ["Today is a wonderful day", "The weather outside is wonderful."]
]
scores = scorer(predictions=predictions, references=references)

Output:

{'empty_predictions': 0,
 'total_items': 2,
 'bleu_1': {'score': 0.42370250917168295,
  'precisions': [0.8823529411764706,
   0.6428571428571429,
   0.45454545454545453,
   0.125],
  'brevity_penalty': 1.0,
  'length_ratio': 1.0,
  'translation_length': 11,
  'reference_length': 11},
 'bleu_2': {'score': 0.42370250917168295,
  'precisions': [0.8823529411764706,
   0.6428571428571429,
   0.45454545454545453,
   0.125],
  'brevity_penalty': 1.0,
  'length_ratio': 1.0,
  'translation_length': 11,
  'reference_length': 11},
 'bleu_3': {'score': 0.42370250917168295,
  'precisions': [0.8823529411764706,
   0.6428571428571429,
   0.45454545454545453,
   0.125],
  'brevity_penalty': 1.0,
  'length_ratio': 1.0,
  'translation_length': 11,
  'reference_length': 11},
 'bleu_4': {'score': 0.42370250917168295,
  'precisions': [0.8823529411764706,
   0.6428571428571429,
   0.45454545454545453,
   0.125],
  'brevity_penalty': 1.0,
  'length_ratio': 1.0,
  'translation_length': 11,
  'reference_length': 11},
 'meteor': {'score': 0.5420511682934044},
 'rouge': {'rouge1': 0.7783882783882783,
  'rouge2': 0.5925324675324675,
  'rougeL': 0.7426739926739926,
  'rougeLsum': 0.7426739926739926}}

Hey, why does computing the BLEU score more than once, affect the key value of the score dict. e.g. 'bleu_1', 'bleu_1_1', 'bleu_1_1_1'

Overall I find the library quite user-friendly, but unsure about this behavior.

New metrics structure allows user to create and define params for metrics as desired. Current metric classes in metrics/ can be extended or completely new custom metric can be defined inheriting jury.metrics.Metric.

Jury constructor accepts metrics as a string, an object from Metric class or list of metric configurations inside a dict. In addition, BLEURT metric checks for config_namekey instead of checkpoint key. Thus, this warning message misleads if default model is not used.

Here is an example of incorrect initialization and warning message:

checkpoint is ignored:

The wrapped function expects a slightly different reference structure than the one we give in the Single Ref-Pred method. A small structure change fixes the issue.

Fixes #72

Currently, jury allows usage of input metrics to be passed in Jury(metrics=metrics) to be either list of jury.metrics.Metric or str, but it does not allow to use both str and Metric object together as,

from jury import Jury
from jury.metrics import load_metric

metrics = ["bleu", load_metric("meteor")]
jury = Jury(metrics=metrics)

raises an error as metrics parameter expects a NestedSingleType of object which is either list<str> or list<jury.metrics.Metric.

I was trying to check with the same example mentioned in the readme file for Bleurt. It is failing by throwing an error. Please let me know the issue.

Error :

ImportError                               Traceback (most recent call last)
<ipython-input-16-ed14e2ab4c7e> in <module>
----> 1 bleurt = Bleurt.construct()
      2 score = bleurt.compute(predictions=predictions, references=references)

~\anaconda3\lib\site-packages\jury\metrics\_core\auxiliary.py in construct(cls, task, resulting_name, compute_kwargs, **kwargs)
     99         subclass = cls._get_subclass()
    100         resulting_name = resulting_name or cls._get_path()
--> 101         return subclass._construct(resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
    102 
    103     @classmethod

~\anaconda3\lib\site-packages\jury\metrics\_core\base.py in _construct(cls, resulting_name, compute_kwargs, **kwargs)
    235         cls, resulting_name: Optional[str] = None, compute_kwargs: Optional[Dict[str, Any]] = None, **kwargs
    236     ):
--> 237         return cls(resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
    238 
    239     @staticmethod

~\anaconda3\lib\site-packages\jury\metrics\_core\base.py in __init__(self, resulting_name, compute_kwargs, **kwargs)
    220     def __init__(self, resulting_name: Optional[str] = None, compute_kwargs: Optional[Dict[str, Any]] = None, **kwargs):
    221         compute_kwargs = self._validate_compute_kwargs(compute_kwargs)
--> 222         super().__init__(task=self._task, resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
    223 
    224     def _validate_compute_kwargs(self, compute_kwargs: Dict[str, Any]) -> Dict[str, Any]:

~\anaconda3\lib\site-packages\jury\metrics\_core\base.py in __init__(self, task, resulting_name, compute_kwargs, config_name, keep_in_memory, cache_dir, num_process, process_id, seed, experiment_id, max_concurrent_cache_files, timeout, **kwargs)
    100         self.resulting_name = resulting_name if resulting_name is not None else self.name
    101         self.compute_kwargs = compute_kwargs or {}
--> 102         self.download_and_prepare()
    103 
    104     @abstractmethod

~\anaconda3\lib\site-packages\evaluate\module.py in download_and_prepare(self, download_config, dl_manager)
    649             )
    650 
--> 651         self._download_and_prepare(dl_manager)
    652 
    653     def _download_and_prepare(self, dl_manager):

~\anaconda3\lib\site-packages\jury\metrics\bleurt\bleurt_for_language_generation.py in _download_and_prepare(self, dl_manager)
    120         global bleurt
    121         try:
--> 122             from bleurt import score
    123         except ModuleNotFoundError:
    124             raise ModuleNotFoundError(

ImportError: cannot import name 'score' from 'bleurt' (unknown location)

Referring this issue https://github.com/thompsonb/prism/issues/13, since it seems like no activate maintanance is going on, we can add this support on a public fork.

Due to the nature of the Jury API, all input strings must be a whole (not tokenized), the current implementation of BLEU score is tokenized by white spaces. However, one might want results for smaller tokens, morphemes, or even character level rather than BLEU score of the words. Thus, it'd be great to support this with adding a support for tokenizer in the score computation for BLEU.