Skip to content

euroeval.metrics.huggingface

source module euroeval.metrics.huggingface

All the Hugging Face metrics used in EuroEval.

Classes

source class HuggingFaceMetric(name: str, pretty_name: str, huggingface_id: str, results_key: str, compute_kwargs: dict[str, t.Any] | None = None, postprocessing_fn: t.Callable[[float], tuple[float, str]] | None = None)

Bases : Metric

A metric which is implemented in the evaluate package.

Initialise the Hugging Face metric.

Attributes

  • name — The name of the metric in snake_case.

  • pretty_name — The pretty name of the metric, used for display purposes.

  • huggingface_id — The Hugging Face ID of the metric.

  • results_key — The name of the key used to extract the metric scores from the results dictionary.

  • compute_kwargs : dict[str, t.Any] — Keyword arguments to pass to the metric's compute function. Defaults to an empty dictionary.

Parameters

  • name : str — The name of the metric in snake_case.

  • pretty_name : str — The pretty name of the metric, used for display purposes.

  • huggingface_id : str — The Hugging Face ID of the metric.

  • results_key : str — The name of the key used to extract the metric scores from the results dictionary.

  • compute_kwargs : dict[str, t.Any] | None — Keyword arguments to pass to the metric's compute function. Defaults to an empty dictionary.

  • postprocessing_fn : t.Callable[[float], tuple[float, str]] | None — A function to apply to the metric scores after they are computed, taking the score to the postprocessed score along with its string representation. Defaults to x -> (100 * x, f"{x:.2%}").

Methods

  • download — Initiates the download of the metric if needed.

source method HuggingFaceMetric.download(cache_dir: str) → HuggingFaceMetric

Initiates the download of the metric if needed.

Parameters

  • cache_dir : str — The directory where the metric will be downloaded to.

Returns