euroeval.benchmark_config_factory

source module euroeval.benchmark_config_factory

Factory class for creating dataset configurations.

Functions

• build_benchmark_config — Create a benchmark configuration.

• get_correct_language_codes — Get correct language code(s).

• prepare_languages — Prepare language(s) for benchmarking.

• prepare_dataset_configs — Prepare dataset config(s) for benchmarking.

• prepare_device — Prepare device for benchmarking.

source build_benchmark_config(benchmark_config_params: BenchmarkConfigParams) → BenchmarkConfig

Create a benchmark configuration.

Parameters

• benchmark_config_params : BenchmarkConfigParams — The parameters for creating the benchmark configuration.

Returns

• BenchmarkConfig — The benchmark configuration.

source get_correct_language_codes(language_codes: str | c.Sequence[str]) → c.Sequence[str]

Get correct language code(s).

Parameters

• language_codes : str | c.Sequence[str] — The language codes of the languages to include, both for models and datasets. Here 'no' means both Bokmål (nb) and Nynorsk (nn). Set this to 'all' if all languages should be considered.

Returns

• c.Sequence[str] — The correct language codes.

source prepare_languages(language_codes: str | c.Sequence[str] | None, default_language_codes: c.Sequence[str]) → c.Sequence['Language']

Prepare language(s) for benchmarking.

Parameters

• language_codes : str | c.Sequence[str] | None — The language codes of the languages to include for models or datasets. If specified then this overrides the language parameter for model or dataset languages.

• default_language_codes : c.Sequence[str] — The default language codes of the languages to include.

Returns

• c.Sequence['Language'] — The prepared dataset languages.

source prepare_dataset_configs(task: str | Task | c.Sequence[str | Task] | None, languages: c.Sequence['Language'], dataset: str | DatasetConfig | c.Sequence[str | DatasetConfig] | None, custom_datasets_file: Path, api_key: str | None, cache_dir: Path, trust_remote_code: bool, run_with_cli: bool) → list['DatasetConfig']

Prepare dataset config(s) for benchmarking.

Parameters

• task : str | Task | c.Sequence[str | Task] | None — The tasks to include for dataset. If None then datasets will not be filtered based on their task.

• languages : c.Sequence['Language'] — The languages of the datasets in the benchmark.

• dataset : str | DatasetConfig | c.Sequence[str | DatasetConfig] | None — The datasets to include for task. If None then all datasets will be included, limited by the task and languages parameters.

• custom_datasets_file : Path — A path to a Python file containing custom dataset configurations.

• api_key : str | None — The API key to use for accessing the Hugging Face Hub.

• cache_dir : Path — The directory to store the cache in.

• trust_remote_code : bool — Whether to trust remote code.

• run_with_cli : bool — Whether to run the benchmark with the CLI.

Returns

• list['DatasetConfig'] — The prepared dataset configs.

source prepare_device(device: Device | None) → torch.device

Prepare device for benchmarking.

Parameters

• device : Device | None — The device to use for running the models. If None then the device will be set automatically.

Returns

• torch.device — The prepared device.