euroeval.benchmark_config_factory

source module euroeval.benchmark_config_factory

Factory class for creating dataset configurations.

Functions

• build_benchmark_config — Create a benchmark configuration.

• prepare_languages — Prepare language(s) for benchmarking.

• prepare_dataset_configs — Prepare dataset config(s) for benchmarking.

• prepare_device — Prepare device for benchmarking.

source build_benchmark_config(benchmark_config_params: BenchmarkConfigParams) → BenchmarkConfig

Create a benchmark configuration.

Parameters

• benchmark_config_params : BenchmarkConfigParams — The parameters for creating the benchmark configuration.

Returns

• BenchmarkConfig — The benchmark configuration.

source prepare_languages(language_codes: str | c.Sequence[str] | None, default_language_codes: c.Sequence[str]) → c.Sequence['Language']

Prepare language(s) for benchmarking.

Parameters

• language_codes : str | c.Sequence[str] | None — The language codes of the languages to include for models or datasets. If specified then this overrides the language parameter for model or dataset languages.

• default_language_codes : c.Sequence[str] — The default language codes of the languages to include.

Returns

• c.Sequence['Language'] — The prepared dataset languages.

source prepare_dataset_configs(task: str | Task | c.Sequence[str | Task] | None, languages: c.Sequence['Language'], dataset: str | DatasetConfig | c.Sequence[str | DatasetConfig] | None, custom_datasets_file: Path, api_key: str | None, cache_dir: Path, trust_remote_code: bool, run_with_cli: bool) → list['DatasetConfig']

Prepare dataset config(s) for benchmarking.

Parameters

• task : str | Task | c.Sequence[str | Task] | None — The tasks to include for dataset. If None then datasets will not be filtered based on their task.

• languages : c.Sequence['Language'] — The languages of the datasets in the benchmark.

• dataset : str | DatasetConfig | c.Sequence[str | DatasetConfig] | None — The datasets to include for task. If None then all datasets will be included, limited by the task and languages parameters.

• custom_datasets_file : Path — A path to a Python file containing custom dataset configurations.

• api_key : str | None — The API key to use for accessing the Hugging Face Hub.

• cache_dir : Path — The directory to store the cache in.

• trust_remote_code : bool — Whether to trust remote code.

• run_with_cli : bool — Whether to run the benchmark with the CLI.

Returns

• list['DatasetConfig'] — The prepared dataset configs.

source prepare_device(device: Device | None) → torch.device

Prepare device for benchmarking.

Parameters

• device : Device | None — The device to use for running the models. If None then the device will be set automatically.

Returns

• torch.device — The prepared device.