euroeval.exceptions

source module euroeval.exceptions

Exceptions to used by other functions.

Classes

• InvalidBenchmark — The (model, dataset) combination cannot be benchmarked.

• InvalidModel — The model cannot be benchmarked on any datasets.

• HuggingFaceHubDown — The Hugging Face Hub seems to be down.

• NoInternetConnection — There seems to be no internet connection.

• NaNValueInModelOutput — There is a NaN value in the model output.

• FlashAttentionNotInstalled — The flash-attn package has not been installed.

• NeedsExtraInstalled — The evaluation requires extra to be installed.

• NeedsManualDependency — The evaluation requires a dependency to be manually installed.

• NeedsAdditionalArgument — The evaluation requires additional arguments to the euroeval command.

• NeedsEnvironmentVariable — The evaluation requires an environment variable to be set.

source class InvalidBenchmark(message: str = 'This model cannot be benchmarked on the given dataset.')

Bases : Exception

The (model, dataset) combination cannot be benchmarked.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class InvalidModel(message: str = 'The model cannot be benchmarked on any datasets.')

Bases : Exception

The model cannot be benchmarked on any datasets.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class HuggingFaceHubDown(message: str = 'The Hugging Face Hub is currently down.')

Bases : Exception

The Hugging Face Hub seems to be down.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NoInternetConnection(message: str = 'There is currently no internet connection.')

Bases : Exception

There seems to be no internet connection.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NaNValueInModelOutput(message: str = 'There is a NaN value in the model output.')

Bases : Exception

There is a NaN value in the model output.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class FlashAttentionNotInstalled(message: str = 'The model you are trying to load requires Flash Attention. To use Flash Attention, please install the flash-attn package, which can be done by running pip install -U wheel && FLASH_ATTENTION_SKIP_CUDA_BUILD=TRUE pip install flash-attn --no-build-isolation.')

Bases : Exception

The flash-attn package has not been installed.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NeedsExtraInstalled(extra: str)

Bases : InvalidModel

The evaluation requires extra to be installed.

Initialize the exception.

Parameters

• extra : str —

  The extra that needs to be installed.

source class NeedsManualDependency(package: str)

Bases : InvalidModel

The evaluation requires a dependency to be manually installed.

Initialize the exception.

Parameters

• package : str —

  The package that needs to be manually installed.

source class NeedsAdditionalArgument(cli_argument: str, script_argument: str, run_with_cli: bool)

Bases : InvalidModel

The evaluation requires additional arguments to the euroeval command.

Initialize the exception.

Parameters

• cli_argument : str —

  The argument that needs to be passed to the euroeval command.

• script_argument : str —

  The argument that needs to be passed to the Benchmarker class.

• run_with_cli : bool —

  Whether the benchmark is being run with the CLI.

source class NeedsEnvironmentVariable(env_var: str)

Bases : InvalidModel

The evaluation requires an environment variable to be set.

Initialize the exception.

Parameters

• env_var : str —

  The environment variable that needs to be set.