euroeval.enums

source module euroeval.enums

Enums used in the project.

Classes

• AutoStrEnum — StrEnum where auto() returns the field name in lower case.

• Device — The compute device to use for the evaluation.

• InferenceBackend — The backend used for model inference.

• ModelType — The type of a model.

• GenerativeType — The type of a generative model.

• DataType — The data type of the model weights.

• BatchingPreference — The preference for batching.

• TaskGroup — The overall task group of a task.

source enum AutoStrEnum(*args, **kwds)

Bases : str, Enum

StrEnum where auto() returns the field name in lower case.

source enum Device(*args, **kwds)

Bases : AutoStrEnum

The compute device to use for the evaluation.

Attributes

• CPU — CPU device.

• MPS — MPS GPU, used in M-series MacBooks.

• CUDA — CUDA GPU, used with NVIDIA GPUs.

source enum InferenceBackend(*args, **kwds)

Bases : AutoStrEnum

The backend used for model inference.

Attributes

• TRANSFORMERS — Hugging Face transformers library.

• VLLM — VLLM library.

• LITELLM — LiteLLM library.

• NONE — No inference backend used (e.g., for human evaluation).

source enum ModelType(*args, **kwds)

Bases : AutoStrEnum

The type of a model.

Attributes

• ENCODER — An encoder (i.e., BERT-style) model.

• GENERATIVE — A generative model. Can be either decoder or encoder-decoder (aka seq2seq).

• HUMAN — Human evaluator.

source enum GenerativeType(*args, **kwds)

Bases : AutoStrEnum

The type of a generative model.

Attributes

• BASE — A base (i.e., pretrained) generative model.

• INSTRUCTION_TUNED — An instruction-tuned generative model.

• REASONING — A generative reasoning model.

source enum DataType(*args, **kwds)

Bases : AutoStrEnum

The data type of the model weights.

Attributes

• FP32 — 32-bit floating point.

• FP16 — 16-bit floating point.

• BF16 — 16-bit bfloat.

source enum BatchingPreference(*args, **kwds)

Bases : AutoStrEnum

The preference for batching.

Attributes

• NO_PREFERENCE — No preference for batching.

• SINGLE_SAMPLE — Single sample batching.

• ALL_AT_ONCE — All samples at once batching.

source enum TaskGroup(*args, **kwds)

Bases : AutoStrEnum

The overall task group of a task.

Attributes

• SEQUENCE_CLASSIFICATION — Classification of documents.

• MULTIPLE_CHOICE_CLASSIFICATION — Classification of documents with multiple-choice options.

• TOKEN_CLASSIFICATION — Token-level classification.

• QUESTION_ANSWERING — Extractive question answering.

• TEXT_TO_TEXT — Text-to-text generation.

• SPEED — Speed benchmark.