euroeval.metrics.ifeval.constraints¶

source module euroeval.metrics.ifeval.constraints

Constraint functions used for the IFEval metric.

Classes

Constraint — An instruction-following constraint.

Functions

register — Decorator that registers a function under the given name.
check_keyword_existence — Check that all keywords exist in the response.
check_keyword_frequency — Check keyword appears with required frequency.
check_forbidden_words — Check that forbidden words don't appear.
check_letter_frequency — Check letter appears with required frequency.
check_number_sentences — Check number of sentences.
check_number_paragraphs — Check number of paragraphs (separated by ***).
check_number_words — Check number of words.
check_nth_paragraph_first_word — Check paragraph count and first word of nth paragraph.
check_number_placeholders — Check minimum number of [placeholder] brackets.
check_postscript — Check for postscript marker.
check_number_bullet_lists — Check exact number of bullet points.
check_constrained_response_english — Check response contains one of the constrained options.
check_constrained_response_spanish — Check response contains one of the constrained options.
check_constrained_response_catalan — Check response contains one of the constrained options.
check_constrained_response_french — Check response contains one of the constrained options.
check_constrained_response_with_argument — Check response contains one of the constrained options.
check_number_highlighted_sections — Check minimum highlighted sections.
check_multiple_sections — Check for Section X markers.
check_json_format — Check response is valid JSON.
check_title — Check for <> format. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_two_responses" title="euroeval.metrics.ifeval.constraints.check_two_responses">check_two_responses</a> — Check for two different responses separated by **. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_repeat_prompt" title="euroeval.metrics.ifeval.constraints.check_repeat_prompt">check_repeat_prompt</a> — Check response starts with the prompt. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_end_phrase" title="euroeval.metrics.ifeval.constraints.check_end_phrase">check_end_phrase</a> — Check response ends with exact phrase. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_capital_word_frequency" title="euroeval.metrics.ifeval.constraints.check_capital_word_frequency">check_capital_word_frequency</a> — Check frequency of ALL CAPS words. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_english_capital" title="euroeval.metrics.ifeval.constraints.check_english_capital">check_english_capital</a> — Check response is English and all caps. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_spanish_capital" title="euroeval.metrics.ifeval.constraints.check_spanish_capital">check_spanish_capital</a> — Check response is Spanish and all caps. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_french_capital" title="euroeval.metrics.ifeval.constraints.check_french_capital">check_french_capital</a> — Check response is French and all caps. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_catalan_capital" title="euroeval.metrics.ifeval.constraints.check_catalan_capital">check_catalan_capital</a> — Check response is Catalan and all caps. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_english_lowercase" title="euroeval.metrics.ifeval.constraints.check_english_lowercase">check_english_lowercase</a> — Check response is English and all lowercase. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_spanish_lowercase" title="euroeval.metrics.ifeval.constraints.check_spanish_lowercase">check_spanish_lowercase</a> — Check response is Spanish and all lowercase. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_french_lowercase" title="euroeval.metrics.ifeval.constraints.check_french_lowercase">check_french_lowercase</a> — Check response is French and all lowercase. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_catalan_lowercase" title="euroeval.metrics.ifeval.constraints.check_catalan_lowercase">check_catalan_lowercase</a> — Check response is Catalan and all lowercase. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_no_comma" title="euroeval.metrics.ifeval.constraints.check_no_comma">check_no_comma</a> — Check response has no commas. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_quotation" title="euroeval.metrics.ifeval.constraints.check_quotation">check_quotation</a> — Check response is wrapped in double quotes. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_response_language" title="euroeval.metrics.ifeval.constraints.check_response_language">check_response_language</a> — Check response is in specified language. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_lowercase_letters" title="euroeval.metrics.ifeval.constraints.check_lowercase_letters">check_lowercase_letters</a> — Check response has no uppercase letters. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_capital_letters" title="euroeval.metrics.ifeval.constraints.check_capital_letters">check_capital_letters</a> — Check response has no lowercase letters. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_no_digits" title="euroeval.metrics.ifeval.constraints.check_no_digits">check_no_digits</a> — Check response contains no digits. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_ethel_or_cedilla_not_present" title="euroeval.metrics.ifeval.constraints.check_ethel_or_cedilla_not_present">check_ethel_or_cedilla_not_present</a> — Check response contains no forbidden character. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_no_accents" title="euroeval.metrics.ifeval.constraints.check_no_accents">check_no_accents</a> — Check response contains no accents. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_accents" title="euroeval.metrics.ifeval.constraints.check_accents">check_accents</a> — Check response contains accents. </li> <li> <a href="./#euroeval.metrics.ifeval.constraints.check_informal_address" title="euroeval.metrics.ifeval.constraints.check_informal_address">check_informal_address</a> — Check response contains informal address. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.Constraint" title="Go to source">source</a> class <a href="./#euroeval.metrics.ifeval.constraints.Constraint" title="euroeval.metrics.ifeval.constraints.Constraint">Constraint</a>() <div class="mkapi-document"> Bases : t.Protocol An instruction-following constraint. </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.register" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.register" title="euroeval.metrics.ifeval.constraints.register">register</a>(name: str, **desired_keys_and_types) → c.Callable[[<a href="./#euroeval.metrics.ifeval.constraints.Constraint" title="euroeval.metrics.ifeval.constraints.Constraint">Constraint</a>], <a href="./#euroeval.metrics.ifeval.constraints.Constraint" title="euroeval.metrics.ifeval.constraints.Constraint">Constraint</a>] <div class="mkapi-document"> Decorator that registers a function under the given name. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> name : str — The name under which to register the function. </li> <li> **desired_keys_and_types — The keyword arguments and their types that the function expects. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> c.Callable[[<a href="./#euroeval.metrics.ifeval.constraints.Constraint" title="euroeval.metrics.ifeval.constraints.Constraint">Constraint</a>], <a href="./#euroeval.metrics.ifeval.constraints.Constraint" title="euroeval.metrics.ifeval.constraints.Constraint">Constraint</a>] — The decorator function. </li> </ul> </div> Raises <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> <a href="../../../exceptions/#euroeval.exceptions.InvalidBenchmark" title="euroeval.exceptions.InvalidBenchmark">InvalidBenchmark</a> </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_keyword_existence" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_keyword_existence" title="euroeval.metrics.ifeval.constraints.check_keyword_existence">check_keyword_existence</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check that all keywords exist in the response. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to be checked. </li> <li> **constraint_kwargs — Keyword arguments containing <code>keywords</code> – a list of keyword patterns (case‑insensitive) to search for. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if all keywords are found in the response, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_keyword_frequency" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_keyword_frequency" title="euroeval.metrics.ifeval.constraints.check_keyword_frequency">check_keyword_frequency</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check keyword appears with required frequency. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>keyword</code>, <code>frequency</code> and <code>relation</code> – the keyword pattern (case‑insensitive), the required frequency, and the comparison operator ("less than" or "at least"). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the keyword appears with the required frequency, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_forbidden_words" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_forbidden_words" title="euroeval.metrics.ifeval.constraints.check_forbidden_words">check_forbidden_words</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check that forbidden words don't appear. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>forbidden_words</code> – a list of words that must not appear (case‑insensitive, whole‑word match). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if none of the forbidden words are found, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_letter_frequency" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_letter_frequency" title="euroeval.metrics.ifeval.constraints.check_letter_frequency">check_letter_frequency</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check letter appears with required frequency. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>letter</code>, <code>let_frequency</code> and <code>let_relation</code> – the single character to count (case‑insensitive), the frequency threshold, and the comparison operator ("less than" or "at least"). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the letter frequency satisfies the relation, False otherwise. </li> </ul> </div> Raises <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> InvalidBenchmark — If letter is not a single character. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_sentences" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_sentences" title="euroeval.metrics.ifeval.constraints.check_number_sentences">check_number_sentences</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check number of sentences. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_sentences</code> and <code>relation</code>. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the sentence count satisfies the relation, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_paragraphs" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_paragraphs" title="euroeval.metrics.ifeval.constraints.check_number_paragraphs">check_number_paragraphs</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check number of paragraphs (separated by ***). Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_paragraphs</code> – the exact number of paragraphs expected. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly num_paragraphs non‑empty paragraphs, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_words" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_words" title="euroeval.metrics.ifeval.constraints.check_number_words">check_number_words</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check number of words. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_words</code> and <code>relation</code>. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the word count satisfies the relation, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_nth_paragraph_first_word" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_nth_paragraph_first_word" title="euroeval.metrics.ifeval.constraints.check_nth_paragraph_first_word">check_nth_paragraph_first_word</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check paragraph count and first word of nth paragraph. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_paragraphs</code>, <code>nth_paragraph</code>, and <code>first_word</code> – the expected first word of the nth paragraph (case‑insensitive). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response has exactly num_paragraphs paragraphs and the nth paragraph starts with first_word, False otherwise. </li> </ul> </div> Raises <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> InvalidBenchmark — If the n'th paragraph is greater than the number of paragraphs in the response. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_placeholders" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_placeholders" title="euroeval.metrics.ifeval.constraints.check_number_placeholders">check_number_placeholders</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check minimum number of [placeholder] brackets. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_placeholders</code> – the minimum number of placeholder brackets expected. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains at least num_placeholders placeholders of the form [placeholder], False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_postscript" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_postscript" title="euroeval.metrics.ifeval.constraints.check_postscript">check_postscript</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check for postscript marker. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>postscript_marker</code> – the postscript label to look for (e.g. "P.S.", "P.P.S"). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the postscript marker is found in the response, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_bullet_lists" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_bullet_lists" title="euroeval.metrics.ifeval.constraints.check_number_bullet_lists">check_number_bullet_lists</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check exact number of bullet points. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_bullets</code> – the exact number of bullet points expected. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly num_bullets bullet points, where bullet points are lines starting with <code>*</code> or <code>-</code>, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_constrained_response_english" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_constrained_response_english" title="euroeval.metrics.ifeval.constraints.check_constrained_response_english">check_constrained_response_english</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains one of the constrained options. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly one of "My answer is yes.", "My answer is no.", or "My answer is maybe.", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_constrained_response_spanish" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_constrained_response_spanish" title="euroeval.metrics.ifeval.constraints.check_constrained_response_spanish">check_constrained_response_spanish</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains one of the constrained options. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly one of "Mi respuesta es sí" or "Mi respuesta es no" or "Mi respuesta es tal vez", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_constrained_response_catalan" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_constrained_response_catalan" title="euroeval.metrics.ifeval.constraints.check_constrained_response_catalan">check_constrained_response_catalan</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains one of the constrained options. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly one of "La meva resposta és sí.", "La meva resposta és no.", or "La meva resposta és potser.", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_constrained_response_french" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_constrained_response_french" title="euroeval.metrics.ifeval.constraints.check_constrained_response_french">check_constrained_response_french</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains one of the constrained options. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly one of "Oui.", "Non.", or "Peut-être.", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_constrained_response_with_argument" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_constrained_response_with_argument" title="euroeval.metrics.ifeval.constraints.check_constrained_response_with_argument">check_constrained_response_with_argument</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check response contains one of the constrained options. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>options</code> – a list of strings to check for. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly one of the options, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_number_highlighted_sections" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_number_highlighted_sections" title="euroeval.metrics.ifeval.constraints.check_number_highlighted_sections">check_number_highlighted_sections</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check minimum highlighted sections. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>num_highlights</code> – the minimum number of highlighted sections expected. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains at least num_highlights non‑empty highlighted sections, where highlights are text wrapped in single (text) or double (text) asterisks, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_multiple_sections" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_multiple_sections" title="euroeval.metrics.ifeval.constraints.check_multiple_sections">check_multiple_sections</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check for Section X markers. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>section_spliter</code> and <code>num_sections</code>. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains at least num_sections sections delimited by markers of the form "<section_spliter> <number>", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_json_format" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_json_format" title="euroeval.metrics.ifeval.constraints.check_json_format">check_json_format</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is valid JSON. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response (after stripping optional <code>json</code> fences) is valid JSON, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_title" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_title" title="euroeval.metrics.ifeval.constraints.check_title">check_title</a>(response: str, **_) → bool <div class="mkapi-document"> Check for <<title>> format. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains at least one non‑empty title wrapped in double angle brackets (e.g. <<My Title>>), False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_two_responses" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_two_responses" title="euroeval.metrics.ifeval.constraints.check_two_responses">check_two_responses</a>(response: str, **_) → bool <div class="mkapi-document"> Check for two different responses separated by **. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains exactly two non‑empty, non‑identical sections separated by "**", False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_repeat_prompt" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_repeat_prompt" title="euroeval.metrics.ifeval.constraints.check_repeat_prompt">check_repeat_prompt</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check response starts with the prompt. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>prompt_to_repeat</code> – the exact prompt text the response must begin with (case‑insensitive). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response starts with prompt_to_repeat, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_end_phrase" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_end_phrase" title="euroeval.metrics.ifeval.constraints.check_end_phrase">check_end_phrase</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check response ends with exact phrase. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>end_phrase</code> – the exact phrase the response must end with (case‑insensitive). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response ends with end_phrase, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_capital_word_frequency" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_capital_word_frequency" title="euroeval.metrics.ifeval.constraints.check_capital_word_frequency">check_capital_word_frequency</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check frequency of ALL CAPS words. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>capital_frequency</code> and <code>capital_relation</code>. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the count of fully uppercased words satisfies the relation, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_english_capital" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_english_capital" title="euroeval.metrics.ifeval.constraints.check_english_capital">check_english_capital</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is English and all caps. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely uppercase and detected as English, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_spanish_capital" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_spanish_capital" title="euroeval.metrics.ifeval.constraints.check_spanish_capital">check_spanish_capital</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is Spanish and all caps. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely uppercase and detected as Spanish, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_french_capital" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_french_capital" title="euroeval.metrics.ifeval.constraints.check_french_capital">check_french_capital</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is French and all caps. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely uppercase and detected as French, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_catalan_capital" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_catalan_capital" title="euroeval.metrics.ifeval.constraints.check_catalan_capital">check_catalan_capital</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is Catalan and all caps. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely uppercase and detected as Catalan, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_english_lowercase" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_english_lowercase" title="euroeval.metrics.ifeval.constraints.check_english_lowercase">check_english_lowercase</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is English and all lowercase. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely lowercase and detected as English, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_spanish_lowercase" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_spanish_lowercase" title="euroeval.metrics.ifeval.constraints.check_spanish_lowercase">check_spanish_lowercase</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is Spanish and all lowercase. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely lowercase and detected as Spanish, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_french_lowercase" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_french_lowercase" title="euroeval.metrics.ifeval.constraints.check_french_lowercase">check_french_lowercase</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is French and all lowercase. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely lowercase and detected as French, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_catalan_lowercase" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_catalan_lowercase" title="euroeval.metrics.ifeval.constraints.check_catalan_lowercase">check_catalan_lowercase</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is Catalan and all lowercase. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is entirely lowercase and detected as Catalan, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_no_comma" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_no_comma" title="euroeval.metrics.ifeval.constraints.check_no_comma">check_no_comma</a>(response: str, **_) → bool <div class="mkapi-document"> Check response has no commas. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains no comma characters, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_quotation" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_quotation" title="euroeval.metrics.ifeval.constraints.check_quotation">check_quotation</a>(response: str, **_) → bool <div class="mkapi-document"> Check response is wrapped in double quotes. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response (after stripping whitespace) begins and ends with a double quote character, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_response_language" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_response_language" title="euroeval.metrics.ifeval.constraints.check_response_language">check_response_language</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check response is in specified language. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>language</code> – the language code of the language the response must be in. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response is detected as the specified language, False otherwise. Returns True if language detection fails. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_lowercase_letters" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_lowercase_letters" title="euroeval.metrics.ifeval.constraints.check_lowercase_letters">check_lowercase_letters</a>(response: str, **_) → bool <div class="mkapi-document"> Check response has no uppercase letters. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains no uppercase letters, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_capital_letters" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_capital_letters" title="euroeval.metrics.ifeval.constraints.check_capital_letters">check_capital_letters</a>(response: str, **_) → bool <div class="mkapi-document"> Check response has no lowercase letters. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains no lowercase letters, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_no_digits" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_no_digits" title="euroeval.metrics.ifeval.constraints.check_no_digits">check_no_digits</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains no digits. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains no digits, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_ethel_or_cedilla_not_present" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_ethel_or_cedilla_not_present" title="euroeval.metrics.ifeval.constraints.check_ethel_or_cedilla_not_present">check_ethel_or_cedilla_not_present</a>(response: str, **constraint_kwargs) → bool <div class="mkapi-document"> Check response contains no forbidden character. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> <li> **constraint_kwargs — Keyword arguments containing <code>forbidden_char</code> – the character that is forbidden in the response (must not be present). </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the forbidden character is not present in the response, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_no_accents" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_no_accents" title="euroeval.metrics.ifeval.constraints.check_no_accents">check_no_accents</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains no accents. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains no accents, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_accents" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_accents" title="euroeval.metrics.ifeval.constraints.check_accents">check_accents</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains accents. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains accents, False otherwise. </li> </ul> </div> </div> <a href="../../../../../src/euroeval/metrics/ifeval/constraints/#euroeval.metrics.ifeval.constraints.check_informal_address" title="Go to source">source</a> <a href="./#euroeval.metrics.ifeval.constraints.check_informal_address" title="euroeval.metrics.ifeval.constraints.check_informal_address">check_informal_address</a>(response: str, **_) → bool <div class="mkapi-document"> Check response contains informal address. Parameters <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> response : str — The response string to check. </li> </ul> </div> Returns <div class="mkapi-section-content"> <ul class="mkapi-item-list"> <li> bool — True if the response contains informal address, False otherwise. </li> </ul> </div> </div> </article> </div> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type="button" class="md-top md-icon" data-md-component="top" hidden> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class="md-footer"> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class="md-copyright"> Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"> Material for MkDocs </a> </div> </div> </div> </footer> </div> <div class="md-dialog" data-md-component="dialog"> <div class="md-dialog__inner md-typeset"></div> </div> <div class="md-progress" data-md-component="progress" role="progressbar"></div> <script id="__config" type="application/json">{"annotate": null, "base": "../../../../..", "features": ["navigation.top", "navigation.path", "navigation.prune", "navigation.indexes", "navigation.tabs", "navigation.tabs.sticky", "navigation.instant", "navigation.instant.prefetch", "navigation.instant.progress", "navigation.tracking", "navigation.sections", "content.code.copy", "content.tooltips", "toc.follow"], "search": "../../../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script src="../../../../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../../../javascript/mkapi.js"></script> </body> </html>