NLP_Metrics
Install extra dependencies to compute LLM evaluation metrics
Install extra dependencies in the SDK:
Import metrics from arize.pandas.generative.nlp_metrics
Metrics
bleu
bleu
BLEU score is typically used to evaluate the quality of machine-translated text from one natural language to another. BLEU calculates scores for individual translated segments, typically sentences, by comparing them to a set of high-quality reference translations. These scores are then averaged over the entire corpus to obtain an estimate of the overall quality of the translation.
Argument | Expected Type | Description |
---|---|---|
| pd.Series | [Required] Pandas Series containing translations (as strings) to score |
| pd.Series | [Required] Pandas Series containing a reference, or list of several references, per prediction |
| int | [Optional] Maximum n-gram order to use when computing BLEU score. Defaults to |
| bool | [Optional] Whether or not to apply Lin et al. 2004 smoothing. Defaults to |
Code Example
sacre_bleu
sacre_bleu
A hassle-free computation of shareable, comparable, and reproducible BLEU scores. Inspired by Rico Sennrich’s multi-bleu-detok.perl, it produces the official Workshop on Machine Translation (WMT) scores but works with plain text.
Argument | Expected Type | Description |
---|---|---|
| pd.Series | [Required] Pandas Series containing translations (as strings) to score |
| pd.Series | [Required] Pandas Series containing a reference, or list of several references, per prediction Note: There must be the same number of references for each prediction (i.e. all sub-lists must be of the same length) |
| str | [Optional] The smoothing method to use, defaults to |
| float | [Optional] The smoothing value. Defaults to |
| bool | [Optional] If True, lowercases the input, enabling case-insensitivity. Defaults to |
| bool | [Optional] If True, insists that your tokenized input is actually de-tokenized. Defaults to |
| bool | [Optional] If True, stops including n-gram orders for which precision is 0. This should be True, if sentence-level BLEU will be computed. Defaults to |
Code Example
google_bleu
google_bleu
BLEU score is typically used as a corpus measure, and it has some limitations when applied to single sentences. To overcome this issue in RL experiments, there exists a variation called the GLEU score. The GLEU score is the minimum of recall and precision.
Argument | Expected Type | Description |
---|---|---|
| pd.Series | Pandas Series containing translations (as strings) to score |
| pd.Series | Pandas Series containing a reference, or list of several references, for each translation |
| int | The minimum order of n-gram this function should extract. Defaults to |
| int | The maximum order of n-gram this function should extract. Defaults to |
Code Example
rouge
rouge
A software package and a set of metrics commonly used to evaluate machine translation and automatic summarization software in natural language processing. These metrics involve comparing a machine-produced summary or translation with a reference or set of references that have been human-produced.
Argument | Expected Type | Description |
---|---|---|
| pd.Series | [Required] Pandas Series containing predictions (as strings) to score |
| pd.Series | [Required] Pandas Series containing a reference, or list of several references, per prediction |
| List[str] | [Optional] A list of rouge types to calculate. Defaults to - |
| bool | [Optional] If True, uses Porter stemmer to strip word suffixes. Defaults to |
Code Example
meteor
meteor
An automatic metric typically used to evaluate machine translation, which is based on a generalized concept of unigram matching between the machine-produced translation and the reference human-produced translations.
Argument | Expected Type | Description |
---|---|---|
| pd.Series | [Required] Pandas Series containing predictions (as strings) to score |
| pd.Series | [Required] Pandas Series containing a reference, or list of several references, per prediction |
| float | [Optional] Parameter for controlling relative weights of precision and recall. Default is |
| float | [Optional] Parameter for controlling shape of penalty as a function of fragmentation. Default is |
| float | [Optional] The relative weight assigned to fragmentation penalty. Default is |
Code Example
Last updated