Package evaluation :: Package ranking :: Module set

Module set

This module allows for the calculation of the basic rank metrics that evaluate on a segment level (i.e. one ranking list at a time)

Created on 18 Dec 2012

Author: Eleftherios Avramidis

Functions

[hide private]

kendall_tau_set_no_ties(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

{string:float, string:float, string:int, string:int, string:int, string:int, string:int, string:int}

kendall_tau_set(predicted_rank_vectors, original_rank_vectors, **kwargs)
This is the refined calculation of set-level Kendall tau of predicted vs human ranking according to WMT12 (Birch et.

source code

{string, float}

mrr(predicted_rank_vectors, original_rank_vectors, **kwargs)
Calculation of mean reciprocal rank based on Radev et.

source code

{string, float}

best_predicted_vs_human(predicted_rank_vectors, original_rank_vectors)
For each sentence, the item selected as best by our system, may have been ranked lower by the humans.

source code

{string, float}

avg_predicted_ranked(predicted_rank_vectors, original_rank_vectors, **kwargs)
It will provide the average human rank of the item chosen by the system as best

source code

{string, float}

avg_ndgc_err(predicted_rank_vectors, original_rank_vectors, **kwargs)
Returns normalize Discounted Cumulative Gain and the Expected Reciprocal Rank, both averaged over number of sentences

source code

allmetrics(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

Variables

[hide private]

__package__ = 'evaluation.ranking'

Function Details

[hide private]

kendall_tau_set(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

This is the refined calculation of set-level Kendall tau of predicted vs human ranking according to WMT12 (Birch et. al 2012) It returns both set-level Kendall tau and average segment-level Kendall tau

Parameters:

predicted_rank_vectors ([Ranking, ..]) - a list of lists containing integers representing the predicted ranks, one ranking for each segment
original_rank_vectors ([Ranking, ..]) - a list of the names of the attribute containing the human rank, one ranking for each segment

Returns: {string:float, string:float, string:int, string:int, string:int, string:int, string:int, string:int}

overall Kendall tau score,

average segment Kendall tau score,
the probability for the null hypothesis of X and Y being independent
the count of concordant pairs,
the count of discordant pairs,
the count of pairs used for calculating tau (excluding "invalid" pairs)
the count of original ties,
the count of predicted ties,
the count of all pairs

mrr(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

Calculation of mean reciprocal rank based on Radev et. all (2002)

Parameters:

predicted_rank_vectors ([Ranking, ..]) - a list of lists containing integers representing the predicted ranks, one ranking for each segment
original_rank_vectors ([Ranking, ..]) - a list of the names of the attribute containing the human rank, one ranking for each segment

Returns: {string, float}

mean reciprocal rank

best_predicted_vs_human(predicted_rank_vectors, original_rank_vectors)

source code

For each sentence, the item selected as best by our system, may have been ranked lower by the humans. This statistic counts how many times the item predicted as best has fallen into each of the human ranks. This is useful for plotting.

Parameters:

predicted_rank_vectors ([Ranking, ..]) - a list of lists containing integers representing the predicted ranks, one ranking for each segment
original_rank_vectors ([Ranking, ..]) - a list of the names of the attribute containing the human rank, one ranking for each segment

Returns: {string, float}

a dictionary with percentages for each human rank

avg_predicted_ranked(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

It will provide the average human rank of the item chosen by the system as best

Parameters:

predicted_rank_vectors ([Ranking, ..]) - a list of lists containing integers representing the predicted ranks, one ranking for each segment
original_rank_vectors ([Ranking, ..]) - a list of the names of the attribute containing the human rank, one ranking for each segment

Returns: {string, float}

a dictionary with the name of the metric and its value

avg_ndgc_err(predicted_rank_vectors, original_rank_vectors, **kwargs)

source code

Returns normalize Discounted Cumulative Gain and the Expected Reciprocal Rank, both averaged over number of sentences

Parameters:

predicted_rank_vectors ([Ranking, ..]) - a list of lists containing integers representing the predicted ranks, one ranking for each segment
original_rank_vectors ([Ranking, ..]) - a list of the names of the attribute containing the human rank, one ranking for each segment
k (int) - cut-off passed to the segment ndgc_err function

Returns: {string, float}

a dictionary with the name of the metric and the respective result