Trees       Indices       Help   
Qualitative
Package sentence :: Module scoring :: Class Scoring
[hide private]
[frames] | no frames]

Class Scoring

source code

                       object --+        
                                |        
                  dataset.DataSet --+    
                                    |    
multirankeddataset.MultiRankedDataset --+
                                        |
                                       Scoring

classdocs

Instance Methods [hide private]
 
__init__(self, *params, **kwargs)
x.__init__(...) initializes x; see help(type(x)) for signature		 source code
 
get_systems_scoring_from_segment_ranks(self, rank_attribute_name)
Provides a performance score for every system.		 source code
 
get_spearman_correlation(self, rank_name_1, rank_name_2) source code
tuple(float, float)
get_metrics_scores(self, predicted_rank_name, original_rank_name, **kwargs)
Calculates a metric		 source code
 
get_kendall_tau_vector(self, rank_name_1, rank_name_2) source code
 
get_kendall_tau_avg(self, rank_name_1, rank_name_2) source code
 
get_kendall_tau_freq(self, rank_name_1, rank_name_2) source code
 
selectbest_accuracy(self, estimated_rank_name, original_rank_name) source code
 
normalize_rank_list(self, rank_list)
Normalizes a rank list so that it doesn't contain gaps.		 source code
 
best_predicted_vs_human(self, predicted_rank_name, original_rank_name) source code
 
mrr(self, predicted_rank_name, original_rank_name) source code
 
avg_predicted_ranked(self, predicted_rank_name, original_rank_name) source code
 
avg_first_ranked(self, predicted_rank_name, original_rank_name)
Provide an integer that shows the predicted rank of the best system It is averaged over all segments.		 source code
tuple(float, float)
get_kendall_tau(self, predicted_rank_name, original_rank_name, **kwargs)
Calculates average Kendall tau of predicted vs human ranking according to WMT12 (Birch et.		 source code
tuple(float, float)
get_kendall_tau_b(self, predicted_rank_name, original_rank_name)
Calculates Kendall tau beta of predicted vs human ranking according to the Knight (1966) [scipy implementation] taking account of ties		 source code

Inherited from dataset.DataSet: __eq__, __iter__, add_attribute_vector, append_dataset, clone, compare, confirm_attributes, ensure_judgment_ids, get_all_attribute_names, get_annotations, get_attribute_names, get_discrete_attribute_values, get_head_sentences, get_multisource_strings, get_nested_attribute_names, get_parallelsentences, get_parallelsentences_per_sentence_id, get_parallelsentences_with_judgment_ids, get_singlesource_strings, get_size, get_tail_sentences, get_target_strings, get_translations_count_vector, import_target_attributes_onsystem, merge_dataset, merge_dataset_symmetrical, merge_references_symmetrical, modify_singlesource_strings, modify_target_strings, remove_ties, select_attribute_names, split, write_singlesource_strings_file

Inherited from dataset.DataSet (private): _retrieve_attribute_names

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Instance Variables [hide private]

Inherited from dataset.DataSet: attribute_names, attribute_names_found, parallelsentences

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, *params, **kwargs)
(Constructor)

source code 

x.__init__(...) initializes x; see help(type(x)) for signature

Parameters:
  • parallelsentence_list - the parallelsentences to be wrapped in the dataset
  • attributes_list - if the names of the attributes for the parallelsentences are known, they can be given here, in order to avoid extra processing. Otherwise they will be computed when needed. @type [str, ...]
  • annotations - Not implemented @type list
Overrides: object.__init__
(inherited documentation)

get_systems_scoring_from_segment_ranks(self, rank_attribute_name)

source code 

Provides a performance score for every system. The score is the percentage of times the system performed better than all other systems or equally to the systems that performed better than all other systems

Parameters:
  • rank_attribute_name (string @return A map of the system names and their performance percentage) - the name of the target sentence attribute which contains the rank value, that we compare upon Smaller rank means better system.

get_metrics_scores(self, predicted_rank_name, original_rank_name, **kwargs)

source code 

Calculates a metric

Parameters:
  • predicted_rank_name (str) - the name of the attribute containing the predicted rank
  • original_rank_name (str) - the name of the attribute containing the human rank
  • filter_ref (boolean) - don't include reference sentences when existing in the pairs
  • exclude_ties (boolean) - don't include human ties in the calculation, even if correctly predicted
Returns: tuple(float, float)
the Kendall tau score and the probability for the null hypothesis of X and Y being independent

normalize_rank_list(self, rank_list)

source code 

Normalizes a rank list so that it doesn't contain gaps. E.g [1,3,3,4] will be converted to [1,2,2,3]

avg_first_ranked(self, predicted_rank_name, original_rank_name)

source code 

Provide an integer that shows the predicted rank of the best system It is averaged over all segments. Tied predictions are penalized

get_kendall_tau(self, predicted_rank_name, original_rank_name, **kwargs)

source code 

Calculates average Kendall tau of predicted vs human ranking according to WMT12 (Birch et. al 2012)

Parameters:
  • predicted_rank_name (str) - the name of the attribute containing the predicted rank
  • original_rank_name (str) - the name of the attribute containing the human rank
  • filter_ref (boolean) - don't include reference sentences when existing in the pairs
  • exclude_ties (boolean) - don't include human ties in the calculation, even if correctly predicted
Returns: tuple(float, float)
the Kendall tau score and the probability for the null hypothesis of X and Y being independent

get_kendall_tau_b(self, predicted_rank_name, original_rank_name)

source code 

Calculates Kendall tau beta of predicted vs human ranking according to the Knight (1966) [scipy implementation] taking account of ties

Parameters:
  • predicted_rank_name (str) - the name of the attribute containing the predicted rank
  • original_rank_name (str) - the name of the attribute containing the human rank
Returns: tuple(float, float)
the Kendall tau score and the probability for the null hypothesis of X and Y being independent

   Trees       Indices       Help   
Qualitative
Generated by Epydoc 3.0.1 on Fri Jul 18 11:46:13 2014 http://epydoc.sourceforge.net