Package featuregenerator :: Module preprocessor :: Class Tokenizer

Class Tokenizer

                       object --+            
                                |            
featuregenerator.FeatureGenerator --+        
                                    |        
                         Preprocessor --+    
                                        |    
                  CommandlinePreprocessor --+
                                            |
                                           Tokenizer

Instance Methods

[hide private]

__init__(self, lang)
x.__init__(...) initializes x; see help(type(x)) for signature

source code

Inherited from CommandlinePreprocessor: __del__, close, process_string

Inherited from CommandlinePreprocessor (private): _enqueue_output, _get_temporary_file, _get_tool_output

Inherited from Preprocessor: add_features_src, add_features_tgt

Inherited from featuregenerator.FeatureGenerator: add_features_batch, add_features_dataset, add_features_parallelsentence, add_features_simplesentence, get_annotation_name, get_features_parallelsentence, get_features_simplesentence, get_features_src, get_features_string, get_features_tgt, process_dataset

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

init(self, lang)
(Constructor)

source code

x.__init__(...) initializes x; see help(type(x)) for signature

Overrides: object.__init__: (inherited documentation)

Class Tokenizer

__init__(self, lang) (Constructor)

init(self, lang)
(Constructor)