[mary-dev] WikipediaProcessor: Japanese Processing Exception

[Marc Schroeder]

Thanks for this update -- good to hear that more memory solves this 
problem. Of course it seems curious that 2GB of RAM should be required 
for running this code; if anyone would like to try and reduce the 
footprint, let me know.

Best,
Marc

Hind Abdul-Khaleq schrieb:
> The problem solved with "-Xmx2000m"   given the vm and without other 
> changes to the source .
> Thanks a lot and All the Best.
> 
> 
> 
>     --- On *Wed, 10/28/09, Hind Abdul-Khaleq /<habdolkhaleq at yahoo.com>/*
>     wrote:
> 
> 
>         From: Hind Abdul-Khaleq <habdolkhaleq at yahoo.com>
>         Subject: Re: [mary-dev] WikipediaProcessor: Japanese Processing
>         Exception
>         To: mary-dev at dfki.de
>         Date: Wednesday, October 28, 2009, 11:45 AM
> 
>         I'm getting this exception while processing Japanese
>         I changed the encoding to "EUC_JP" at the line
> 
>                     word = new String(wordBytes, "UTF8"); 
>         in 
>         marytts.tools.dbselection.DBHandler.getMostFrequentWords(DBHandler.java:1366)
> 
>         but it produced another exception at the next line:
>                  wordList.put(word, new Integer(rs.getInt(2)));
>         Exception in thread "main" java.lang.OutOfMemoryError: Java heap
>         space
>             at java.util.HashMap.resize(HashMap.java:462)
>             at java.util.HashMap.addEntry(HashMap.java:755)
>             at java.util.HashMap.put(HashMap.java:385)
>             at
>         marytts.tools.dbselection.DBHandler.getMostFrequentWords(DBHandler.java:1367)
>             at
>         marytts.tools.dbselection.WikipediaMarkupCleaner.updateWordList(WikipediaMarkupCleaner.java:953)
>             at
>         marytts.tools.dbselection.WikipediaMarkupCleaner.processWikipediaPages(WikipediaMarkupCleaner.java:1133)
>             at
>         marytts.tools.dbselection.WikipediaProcessor.main(WikipediaProcessor.java:368)
> 
>            
>         also I do "-Xmx1000m",... so what to do?
> 
>         --- On *Wed, 10/28/09, Hind Abdul-Khaleq
>         /<habdolkhaleq at yahoo.com>/* wrote:
> 
> 
>             From: Hind Abdul-Khaleq <habdolkhaleq at yahoo.com>
>             Subject: [mary-dev] WikipediaProcessor: Japanese Exception
>             To: mary-dev at dfki.de
>             Date: Wednesday, October 28, 2009, 11:34 AM
> 
>             Exception in thread "main" java.lang.OutOfMemoryError: GC
>             overhead limit exceeded
>                 at java.util.Arrays.copyOf(Arrays.java:2882)
>                 at java.lang.StringCoding.safeTrim(StringCoding.java:75)
>                 at java.lang.StringCoding.access$100(StringCoding.java:34)
>                 at
>             java.lang.StringCoding$StringDecoder.decode(StringCoding.java:151)
>                 at java.lang.StringCoding.decode(StringCoding.java:173)
>                 at java.lang.String.<init>(String.java:443)
>                 at java.lang.String.<init>(String.java:515)
>                 at
>             marytts.tools.dbselection.DBHandler.getMostFrequentWords(DBHandler.java:1366)
>                 at
>             marytts.tools.dbselection.WikipediaMarkupCleaner.updateWordList(WikipediaMarkupCleaner.java:953)
>                 at
>             marytts.tools.dbselection.WikipediaMarkupCleaner.processWikipediaPages(WikipediaMarkupCleaner.java:1133)
>                 at
>             marytts.tools.dbselection.WikipediaProcessor.main(WikipediaProcessor.java:368)
> 
> 
> 
>             -----Inline Attachment Follows-----
> 
>             _______________________________________________
>             Mary-dev mailing list
>             Mary-dev at dfki.de
>             http://www.dfki.de/mailman/cgi-bin/listinfo/mary-dev
> 
> 
> 
>         -----Inline Attachment Follows-----
> 
>         _______________________________________________
>         Mary-dev mailing list
>         Mary-dev at dfki.de
>         http://www.dfki.de/mailman/cgi-bin/listinfo/mary-dev
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Mary-dev mailing list
> Mary-dev at dfki.de
> http://www.dfki.de/mailman/cgi-bin/listinfo/mary-dev

-- 
Dr. Marc Schröder, Senior Researcher at DFKI GmbH
Coordinator EU FP7 Project SEMAINE http://www.semaine-project.eu
Portal Editor http://emotion-research.net
Team Leader DFKI Speech Group http://mary.dfki.de

Homepage: http://www.dfki.de/~schroed
Email: schroed at dfki.de
Phone: +49-681-302-5303
Postal address: DFKI GmbH, Campus D3_2, Stuhlsatzenhausweg 3, D-66123 
Saarbrücken, Germany
--
Official DFKI coordinates:
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313