punctuator2/example · pu/punctuator2

552adb8b创建于 2018年6月8日历史提交

文件	最后提交记录	最后更新时间
README.md	Europarl score update for new tokenization. Links to game	9 年前
dont_run_me_run_the_other_script_instead.py	Python 3 compatibility: - Use io.open instead of codecs.open/codecs.getwriter - Use input() if raw_input is not available - Use _pickle if cPickle is not available - Use print_function from __future__	8 年前
run.sh	Europarl example	9 年前

This example produces the preprocessed Europarl English corpus that can be then used for training a model.

Requires nltk

Usage example: ./run.sh

cd ..

python data.py ./example/out

python main.py ep 256 0.02

python play_with_model.py Model_ep_h256_lr0.02.pcl

The input text to play_with_model.py should be similar to the contents of the preprocessed files in ./example/out (i.e. lowercased, numeric tokens replaced with ), but should not contain punctuation tokens.

Training time on this dataset with a Nvidia Tesla K20 GPU was about 15 hours (~3500 samples per second)