Python-Byte_Pair_Encoding

Byte Pair Encoding (BPE)

Env

• Python 3
• Numpy 1.15
• tqdm
• multiprocessing

Paper

• Byte-Pair Encoding (BPE): https://arxiv.org/abs/1508.07909

Command

• learn BPE from document

python bpe_learn.py 
	-train_path 1_document 2_document ... K_document
	-voca_out_path voca_path/voca_file_name
	-bpe_out_path 1_BPE_document 2_BPE_document ... K_BPE_document
	-train_voca_threshold 1 
	-num_merges 30000 
	-multi_proc=-1 (-1:use all process, 1:not use)
	-final_voca_size 30000 or -final_voca_threshold 50

• apply BPE to document

python bpe_apply.py
	-data_path 1_document 2_document ... K_document
	-voca_path voca_path/voca_file_name
	-bpe_out_path 1_BPE_document 2_BPE_document ... K_BPE_document

Reference

• https://lovit.github.io/nlp/2018/04/02/wpm/