tokenizer.py script can be used to train the vocabulary from the data that is going to be used to train BERT & HAN.
Here is a detailed description:
1.frequent_bichar functions generates bigrams that maximizes the likelihood of occurence in the corpus.
2. decode function uses the vocabulary which is generated by the frequent_bichar function in order to tokenize the word.
The script expects three user arguments:
-complete path to the .txt file containing the data
-maximum number of words that the dictionary should have.
-minimum number of frequency that a bigram should have.
python tokenizer.py "sample.txt" 100 7