Biterm Topic Model (BTM) is a word co-occurrence based topic model that learns topics by modeling word-word co-occurrences patterns (e.g., biterms). (In constrast, LDA and PLSA are word-document co-occurrence topic models, since they model word-document co-occurrences.)
A biterm consists of two words co-occurring in the same context, for example, in the same short text window. Unlike LDA models the word occurrences, BTM models the biterm occurrences in a corpus. In generation procedure, a biterm is generated by drawn two words independently from a same topic. In other words, the distribution of a biterm b=(wi,wj) is defined as:
P(b) = \sum_k{P(wi|z)*P(wj|z)*P(z)}.
With Gibbs sampling algorithm, we can learn topics by estimate P(w|k) and P(z).
More detail can be referred to the following paper:
Xiaohui Yan, Jiafeng Guo, Yanyan Lan, Xueqi Cheng. A Biterm Topic Model For Short Text. WWW2013.
The code has been test on linux. If you on windows, please install cygwin (with bc, wc, make).
To run the code, first config your own data and resources path in scripts/config.py
. The format of your training data can be arbitary, note that you should modify the corresponding preporcessing step in scripts/indexDocs.py
.
Then you can run it by:
$ cd script
$ sh run.sh
Indeed, the run.sh processes the input documents in 4 steps.
1. Index the words in the documents
To simplify the main code, we provide a python script to map each word to a unique ID (starts from 0) in the documents.
$ python script/indexDocs.py <doc_pt> <dwid_pt> <voca_pt>
doc_pt input docs to be indexed, each line is a doc with the format "word word ..."
dwid_pt output docs after indexing, each line is a doc with the format "wordId wordId ..."
voca_pt output vocabulary file, each line is a word with the format "wordId word"
fstop 1 if to filter stopwords
2. Topic learning
The next step is to train the model using the documents represented by word ids.
$ ./src/btm est <K> <W> <P> <alpha> <beta> <n_iter> <save_step> <docs_pt> <model_dir> <has_b>
K int, number of topics
P int, number of threads to run in multi-threaded program (Almost linear speed up)
W int, size of vocabulary
alpha double, Symmetric Dirichlet prior of P(z), like 1
beta double, Symmetric Dirichlet prior of P(w|z), like 0.01
n_iter int, number of iterations of Gibbs sampling
save_step int, steps to save the results
docs_pt string, path of training docs
model_dir string, output directory
has_b whether to include a background topic
The results will be written into the directory "model_dir":
- k20.bs: a B vector as the intermediate result for Gibbs sampling, suppose K=20
- k20.pw_z.: a K*M matrix for P(w|z), suppose K=20. Note that the file is created every saved step
- k20.pz.: a K*1 matrix for P(z), suppose K=20. Note that the file is created every saved step
Note that our model can support incremental training by storing the bs file.
3. Inference topic proportions for documents, i.e., P(z|d)
If you need to analysis the topic proportions of each documents, just run the following common to infer that using the model estimated.
$ ./src/btm inf <type> <K> <docs_pt> <model_dir> <suffix> <infer_type>
K int, number of topics, like 20
type string, 4 choices:sum_w, sum_b, lda, mix. sum_b is used in our paper.
docs_pt string, path of docs to be inferred
model_dir string, output directory
suffix the suffix for the output file
infer_type whether `prob` to get the whole distribution, or `max_idx` to get a single topic idx
The result will be output to "model_dir":
- k20.: a N*K matrix for P(z|d), suppose K=20
4. Results display
Finally, we also provide a python script to illustrate the top words of the topics and their proportions in the collection.
$ python script/btm.py <model_dir> <iter>
model_dir the output dir of BTM
iter the number of iteration
The script `btm.py` includes a number of different evaluation methods for topic display and document display. Also, the automatic metric calculation is done. For some other examples, see `testcase.py`.