Skip to content

Latest commit

 

History

History
49 lines (43 loc) · 1.83 KB

README.md

File metadata and controls

49 lines (43 loc) · 1.83 KB
Raw

GPT_implementation

Here is pytorch implementation of a decoder-only model for poem generation. It was trained on truongpdd/vietnamese_poetry - a dataset of Vietnamese poem. This repo may be useful to understand the architecture of decoder block in transfomer and how a decoder-only model work.

Result

This still a very simple network and train in just about 20000 iterations with Kaggle GPU P100 and it tooks about 30 mins to train. Therefore, the results are not good but you can still see the improvement in performance after some iterations.

After 100 iterations


Figure_1


Figure_2


Figure_3

After 20000 iterations


Figure_4


Figure_5


Figure_6

with the same input, we can see the difference between the output of model after 100 and after 200 iterations.

How to use

Run the following command to install modules needed

pip install -r requirements.txt

The code was organized:

  • In the notebook directory, you can use gpt-pytorch-implementation.ipynb and adjust some things accordingly your conditions to train on notebook or Kaggle.
  • set up paramter for dataset and model: config.py
  • download data from hugging face, get batched data: data.py
  • modules needed to build model: modules.py
  • script to train model: train.py
  • test model with different input: test.py