Markus Konrad markus.konrad@wzb.eu, November 2020
This is a small project to showcase topic modeling with the tmtoolkit Python package via LDA, where I use a corpus of articles from the German online news website Spiegel Online (SPON) to create a topic model for before and during the COVID-19 pandemic. This topic model is then used to analyze the volume of media coverage regarding the pandemic and how it changed over time. Currently, a time span from Oct. 2019 to end of Aug. 2020 is covered but I plan to give an update for a time span until end of Nov. 2020.
For an introduction to topic modeling via LDA see Introduction to Probabilistic Topic Models (Blei 2012) or Topic modeling made just simple enough (Underwoord 2012).
The main analysis is done in the notebook tm_analysis.ipynb. Head over there for an application of topic models. However, data retrieval, preparation and topic modeling is just as important and is done in the following scripts:
- scraping news from SPON:
fetch_news/spon.py - text data preparation for topic modeling with tmtoolkit:
prepare.py - topic model evaluation with tmtoolkit:
tm_evaluation.py - generation of final candidate topic models with tmtoolkit:
tm_final.py
Most raw data files are too big for git. I provide the document-term matrix, corpus metadata and generated topic models as separate ZIP file for download. Simply unzip the file to the cloned repository folder. You may contact me for access to the raw text data.
Licensed under Apache License 2.0. See LICENSE file.