Papers about Causal Inference and Language

A collection of papers and codebases about influence, causality, and language.

Pull requests welcome!

Table of Contents

1. Datasets and Simulations
2. Learning resources and blog posts
3. Causal Inference with Text Variables
   1. Text as treatment
   2. Text as mediator
   3. Text as outcome
   4. Text as confounder
4. Causality to improve NLP
   1. Causal interpretations and explanations
   2. Sensitivity and rhobustness
5. Applications in the Social Sciences
   1. Linguistics
   2. Marketing
   3. Persuasion & Argumentation
   4. Mental health
   5. Psychology
   6. Economics
   7. Bias and Fairness
   8. Social Media
   9. Law
   10. Online Hate Speech
6. Potential Connections to Language
   1. Vectorized treatments

Datasets and Simulations

Type	Description	Code
Semi-simulated	Given text (amazon reviews), extracts treatments (0 or 5 stars) and confounds (product type), then samples outcomes (sales) conditioned on the extracted treatments and confounds.	git
Fully synthetic	Samples outcomes, treatments, and confounds from binomial distributions, then words from a uniform distribution conditioned on those sampled variables.	git

Learning resources and blog posts

Title	Description	Code
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates Katherine A. Keith, David Jensen, and Brendan O’Connor	Survey of studies that use text to remove confouding. Also highlights numerous open problems in the space of text and causal inference.
Text Feature Selection for Causal Inference Reid Pryzant and Dan Jurafsky	Blog post about text as treatment (operationalized through lexicons)	git
Econometrics Meets Sentiment: An Overview of Methodology and Applications Andres Algaba, David Ardia, Keven Bluteau, Samuel Borms, and Kris Boudt	Survey summarizing various methods to transform alternative data (with a focus on text) into a variable, and use it in econometric models. Includes applications throughout.	git

Causal Inference with Text Variables

Text as treatment

Title	Description	Code
Causal Effects of Linguistic Properties Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, Dhanya Sridhar	Develops an adjustment procedure for text-based causal inference with classifier-based treatments. Proves bounds on the bias	git
Challenges of Using Text Classifiers for Causal Inference Zach Wood-Doughty, Ilya Shpitser, Mark Dredze	Looks at different errors that can stem from estimating treatment labels with classifiers, proposes adjustments to account for said errors	git
Deconfounded Lexicon Induction for Interpretable Social Science Reid Pryzant, Kelly Shen, Dan Jurafsky, Stefan Wager	Looks at effect of text as manifested in lexicons or individual words, proposes algorithms for estimating effects and evaluating lexicons	git
How to Make Causal Inferences Using Texts Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart	(Also text as outcome). Covers assumptions needed for text as treatment and concludes that you should use a train/test set.
Discovery of treatments from text corpora Christian Fong, Justin Grimmer	Propose a new experimental design and statistical model to simultaneously discover treatments in a corpora and estimate causal effects for these discovered treatments.
The effect of wording on message propagation: Topic and author-controlled natural experiments on twitter Chenhao Tan, Lillian Lee, and Bo Pang	Controls for confouding by looking at Tweets containing the same url and written by the same user but employing different wording.
When do Words Matter? Understanding the Impact of Lexical Choice on Audience Perception using Individual Treatment Effect Estimation Zhao Wang and Aron Culotta	Measure effect of words on reader's perception. Multiple quasi-experimental methods compared.	git

Text as mediator

Title	Description	Code
Adapting Text Embeddings for Causal Inference Victor Veitch, Dhanya Sridhar, and David Blei	(also text as confounder) Adapts BERT embeddings for causal inference by predicting propensity scores and potential outcomes alongside masked language modeling objective.	tensorflow pytorch
Operationalizing Complex Causes: A Pragmatic View of Mediation Limor Gultchin, David Watson, Matt Kusner and Ricardo Silva	(can also be viewed as text as treatment) Develops a notion of pragmatic mediation which helps make causel effect estimation when complex objects, such as text, image or genomics are involved, across various intervention regimes. Identification of prgamtic mediators has an interpretability benefit which could guide the development of new interventions.	git
Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects Katherine A. Keith, Douglas Rice, and Brendan O’Connor	Proposes a causal research design for observational (nonexperimental) data to estimate the natural direct and indirect effects of social group signals (e.g. race or gender) on speakers’ responses with separate aspects of language as causal mediators

Text as outcome

Title	Description	Code
Estimating Causal Effects of Tone in Online Debates Dhanya Sridhar and Lise Getoor	(Also text as confounder). Looks at effect of reply tone on the sentiment of subsiquent responses in online debates.	git
How Judicial Identity Changes the Text of Legal Rulings Michael Gill and Andrew Hall	Looks at how the random assignment of a female judge or a non-white judge affects the language of legal rulings.
Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations Anna Koroleva, Sanjay Kamath, Patrick Paroubek

Text as confounder

Title	Description	Code
CausalNLP: A Practical Toolkit for Causal Inference with Text Arun S. Maiya	(also text as treatment). Describes a toolkit for causal inference with text largely based on meta-learners. Includes support for encoding text as a "controlled-for" variable using traditional BOW features in addition to a PyTorch implementation of Causal Bert (originally from R. Pryzant). Also includes convenience methods for easily transforming text into traditional numerical or categorical variables for use as treatment/confounder/outcome in a causal analyses (e.g., sentiment, topic, emotion, etc.)	git
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates Katherine A. Keith, David Jensen, and Brendan O’Connor	Survey of studies that use text to remove confouding. Also highlights numerous open problems in the space of text and causal inference.
Adjusting for confounding with text matching Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen	Estimate a low-dimensional summary of the text and condition on this summary via matching to remove confouding. Proposes a method of text matching, topical inverse regression matching, that matches on both on the topical content and propensity score.
Matching with text data: An experimental evaluation of methods for matching documents and of measuring match quality Reagan Mozer, Luke Miratrix, Aaron Russell Kaufman, L Jason Anastasopoulos	Characterizes and empirically evaluates a framework for matching text documents that decomposes existing methods into: the choice of text representation, and the choice of distance metric.
Learning representations for counterfactual inference Fredrik Johansson, Uri Shalit, David Sontag	One of their semi-synthetic experiments has news content as a confounder.
Conceptualizing Treatment Leakage in Text-based Causal Inference Adel Daoud, Connor T. Jerzak, and Richard Johansson	Characterize the problem of the leakage of treatment signal when controlling for text-based confounders which may lead to issues in identification and estimation. Simulation study on how treatment-leakage leads to issues with the estimation of the Average Treatment Effect (ATE) and how to mitigate this bias with text pre-processing by assuming separability.

Causality to Improve NLP

Causal interpretations and explanations

Title	Description	Code
CausaLM: Causal Model Explanation Through Counterfactual Language Models Amir Feder, Nadav Oved, Uri Shalit and Roi Reichart	Suggested a method for generating causal explanations through counterfactual language representations.	git
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer and Stuart Shieber	Uses causal mediation analysis to interpret NLP models.	git
Causal BERT: Language Models for Causality Detection Between Events Expressed in Text Vivek Khetan, Roshni Ramnani, Mayuresh Anand, Subhashis Sengupta, Andrew E. Fano	This paper investigate the language model’s capabilities for identification of causal association among events expressed in natural language text using only the sentence context, sentence context combined with event information, and by leveraging masked event context with in-domain and out-of-domain data distribution.

Sensitivity and Robustness

Title	Description	Code
Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals Zhao Wang and Aron Culotta	Matching to identify causal terms, then generate counterfactuals for training.	git
Identifying Spurious Correlations for Robust Text Classification Zhao Wang and Aron Culotta	Matching to identify spurious word features	git
Discovering and Controlling for Latent Confounds in Text Classification Using Adversarial Domain Adaptation Virgile Landeiro, Tuan Tran, and Aron Culotta	Control for unobserved confounders in text classification
Robust Text Classification under Confounding Shift Virgile Landeiro and Aron Culotta	Control for changing confounders in text classification	git
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data Divyansh Kaushik, Eduard Hovy, Zachary C. Lipton	Introducing methods and resources for training models less sensitive to spurious patterns	git
Explaining The Efficacy of Counterfactually-Augmented Data Divyansh Kaushik, Amrith Setlur, Eduard Hovy, Zachary C. Lipton	Explaining the efficacy of counterfactually-augmented data for training models less sensitive to spurious patterns	git

Applications in the Social Sciences

Linguistics

Title	Description	Code
Decoupling entrainment from consistency using deep neural networks Andreas Weise, Rivka Levitan	Isolated the individual style of a speaker when modeling entrainment in speech.
Estimating causal effects of exercise from mood logging data Dhanya Sridhar, Aaron Springer, Victoria Hollis, Steve Whittaker, Lise Getoor	Confouder: Text of mood triggers. Confounding adjustment method: Propensity score matching

Marketing

Title	Description	Code
Predicting Sales from the Language of Product Descriptions Reid Pryzant, Young-Joo Chung, and Dan Jurafsky	Found features of product descriptions most predictive of sales while controlling for brand & price.	git
Interpretable Neural Architectures for Attributing an Ad’s Performance to its Writing Style Reid Pryzant, Kazoo Sone, and Sugato Basu	Found features of ad copy most predictive of high CTR while controlling for advertiser and targeting.	git

Persuasion & Argumentation

Title	Description	Code
Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online Emaad Manzoor, George H. Chen, Dokyun Lee, Michael D. Smith	Controls for unstructured argument text using neural models of language in the double machine-learning framework.

HealthCare

Title	Description	Code
MIMICause: Representation and automatic extraction of causal relation types from clinical notes Vivek Khetan, Md Imbesat Rizvi, Jessica Huber, Paige Bartusiak, Bogdan Sacaleanu, Andrew Fano	This work proposed annotation guidelines, develop an annotated corpus, and provided baseline scores to identify types and direction of causal relations between a pair of biomedical concepts in clinical notes; communicated implicitly or explicitly identified either in a single sentence or across multiple sentences.

Mental Health

Title	Description	Code
The language of social support in social media and its effect on suicidal ideation risk Munmun De Choudhury and Emre Kiciman	Confouder: previous text written in a Reddit forum. Confounding adjustment method: stratified propensity scores matching.
Discovering shifts to suicidal ideation from mental health content in social media Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, Mrinal Kumar	Confouder: User’s previous posts and comments received. Confounding adjustment method: stratified propensity scores matching

Psychology

Title	Description	Code
Increasing vegetable intake by emphasizing tasty and enjoyable attributes: A randomized controlled multisite intervention for taste-focused labeling Bradley Turnwald, Jaclyn Bertoldo, Margaret Perry, Peggy Policastro, Maureen Timmons, Christopher Bosso, Priscilla Connors, Robert Valgenti, Lindsey Pine, Ghislaine Challamel, Christopher Gardner, Alia Crum	Did RCT on cafeteria food labels, observing effect on how much of those foods students took.
A social media study on the effects of psychiatric medication use Koustuv Saha, Benjamin Sugar, John Torous, Bruno Abrahao, Emre Kıcıman, Munmun De Choudhury	Confounder: users' previous posts on Twitter. Confounding adjustment method: Stratified propensity score matching.

Economics

Title	Description	Code
A deep causal inference approach to measuring the effects of forming group loans in online non-profit microfinance platform Thai T Pham and Yuanyuan Shen	Confounder: Microloan descriptions on Kiva. Confounding adjustment method: A-IPTW, TMLE on embeddings.

Bias and Fairness

Title	Description	Code
Unsupervised Discovery of Implicit Gender Bias	Propensity score matching and adversarial learning to get a model to focus on bias instead of other artifacts.
Tweetment Effects on the Tweeted: Experimentally Reducing Racist Harassment Kevin Munger	Did RCT sending de-escalation messages to racist twitter users, changing the "from" user and observing effects on downstream behavior.

Social Media

Title	Description	Code
Estimating the effect of exercising on users online behavior Seyed Amin Mirlohi Falavarjani, Hawre Hosseini, Zeinab Noorian, Ebrahim Bagheri	Confounder: Pre-treatment topical interest shift. Confounding adjustment method: Matching on topic models.
Distilling the outcomes of personal experiences: A propensity-scored analysis of social media Alexandra Olteanu, Onur Varol, Emre Kiciman	Confounder: Past word use on Twitter. Confoundig adjustment method: Stratified propensity score matching.
Using longitudinal social media analysis to understand the effects of early college alcohol use Emre Kiciman, Scott Counts, Melissa Gasser	Confounder: Previous posts on Twitter. Confounding adjustment method: Stratified propensity score matching.
Using Matched Samples to Estimate the Effects of Exercise on Mental Health from Twitter Virgile Landeiro and Aron Culotta	Confounder: Gender, location, profile. Confounding adjustment method: Matching.	git
Extraction of Explicit and Implicit Cause-Effect Relationships in Patient-Reported Diabetes-Related Tweets From 2017 to 2021: Deep Learning Approach Adrian Ahne, Vivek Khetan, Xavier Tanner, Md Imbessat Hasan Rizvi, Thomas Czernichow, Francisco Orchard , Charline Bour, Andrew Fano, Guy Fagherazzi	A cause-effect dataset was manually labeled and augmented using active learning. First,sentences containing causal information (causal sentences) were detected by fine-tuning a BERTweet model. Secondly, cause-effect pairs were identified in causal sentences with several models tested. Lastly, in a semi-supervised approach cause-effect pairs were aggregated to form a cause-effect network, which was visualised in D3.

Law

Title	Description	Code
Everything Has a Cause: Leveraging Causal Inference in Legal Text Analysis Xiao Liu, Da Yin, Yansong Feng, Yuting Wu, Dongyan Zhao	Built causal graphs from legal descriptions automatically, and disambiguated similar charges with the built graphs. Treatment & Confounders: factors from legal descriptions.	git

Online Hate Speech

Title	Description	Code
A Survey of Online Hate Speech through the Causal Lens Antigoni M. Founta, Lucia Specia	A survey of studies that measure causal effects related to online hate speech. The survey also highlights potential knowledge gaps and issues, and provides suggestions on how to further extend the causal perspective of hate speech.
Robust Cyberbullying Detection with Causal Interpretation Lu Cheng, Ruocheng Guo, Huan Liu	Proposes a principled framework to identify and block the influence of plausible latent confounders on cyberbullying detection.
Prevalence and Psychological Effects of Hateful Speech in Online College Communities Koustuv Saha, Eshwar Chandrasekharan, Munmun De Choudhury	Measure psychological effect of exposure to hate speech on Reddit communities, as increase in levels of stress. Confounders: Subreddit & User activity. Confounding adjustment method: Propensity score matching.

Potential Connections to Language

Vectorized treatments

Title	Description	Code
Graph Intervention Networks for Causal Effect Estimation Jean Kaddour, Qi Liu, Yuchen Zhu, Matt J. Kusner, Ricardo Silva	Generalizes the Robinson decomposition, as used in R-learner or generalized random forests, to vectorized treatments (e.g. text, images, graphs).	git