Skip to content

Study on stereotype transfer accross Multilingual Language Models for English, Spanish, French, Greek and Croatian. The emotion profiles for different social groups are obtained from pre-trained XLM-RoBERTa and fine-tuned versions of it.

Notifications You must be signed in to change notification settings

gerardPlanella/MultiLingual_Stereotypes

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Equal Contributors : ADAM VALIN, VELJKO KOVAC, GERARD PLANELLA

STEREOTYPES IN MULTILINGUAL LANGUAGE MODELS

Description

This repository contains all the code used for the realisation of the ATCS Project : Stereotypes In MultiLingual Language Models. this project, we study stereotypes that emerge within pre-trained Multilingual Language Models(MLMs). These models are typically trained on large-scale multilingual text corpora, learning torepresent the different languages in a shared embedding space. This shared representation allows the model to generalize knowledge learned in one language to other languages, a phenomenon known as cross-lingual transfer (Conneau et al., 2019). Our objective is to build upon the work of Choenni et al., 2021 by initially contrasting emotion profilesof identical social groups across diverse languages within these MLMs.

Installing the Project Environment

This project uses an Anaconda environment for managing dependencies. Follow the instructions below to set up the environment:

First, ensure that you have Anaconda or Miniconda installed on your system. If not, you can download Anaconda from https://www.anaconda.com/products/distribution.

Clone the repository and navigate to the project directory:

'''python
git clone https://github.com/username/project.git
cd project
'''

The environment.yaml file in the project root contains the specifications for the project's conda environment. Create the environment using the following command:

'''python
conda env create -f environment.yaml
'''

Once the environment is created, you can activate it using:

'''python
conda activate env_name
'''

Replace env_name with the name of the environment specified in the environment.yaml file.

To check that the environment was installed correctly, you can list the environments:

'''python
conda env list
'''

You should see your new environment in the list.

Now you are ready to start working on the project!

Remember to replace username, project, and env_name with the appropriate values for your project.

Generating the priors

The priors are already generated and pushed to the repository so this step can be skipped. However, if wanted, one could run create_priors.py with the right arguments to regenrate them.

Creating the emotion profiles

The first thing to do is to generate emotion profiles for given social groups for given languages. You can run the file run_test_normalization.py, it will generate emotion profiles for each social group and each language for a given model. Modify the arguments to modify either the used model or the top_k which corresponds to the number of predictions taken into account to generate the emotion profiles per social group per prompt.

You can run the emotion profiles for specific fine-tuned models by changing the --finetuned_model flag to either 'french', 'english', 'greek', 'spanish' or 'base'.

Correlations

Ideally, you want to compare the baseline emotion profiles with fine-tuned emotion profiles, to see how the outputs of the model changed with fine-tuning. For this purpose, first run emotion profiles with the pretrained xlm-R model and with a fine-tuned model. Then run the run_correlations.py file with the right fine_tuned model.

Fine-tuning of the models

Fine-tuning is possible through the use of all finetuning/train_*.py file. Running them directly is possible. You can also change the flags for parameters such as batch_size, number of epochs, output_directory ...

Fine-tuned Models

The weights of the already trained models that are discussed in our paper can be found here: https://drive.google.com/drive/folders/1VdK-m7a8uM2A6pQref9VNrMq_lryzCEV?usp=sharing

Emotion Profile Analysis

To perform similar analysis on the shifts of emotion profiles before and after finetuning as in our paper, you can follow the "Emotion Profile Analysis.ipynb".

About

Study on stereotype transfer accross Multilingual Language Models for English, Spanish, French, Greek and Croatian. The emotion profiles for different social groups are obtained from pre-trained XLM-RoBERTa and fine-tuned versions of it.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published