The Epic Road to Fairness

[Hamedloghmani]

The plan for both short and long term future development of this project will be discussed here. This is just a road map to have a bird's-eye view on the project and each task should be defined in details in another issue page if needed. In case a separate issue page is created for a task, it should be linked here by the number of created issue.

1- Experiments

• Equality of Opportunity.popularity.0.01.100.fa-ir.uspt
• Demographic Parity.popularity.0.01.100.fa-ir.uspt
• Equality of Opportunity.gender.0.01.100.fa-ir.uspt
• Demographic Parity.gender.0.01.100.fa-ir.uspt
• Equality of Opportunity.popularity.0.05.100.fa-ir.uspt
• Demographic Parity.popularity.0.05.100.fa-ir.uspt
• Equality of Opportunity.gender.0.05.100.fa-ir.uspt
• Demographic Parity.gender.0.05.100.fa-ir.uspt
  #######################

2- Implementation:

• Optimize fair_greedy
• Add argument in get_stats to choose between avg or auc for pupolarity
• Add det_const_sort to the pipeline

3- Refactor

• Edit Adila's Readme and add link to new full experiments ( for undergrad mentees)
• 2d scatter diagram for fairness vs utility Bubble Plot for Fairness vs Utility #67
• Change popularity label files from csv to binary
• f'{new_output}.{algorithm}.{k_max}.rerank.csv' => binary
• Separate the layers of the pipeline (dal, rerank, eval layers)
• removing torch from pipeline and using pickle instead. Now, we add torch only to load *.pred file of a team formation model Removing torch from Adila's pipeline #64

4- Write up and literature review

Done:

• Equality of Opportunity.popularity.0.1.100.fa-ir.uspt
• Demographic Parity.popularity.0.1.100.fa-ir.uspt
• Equality of Opportunity.gender.0.1.100.fa-ir.uspt
• Demographic Parity.gender.0.1.100.fa-ir.uspt
• Equality of Opportunity.popularity.100.det.uspt
• Demographic Parity.popularity.100.det.uspt
• Equality of Opportunity.gender.100.det.uspt
• Demographic Parity.gender.100.det.uspt
• Run all the gender related experiments on dblp after fixing the bug in dblp gender-to-index file

#######################

• Equality of Opportunity.gender.det.imdb
• Equality of Opportunity.gender.det.dblp
• Demographic Parity.popularity.det.imdb
• Equality of Opportunity.popularity.det.imdb
  ###################################################################
• Equality of Opportunity.popularity.0.1.100.fa-ir.ndkl.imdb
• Equality of Opportunity.popularity.0.1.100.fa-ir.skew.imdb
• Equality of Opportunity.popularity.0.1.100.fa-ir.ndkl.dblp
• Equality of Opportunity.popularity.0.1.100.fa-ir.skew.dblp
• Equality of Opportunity.gender.0.1.100.fa-ir.ndkl.imdb
• Equality of Opportunity.gender.0.1.100.fa-ir.skew.imdb
• Equality of Opportunity.gender.0.1.100.fa-ir.ndkl.dblp
• Equality of Opportunity.gender.0.1.100.fa-ir.skew.dblp
• Demographic Parity.popularity.0.1.100.fa-ir.ndkl.imdb
• Demographic Parity.popularity.0.1.100.fa-ir.skew.imdb
• Demographic Parity.popularity.0.1.100.fa-ir.ndkl.dblp
• Demographic Parity.popularity.0.1.100.fa-ir.skew.dblp
• Demographic Parity.gender.0.1.100.fa-ir.ndkl.imdb
• Demographic Parity.gender.0.1.100.fa-ir.skew.imdb
• Demographic Parity.gender.0.1.100.fa-ir.ndkl.dblp
• Demographic Parity.gender.0.1.100.fa-ir.skew.dblp

###################################################################

• Equality of Opportunity.popularity.0.05.100.fa-ir.ndkl.imdb
• Equality of Opportunity.popularity.0.05.100.fa-ir.skew.imdb
• Equality of Opportunity.popularity.0.05.100.fa-ir.ndkl.dblp
• Equality of Opportunity.popularity.0.05.100.fa-ir.skew.dblp
• Equality of Opportunity.gender.0.05.100.fa-ir.ndkl.imdb
• Equality of Opportunity.gender.0.05.100.fa-ir.skew.imdb
• Equality of Opportunity.gender.0.05.100.fa-ir.ndkl.dblp
• Equality of Opportunity.gender.0.05.100.fa-ir.skew.dblp
• Equality of Opportunity.popularity.0.01.100.fa-ir.ndkl.imdb
• Equality of Opportunity.popularity.0.01.100.fa-ir.skew.imdb
• Equality of Opportunity.popularity.0.01.100.fa-ir.ndkl.dblp
• Equality of Opportunity.popularity.0.01.100.fa-ir.skew.dblp
• Equality of Opportunity.gender.0.01.100.fa-ir.ndkl.imdb
• Equality of Opportunity.gender.0.01.100.fa-ir.skew.imdb
• Equality of Opportunity.gender.0.01.100.fa-ir.ndkl.dblp
• Equality of Opportunity.gender.0.01.100.fa-ir.skew.dblp

###################################################################

• Demographic Parity.popularity.0.05.100.fa-ir.ndkl.imdb

• Demographic Parity.popularity.0.05.100.fa-ir.skew.imdb

• Demographic Parity.popularity.0.05.100.fa-ir.ndkl.dblp

• Demographic Parity.popularity.0.05.100.fa-ir.skew.dblp

• Demographic Parity.gender.0.05.100.fa-ir.ndkl.imdb

• Demographic Parity.gender.0.05.100.fa-ir.skew.imdb

• Demographic Parity.gender.0.05.100.fa-ir.ndkl.dblp (compute canada)

• Demographic Parity.gender.0.05.100.fa-ir.skew.dblp (compute canada)

• Demographic Parity.popularity.0.01.100.fa-ir.ndkl.imdb

• Demographic Parity.popularity.0.01.100.fa-ir.skew.imdb

• Demographic Parity.popularity.0.01.100.fa-ir.ndkl.dblp

• Demographic Parity.popularity.0.01.100.fa-ir.skew.dblp

• Demographic Parity.gender.0.01.100.fa-ir.ndkl.imdb

• Demographic Parity.gender.0.01.100.fa-ir.skew.imdb

• Demographic Parity.gender.0.01.100.fa-ir.ndkl.dblp

• Demographic Parity.gender.0.01.100.fa-ir.skew.dblp
  #########################################################################

• Demographic Parity.Popularity.Greedy.IMDB ( due 4/28/2023)

• Demographic Parity.Popularity.Greedy_Conservative.IMDB ( due 4/28/2023)

• Demographic Parity.Popularity.Greedy_Relaxed.IMDB ( due 4/28/2023)

• Equality of Opportunity.Popularity.Greedy.IMDB ( due 4/28/2023)

• Equality of Opportunity.Popularity.Greedy_Conservative.IMDB

• Equality of Opportunity.Popularity.Greedy_Relaxed.IMDB ( due 4/28/2023)

• Demographic Parity.Gender.Greedy.IMDB

• Demographic Parity.Gender.Greedy_Conservative.IMDB

• Demographic Parity.Gender.Greedy_Relaxed.IMDB

• Demographic Parity.Popularity.Greedy.DBLP

• Demographic Parity.Popularity.Greedy_Conservative.DBLP

• Demographic Parity.Popularity.Greedy_Relaxed.DBLP

• Equality of Opportunity.Popularity.Greedy.DBLP

• Equality of Opportunity.Popularity.Greedy_Conservative.DBLP

• Equality of Opportunity.Popularity.Greedy_Relaxed.DBLP

• averaging methods for fairness outputs such as skew and ndkl for different folds

• Full paper writeup for ECIR ir for good

• Resource paper writeup

• Fair Top-k Ranking with multiple protected groups

• Add FA*IR: A Fair Top-k Ranking Algorithm to the pipeline FA*IR Method for Neural Team Formation #80

• Add another fairness metric e.g. skew (or from normalized discounted difference (rND), ratio (rRD), and KL-divergence (rKL) proposed in Measuring Fairness in Ranked Outputs)

• FA*IR: A Fair Top-k Ranking Algorithm ( due 4/28/2023)

• Clean and push equality of opportunity function ( due 4/28/2023)

• Inferring the gender labels for DBLP dataset Gender Prediction Methods Based on Name #36

• Inferring the gender labels for IMDB dataset Gender Prediction Methods Based on Name #36

• Collect and process GitHub dataset

• Explore papers for other state-of-the-art score-based ranking and learning to rank methods

[hosseinfani]

@Hamedloghmani
Any progress with any of these tasks?

[Hamedloghmani]

@hosseinfani
Yes, these are the stuff that are going to be reported by the end of the day:

• Summary of FA*IR: A Fair Top-k Ranking Algorithm will be finished and uploaded
• Our labeling procedure for DBLP has 3 phases : 1) Analysis and automatically extract unique names from DBLB 2) API purchase and label unique names 3)Patch back the names to the dataset ( log is in Gender Prediction Methods Based on Name #36 )
  Phase 1 has finished last night we are on phase 2, I will purchase the API today.
• The write-up of the summary of deterministic reranking algorithms will be finished and uploaded
• Analysis and unique name results for the whole DBLP dataset is available in Teams -> Adila -> DBLP Labeling files
• Github crawler is on 800,000 record. About 200,000 to go

These are next in the queue:

1. Adding another fairness metric ( my choice is normalized discounted difference (rND) )
2. Finishing gender labeling for DBLP
3. adding dblp to the pipeline

[hosseinfani]

@Hamedloghmani
how about equality of odds implementation?

[Hamedloghmani]

@hosseinfani
My understanding from demographic parity, equality of opportunity and equalized odds is that if we want to include equalized odds or equality of opportunity, we have to somehow enter a label to our problem to define whether an expert is 'qualified' or not. And then examine how qualified/non-qualified experts place into the team. I think ( with my knowledge at this point, I might be wrong) these are not suitable for our problem and demographic parity which falls under group fairness criteria is more helpful.

My personal suggestion is going with Group/Individual fairness criteria instead of the 3 that I mentioned at the beginning since most of the significant work in fairness did the same.

I would be happy to hear your input on this. I can dedicate more time to equalized odds if you believe it would be helpful for us.

[hosseinfani]

@Hamedloghmani
I update the tasks. Please come up with a timeline in this issue page (don't create another doc/issue).
Also, drop by during office hour every week to discuss your progress.

[hosseinfani]

@Hamedloghmani
pls keep this issue page updated with your fall23 plan.

[Hamedloghmani]

@hosseinfani
Sure, the page has been successfully updated with the tasks that we discussed and you kindly mentioned in our Excel sheet.