From fe42658e58c215b11cafa32709a21a31680d2095 Mon Sep 17 00:00:00 2001 From: IshaShah27 Date: Wed, 13 Mar 2024 11:35:08 -0400 Subject: [PATCH 01/12] added examples of harm related to llms --- deon/assets/examples_of_ethical_issues.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 653a248..e8890db 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -72,6 +72,10 @@ url: https://www.bbc.com/news/magazine-22223190 - line_id: D.1 links: + - text: OpenAI's GPT models show racial bias in ranking job applications based on candidate names. + url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTcwOTg1NjE0OCwiZXhwIjoxNzEwNDYwOTQ4LCJhcnRpY2xlSWQiOiJTQTA1Q1FUMEFGQjQwMCIsImJjb25uZWN0SWQiOiI2NDU1MEM3NkRFMkU0QkM1OEI0OTI5QjBDQkIzRDlCRCJ9.MdkSGC3HMwwUYtltWq6WxWg3vULNeCTJcjacB-DNi8k + - text: In hypothetical trials, language models assign the death penalty more frequently for defendants using African American English than those using Standard American English. + url: https://arxiv.org/abs/2403.00742 - text: Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny. url: https://www.wired.com/story/excerpt-from-automating-inequality/ - text: Amazon scraps AI recruiting tool that showed bias against women. @@ -126,6 +130,8 @@ url: https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy - line_id: E.3 links: + - text: Google stops Gemini, its generative AI chatboth, from generating all images from people after it began generating historically inaccurate images in an attempt to combat amplification of racial stereotypes. + url: https://apnews.com/article/google-gemini-ai-chatbot-image-generation-1bd45f1e67dfe0f88e5419a6efe3e06f - text: Google "fixes" racist algorithm by removing gorillas from image-labeling technology. url: https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai - line_id: E.4 From c04015f7032b38fc399a9730d13b4b30c80dd7f0 Mon Sep 17 00:00:00 2001 From: IshaShah27 Date: Wed, 13 Mar 2024 11:36:38 -0400 Subject: [PATCH 02/12] typo fix --- deon/assets/examples_of_ethical_issues.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index e8890db..56442f2 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -130,7 +130,7 @@ url: https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy - line_id: E.3 links: - - text: Google stops Gemini, its generative AI chatboth, from generating all images from people after it began generating historically inaccurate images in an attempt to combat amplification of racial stereotypes. + - text: Google stops Gemini, its generative AI chatbot, from generating any images from people after it began generating historically inaccurate images in an attempt to combat amplification of racial stereotypes. url: https://apnews.com/article/google-gemini-ai-chatbot-image-generation-1bd45f1e67dfe0f88e5419a6efe3e06f - text: Google "fixes" racist algorithm by removing gorillas from image-labeling technology. url: https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai From 2f9f4b6f16287ea71581b653f2a053a2b149a843 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 11:01:25 -0400 Subject: [PATCH 03/12] fix link to add example --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fc99055..8546893 100644 --- a/README.md +++ b/README.md @@ -270,7 +270,7 @@ We're excited to see so many articles popping up on data ethics! The short list To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. -We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/wiki/Add-a-new-item-to-the-examples-table) to add an example. +We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md#1-adding-a-new-item-to-the-examples-table) to add an example. ## Related tools From a31bcc6477aaa733defe0e45f2c7e401d35f3f0a Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 11:16:50 -0400 Subject: [PATCH 04/12] add examples --- deon/assets/examples_of_ethical_issues.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 56442f2..5b7e78e 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -102,12 +102,16 @@ url: https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/ - text: -- Related academic study. url: https://arxiv.org/abs/1301.6822 + - text: When screening resumes to select top candidates for a job posting, ChatGPT 3.5 favored certain names based on their demographics to an extent that would fail job discrimination benchmarks. + url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/ - line_id: D.3 links: - text: Facebook seeks to optimize "time well spent", prioritizing interaction over popularity. url: https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/ - text: YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos. url: https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240 + - text: A widely used health system model underpredicts the need of Black patients because it uses spending as a proxy for health need. Cost as a metric introduces racial bias because of unequal access to care. + url: https://www.science.org/doi/10.1126/science.aax2342 - line_id: D.4 links: - text: Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die. @@ -130,8 +134,6 @@ url: https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy - line_id: E.3 links: - - text: Google stops Gemini, its generative AI chatbot, from generating any images from people after it began generating historically inaccurate images in an attempt to combat amplification of racial stereotypes. - url: https://apnews.com/article/google-gemini-ai-chatbot-image-generation-1bd45f1e67dfe0f88e5419a6efe3e06f - text: Google "fixes" racist algorithm by removing gorillas from image-labeling technology. url: https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai - line_id: E.4 From 918c859f4d0de5407379e30a24ed1b1df505a592 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 13:17:09 -0400 Subject: [PATCH 05/12] rephrase dialects --- deon/assets/examples_of_ethical_issues.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 5b7e78e..043b51e 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -74,7 +74,7 @@ links: - text: OpenAI's GPT models show racial bias in ranking job applications based on candidate names. url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTcwOTg1NjE0OCwiZXhwIjoxNzEwNDYwOTQ4LCJhcnRpY2xlSWQiOiJTQTA1Q1FUMEFGQjQwMCIsImJjb25uZWN0SWQiOiI2NDU1MEM3NkRFMkU0QkM1OEI0OTI5QjBDQkIzRDlCRCJ9.MdkSGC3HMwwUYtltWq6WxWg3vULNeCTJcjacB-DNi8k - - text: In hypothetical trials, language models assign the death penalty more frequently for defendants using African American English than those using Standard American English. + - text: In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects. url: https://arxiv.org/abs/2403.00742 - text: Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny. url: https://www.wired.com/story/excerpt-from-automating-inequality/ From 0ec2fd1ccfc9d8976081830bf28995d07b94cd69 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 13:19:16 -0400 Subject: [PATCH 06/12] move job ranking example --- deon/assets/examples_of_ethical_issues.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 043b51e..c03ec32 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -72,8 +72,6 @@ url: https://www.bbc.com/news/magazine-22223190 - line_id: D.1 links: - - text: OpenAI's GPT models show racial bias in ranking job applications based on candidate names. - url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTcwOTg1NjE0OCwiZXhwIjoxNzEwNDYwOTQ4LCJhcnRpY2xlSWQiOiJTQTA1Q1FUMEFGQjQwMCIsImJjb25uZWN0SWQiOiI2NDU1MEM3NkRFMkU0QkM1OEI0OTI5QjBDQkIzRDlCRCJ9.MdkSGC3HMwwUYtltWq6WxWg3vULNeCTJcjacB-DNi8k - text: In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects. url: https://arxiv.org/abs/2403.00742 - text: Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny. @@ -102,7 +100,7 @@ url: https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/ - text: -- Related academic study. url: https://arxiv.org/abs/1301.6822 - - text: When screening resumes to select top candidates for a job posting, ChatGPT 3.5 favored certain names based on their demographics to an extent that would fail job discrimination benchmarks. + - text: OpenAI's GPT models show racial bias in ranking job applications based on candidate names. url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/ - line_id: D.3 links: From 039fd5212bcab48e4268dc5e279b5d05cb826369 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:08:57 -0400 Subject: [PATCH 07/12] update examples --- deon/assets/examples_of_ethical_issues.yml | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index c03ec32..dc8db64 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -4,6 +4,8 @@ url: https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/ - text: African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld. url: https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment + - text: OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions. + url: https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems - line_id: A.2 links: - text: StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population. @@ -22,8 +24,6 @@ url: https://www.bloomberg.com/graphics/2016-amazon-same-day/ - text: Facial recognition software is significanty worse at identifying people with darker skin. url: https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/ - - text: -- Related academic study. - url: http://proceedings.mlr.press/v81/buolamwini18a.html - line_id: B.1 links: - text: Personal and financial data for more than 146 million people was stolen in Equifax data breach. @@ -46,14 +46,8 @@ url: https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track - line_id: C.2 links: - - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients, assigning them lower risk scores compared to equivalently sick white patients. - url: https://www.nature.com/articles/d41586-019-03228-6 - - text: -- Related academic study. - url: https://science.sciencemag.org/content/366/6464/447 - text: word2vec, trained on Google News corpus, reinforces gender stereotypes. url: https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/ - - text: -- Related academic study. - url: https://arxiv.org/abs/1607.06520 - text: Women are more likely to be shown lower-paying jobs than men in Google ads. url: https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study - line_id: C.3 @@ -134,9 +128,11 @@ links: - text: Google "fixes" racist algorithm by removing gorillas from image-labeling technology. url: https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai -- line_id: E.4 - links: - text: Microsoft's Twitter chatbot Tay quickly becomes racist. url: https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay +- line_id: E.4 + links: + - text: Generative AI can be exploited to create more convincing scams, such as "virtual kidnapping". + url: https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt - text: Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements. url: http://theweek.com/articles/777592/rise-deepfakes From 92ae55ef0cb5a964b08a55ed2cd7c3947461ef03 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:11:16 -0400 Subject: [PATCH 08/12] update obermeyer language --- deon/assets/examples_of_ethical_issues.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index dc8db64..607ab9b 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -103,6 +103,7 @@ - text: YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos. url: https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240 - text: A widely used health system model underpredicts the need of Black patients because it uses spending as a proxy for health need. Cost as a metric introduces racial bias because of unequal access to care. + - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients compared to equivalently sick white patients. The model uses spending as a proxy for health need, introducing racial bias due to unequal access to care. url: https://www.science.org/doi/10.1126/science.aax2342 - line_id: D.4 links: From 8722766d12d684bbad577091d72f99420bda411e Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:15:59 -0400 Subject: [PATCH 09/12] remove extra bullet --- deon/assets/examples_of_ethical_issues.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 607ab9b..b4010f0 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -102,7 +102,6 @@ url: https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/ - text: YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos. url: https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240 - - text: A widely used health system model underpredicts the need of Black patients because it uses spending as a proxy for health need. Cost as a metric introduces racial bias because of unequal access to care. - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients compared to equivalently sick white patients. The model uses spending as a proxy for health need, introducing racial bias due to unequal access to care. url: https://www.science.org/doi/10.1126/science.aax2342 - line_id: D.4 @@ -133,7 +132,7 @@ url: https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay - line_id: E.4 links: - - text: Generative AI can be exploited to create more convincing scams, such as "virtual kidnapping". + - text: Generative AI can be exploited to create convincing scams like "virtual kidnapping". url: https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt - text: Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements. url: http://theweek.com/articles/777592/rise-deepfakes From c39f80f45c89cbb8569ff10ecc69029e253f9785 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:21:47 -0400 Subject: [PATCH 10/12] shorten example --- deon/assets/examples_of_ethical_issues.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index b4010f0..a9fad18 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -102,7 +102,7 @@ url: https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/ - text: YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos. url: https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240 - - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients compared to equivalently sick white patients. The model uses spending as a proxy for health need, introducing racial bias due to unequal access to care. + - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care. url: https://www.science.org/doi/10.1126/science.aax2342 - line_id: D.4 links: From d21c43fddc03955f30d2791d710e80d7512d6944 Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:37:02 -0400 Subject: [PATCH 11/12] run make docs --- docs/docs/examples.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/docs/examples.md b/docs/docs/examples.md index 0cd51fa..779c881 100644 --- a/docs/docs/examples.md +++ b/docs/docs/examples.md @@ -7,28 +7,28 @@ To make the ideas contained in the checklist more concrete, we've compiled examp
Checklist Question
|
Examples of Ethical Issues
--- | --- |
**Data Collection**
-**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? |
  • [Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)
  • [African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)
+**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? |
  • [Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)
  • [African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)
  • [OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)
**A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? |
  • [StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)
  • [Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)
**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? |
  • [Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)
  • [Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)
-**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? |
  • [In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)
  • [Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)
  • [-- Related academic study.](http://proceedings.mlr.press/v81/buolamwini18a.html)
+**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? |
  • [In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)
  • [Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)
|
**Data Storage**
**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? |
  • [Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)
  • [Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)
  • [AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)
**B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? |
  • [The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)
**B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? |
  • [FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)
|
**Analysis**
**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? |
  • [When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? |
  • [A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients, assigning them lower risk scores compared to equivalently sick white patients.](https://www.nature.com/articles/d41586-019-03228-6)
  • [-- Related academic study.](https://science.sciencemag.org/content/366/6464/447)
  • [word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)
  • [-- Related academic study.](https://arxiv.org/abs/1607.06520)
  • [Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? |
  • [word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)
  • [Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)
**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? |
  • [Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)
  • [Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)
**C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? |
  • [Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)
**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? |
  • [Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)
|
**Modeling**
-**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? |
  • [Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)
  • [Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)
  • [Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)
  • [Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)
-**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? |
  • [Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)
  • [Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)
  • [With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)
  • [-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)
  • [-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)
  • [Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)
  • [Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)
  • [-- Related academic study.](https://arxiv.org/abs/1301.6822)
-**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? |
  • [Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)
  • [YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)
+**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? |
  • [In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)
  • [Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)
  • [Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)
  • [Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)
  • [Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)
+**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? |
  • [Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)
  • [Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)
  • [With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)
  • [-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)
  • [-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)
  • [Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)
  • [Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)
  • [-- Related academic study.](https://arxiv.org/abs/1301.6822)
  • [OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)
+**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? |
  • [Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)
  • [YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)
  • [A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)
**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? |
  • [Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)
  • [GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)
**D.5 Communicate bias**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? |
  • [Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)
|
**Deployment**
**E.1 Monitoring and evaluation**: How are we planning to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? |
  • [Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)
  • [Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)
**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? |
  • [Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)
-**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? |
  • [Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)
-**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? |
  • [Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)
  • [Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)
+**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? |
  • [Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)
  • [Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)
+**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? |
  • [Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)
  • [Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)
From a85f21074ad78e0c09e42068e1573e5220bced2e Mon Sep 17 00:00:00 2001 From: Katie Wetstone Date: Fri, 12 Apr 2024 14:38:43 -0400 Subject: [PATCH 12/12] update contribution link everywhere --- README.md | 2 +- docs/docs/index.md | 2 +- docs/md_templates/_common_body.tpl | 2 +- docs/md_templates/_common_body_pt-BR.tpl | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 8546893..aed82c1 100644 --- a/README.md +++ b/README.md @@ -270,7 +270,7 @@ We're excited to see so many articles popping up on data ethics! The short list To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. -We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md#1-adding-a-new-item-to-the-examples-table) to add an example. +We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example. ## Related tools diff --git a/docs/docs/index.md b/docs/docs/index.md index 1d9dedf..5c15871 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -263,7 +263,7 @@ We're excited to see so many articles popping up on data ethics! The short list To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. -We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/wiki/Add-a-new-item-to-the-examples-table) to add an example. +We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example. ## Related tools diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl index b107c78..c46a089 100644 --- a/docs/md_templates/_common_body.tpl +++ b/docs/md_templates/_common_body.tpl @@ -199,7 +199,7 @@ We're excited to see so many articles popping up on data ethics! The short list To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. -We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/wiki/Add-a-new-item-to-the-examples-table) to add an example. +We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example. ## Related tools diff --git a/docs/md_templates/_common_body_pt-BR.tpl b/docs/md_templates/_common_body_pt-BR.tpl index 3fee1fe..a0f246b 100644 --- a/docs/md_templates/_common_body_pt-BR.tpl +++ b/docs/md_templates/_common_body_pt-BR.tpl @@ -175,7 +175,7 @@ Estamos entusiasmados de ver tantos artigos surgindo sobre ética de dados! A cu Para tornar as ideias contidas na checklist mais concretas, compilamos [exemplos](http://deon.drivendata.org/examples/) de situações em que as coisas deram errado. Estão vinculadas a questões da checklist para ajudar a iluminar onde no processo as discussões éticas poderiam ter ajudado a criar uma correção no curso. -Nós aceitamos contribuições! Siga [estas instruções](https://github.com/drivendataorg/deon/wiki/Add-a-new-item-to-the-examples-table) para acrescentar um exemplo. +Nós aceitamos contribuições! Siga [estas instruções](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) para acrescentar um exemplo. ## Ferramentas relacionadas