{"id":126,"date":"2020-08-05T14:40:01","date_gmt":"2020-08-05T14:40:01","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/08\/05\/word-embedding-fairness-evaluation\/"},"modified":"2020-08-05T14:40:01","modified_gmt":"2020-08-05T14:40:01","slug":"word-embedding-fairness-evaluation","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/08\/05\/word-embedding-fairness-evaluation\/","title":{"rendered":"Word Embedding Fairness Evaluation"},"content":{"rendered":"
\n

By Pablo Badilla<\/a>\u00a0and Felipe Bravo-Marquez<\/a>.<\/strong><\/p>\n

\u00a0<\/p>\n

About WEFE<\/h3>\n

\u00a0<\/p>\n

Word embeddings<\/a>\u00a0are dense vector representations of words trained from document corpora. They have become a core component of natural language processing (NLP) downstream systems because of their ability to efficiently capture semantic and syntactic relationships between words. A widely reported shortcoming of word embeddings is that they are prone to inherit stereotypical social biases exhibited in the corpora on which they are trained.<\/p>\n

The problem of how to quantify the mentioned biases is currently an active area of research, and several different\u00a0fairness metrics<\/em>\u00a0have been proposed in the literature in the past few years.<\/p>\n

Although all metrics have a similar objective, the relationship between them is by no means clear. Two issues that prevent a clean comparison is that they operate with different inputs (pairs of words, sets of words, multiple sets of words, and so on) and that their outputs are incompatible with each other (reals, positive numbers,\u00a0\u00a0range, etc.). This leads to a lack of consistency between them, which causes several problems when trying to compare and validate their results.<\/p>\n

We propose the\u00a0Word Embedding Fairness Evaluation<\/strong>\u00a0(WEFE)<\/a>\u00a0as a framework for measuring fairness in word embeddings, and we released its implementation as an\u00a0open-source library<\/a>.<\/p>\n

\u00a0<\/p>\n

Framework<\/h3>\n

\u00a0<\/p>\n

We propose an abstract view of a\u00a0fairness metric<\/em>\u00a0as a function that receives\u00a0queries<\/em>\u00a0as input, with each query formed by a\u00a0target<\/em>\u00a0and\u00a0attribute<\/em>\u00a0words. The\u00a0target words<\/em>\u00a0describe the social groups in which fairness is intended to be measured (e.g., women, white people, Muslims), and the\u00a0attribute words<\/em>\u00a0describe traits or attitudes by which a bias towards one of the social groups may be exhibited (e.g., pleasant vs. unpleasant terms). For more details on the framework, you can read our recently accepted paper IJCAI paper [1].<\/p>\n

WEFE implements the following metrics:<\/p>\n

\u00a0<\/p>\n

Standard usage pattern of WEFE<\/h3>\n

\u00a0<\/p>\n

The standard process for measuring bias using WEFE is shown in the following diagram:<\/p>\n

<\/p>\n

\u00a0<\/p>\n

Installation<\/h3>\n

\u00a0<\/p>\n

There are two different ways to install WEFE:<\/p>\n

or<\/p>\n

\n
conda install -c pbadilla wefe\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
Running a Query<\/h3>\n
\u00a0<\/p>\n
In the following code, we measure the\u00a0gender bias<\/em>\u00a0of\u00a0word2vec<\/em>\u00a0using:<\/p>\n
Given a word embedding \"w\", WEAT defines first the measure\u00a0\"d(w,A_1,A_2)\u00a0where\u00a0\"cos(w,\u00a0is the cosine similarity of the word embedding vectors.<\/p>\n
Then for a query\u00a0\"Q=({T_1,T_2},{A_1,A_2})\"\u00a0the WEAT metric is defined over the embeddings of the query word sets as:<\/p>\n
\"F_WEAT(M,<\/p>\n
The idea is that the more positive the value given by\u00a0\"F_WEAT\", the more the target\u00a0\"T_1\"\u00a0will be related to attribute\u00a0\"A_1\"\u00a0and target\u00a0\"T_2\"\u00a0to attribute\u00a0\"A_2\". On the other hand, the more negative the value, the more target \"T_1\"\u00a0will be related to attribute\u00a0\"A_2\"\u00a0and target\u00a0\"T_2\"\u00a0to attribute\u00a0\"A_1\". Commonly these values are between\u00a0\"+-0.5\"\u00a0and\u00a0\"+-2\". The ideal score is\u00a0\"+-0.5\".<\/p>\n
1. We first load a word embedding model using the\u00a0gensim<\/a>\u00a0API.<\/p>\n
\n
from wefe import WordEmbeddingModel, Query, WEAT\r\nimport gensim.downloader as api\r\n\r\nword2vec_model = WordEmbeddingModel(api.load('word2vec-google-news-300'),\r\n                                    'word2vec-google-news-300')\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
2. Then, we create the Query object using the target words (Male names<\/em>\u00a0and\u00a0Female names<\/em>) and two attribute words sets (Career<\/em>\u00a0and\u00a0Family<\/em>\u00a0terms).<\/p>\n
\n
# target sets (sets of popular names in the US)\r\nmale_names = ['John', 'Paul', 'Mike', 'Kevin', 'Steve', 'Greg', 'Jeff', 'Bill']\r\nfemale_names = ['Amy', 'Joan', 'Lisa', 'Sarah', 'Diana', 'Kate', 'Ann', 'Donna']\r\n\r\n#attribute sets\r\ncareer = ['executive', 'management', 'professional', 'corporation',\r\n         'salary', 'office', 'business', 'career']\r\nfamily = ['home', 'parents', 'children', 'family', 'cousins', 'marriage',\r\n         'wedding', 'relatives']\r\n\r\ngender_occupation_query = Query([male_names, female_names],\r\n                                [career, family],\r\n                                ['Male names', 'Female names'],\r\n                                ['Career', 'Family'])\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
3. Finally, we run the Query using WEAT as the metric.<\/p>\n
\n
weat = WEAT()\r\nresults = weat.run_query(gender_occupation_query, word2vec_model)\r\n\r\nprint(results\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
\n
{'query_name': 'Male names and Female names wrt Career and Family',\r\n 'result': 1.251}\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
As we can see, the execution returns a\u00a0dict<\/em>\u00a0with the name of the executed query and its score. The score being positive and higher than one indicates that\u00a0word2vec<\/em>\u00a0exhibits a moderately strong relationship between men’s names and careers and women’s names and family.<\/p>\n
Running multiple Queries<\/strong><\/p>\n
In WEFE, we can easily test multiple queries in one single call:<\/p>\n
1. Create the queries:<\/p>\n
\n
from wefe.utils import run_queries\r\nfrom wefe.datasets import load_weat\r\n\r\n# Load the sets used in the weat case study\r\nweat_wordsets = load_weat()\r\n\r\ngender_math_arts_query = Query(\r\n    [male_names, female_names],\r\n    [weat_wordsets['math'], weat_wordsets['arts']],\r\n    ['Male names', 'Female names'],\r\n    ['Math', 'Arts'])\r\n\r\ngender_science_arts_query = Query(\r\n    [male_names, female_names],\r\n    [weat_wordsets['science'], weat_wordsets['arts_2']],\r\n    ['Male names', 'Female names'],\r\n    ['Science', 'Arts'])\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
2. Add the queries to an array:<\/p>\n
\n
queries = [\r\n    gender_occupation_query,\r\n    gender_math_arts_query,\r\n    gender_science_arts_query,\r\n]\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
3. Run the queries using WEAT:<\/p>\n
\n
weat_results = run_queries(WEAT, queries, [word2vec_model])\r\nweat_results\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
Note that these results are returned as DataFrame objects.<\/p>\n\n\n\n\n\n
model_name<\/strong><\/td>\nMale names and Female names wrt Career and Family<\/strong><\/td>\nMale names and Female names wrt Math and Arts<\/strong><\/td>\nMale names and Female names wrt Science and Arts<\/strong><\/td>\n<\/tr>\n<\/thead>\n
word2vec-google-news-300<\/td>\n1.25161<\/td>\n0.631749<\/td>\n0.535707<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u00a0<\/p>\n
We can see that in all cases, male names are positively associated with career, science and math words, whereas female names are more associated with family and art terms.<\/p>\n
While the above results give us an idea of the gender bias that\u00a0word2vec\u00a0exhibits, we would also like to know how these biases occur in other Word Embeddings models.<\/p>\n
We run the same queries on two other embedding models: “glove-wiki” and “glove-twitter”.<\/p>\n
1. Load glove models and execute the queries again:<\/p>\n
\n
# load the models.\r\nglove_wiki = WordEmbeddingModel(api.load('glove-wiki-gigaword-300'),\r\n                                'glove-wiki-gigaword-300')\r\nglove_twitter = WordEmbeddingModel(api.load('glove-twitter-200'),\r\n                                'glove-twitter-200')\r\n\r\n# add the models to an array.\r\nmodels = [word2vec_model, glove_wiki, glove_twitter]\r\n\r\n# run the queries.\r\nresults = run_queries(WEAT, queries, models)\r\nresults\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n\n\n\n\n\n\n\n
Model Name<\/strong><\/td>\nMale names and Female names wrt Career and Family<\/strong><\/td>\nMale names and Female names wrt Math and Arts<\/strong><\/td>\nMale names and Female names wrt Science and Arts<\/strong><\/td>\n<\/tr>\n<\/thead>\n
word2vec-google-news-300<\/td>\n1.25161<\/td>\n0.631749<\/td>\n0.535707<\/td>\n<\/tr>\n
glove-wiki-gigaword-300<\/td>\n1.31949<\/td>\n0.536996<\/td>\n0.549819<\/td>\n<\/tr>\n
glove-twitter-200<\/td>\n0.537437<\/td>\n0.445879<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u00a0<\/p>\n
2. We can also plot the results:<\/p>\n
\n
from wefe.utils import plot_queries_results\r\nplot_queries_results(results)\r\n<\/pre>\n<\/div>\n
<\/p>\n
\u00a0<\/p>\n
Aggregating Results<\/h3>\n
\u00a0<\/p>\n
The execution of\u00a0run_queries\u00a0in the previous step gave us various result scores. However, these do not tell us much about the overall fairness of the embedding models.<\/p>\n
We would like to have some mechanism to aggregate these results into a single score.<\/p>\n
To do this, when using\u00a0run_queries, you can set the\u00a0add_results\u00a0parameter to\u00a0True. This will activate the option to add the results by averaging the absolute values of the results and putting them in the last column of the result table.<\/p>\n
It is also possible to ask the function\u00a0run_queries\u00a0to return only the aggregated results by setting the\u00a0return_only_aggregation\u00a0parameter to\u00a0True.<\/p>\n
\n
weat_results = run_queries(WEAT,\r\n                        queries,\r\n                        models,\r\n                        aggregate_results=True,\r\n                        return_only_aggregation=True,\r\n                        queries_set_name='Gender bias')\r\nweat_results\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n\n\n\n\n\n\n\n
model_name<\/strong><\/td>\nWEAT: Gender bias average of abs values score<\/strong><\/td>\n<\/tr>\n<\/thead>\n
word2vec-google-news-300<\/td>\n0.806355<\/td>\n<\/tr>\n
glove-wiki-gigaword-300<\/td>\n0.802102<\/td>\n<\/tr>\n
glove-twitter-200<\/td>\n0.438586<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u00a0<\/p>\n
The idea of this type of aggregation is to quantify the amount of bias of the embedding model according to various queries. In this case, we can see that glove-twitter has a lower amount of gender bias than the other models.<\/p>\n
\u00a0<\/p>\n
Rank Word Embeddings<\/h3>\n
\u00a0<\/p>\n
Finally, we would like to rank these embedding models according to the overall amount of bias they contain. This can be done using the\u00a0create_ranking\u00a0function, which calculates a fairness ranking from one or more query result.<\/p>\n
\n
from wefe.utils import create_ranking\r\n\r\nranking = create_ranking([weat_results])\r\nranking\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n\n\n\n\n\n\n\n
model_name<\/strong><\/td>\nWEAT: Gender bias average of abs values score<\/strong><\/td>\n<\/tr>\n<\/thead>\n
word2vec-google-news-300<\/td>\n3<\/td>\n<\/tr>\n
glove-wiki-gigaword-300<\/td>\n2<\/td>\n<\/tr>\n
glove-twitter-200<\/td>\n1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u00a0<\/p>\n
You can see this tutorial code in this\u00a0notebook<\/a>\u00a0and the complete reference documentation including a user guide, examples and replication of previous studies at the following\u00a0link<\/a>.<\/p>\n
If you like the project, you are more than welcome to “star” it on\u00a0Github<\/a>.<\/p>\n
[1] P. Badilla, F. Bravo-Marquez, and J. P\u00e9rez\u00a0WEFE: The Word Embeddings Fairness Evaluation Framework<\/a>\u00a0In\u00a0Proceedings of the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence (IJCAI-PRICAI 2020)<\/em>, Yokohama, Japan.<\/p>\n
\u00a0<\/p>\n
Related:<\/b><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
https:\/\/www.kdnuggets.com\/2020\/08\/word-embedding-fairness-evaluation.html<\/p>\n","protected":false},"author":0,"featured_media":127,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/126"}],"collection":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/comments?post=126"}],"version-history":[{"count":0,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/126\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media\/127"}],"wp:attachment":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media?parent=126"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/categories?post=126"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/tags?post=126"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}