# Fairness in AI: Removing word embeddings

#### Kylian van Geijtenbeek, Thom Visser, Martine Toering, Iulia Ionescu

$\textbf{Abstract:}$ This report attempts to reproduce the word embedding debiasing algorithm and replicate experiments from Bolukbasi, Chang, Zou, Saligrama, and Kalai (2016). We adapt the publicly available implementation (https://github.com/tolga-b/debiaswe) and extend it with the soft debiasing method described in their paper. Several popular benchmarks are integrated in order to evaluate the word embeddings before and after debiasing. Besides replicating results on Word2vec, the effectiveness of the debiasing algorithms is investigated on GloVe and fastText embeddings. We show that the removal of direct bias for all the embeddings barely affects their expressiveness through a comparison of benchmark scores. However, we fail to reproduce large scale soft debiasing results as the method described by the authors is not realistically applicable.

```python
from __future__ import print_function, division
%matplotlib inline
from matplotlib import pyplot as plt
import json
import random
import numpy as np
import copy

import embetter
import embetter.we as we
from embetter.we import WordEmbedding
import embetter.data as data

from embetter.debias import hard_debias
from embetter.benchmarks import Benchmark

from compare_bias import *
```

# 1 - Gender Bias in Word2vec, Glove and FastText

### Load data

In this notebook, we will use one of the three different word embeddings: $\textbf{Word2vec}$ (Mikolov et al. 2013). $\textbf{GloVe}$ (Pennington et al. 2014) and $\textbf{FastText}$ (Bojanowski et al. 2016) are also available.

The Word2vec embedding we use is learned from a corpus of Google News articles (https://code.google.com/archive/p/word2vec/). The embeddings are 300-dimensional for 3 million words. For GloVe we make use of the 300-dimensional vectors trained on Common Crawl (https://nlp.stanford.edu/projects/glove/). Lastly, FastText is a word embedding from Facebook AI Research lab trained on Wikipedia corpus and Common Crawl and also consists of 300-dimensional vectors (https://fasttext.cc/docs/en/english-vectors.html).

We start by loading in the data.

```python
# Load google news word2vec
E = WordEmbedding("word2vec_small")
# Load soft debiased word2vec (for later)
E_soft = WordEmbedding("word2vec_small_soft_debiased")
```

    Embedding shape: (26423, 300)
    26423 words of dimension 300 : in, for, that, is, ..., Jay, Leroy, Brad, Jermaine
    Embedding shape: (26423, 300)
    26423 words of dimension 300 : in, for, that, is, ..., Jay, Leroy, Brad, Jermaine

All the other embeddings can be loaded in a similar fashion, using the embedding names in the table on the GitHub Repository *(word2vec_small_hard_debiased, glove_small, glove_small_hard_debiased, glove_small_soft_debiased, word2vec_large, fasttext_small, fasttext_small_hard_debiased, fasttext_small_soft_debiased, glove_large, fasttext_large, or custom embedding file)*.

## Word2vec

```python
# Load professions and gender related lists from Bolukbasi et al. for Word2vec

gender_seed, defs, equalize_pairs, profession_words = data.load_data(E.words)
```

### Define gender direction

We define the gender direction by either PCA or by the words "she" and "he" for Word2vec. \
The PCA method is generally more robust by incorporating all definitional word pairs.

```python
# Define gender direction with the words "she" and "he"
# v_gender = E.diff('she', 'he')

# Define gender direction with PCA
v_gender = we.doPCA(defs, E).components_[0]
```

### Generating analogies

Below, we show some of the gender analogies that we can create from the embeddings. \
This method is based on "she is to X as he is to Y" analogies, with X looping through all the words and then finding the appropriate Y. \
"she" and "he" are either the embeddings of these words, or the extremes of the first principal component, depending on the method used above.

```python
# Analogies gender
a_gender = E.best_analogies_dist_thresh(v_gender, thresh=1)
we.viz(a_gender)
```

    Computing neighbors...
    Mean number of neighbors per word: 10.219732808538016
    Median of number of neighbors per word: 7.0
    Index                          Analogy                   Gender score
    ---------------------------------------------------------------------
       0                      herself | himself                      0.94
       1                          she | he                           0.94
       2                          her | his                          0.91
       3                        woman | man                          0.82
       4                     daughter | son                          0.74
       5                         girl | boy                          0.74
       6                      actress | actor                        0.72
       7                businesswoman | businessman                  0.70
       8                       sister | brother                      0.69
       9                       mother | father                       0.69
      10                  spokeswoman | spokesman                    0.67
      11                      heroine | hero                         0.67
      12                   chairwoman | chairman                     0.67
      13                      sisters | brothers                     0.67
      14                        girls | boys                         0.65
      15                        niece | nephew                       0.64
      16                 councilwoman | councilman                   0.62
      17                          gal | dude                         0.61
      18                         aunt | uncle                        0.60
      19                granddaughter | grandson                     0.59
      20                        queen | king                         0.59
      21                   motherhood | fatherhood                   0.59
      22                   schoolgirl | schoolboy                    0.59
      23                         Anne | John                         0.58
      24                        women | men                          0.58
      25                  twin_sister | twin_brother                 0.58
      26                       petite | lanky                        0.58
      27                    matriarch | patriarch                    0.56
      28               ovarian_cancer | prostate_cancer              0.56
      29                          mom | dad                          0.55
      30                      soprano | baritone                     0.55
      31                      lesbian | gay                          0.53
      32                    daughters | sons                         0.52
      33                      mothers | fathers                      0.52
      34                  grandmother | grandfather                  0.52
      35                      husband | younger_brother              0.51
      36                     sorority | fraternity                   0.51
      37                         lady | gentleman                    0.49
      38                       blouse | shirt                        0.49
      39                       queens | kings                        0.48
      40                      grandma | grandpa                      0.48
      41                         diva | superstar                    0.47
      42                         gals | dudes                        0.46
      43                      goddess | god                          0.46
      44               granddaughters | grandsons                    0.46
      45                        mommy | kid                          0.45
      46                        Aisha | Jamal                        0.45
      47                     waitress | waiter                       0.44
      48                        Sarah | Matthew                      0.44
      49                     softball | baseball                     0.44
      50                         mare | gelding                      0.44
      51                        filly | colt                         0.44
      52                         Jill | Greg                         0.43
      53                   volleyball | football                     0.43
      54                  hairdresser | barber                       0.43
      55                  girlfriends | buddies                      0.43
      56                        aunts | uncles                       0.42
      57                    actresses | actors                       0.42
      58                         hair | beard                        0.42
      59                      heiress | magnate                      0.42
      60                     feminine | manly                        0.42
      61                     princess | prince                       0.41
      62            interior_designer | architect                    0.41
      63                       ladies | gentlemen                    0.41
      64                          bra | pants                        0.41
      65                         moms | dads                         0.41
      66                congresswoman | congressman                  0.39
      67                          nun | priest                       0.39
      68                       nieces | nephews                      0.39
      69                         babe | fella                        0.39
      70                     fabulous | terrific                     0.39
      71                     vocalist | guitarist                    0.39
      72                        blond | burly                        0.38
      73                    housewife | shopkeeper                   0.38
      74                     feminism | conservatism                 0.38
      75                      starlet | youngster                    0.38
      76                       sewing | carpentry                    0.38
      77                        sassy | snappy                       0.38
      78                    glamorous | flashy                       0.38
      79                         hers | yours                        0.38
      80                 stepdaughter | stepson                      0.38
      81                        bitch | bastard                      0.38
      82                       feisty | mild_mannered                0.38
      83                      topless | shirtless                    0.37
      84                    childhood | boyhood                      0.37
      85                     gorgeous | brilliant                    0.37
      86                        nurse | physician                    0.37
      87                   compatriot | countryman                   0.37
      88                wedding_dress | tuxedo                       0.37
      89                     charming | affable                      0.36
      90                       breast | prostate                     0.36
      91                      fillies | colts                        0.36
      92                      females | males                        0.35
      93                      handbag | briefcase                    0.35
      94                       lovely | magnificent                  0.35
      95                        kinda | guy                          0.35
      96                        whore | coward                       0.35
      97                     siblings | elder_brother                0.35
      98                        salon | barbershop                   0.35
      99                       female | male                         0.35
     100                    headscarf | turban                       0.35
     101                        hubby | pal                          0.35
     102                         mums | blokes                       0.34
     103                       singer | frontman                     0.34
     104                     cupcakes | pizzas                       0.34
     105                    vocalists | trumpeter                    0.34
     106                     feminist | liberal                      0.34
     107                         kids | guys                         0.34
     108                  salesperson | salesman                     0.33
     109                         sexy | nerdy                        0.33
     110                breast_cancer | lymphoma                     0.33
     111                     estrogen | testosterone                 0.33
     112                     children | youngsters                   0.33
     113                       vagina | penis                        0.33
     114                     adorable | goofy                        0.33
     115                     cheerful | jovial                       0.33
     116                        witch | demon                        0.32
     117                       skirts | shorts                       0.32
     118                       blonde | blond                        0.32
     119                      convent | cathedral                    0.32
     120                         bras | trousers                     0.32
     121                      dresses | shirts                       0.32
     122                     luscious | crisp                        0.32
     123                       giggle | chuckle                      0.32
     124                         gown | blazer                       0.32
     125                     giggling | grinning                     0.32
     126                      Latonya | Leroy                        0.31
     127                    feminists | liberalism                   0.31
     128                       Laurie | Brett                        0.31
     129                   stepmother | eldest_son                   0.31
     130                    cosmetics | pharmaceuticals              0.31
     131                      netball | rugby                        0.31
     132                    beautiful | majestic                     0.31
     133                          wig | mustache                     0.31
     134                         pink | blue                         0.31
     135                         them | him                          0.30
     136                       uterus | intestine                    0.30
     137                    violinist | virtuoso                     0.30
     138                      midwife | doctor                       0.30
     139                        dolls | replicas                     0.30
     140                    wonderful | great                        0.30
     141                         nuns | monk                         0.30
     142             registered_nurse | paramedic                    0.30
     143                      breasts | genitals                     0.30
     144                        boobs | ass                          0.29
     145                        nanny | chauffeur                    0.29
     146                       beauty | grandeur                     0.29
     147                  middle_aged | bearded                      0.29
     148                   delightful | superb                       0.29
     149           physical_therapist | orthopedic_surgeon           0.29
     150                teenage_girls | youths                       0.29
     151                        dress | wearing                      0.29
     152                    rebounder | playmaker                    0.29
     153                     lingerie | menswear                     0.29
     154                      hostess | bartender                    0.29
     155                      friends | buddy                        0.29
     156                    mammogram | colonoscopy                  0.29
     157                    boyfriend | stepfather                   0.29
     158                 spokespeople | spokesmen                    0.29
     159                     alluring | intriguing                   0.29
     160                     swimwear | sportswear                   0.29
     161                     implants | stents                       0.29
     162                schoolteacher | carpenter                    0.29
     163                       sultry | mellow                       0.28
     164                       sexism | racism                       0.28
     165                      singers | guitarists                   0.28
     166                      sensual | moody                        0.28
     167                   enchanting | splendid                     0.28
     168                 practicality | durability                   0.28
     169                  captivating | masterful                    0.28
     170                         cute | clever                       0.28
     171                     gymnasts | wrestlers                    0.28
     172                        bride | groom                        0.28
     173                     lesbians | homosexuals                  0.28
     174                   cheesecake | pizza                        0.28
     175            singer_songwriter | musician                     0.28
     176                     ethereal | brooding                     0.28
     177                      gymnast | wrestler                     0.28
     178                       libero | midfielders                  0.28
     179           guidance_counselor | headmaster                   0.28
     180                 teenage_girl | teenager                     0.27
     181                     rehearse | improvise                    0.27
     182                   midfielder | winger                       0.27
     183                   hysterical | comical                      0.27
     184                   judgmental | arrogant                     0.27
     185                      giggles | chuckles                     0.27
     186                        gowns | robes                        0.27
     187                   servicemen | veterans                     0.27
     188                   gymnastics | judo                         0.27
     189                   boyfriends | friends                      0.27
     190                    cigarette | cigar                        0.26
     191                       baking | roasting                     0.26
     192                       lovers | aficionados                  0.26
     193                 choreography | footwork                     0.26
     194                    burlesque | rock_n_roll                  0.26
     195                     kindness | humility                     0.26
     196                      pianist | maestro                      0.26
     197                       sexist | racist                       0.26
     198                      kittens | neutered                     0.26
     199                        mares | stallion                     0.26
     200                     silicone | polymer                      0.26
     201              eating_disorder | alcoholism                   0.26
     202                      cupcake | donut                        0.26
     203                  housekeeper | janitor                      0.26
     204                    designers | architects                   0.26
     205                         mama | daddy                        0.26
     206                      panties | socks                        0.26
     207                       hurler | starter                      0.26
     208                  chairperson | chaired                      0.26
     209                     novelist | philosopher                  0.26
     210                    backcourt | swingman                     0.25
     211                     stroller | bicycle                      0.25
     212            male_counterparts | counterparts                 0.25
     213                        flute | trombone                     0.25
     214                         cubs | lions                        0.25
     215                     crafting | drafting                     0.25
     216                        manga | comic_books                  0.25
     217                    presenter | broadcaster                  0.25
     218                     husbands | wives                        0.25
     219                      kissing | shaking_hands                0.25
     220                        lupus | multiple_myeloma             0.25
     221                        cakes | sausages                     0.25
     222                        latex | rubber                       0.25
     223                       hooker | fullback                     0.25
     224                        bingo | gambling                     0.25
     225                         veil | cloak                        0.25
     226                      dancers | drummers                     0.25
     227                       gender | racial                       0.25
     228                       sobbed | grinned                      0.25
     229                 prostitution | drug_trafficking             0.25
     230                     antiques | memorabilia                  0.25
     231                   hair_salon | pizzeria                     0.25
     232                     costumes | props                        0.25
     233                       vocals | drummer                      0.25
     234                       fiance | cousin                       0.25
     235                    intuition | instincts                    0.25
     236             eating_disorders | substance_abuse              0.24
     237                       cougar | wolves                       0.24
     238                       Carrie | Laurie                       0.24
     239                     freshmen | rookies                      0.24
     240                      amazing | unbelievable                 0.24
     241                     artistry | genius                       0.24
     242             flight_attendant | pilots                       0.24
     243                         teen | youth                        0.24
     244                     sweaters | jerseys                      0.24
     245                        smile | grin                         0.24
     246                       layups | downfield                    0.24
     247                    harmonies | riffs                        0.24
     248                 osteoporosis | atherosclerosis              0.24
     249                       cheery | amiable                      0.24
     250                      outfits | garb                         0.24
     251                         beau | pals                         0.24
     252                    hairstyle | facial_hair                  0.24
     253                   terrifying | fearsome                     0.24
     254           child_endangerment | aggravated_assault           0.24
     255               figure_skating | hockey                       0.24
     256                     teenager | lad                          0.24
     257                      jewelry | collectibles                 0.24
     258                         chic | minimalist                   0.24
     259                        duets | saxophonist                  0.24
     260                       worker | foreman                      0.24
     261                         glam | retro                        0.24
     262                      uptight | cocky                        0.24
     263                      sweater | jersey                       0.24
     264                       rapist | thug                         0.24
     265              cervical_cancer | colorectal_cancer            0.24
     266                         hugs | shook_hands                  0.24
     267                    masculine | macho                        0.24
     268                        scarf | jacket                       0.24
     269                          hen | cock                         0.24
     270                      glitter | confetti                     0.24
     271                         grad | alumnus                      0.24
     272              romantic_comedy | flick                        0.24
     273                meter_hurdles | yard_dash                    0.24
     274                      cooking | grill                        0.24
     275                        maids | servants                     0.24
     276                    exclaimed | quipped                      0.24
     277               unconventional | unorthodox                   0.24
     278                         foal | thoroughbred                 0.23
     279                    seductive | mesmerizing                  0.23
     280          assistant_professor | professor_emeritus           0.23
     281                   classmates | teammates                    0.23
     282                      putback | yard_touchdown               0.23
     283                        retro | throwback                    0.23
     284                underclassmen | players                      0.23
     285                      baseman | offensive_lineman            0.23
     286                  transgender | homosexual                   0.23
     287              unassisted_goal | deflected                    0.23
     288                   inadequate | ineffective                  0.23
     289                     elegance | style                        0.23
     290                    tamoxifen | statins                      0.23
     291                        anime | videogames                   0.23
     292                 giant_slalom | skied                        0.23
     293                    menopause | heart_disease                0.23
     294                    sophomore | pounder                      0.23
     295                        lover | enthusiast                   0.23
     296                       plucky | hapless                      0.23
     297                   dispatcher | patrolman                    0.23
     298                    classmate | teammate                     0.23
     299                         cats | pigeons                      0.23
     300                     midwives | doctors                      0.23
     301             cosmetic_surgery | surgery                      0.23
     302                       feline | bulldog                      0.23
     303                      apparel | sporting_goods               0.23
     304                      warship | destroyer                    0.23
     305                  breast_milk | sperm                        0.23
     306                 delightfully | brilliantly                  0.23
     307                   cigarettes | cigars                       0.23
     308                       choral | composer                     0.23
     309                 ex_boyfriend | friend                       0.23
     310                heartbreaking | humbling                     0.23
     311                     seatbelt | wearing_helmet               0.23
     312                     sprinter | speedster                    0.23
     313                    regionals | playoffs                     0.22
     314                 spokesperson | statement                    0.22
     315                      realtor | builder                      0.22
     316                         lamb | butcher                      0.22
     317                   ultrasound | x_ray                        0.22
     318                       dancer | entertainer                  0.22
     319                       gossip | rumor                        0.22
     320                      stalker | pedophile                    0.22
     321                        cried | chuckled                     0.22
     322                  self_esteem | morale                       0.22
     323                    elegantly | superbly                     0.22
     324                   housewives | middle_aged                  0.22
     325                    librarian | curator                      0.22
     326                   frontcourt | playmakers                   0.22
     327                  free_throws | touchdown_pass               0.22
     328                      singing | rapping                      0.22
     329                    bartender | bouncer                      0.22
     330                      pitcher | lefthander                   0.22
     331               transformative | visionary                    0.22
     332                    paralegal | accountant                   0.22
     333                     clarinet | saxophone                    0.22
     334                   empowering | motivating                   0.22
     335                    auditions | tryout                       0.22
     336                     wellness | fitness                      0.22
     337                     neurotic | eccentric                    0.22
     338                   internship | apprenticeship               0.22
     339                 pediatrician | cardiologist                 0.22
     340                      sobbing | shouted                      0.22
     341                      vampire | superhero                    0.22
     342                          bun | burgers                      0.22
     343                       blokes | bloke                        0.22
     344                     anorexia | depression                   0.22
     345                   incumbents | incumbent                    0.22
     346                        raped | sexually_abusing             0.22
     347                    mentoring | mentor                       0.22
     348                  provocative | incendiary                   0.22
     349                      cashier | robber                       0.22
     350                         gosh | heck                         0.22
     351                        rapes | crimes                       0.22
     352                       nurses | physicians                   0.21
     353                     collagen | cartilage                    0.21
     354                       hitter | southpaw                     0.21
     355                    pregnancy | gestation                    0.21
     356                        layup | yarder                       0.21
     357                    sexuality | homosexuality                0.21
     358                  resourceful | wily                         0.21
     359                     plunging | slumping                     0.21
     360                    badminton | snooker                      0.21
     361                       shrill | louder                       0.21
     362                      romance | friendship                   0.21
     363                       clique | cronies                      0.21
     364                   delectable | sublime                      0.21
     365                    terrified | mad                          0.21
     366                      Allison | Todd                         0.21
     367                     thirteen | eleven                       0.21
     368                      campers | camp                         0.21
     369                       curves | curved                       0.21
     370                      beloved | legendary                    0.21
     371                       cloned | clone                        0.21
     372                    exercises | drills                       0.21
     373                       steals | tackles                      0.21
     374                   starvation | famine                       0.21
     375                       vulgar | profane                      0.21
     376              silver_medalist | compatriot                   0.21
     377                entrepreneurs | businessmen                  0.21
     378                       walker | crutches                     0.21
     379                      closets | lockers                      0.21
     380             graphic_designer | cartoonist                   0.21
     381                        rower | skipper                      0.21
     382                     artisans | craftsmen                    0.21
     383                experimenting | tinkering                    0.21
     384                    shortstop | defensive_lineman            0.21
     385                      couples | married_couples              0.21
     386                    horrified | incensed                     0.21
     387                       memoir | autobiography                0.21
     388                    prettiest | finest                       0.21
     389                        fairy | magical                      0.21
     390                     folklore | legend                       0.21
     391                    scripture | disciples                    0.21
     392                        satin | leather                      0.21
     393                  coordinator | manager                      0.21
     394                    counselor | adviser                      0.21
     395                       chicks | ducks                        0.21
     396               middle_blocker | leadoff_hitter               0.21
     397                      minivan | pickup_truck                 0.21
     398                    activists | supporters                   0.21
     399                       caring | selfless                     0.21
     400                       thighs | biceps                       0.21
     401                     designed | engineered                   0.21
     402                love_triangle | saga                         0.21
     403                 insecurities | frustrations                 0.21
     404                     cookbook | tome                         0.21
     405                        mills | steel_mills                  0.20
     406                  ice_skating | skateboarding                0.20
     407             critical_acclaim | plaudits                     0.20
     408                      skirted | evaded                       0.20
     409                 resignations | sacking                      0.20
     410               animal_cruelty | dogfighting                  0.20
     411                        purse | wallet                       0.20
     412                     abducted | gunned_down                  0.20
     413                    energetic | charismatic                  0.20
     414                    rehearsal | improvisation                0.20
     415                     beautify | rehabilitate                 0.20
     416                  celebrities | superstars                   0.20
     417                    turquoise | navy_blue                    0.20
     418                       celebs | stars                        0.20
     419            holistic_approach | approach                     0.20
     420                    cocktails | beers                        0.20
     421                       adored | revered                      0.20
     422                   shimmering | gleaming                     0.20
     423                      interns | fellows                      0.20
     424                  contestants | hopefuls                     0.20
     425                      smitten | enamored                     0.20
     426                      ribbons | plaques                      0.20
     427                       golfer | sportsman                    0.20
     428                         sang | chanted                      0.20
     429                    demeaning | disrespectful                0.20
     430                     clothing | merchandise                  0.20
     431                       flower | ornamental                   0.20
     432                   strawberry | potato                       0.20
     433                     makeover | revamp                       0.20
     434                   courageous | honorable                    0.20
     435                 demonstrates | underlines                   0.20
     436                   sophomores | offensive_linemen            0.20
     437                       unsafe | dangerous                    0.20
     438                  storyteller | magician                     0.20
     439                    thousands | tens                         0.20
     440                 doubleheader | game                         0.20
     441                     hormonal | metabolic                    0.20
     442                     watchers | pundits                      0.20
     443                       bubbly | champagne                    0.20
     444                      thyroid | inflammation                 0.20
     445               anthropologist | historian                    0.20
     446                  overwhelmed | humbled                      0.20
     447                    boutiques | retail_outlets               0.20
     448                         poem | eulogy                       0.20
     449               nerve_wracking | frustrating                  0.20
     450                       export | import                       0.20
     451                     freshman | redshirt_freshman            0.20
     452                       creepy | menacing                     0.20
     453                      bondage | shackles                     0.20
     454                 foul_trouble | defensively                  0.20
     455                    therapist | neurologist                  0.20
     456                       horrid | mediocre                     0.20
     457                    workshops | sessions                     0.20
     458                   articulate | thinker                      0.20
     459                    tenacious | hard_nosed                   0.20
     460                       drinks | beer                         0.20
     461                       violin | guitar                       0.20
     462                     quilting | stained_glass                0.20
     463                     boutique | shop                         0.20
     464                   mysterious | enigmatic                    0.20
     465                     soldiers | platoon                      0.20
     466                      catcher | lineman                      0.20
     467                     luminous | dazzling                     0.20
     468                        dayer | seamer                       0.20
     469                  backstretch | straightaway                 0.20
     470                    behaviors | tendencies                   0.20
     471               attractiveness | competitiveness              0.20
     472                      vibrant | dynamic                      0.20
     473                   incredibly | obviously                    0.20
     474                   scandalous | disgraceful                  0.20
     475                      firings | firing                       0.19
     476                    sergeants | lieutenants                  0.19
     477                 complainants | accusers                     0.19
     478                    homemaker | schoolteacher                0.19
     479              gender_equality | poverty_alleviation          0.19
     480                    underpass | tunnel                       0.19
     481                   characters | villain                      0.19
     482                   delicately | deftly                       0.19
     483                     critters | beasts                       0.19
     484                     advocate | proponent                    0.19
     485                        soaps | sitcoms                      0.19
     486                    complicit | culpable                     0.19
     487                           ok | alright                      0.19
     488             interim_dividend | dividends                    0.19
     489                     garments | apparel                      0.19
     490                     downsize | restructure                  0.19
     491             oncoming_traffic | swerving                     0.19
     492                           ex | former                       0.19
     493                 manipulative | cunning                      0.19
     494            sexual_harassment | misconduct                   0.19
     495                       embryo | embryonic                    0.19
     496                    endearing | likeable                     0.19
     497                         rape | attempted_murder             0.19
     498                       blooms | buds                         0.19
     499                     stripper | strippers                    0.19

These analogies offer an insight in potential biases along the specified bias axis (in this case gender). This is useful for a qualitative analysis.

### Analyzing occupational gender bias

The projection of occupations on the bias axis serves as another useful source for qualitative analysis.

```python
# Analysis of extreme male and extreme female professions
sp = E.profession_stereotypes(profession_words, v_gender)
```

                    Female                |                 Male
    -----------------------------------------------------------------------------
    0.38                    businesswoman | maestro                        -0.244
    0.379                         actress | protege                        -0.236
    0.378                       housewife | statesman                      -0.222
    0.323                       homemaker | businessman                    -0.219
    0.308                           nurse | sportsman                      -0.209
    0.302                registered_nurse | philosopher                    -0.196
    0.297                        waitress | marksman                       -0.192
    0.28                     receptionist | skipper                        -0.187
    0.278                       socialite | financier                      -0.183
    0.277                       librarian | architect                      -0.177
    0.272                            maid | magician                       -0.172
    0.265                             nun | trumpeter                      -0.172
    0.261                       ballerina | major_leaguer                   -0.16
    0.233                           nanny | salesman                       -0.157
    0.232                       paralegal | captain                        -0.154
    0.218                     hairdresser | mechanic                       -0.153
    0.211                     housekeeper | warrior                        -0.152
    0.203                      bookkeeper | lieutenant                     -0.152
    0.199                         stylist | gangster                        -0.15
    0.198               interior_designer | fighter_pilot                  -0.148

# 2 - Comparing Bias of Word2vec and FastText

We will compare the gender bias between word embeddings Word2vec and FastText. We do this by following Bolukbasi et al.'s approach shown in Figure 4 in their paper. The profession words are projected onto the gender axis for two embeddings. Each datapoint represents a profession word.

Below we compare the bias of Word2vec and FastText.

```python
E_f = WordEmbedding("fasttext_small")
compare_occupational_bias(E, E_f, ["Word2vec", "FastText"])
```

# 3 - Debiasing algorithms on Word2vec

## Hard debiasing

In hard debiasing, the gender neutral words are shifted to zero in the gender subspace (i.e. neutralized) by subtracting the projection of the neutral word embedding vector onto the gender subspace and renormalizing the resulting embedding to unit length.

## Soft debiasing

Soft debiasing is done by solving the following optimization problem as mentioned in their paper:

\begin{equation}
    \underset{T}{\min} || (TW)^T(TW) - W^TW||^2_F + \lambda ||(TN)^T (TB)||^2_F
\end{equation}

where W is the matrix of all embedding vectors, N is the matrix of the embedding vectors of the gender neutral words, B is the gender subspace, and T is the debiasing transformation that minimizes the projection of the neutral words onto the gender subspace but tries to maintain the pairwise inner products between the words.

For the optimization, we adapted specifics from Manzini, Chong, Black, and Tsvetkov (2019), where our code is based on the code they provide (https://github.com/TManzini/DebiasMulticlassWordEmbedding).

### Hard debiasing Word2vec
Firstly, we show how to manually debias embeddings of choice. \
This overwrites the embeddings in the WordEmbedding object, so make sure to re-load the biased embeddings in a new WordEmbedding object for comparison between the benchmarks (as is done in this notebook in the Benchmarks section).

```python
# Pass the WordEmbedding object which contains the embeddings that should be debiased.
hard_debias(E, gender_seed, defs, equalize_pairs)
```

Secondly, we show the effect of hard debiasing on Word2vec.

```python
# Hard debiased Word2vec
# Analysis of extreme male and extreme female professions
sp_hard_debiased = E.profession_stereotypes(profession_words, v_gender)
```

                    Female                |                 Male
    -----------------------------------------------------------------------------
    0.433                   businesswoman | businessman                    -0.433
    0.379                         actress | congressman                    -0.428
    0.378                       housewife | dad                            -0.365
    0.297                        waitress | councilman                     -0.358
    0.272                            maid | statesman                      -0.222
    0.265                             nun | salesman                       -0.157
    0.261                       ballerina | handyman                       -0.107
    0.0                       broadcaster | monk                           -0.082
    0.0                           plumber | librarian                        -0.0
    0.0                     valedictorian | magician                         -0.0
    0.0                      sportswriter | graphic_designer                 -0.0
    0.0                          observer | novelist                         -0.0
    0.0                          sergeant | assistant_professor              -0.0
    0.0                  doctoral_student | officer                          -0.0
    0.0                         fisherman | shopkeeper                       -0.0
    0.0                          landlord | filmmaker                        -0.0
    0.0                          chaplain | homemaker                        -0.0
    0.0                         paralegal | dancer                           -0.0
    0.0                             clerk | industrialist                    -0.0
    0.0                    superintendent | physicist                        -0.0

```python
# Analogies gender
a_gender_hard_debiased = E.best_analogies_dist_thresh(v_gender)
we.viz(a_gender_hard_debiased)
```

    Computing neighbors...
    Mean number of neighbors per word: 10.218597434053665
    Median of number of neighbors per word: 7.0
    Index                          Analogy                   Gender score
    ---------------------------------------------------------------------
       0                 ex_boyfriend | ex_girlfriend                1.00
       1                       mother | father                       1.00
       2                    daughters | sons                         1.00
       3                businesswoman | businessman                  1.00
       4                     daughter | son                          1.00
       5                 councilwoman | councilman                   1.00
       6                      herself | himself                      1.00
       7                        queen | king                         1.00
       8                        filly | colt                         1.00
       9                     husbands | wives                        1.00
      10                        niece | nephew                       1.00
      11                      sisters | brothers                     1.00
      12                      females | males                        1.00
      13                      mothers | fathers                      1.00
      14                  spokeswoman | spokesman                    1.00
      15                     princess | prince                       1.00
      16                   schoolgirl | schoolboy                    1.00
      17                     estrogen | testosterone                 1.00
      18                  twin_sister | twin_brother                 1.00
      19                       queens | kings                        1.00
      20                      grandma | grandpa                      1.00
      21                         girl | boy                          1.00
      22                        women | men                          1.00
      23                       female | male                         1.00
      24               granddaughters | grandsons                    1.00
      25                  grandmother | grandfather                  1.00
      26                          she | he                           1.00
      27               ovarian_cancer | prostate_cancer              1.0
      28                       ladies | gentlemen                    1.0
      29                          her | his                          1.0
      30                   motherhood | fatherhood                   1.0
      31                         aunt | uncle                        1.0
      32                     sorority | fraternity                   1.0
      33                congresswoman | congressman                  1.0
      34                        girls | boys                         1.0
      35                      convent | monastery                    1.0
      36                        woman | man                          1.0
      37                         gals | dudes                        1.0
      38                   chairwoman | chairman                     0.99
      39                granddaughter | grandson                     0.99
      40                          mom | dad                          0.99
      41                         moms | dads                         0.99
      42                         mare | gelding                      0.99
      43                       sister | brother                      0.99
      44                      actress | actor                        0.64
      45                          gal | dude                         0.61
      46                      lesbian | gay                          0.54
      47                   compatriot | countryman                   0.52
      48                      husband | younger_brother              0.51
      49                      heroine | protagonist                  0.49
      50                    actresses | actors                       0.48
      51                    housewife | homemaker                    0.43
      52                     waitress | waiter                       0.43
      53                        aunts | uncles                       0.42
      54                     feminism | feminist                     0.42
      55                     mustache | beard                        0.42
      56                         hers | theirs                       0.41
      57                          kid | guy                          0.40
      58                        fella | gentleman                    0.39
      59                       nieces | nephews                      0.39
      60                teenage_girls | teenagers                    0.39
      61                          nun | monk                         0.38
      62                 stepdaughter | stepson                      0.38
      63                    childhood | boyhood                      0.38
      64                        mommy | daddy                        0.37
      65                           me | him                          0.37
      66                      goddess | god                          0.36
      67                       viagra | cialis                       0.36
      68                         diva | superstar                    0.36
      69                      fillies | colts                        0.36
      70                       brides | bridal                       0.35
      71                    matriarch | patriarch                    0.35
      72                         maid | housekeeper                  0.35
      73                      hostess | bartender                    0.34
      74                       vagina | penis                        0.33
      75                         mama | fella                        0.33
      76                 teenage_girl | teenager                     0.32
      77                   stepmother | eldest_son                   0.31
      78                    ballerina | dancer                       0.30
      79                    maternity | midwives                     0.30
      80                 grandmothers | grandparents                 0.30
      81                  compatriots | countrymen                   0.29
      82                        witch | witchcraft                   0.29
      83                    boyfriend | stepfather                   0.29
      84                       uterus | intestine                    0.28
      85                    menopause | puberty                      0.28
      86                      heiress | socialite                    0.28
      87                        bride | wedding                      0.28
      88                     lesbians | gays                         0.27
      89                       eldest | elder_brother                0.27
      90                   politician | statesman                    0.25
      91                        maids | servants                     0.24
      92                     dictator | strongman                    0.24
      93                    youngster | lad                          0.24
      94                         nuns | priests                      0.24
      95                     maternal | infant_mortality             0.24
      96                        hubby | pal                          0.23
      97                       blokes | bloke                        0.22
      98                         lady | waitress                     0.21
      99                      soprano | baritone                     0.21
     100                  girlfriends | buddies                      0.21
     101                   boyfriends | girlfriend                   0.20
     102                  facial_hair | beards                       0.19
     103                         womb | fetus                        0.18
     104               businesspeople | businessmen                  0.18
     105                       fiance | roommate                     0.18
     106                         beau | lover                        0.18
     107                  salesperson | salesman                     0.18
     108                      witches | vampires                     0.18
     109            estranged_husband | estranged                    0.17
     110                 counterparts | brethren                     0.16
     111                      bastard | chap                         0.16
     112                        widow | deceased                     0.16
     113                   obstetrics | pediatrics                   0.16
     114                 spokespeople | spokesmen                    0.16
     115                   friendship | brotherhood                  0.15
     116                         hens | chickens                     0.15
     117                          hen | cock                         0.15
     118                      replied | sir                          0.14
     119                        colon | prostate                     0.14
     120                     mistress | prostitute                   0.14
     121                     stallion | stud                         0.13
     122                        manly | macho                        0.13
     123                         wife | cousin                       0.13
     124                           ma | na                           0.13
     125                    carpenter | handyman                     0.12
     126                        bulls | bull                         0.12
     127                       widows | families                     0.11
     128                  salespeople | salesmen                     0.10
     129                   girlfriend | friend                       0.10
     130                       suitor | takeover_bid                 0.09
     131                       gaffer | lads                         0.09
     132                        semen | saliva                       0.09
     133                    elephants | lions                        0.09
     134                      suitors | bidders                      0.08
     135                      fiancee | married                      0.08
     136                         guys | fellas                       0.08
     137                   hair_salon | barbershop                   0.07
     138                     elephant | lion                         0.05
     139                        colts | mares                        0.05
     140                           pa | mo                           0.05
     141                        footy | blokes                       0.04
     142                     aldermen | councilmen                   0.04
     143                        monks | monasteries                  0.04
     144                      widower | widowed                      0.03
     145                     bachelor | bachelor_degree              0.02
     146                        sperm | embryos                      0.02
     147                         deer | elk                          0.02
     148              residence_halls | fraternities                 0.01
     149                      wedlock | fathered                     0.01
     150                        penis | genitals                     0.00
     151                      princes | royals                       0.00

# 4 - Benchmarks

This package includes some basic benchmarks, which allow for easy verification of the embedding's quality before and after debiassing (RG-65, WS-353 and MSR), as well as a statistical measure of bias to quantitatively inspect the effect of debiassing (WEAT).

```python
benchmark = Benchmark()
E_before = WordEmbedding("word2vec_small")
```

    Embedding shape: (26423, 300)
    26423 words of dimension 300 : in, for, that, is, ..., Jay, Leroy, Brad, Jermaine

### Word2vec

Below, we show the benchmarks for Word2vec. \
When comparing the WEAT effect size, make sure to use a single Benchmark object per embedding, benchmarking the biased version first, as is done in this notebook. (This is because the first bias axis is saved internally to measure bias along it for subsequent embeddings.)

```python
# Evaluate for Word2vec
before_results = benchmark.evaluate(E_before, "Before")
hard_results = benchmark.evaluate(E, "Hard")
soft_results = benchmark.evaluate(E_soft, "Soft")

```

    100%|██████████| 40/40 [00:05<00:00,  7.86it/s]

    +-------------------------------------------------------+
    |                   Results for Before                  |
    +---------------+-------+-----------+-------------------+
    | Dataset       | Found | Not Found |       Score       |
    +---------------+-------+-----------+-------------------+
    | EN-RG-65      |   53  |     12    | 77.66555804950227 |
    | EN-WS-353-ALL |  318  |     35    | 68.82719646959825 |
    | MSR-analogy   |  5276 |    2724   | 46.79681576952237 |
    | WEAT          |   -   |     -     |     1.4849316     |
    +---------------+-------+-----------+-------------------+

    100%|██████████| 40/40 [00:04<00:00,  8.82it/s]

    +--------------------------------------------------------+
    |                    Results for Hard                    |
    +---------------+-------+-----------+--------------------+
    | Dataset       | Found | Not Found |       Score        |
    +---------------+-------+-----------+--------------------+
    | EN-RG-65      |   53  |     12    | 77.49622028082247  |
    | EN-WS-353-ALL |  318  |     35    | 68.52623098234018  |
    | MSR-analogy   |  5276 |    2724   | 46.967399545109934 |
    | WEAT          |   -   |     -     |     0.36466902     |
    +---------------+-------+-----------+--------------------+

    100%|██████████| 40/40 [00:04<00:00,  8.34it/s]

    +-------------------------------------------------------+
    |                    Results for Soft                   |
    +---------------+-------+-----------+-------------------+
    | Dataset       | Found | Not Found |       Score       |
    +---------------+-------+-----------+-------------------+
    | EN-RG-65      |   53  |     12    | 77.66555804950227 |
    | EN-WS-353-ALL |  318  |     35    | 68.82719646959825 |
    | MSR-analogy   |  5276 |    2724   | 46.79681576952237 |
    | WEAT          |   -   |     -     |    -0.22672768    |
    +---------------+-------+-----------+-------------------+

The results of individual benchmarks can be combined into a single list and passed to benchmark's `pprint_compare` method for easy comparison.

```python
w2v_results = [before_results, hard_results, soft_results]
benchmark.pprint_compare(w2v_results, ["Before", "Hard-debiased", "Soft-debiased"], "word2vec")
```

    +------------------------------------------------------------------------------------------+
    |                                   Results for word2vec                                   |
    +---------------+-------------------+-------------------+--------------------+-------------+
    |     Score     |       RG-65       |       WS-353      |        MSR         |     WEAT    |
    +---------------+-------------------+-------------------+--------------------+-------------+
    |     Before    | 77.66555804950227 | 68.82719646959825 | 46.79681576952237  |  1.4849316  |
    | Hard-debiased | 77.49622028082247 | 68.52623098234018 | 46.967399545109934 |  0.36466902 |
    | Soft-debiased | 77.66555804950227 | 68.82719646959825 | 46.79681576952237  | -0.22672768 |
    +---------------+-------------------+-------------------+--------------------+-------------+

# 5 - Full Experiments

The full range of experiments can be executed using the `experiments.py` file from the repository. \
This is best done through a terminal, to modify the behaviour using command line arguments, but its also available here. \
This line outputs comparison benchmarks for all small embeddings. \
Gender analogies and occupational gender bias are not shown for *word2vec_small*, *glove_small*, and *fasttext_small* before debiasing, after hard-debiasing and after soft-debiasing, through the `--no_show` option.

```python
!python experiments.py --no_show
```

    ################## EXPERIMENT DETAILS ##################
    #                                                      #
    # Analogies and occupations: False                     #
    # Do soft debiasing from scratch: False                #
    # Perform benchmarks: True                             #
    #                                                      #
    # Performing experiments for the following embeddings: #
    #	- word2vec_small                               #
    #	- glove_small                                  #
    #	- fasttext_small                               #
    #                                                      #
    ########################################################

    ########################################################
    #           Doing the word2vec_small embedding         #
    ########################################################
    Embedding shape: (26423, 300)
    26423 words of dimension 300 : in, for, that, is, ..., Jay, Leroy, Brad, Jermaine

    Hard debiasing...

    Soft debiasing...
    Embedding shape: (26423, 300)
    26423 words of dimension 300 : in, for, that, is, ..., Jay, Leroy, Brad, Jermaine

    Running benchmarks...
    100%|███████████████████████████████████████████| 40/40 [00:04<00:00,  8.90it/s]
    100%|███████████████████████████████████████████| 40/40 [00:04<00:00,  9.51it/s]
    100%|███████████████████████████████████████████| 40/40 [00:04<00:00,  9.65it/s]
    +------------------------------------------------------------------------------------------+
    |                                Results for word2vec_small                                |
    +---------------+-------------------+-------------------+--------------------+-------------+
    |     Score     |       RG-65       |       WS-353      |        MSR         |     WEAT    |
    +---------------+-------------------+-------------------+--------------------+-------------+
    |     Before    | 77.66555804950227 | 68.82719646959825 | 46.79681576952237  |  1.4785795  |
    | Hard-debiased | 77.49622028082247 | 68.52623098234018 | 46.967399545109934 |  0.37883297 |
    | Soft-debiased | 77.66555804950227 | 68.82719646959825 | 46.79681576952237  | -0.15358597 |
    +---------------+-------------------+-------------------+--------------------+-------------+

    ########################################################
    #            Doing the glove_small embedding           #
    ########################################################
    Embedding shape: (42982, 300)
    42982 words of dimension 300 : the, and, to, of, ..., cushman, darkside, motherland, chairmen

    Hard debiasing...

    Soft debiasing...
    Embedding shape: (42982, 300)
    42982 words of dimension 300 : the, and, to, of, ..., cushman, darkside, motherland, chairmen

    Running benchmarks...
    100%|███████████████████████████████████████████| 40/40 [00:07<00:00,  5.56it/s]
    100%|███████████████████████████████████████████| 40/40 [00:07<00:00,  5.32it/s]
    100%|███████████████████████████████████████████| 40/40 [00:07<00:00,  5.37it/s]
    +-----------------------------------------------------------------------------------------+
    |                                 Results for glove_small                                 |
    +---------------+-------------------+-------------------+-------------------+-------------+
    |     Score     |       RG-65       |       WS-353      |        MSR        |     WEAT    |
    +---------------+-------------------+-------------------+-------------------+-------------+
    |     Before    | 83.10311410756115 | 66.43748693628271 | 37.47211895910781 |  -1.6155657 |
    | Hard-debiased | 83.35243100494134 | 66.61161322854848 | 37.62081784386617 | -0.39784268 |
    | Soft-debiased |  83.0703755250769 | 66.44364789148896 | 37.43494423791822 | -0.69473237 |
    +---------------+-------------------+-------------------+-------------------+-------------+

    ########################################################
    #           Doing the fasttext_small embedding         #
    ########################################################
    Embedding shape: (27014, 300)
    27014 words of dimension 300 : the, and, of, to, ..., circumscribed, whos, salvaging, anion

    Hard debiasing...

    Soft debiasing...
    Embedding shape: (27014, 300)
    27014 words of dimension 300 : the, and, of, to, ..., circumscribed, whos, salvaging, anion

    Running benchmarks...
    100%|███████████████████████████████████████████| 40/40 [00:03<00:00, 10.44it/s]
    100%|███████████████████████████████████████████| 40/40 [00:03<00:00, 10.90it/s]
    100%|███████████████████████████████████████████| 40/40 [00:03<00:00, 10.88it/s]
    +----------------------------------------------------------------------------------------+
    |                               Results for fasttext_small                               |
    +---------------+-------------------+-------------------+-------------------+------------+
    |     Score     |       RG-65       |       WS-353      |        MSR        |    WEAT    |
    +---------------+-------------------+-------------------+-------------------+------------+
    |     Before    | 83.86622701863348 | 74.10786418997199 | 55.87019429516329 | 1.4813895  |
    | Hard-debiased | 83.50735694621694 | 74.17924052453014 | 55.99421248449773 |  0.436271  |
    | Soft-debiased | 84.26611081361189 | 74.12157140993038 | 54.73336089293096 | 0.38817888 |
    +---------------+-------------------+-------------------+-------------------+------------+

```python

```


# Chapter 5: Resampling Methods
- **Chapter 5 from the book [An Introduction to Statistical Learning](https://www.statlearning.com/).**
- **By Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani.**
- **Pages from $197$ to $198$**
- **By [Mosta Ashour](https://www.linkedin.com/in/mosta-ashour/)**

**Exercises:**
- **[1.](#1)**
- **[2.](#2) [(a)](#2a) [(b)](#2b) [(c)](#2c) [(d)](#2d) [(e)](#2e) [(f)](#2f) [(g)](#2g) [(h)](#2h)**
- **[3.](#3) [(a)](#3a) [(b)](#3b)**
- **[4.](#4)**

#  5.4 Exercises
##  Conceptual

### $1.$ Using basic statistical properties of the variance, as well as single-variable calculus, derive $(5.6)$. In other words, prove that $\alpha$ given by $(5.6)$ does indeed minimize $\mathrm{Var}(\alpha X + (1 − \alpha)Y)$.

**Using the following variance [rules](http://www.kaspercpa.com/statisticalreview.htm):**

$
\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y) + 2 Cov(X,Y)
\\
\text{Var}(cX) = c^2 Var(X)
\\
\text{So}\dots
\\
\text{Var}(cX+dY) = c^2 \text{Var}(X) + d^2 \text{Var}(Y) + 2cd Cov(X,Y)
$

$
\text{thus:}\\
f(\alpha) = \text{Var}(\alpha X + (1 - \alpha)Y)\\
\\
f(\alpha) = \alpha^2 \text{Var}(X) + (1 - \alpha)^2 \text{Var}(Y) + 2 \alpha (1 - \alpha) Cov(X, Y)\\
\\
f(\alpha) = \alpha^2 \sigma_X^2 + (1 - \alpha)^2 \sigma_Y^2 + 2 (\alpha-\alpha^2 ) \sigma_{XY}\\
$

$
\begin{align}
\text{Take the first derivative:}\\
\frac {d} {d\alpha} f(\alpha) &= 0\\
\\
2 \alpha \sigma_X^2 + 2 (1 - \alpha) (-1) \sigma_Y^2 + 2 (1 - 2 \alpha ) \sigma_{XY} &= 0\\
\\
\alpha \sigma_X^2 + (\alpha - 1) \sigma_Y^2 + (-2 \alpha + 1) \sigma_{XY} &= 0\\
\\
\color{red}{\alpha \sigma_X^2} + \color{red}{\alpha\sigma_Y^2} - \sigma_Y^2 \color{red}{-2 \alpha \sigma_{XY}} + \sigma_{XY} &= 0\\
\\
\alpha (\sigma_X^2 + \sigma_Y^2 - 2 \sigma_{XY}) - \sigma_Y^2 + \sigma_{XY} &= 0\\
\\
\alpha (\sigma_X^2 + \sigma_Y^2 - 2 \sigma_{XY}) &= \sigma_Y^2 - \sigma_{XY}\\
\\
\text{therefore:}
\\
\alpha &= \frac {\sigma_Y^2 - \sigma_{XY}}
               {\sigma_X^2 + \sigma_Y^2 - 2 \sigma_{XY}}
\\
\text{As required.}\\
\end{align}
$

### $2.$ We will now derive the probability that a given observation is part of a bootstrap sample. Suppose that we obtain a bootstrap sample from a set of $n$ observations.

**$(a)$ What is the probability that the first bootstrap observation is not the $j$th observation from the original sample? Justify your answer.**

>- **Answer:**
>  - $p(n_{1} \neq n_j) = \frac{n-1}{n}$
>  - Which all the others observations are possible except just the first observation "1".

**$(b)$ What is the probability that the second bootstrap observation is not the $j$th observation from the original sample?**

>- **Answer:**
>  - The same as the previous question:
>  - $p(n_{2} \neq n_j) = \frac{n-1}{n}$
>  - Which all the others observations are possible except just the second observation "1".

**$(c)$ Argue that the probability that the $j$th observation is not in the bootstrap sample is $(1 - 1/n)^n$.**

>- **Answer:**
>  - The probability that the $j$th observation is not in the bootstrap sample is
>  - $(1−1/n)_1⋅(1−1/n)_2⋅(1−1/n)_3\dots⋅(1−1/n)_n$
>  - which is: $(1−1/n)^n$
>  - Since choosing any sample is independent of another but has the same probability value.

**$(d)$ When $n = 5$, what is the probability that the $j$th observation is in the bootstrap sample?**

>- **Answer:**
>  - when $n = 5$ and it is **in** the bootstrap sample:
>  - $p = 1 - (1-1/5)^5$
>  - $p = 0.67232$

**$(e)$ When n = 100, what is the probability that the $j$th observation is in the bootstrap sample?**

>- **Answer:**
>  - when $n = 100$ and it is **in** the bootstrap sample:
>  - $p = 1 - (1-1/100)^{100}$
>  - $p = 0.63397$

**$(f)$ When $n = 10,000$, what is the probability that the $j$th observation is in the bootstrap sample?**

>- **Answer:**
>  - when $n = 10,000$ and it is **in** the bootstrap sample:
>  - $p = 1 - (1-1/10000)^{10000}$
>  - $p = 0.63213$

**$(g)$ Create a plot that displays, for each integer value of $n$ from $1$ to $100,000$, the probability that the $j$th observation is in the bootstrap sample. Comment on what you observe.**

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

def prob_j_in_sample(n):
    return 1 - (1 - 1/n)**n

x = np.arange(1, 100000)
y = np.array([prob_j_in_sample(n) for n in x])

plt.figure(figsize=(12,6))
ax = sns.lineplot(x=x, y=prob_j_in_sample(x))
plt.xlabel('$n$')
plt.ylabel('probability')
plt.ylim((0,1));
```

>- **Comments:**
>  - We are seeing a visual representation of $\lim\limits_{x \to \infty} 1 - (1 - \frac{1}{n})^n = 1 - \frac{1}{\epsilon} \approx 0.632$

**$(h)$ We will now investigate numerically the probability that a bootstrap sample of size $n = 100$ contains the $j$th observation. Here $j = 4$. We repeatedly create bootstrap samples, and each time we record whether or not the fourth observation is contained in the bootstrap sample.**
```
> store=rep (NA , 10000)
> for (i in 1:10000) {
store[i]=sum(sample (1:100 , rep =TRUE)==4) >0
}
> mean(store)
```
**Comment on the results obtained.**

```python
store = []
for i in np.arange(1, 10000):
    store += [np.sum((np.random.randint(low=1, high=101, size=100) == 4)) > 0]

np.mean(store)
```

    0.6335633563356335

>- **Comments:**
>  - If we created more bootstrap samples in $h$, the result observed from a numerical approach above will approximately equal to our probabilistic estimation for a sample size of $100$ in $2.e$ which was $p = 0.63397$.

### 3. We now review $\text{k-fold}$ cross-validation.

**$(a)$ Explain how $\text{k-fold}$ cross-validation is implemented.**
>- **Answer:**
>- This approach involves randomly dividing the set of observations into $k$ groups, or folds, of approximately equal size.
>- The first $k-1$ folds is treated as a validation set, and the remaining fold as test set.
>- The model is then fitted to each of these folds, and testing on the remaining fold.
>- This procedure is repeated $k$ times; each time, a different group of observations.
>- This process results in k estimates of the test error,
$\text{MSE_1},\text{MSE_2}, \dots ,\text{MSE_k}$. The k-fold CV estimate is computed by averaging these values,
$$ CV(k) =\frac{1}{k} \sum_{i=1}^k MSE_i. $$

**$(b)$ What are the advantages and disadvantages of k-fold cross- validation relative to:**

**$i.$ The validation set approach?**

- **Answer:**

>- $\text{Advantages:}$
>  - The validation set approach is **conceptually simple** and is **easy** to implement.
>  - We can use the validation set approach in testing which predictor's transformations provides even better results, as we randomly split the observations into $k$ approximately equal sets or folds, a training set of k-1 folds, and the remaining fold is the validation set.
>  - The validation set error rates that result from fitting various regression models on the training sample and evaluating their performance on the validation sample, using $\text{MSE}$ as a measure of validation set error, comparing the $\text{MSE}$ results will lead us to the better predictor's transformations to use in our model.

>- $\text{Disadvantages:}$
>  - The validation estimate of the test error rate can be **highly variable**, depending on precisely which observations are included in the training and validation sets.
>  - Only a subset of the observations those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to **overestimate the test error rate** for the model fit on the entire data set.
>  - The approach has a computational advantage: a model is trained and tested once. In $k-fold$ CV, $k$ models will be trained, and for standard values of $k$. Means that $k-fold$ CV can be far more computationally expensive for large dataset and for large values of $k$.

**$ii.$ LOOCV?**

- **Answer:**

>- $\text{Advantages:}$
>  - It has far less bias over the validation set approach.
>  - Performing LOOCV multiple times will always yield the same results: there is no randomness in the training/validation set splits.
>  - $\text{Leave-one-out cross-validation}$ involves splitting the set of observations into two parts. However, instead of creating two subsets of comparable size, a single observation $(x_1, y_1)$ is used for the validation set, and the remaining observations ${(x_2, y_2), \dots , (x_n, y_n)}$ make up the training set. The statistical learning method is fit on the $n - 1$ training observations, and a prediction $\hat{y}_1$ is made for the excluded observation, using its value $x_1$. Since $(x_1, y_1)$ was not used in the fitting process, $\text{MSE}_1 = (y_1 - \hat{y}_1)^2$ provides an approximately unbiased estimate for the test error.

>- $\text{Disadvantages:}$
>  - Even though $\text{MSE}_1$ is unbiased for the test error, it is a poor estimate because it is highly variable, since it is based upon a single observation $(x_1, y_1)$.
>  - LOOCV requires fitting the statistical learning method $n$ times. This has the potential to be computationally expensive.

### 4. Suppose that we use some statistical learning method to make a prediction for the response Y for a particular value of the predictor X. Carefully describe how we might estimate the standard deviation of our prediction.

>- **Answer:**
>  - Which we need to estimate the standard deviation from a given $n$ observations of the population. We could might estimate the standard deviation of our prediction by using the [sample standard deviation](https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation) formula:
$$
\hat{\sigma} = \sqrt{\frac{\sum_{i=1}^{n}{(\hat{y_i} - \bar{y})^2}}{n - 1}}
$$
>  - The accuracy of this estimate is limited by its variability, so...
>  - We could improve the accuracy of this estimate by using the bootstrap approach. This works by randomly select $n$ observations from the original dataset to create a $B$ different bootstrap datasets.
>  - This procedure is repeated $B$ times for some large value of $B$, in order to produce $B$ different bootstrap datasets, $Z^{*1}, Z^{*2}, \dots , Z^{*B},$ and $B$ corresponding $\alpha$ estimates, $\hat{α}^{∗1}, \hat{α}^{∗2}, \dots , \hat{α}^{∗B}$. We can compute the standard error of these bootstrap estimates using the formula in $(5.8)$:
$$
SE_B(\hat{\alpha}) =
\sqrt{
\frac{1}{B - 1}
\sum^B_{r=1}
\bigg(\hat{\alpha}^{*r} -
\frac{1}{B}
\sum^B_{r'=1}
\hat{\alpha}^{*r}
\bigg)^2
}.
$$

# Done!