Notebook that shows the computation of the cosine similarity metrics for the consituents

```python
import pandas as pd
import numpy as np
pd.options.mode.chained_assignment = None
pd.set_option('display.float_format', lambda x: '%.3f' % x)
```

2 cosine function were created -

The first function took 5 - 6 hours to compute the cosines and hence another function was written.

```python
#old cosine function

def cosiner1(modifier_head,comcols):
    modifier=modifier_head[0]
    head=modifier_head[1]
    head_sim_numerator=np.sum(compounds.loc[modifier,head][comcols]*constituents.loc[head][comcols])
    #print(head_sim_numerator)
    #print((compounds.loc[modifier,head]['denominator']*constituents.loc[head]['denominator']))
    similarity_with_head=head_sim_numerator/(compounds.loc[modifier,head]['denominator']*constituents.loc[head]['denominator'])
    #print(similarity_with_head)
    modifier_sim_numerator=np.sum(compounds.loc[modifier,head][comcols]*constituents.loc[modifier][comcols])
    #print(modifier_sim_numerator)
    similarity_with_modifier=modifier_sim_numerator/(compounds.loc[modifier,head]['denominator']*constituents.loc[modifier]['denominator'])
    return (similarity_with_head,similarity_with_modifier)
    #print((compounds.loc[modifier,head]['denominator']*constituents.loc[modifier]['denominator']))
    #print(similarity_with_modifier)
    #compounds.loc[modifier,head]['sim_with_head']=similarity_with_head
    #print(compounds.loc[modifier,head]['sim_with_head'])
    #compounds.loc[modifier,head]['sim_with_modifier']=similarity_with_modifier
    #print(compounds.loc[modifier,head]['sim_with_modifier'])
```

```python
def cosiner_head(x):
    #print(x)
    #head_sim_numerator=np.sum(x.loc[compound_cols[:-1]]*x.loc[head_cols[:-1]])
    return np.sum(x[compound_cols[:-1]].values*x[head_cols[:-1]].values)/(x[compound_cols[-1]]*x[head_cols[-1]])
def cosiner_modifier(x):
    #modifier_sim_numerator=np.sum(x.loc[compound_cols[:-1]]*x.loc[modifier_cols[:-1]])
    return np.sum(x[compound_cols[:-1]].values*x[modifier_cols[:-1]].values)/(x[compound_cols[-1]]*x[modifier_cols[-1]])

def cosiner_constituent(x):
    #print(x)
    #head_sim_numerator=np.sum(x.loc[compound_cols[:-1]]*x.loc[head_cols[:-1]])
    return np.sum(x[modifier_cols[:-1]].values*x[head_cols[:-1]].values)/(x[modifier_cols[-1]]*x[head_cols[-1]])
```

# Loading of the datasets -

The **compound** and **constituent** are the datasets that were previously created.

```python
constituents=pd.read_csv('unigrams.csv',sep="\t",encoding='utf-8',index_col=[0])
precols=constituents.columns[:19]
constituents[precols]=constituents[precols].applymap(lambda x:np.log10(x))
constituents.fillna(0,inplace=True)
constituents['presence']=constituents["presence"].astype('category')
constituents['pattern']=constituents["pattern"].astype('category')
#constituents['denominator']=np.sqrt((constituents[precols]**2).sum(axis=1))
constituents.head(20)
```

      1810s
      1820s
      1830s
      1840s
      1850s
      1860s
      1870s
      1880s
      1890s
      1900s
      ...
      1920s
      1930s
      1940s
      1950s
      1960s
      1970s
      1980s
      cf
      presence
      pattern

      word

      a
      0.000
      0.000
      0.000
      0.000
      0.000
      2.190
      1.301
      2.354
      1.580
      1.505
      ...
      1.176
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.687
      6
      1111101000000

      a'r
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      1
      10000000000

      a'ready
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.230
      0.000
      1.114
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.653
      3
      10110000000

      a'ter
      0.000
      0.000
      0.000
      0.000
      1.568
      1.255
      0.000
      0.000
      0.000
      0.000
      ...
      1.362
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.892
      3
      11000001000000

      a-bomb
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      1.763
      1.301
      1.000
      1.204
      2.017
      4
      1111

      a-c
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      1.322
      0.000
      0.000
      0.000
      1.322
      1
      1000

      a-class
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      1.204
      0.000
      0.000
      0.000
      1.204
      1
      1000

      a-crystals
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.398
      1
      10000000

      a-fishing
      0.000
      0.000
      0.000
      0.000
      1.114
      1.000
      0.000
      0.000
      1.041
      1.041
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.653
      4
      11001100000000

      a-frame
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.204
      1.204
      1
      1

      a-gettin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      1.176
      1.204
      0.000
      0.000
      0.000
      1.491
      2
      11000

      a-gittin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.176
      0.000
      ...
      1.041
      0.000
      1.041
      0.000
      0.000
      0.000
      0.000
      1.568
      3
      1001010000

      a-gonna
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.362
      1.301
      0.000
      0.000
      0.000
      0.000
      1.633
      2
      110000

      a-gwine
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.301
      1.204
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.556
      2
      11000000000

      a-holdin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      1
      1000000000

      a-huntin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      1
      100000000

      a-hunting
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      1
      1000000000

      a-layin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      1.041
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.322
      2
      11000000000

      a-leavin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      1
      10000000

      a-lookin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.279
      0.000
      1.176
      1.041
      ...
      0.000
      0.000
      1.230
      0.000
      0.000
      0.000
      0.000
      1.869
      5
      101110010000

20 rows × 21 columns

To the **compound** dataset three new columns are added -

**sim_with_head** - Similarity of the compound with its head constituent

**sim_with_modifier** -  Similarity of the compound with its modifier constituent

**novel** - A boolean compound that is True if the compound is novel and False otherwise.

```python
compounds=pd.read_csv('collocations.csv',sep=" ",index_col=[0,1])
print(compounds.shape)
compounds.index.rename(["modifier","head"],inplace=True)
#postcols=compounds.columns[18:20]
#compounds=compounds[compounds.index.get_level_values(0).isin(constituents.index)]
#compounds=compounds[compounds.index.get_level_values(1).isin(constituents.index)]
compounds.fillna(0,inplace=True)
compounds.columns=constituents.columns
compounds['sim_with_head']=0
compounds['sim_with_modifier']=0
compounds['sim_bw_constituents']=0
#compounds(['compound_cf'],axis=1,inplace=True)
#compounds['novel']=False
#compounds['novel'][compounds.presence==0]=True
compounds.head(20)
```

    (103666, 21)

      1810s
      1820s
      1830s
      1840s
      1850s
      1860s
      1870s
      1880s
      1890s
      1900s
      ...
      1950s
      1960s
      1970s
      1980s
      cf
      presence
      pattern
      sim_with_head
      sim_with_modifier
      sim_bw_constituents

      modifier
      head

      a
      dey
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.828
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      2.016
      1
      10000000000.000
      0
      0
      0

      ef
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      3.562
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      2.593
      1
      10000000000.000
      0
      0
      0

      fur
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      3.069
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      1.683
      1
      10000000000.000
      0
      0
      0

      tho
      0.000
      0.000
      0.000
      0.000
      0.000
      4.178
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      2.441
      1
      1000000000000.000
      0
      0
      0

      youer
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      5.082
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      4.749
      1
      10000000000.000
      0
      0
      0

      a-class
      leady
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      5.946
      0.000
      0.000
      0.000
      5.946
      1
      1000.000
      0
      0
      0

      a-frame
      cottage
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      4.404
      3.050
      1
      1.000
      0
      0
      0

      a-gonna
      git
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      2.613
      1
      10000.000
      0
      0
      0

      ab
      exercise
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0
      0.000
      0
      0
      0

      aba
      convention
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      3.581
      0.000
      2.071
      1
      10.000
      0
      0
      0

      journal
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0
      0.000
      0
      0
      0

      section
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0
      0.000
      0
      0
      0

      abalone
      shell
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      4.351
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      2.714
      3
      1001010000.000
      0
      0
      0

      abbey
      ale
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0
      0.000
      0
      0
      0

      church
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.384
      2.642
      2.704
      ...
      0.000
      2.830
      0.000
      0.000
      0.976
      5
      11100100100.000
      0
      0
      0

      gate
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0
      0.000
      0
      0
      0

      land
      0.000
      0.000
      0.000
      2.454
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.028
      1
      100000000000000.000
      0
      0
      0

      library
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.654
      1
      10000000.000
      0
      0
      0

      lodge
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      4.048
      1.406
      1
      1.000
      0
      0
      0

      poem
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.748
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.732
      1
      10000000000.000
      0
      0
      0

20 rows × 24 columns

# Similarity Calculation

A temporary dataset is contructed to perform faster cosine similarity calculations.

Each row contains the 18 decade columns for each of the compound and its 2 constituents.

Cosine similarity is represented using -

\begin{equation} cos(\pmb x, \pmb y) = \frac {\pmb x \cdot \pmb y}{||\pmb x|| \cdot ||\pmb y||} \end{equation} .

The denominator values are pre-calculated for all values.

```python
cosiner=compounds[precols]
cosiner['denominator']=np.sqrt((cosiner[precols]**2).sum(axis=1))
cosiner.columns=cosiner.columns+'_compound'
cosiner.reset_index(['modifier','head'],inplace=True)
currentcolumns=cosiner.columns.tolist()
#cosiner
cosiner=pd.concat([cosiner.set_index('modifier'),constituents], axis=1, join='inner').reset_index()
cosiner.drop(['presence','pattern'],axis=1,inplace=True)
cosiner['denominator']=np.sqrt((cosiner[precols]**2).sum(axis=1))
temp_columns=cosiner.columns[22:]+'_modifier'
cosiner.columns=currentcolumns+temp_columns.tolist()
cosiner=pd.concat([cosiner.set_index('head'),constituents], axis=1, join='inner').reset_index()
cosiner.drop(['presence','pattern'],axis=1,inplace=True)
cosiner['denominator']=np.sqrt((cosiner[precols]**2).sum(axis=1))
fornow=cosiner.columns.tolist()
temp_columns=cosiner.columns[-20:]+'_head'
cosiner.columns=['head']+[fornow[1]]+fornow[2:-20]+temp_columns.tolist()
cosiner.set_index(['modifier','head'],inplace=True)
cosiner.head(20)
```

      1810s_compound
      1820s_compound
      1830s_compound
      1840s_compound
      1850s_compound
      1860s_compound
      1870s_compound
      1880s_compound
      1890s_compound
      1900s_compound
      ...
      1910s_head
      1920s_head
      1930s_head
      1940s_head
      1950s_head
      1960s_head
      1970s_head
      1980s_head
      cf_head
      denominator_head

      modifier
      head

      a
      dey
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.828
      0.000
      0.000
      ...
      2.286
      2.591
      2.452
      1.857
      1.919
      1.740
      1.000
      1.491
      3.694
      9.863

      ef
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      3.562
      0.000
      0.000
      ...
      1.839
      2.111
      0.000
      0.000
      0.000
      1.146
      1.415
      0.000
      3.215
      7.673

      fur
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      3.069
      0.000
      0.000
      ...
      2.853
      3.026
      2.941
      3.002
      2.803
      2.797
      2.692
      2.836
      4.028
      12.040

      tho
      0.000
      0.000
      0.000
      0.000
      0.000
      4.178
      0.000
      0.000
      0.000
      0.000
      ...
      2.288
      2.530
      2.225
      2.033
      1.820
      2.233
      2.290
      2.100
      3.513
      10.057

      youer
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      5.082
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.204
      1.703

      a-class
      leady
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      1.724
      0.000
      0.000
      0.000
      1.724
      2.438

      a-frame
      cottage
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      2.808
      2.744
      2.724
      2.782
      2.734
      2.555
      2.626
      2.664
      4.018
      12.227

      a-gonna
      git
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      2.998
      3.149
      2.915
      2.947
      2.442
      2.279
      2.272
      2.303
      4.026
      11.705

      ab
      exercise
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.066
      2.922
      2.791
      2.713
      2.774
      2.925
      2.948
      2.980
      4.224
      13.137

      aba
      convention
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.155
      3.327
      3.254
      3.157
      3.281
      3.233
      3.173
      3.029
      4.356
      13.674

      journal
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      2.726
      2.843
      2.792
      2.767
      2.757
      2.780
      2.815
      3.101
      4.064
      12.391

      section
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.287
      3.408
      3.531
      3.529
      3.451
      3.416
      3.393
      3.488
      4.497
      14.069

      abalone
      shell
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      4.351
      0.000
      ...
      3.076
      2.999
      3.041
      3.034
      3.005
      2.997
      2.817
      2.876
      4.090
      12.359

      abbey
      ale
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.833
      2.104
      2.260
      2.149
      2.207
      1.968
      2.127
      2.207
      3.266
      8.844

      church
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.384
      2.642
      2.704
      ...
      3.656
      3.701
      3.728
      3.694
      3.671
      3.734
      3.578
      3.678
      4.917
      16.054

      gate
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.264
      3.274
      3.241
      3.237
      3.201
      3.094
      3.107
      3.213
      4.404
      13.874

      land
      0.000
      0.000
      0.000
      2.454
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.782
      3.779
      3.785
      3.757
      3.688
      3.662
      3.735
      3.709
      4.962
      16.310

      library
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      3.445
      3.139
      3.144
      3.091
      3.138
      3.106
      3.067
      3.116
      4.336
      13.463

      lodge
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      2.502
      2.540
      2.401
      2.528
      2.401
      2.746
      2.605
      2.464
      3.806
      11.123

      poem
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      2.748
      0.000
      0.000
      ...
      2.997
      3.109
      2.892
      2.891
      2.955
      2.853
      3.091
      2.980
      4.258
      13.245

20 rows × 60 columns

```python
compound_cols=cosiner.columns[:20]
#print(compound_cols)
modifier_cols=cosiner.columns[20:40]
#print(modifier_cols)
head_cols=cosiner.columns[40:]
#print(head_cols)
```

```python
#cosiner.loc['a-frame_noun','cottage_noun'][cosiner.columns[:19]]
display(cosiner.columns[:20])
display(cosiner.columns[20:40])
cosiner.columns[40:]
```

    Index(['1810s_compound', '1820s_compound', '1830s_compound', '1840s_compound',
           '1850s_compound', '1860s_compound', '1870s_compound', '1880s_compound',
           '1890s_compound', '1900s_compound', '1910s_compound', '1920s_compound',
           '1930s_compound', '1940s_compound', '1950s_compound', '1960s_compound',
           '1970s_compound', '1980s_compound', 'cf_compound',
           'denominator_compound'],
          dtype='object')

    Index(['1810s_modifier', '1820s_modifier', '1830s_modifier', '1840s_modifier',
           '1850s_modifier', '1860s_modifier', '1870s_modifier', '1880s_modifier',
           '1890s_modifier', '1900s_modifier', '1910s_modifier', '1920s_modifier',
           '1930s_modifier', '1940s_modifier', '1950s_modifier', '1960s_modifier',
           '1970s_modifier', '1980s_modifier', 'cf_modifier',
           'denominator_modifier'],
          dtype='object')

    Index(['1810s_head', '1820s_head', '1830s_head', '1840s_head', '1850s_head',
           '1860s_head', '1870s_head', '1880s_head', '1890s_head', '1900s_head',
           '1910s_head', '1920s_head', '1930s_head', '1940s_head', '1950s_head',
           '1960s_head', '1970s_head', '1980s_head', 'cf_head',
           'denominator_head'],
          dtype='object')

```python
compounds['sim_with_head']=cosiner.apply(cosiner_head,axis=1)
compounds['sim_with_modifier']=cosiner.apply(cosiner_modifier,axis=1)
compounds['sim_bw_constituents']=cosiner.apply(cosiner_constituent,axis=1)
compounds[['sim_with_modifier','sim_with_head','sim_bw_constituents']]
```

    /usr/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in double_scalars
      after removing the cwd from sys.path.
    /usr/lib/python3.6/site-packages/ipykernel_launcher.py:7: RuntimeWarning: invalid value encountered in double_scalars
      import sys

      sim_with_modifier
      sim_with_head
      sim_bw_constituents

      modifier
      head

      a
      dey
      0.690
      0.483
      0.759

      ef
      0.691
      0.518
      0.844

      fur
      0.666
      0.377
      0.656

      tho
      0.644
      0.371
      0.665

      youer
      0.705
      0.999
      0.707

      a-class
      leady
      1.000
      1.000
      1.000

      a-frame
      cottage
      0.984
      0.366
      0.386

      a-gonna
      git
      0.795
      0.398
      0.493

      ab
      exercise
      nan
      nan
      0.788

      aba
      convention
      0.756
      0.360
      0.503

      journal
      nan
      nan
      0.518

      section
      nan
      nan
      0.528

      abalone
      shell
      0.749
      0.446
      0.571

      abbey
      ale
      nan
      nan
      0.991

      church
      0.570
      0.564
      0.985

      gate
      nan
      nan
      0.986

      land
      0.239
      0.232
      0.983

      library
      0.335
      0.327
      0.987

      lodge
      0.330
      0.322
      0.995

      poem
      0.341
      0.319
      0.985

      pub
      nan
      nan
      0.831

      ruin
      nan
      nan
      0.980

      smith
      0.371
      0.312
      0.980

      theater
      0.404
      0.492
      0.932

      theatre
      0.495
      0.474
      0.986

      wall
      0.277
      0.243
      0.984

      abc
      abc
      nan
      nan
      1.000

      news
      0.724
      0.377
      0.580

      sitcom
      nan
      nan
      0.878

      abduction
      case
      nan
      nan
      0.952

      ...
      ...
      ...
      ...
      ...

      zone
      system
      0.375
      0.334
      0.980

      x-ray
      nan
      nan
      0.840

      zoning
      adjustment
      nan
      nan
      0.763

      amendment
      0.588
      0.372
      0.725

      board
      0.708
      0.445
      0.723

      change
      0.764
      0.483
      0.684

      code
      0.714
      0.452
      0.726

      commission
      nan
      nan
      0.723

      committee
      0.560
      0.345
      0.721

      decision
      nan
      nan
      0.721

      law
      0.938
      0.651
      0.663

      official
      0.557
      0.393
      0.794

      ordinance
      0.913
      0.606
      0.662

      regulation
      0.739
      0.503
      0.710

      resolution
      0.774
      0.472
      0.674

      restriction
      nan
      nan
      0.718

      variance
      nan
      nan
      0.654

      zoo
      animal
      0.469
      0.324
      0.764

      director
      0.684
      0.537
      0.829

      keeper
      0.665
      0.469
      0.744

      meter
      0.450
      0.415
      0.946

      monkey
      nan
      nan
      0.812

      official
      0.592
      0.493
      0.869

      opera
      0.453
      0.350
      0.810

      yard
      0.409
      0.305
      0.781

      zoology
      book
      0.385
      0.313
      0.878

      zoom
      lens
      0.788
      0.536
      0.714

      zucchini
      bread
      0.978
      0.368
      0.390

      butter
      0.987
      0.403
      0.418

      zum
      zum
      1.000
      1.000
      1.000

103666 rows × 3 columns

In the dataset above, all the novel compounds have nan representation as expected. Perhaps in the future, other similarity measures could be considered to handle zero vectors.

```python
compounds.to_csv('compounds.csv')
```

```python
compounds[['sim_with_modifier','sim_with_head','sim_bw_constituents']].describe()
```

      sim_with_modifier
      sim_with_head
      sim_bw_constituents

      count
      82978.000
      82978.000
      103666.000

      mean
      0.439
      0.421
      0.914

      std
      0.192
      0.178
      0.146

      min
      0.125
      0.098
      0.227

      25%
      0.283
      0.280
      0.908

      50%
      0.386
      0.370
      0.985

      75%
      0.549
      0.513
      0.997

      max
      1.000
      1.000
      1.000

Modifier on average seem to be more similar to their compounds than the heads.

# Computation of average cosine similarities for the constituents

The modifier cosine values are now added to the constituents in the form of 3 feautures -

**mean_sim_as_modifier** - The average similarity measure of the constituent as a modifier
**sim_with_modifier** - The standard deviation of the similarity of the constituent as a modifier
**presence_as_modifier** - The number of times the constituent appeared as a modifier.

```python
modifier_sim=compounds.groupby(level=['modifier'])['sim_with_modifier'].agg(['mean','std','size'])
modifier_sim.columns=['mean_sim_as_modifier','std_sim_as_modifier','presence_as_modifier']

modifier_sim.presence_as_modifier[(modifier_sim.mean_sim_as_modifier==0) & (modifier_sim.std_sim_as_modifier==0)]=0
modifier_sim.std_sim_as_modifier.fillna(0,inplace=True)
modifier_sim.std_sim_as_modifier[modifier_sim.mean_sim_as_modifier.isnull()]=np.nan
modifier_sim['presence_as_modifier']=modifier_sim["presence_as_modifier"].astype('category')
modifier_sim
```

      mean_sim_as_modifier
      std_sim_as_modifier
      presence_as_modifier

      modifier

      a
      0.679
      0.024
      5

      a-class
      1.000
      0.000
      1

      a-frame
      0.984
      0.000
      1

      a-gonna
      0.795
      0.000
      1

      ab
      nan
      nan
      1

      aba
      0.756
      0.000
      3

      abalone
      0.749
      0.000
      1

      abbey
      0.374
      0.104
      13

      abc
      0.724
      0.000
      3

      abduction
      nan
      nan
      1

      abeyance
      0.382
      0.000
      1

      ability
      nan
      nan
      3

      abimelech
      0.996
      0.000
      1

      ablishnists
      0.968
      0.000
      1

      abner
      0.554
      0.000
      1

      abolition
      0.378
      0.141
      8

      abolitionist
      0.383
      0.000
      3

      abortion
      0.540
      0.108
      33

      abrasion
      0.501
      0.000
      1

      abscam
      0.992
      0.000
      1

      absence
      0.226
      0.000
      1

      absentee
      0.588
      0.170
      10

      absorption
      0.369
      0.110
      7

      abstention
      0.494
      0.000
      1

      abstinence
      0.412
      0.000
      1

      abstraction
      0.311
      0.000
      1

      abuse
      0.270
      0.045
      12

      ac
      0.334
      0.070
      8

      acacia
      0.507
      0.077
      5

      academy
      0.329
      0.116
      16

      ...
      ...
      ...
      ...

      zara
      0.620
      0.000
      1

      zattiany
      0.980
      0.009
      2

      zeb
      0.838
      0.000
      1

      zebra
      0.503
      0.037
      6

      zeiss
      0.707
      0.000
      1

      zelia
      0.945
      0.000
      1

      zenana
      nan
      nan
      1

      zend
      0.966
      0.000
      1

      zenith
      0.333
      0.000
      2

      zephyr
      0.227
      0.000
      1

      zero
      0.467
      0.042
      13

      zeta
      0.683
      0.069
      3

      ziegfeld
      0.785
      0.031
      2

      zigzag
      0.463
      0.166
      9

      zinc
      0.415
      0.090
      23

      zionism
      0.642
      0.003
      2

      zionist
      0.617
      0.000
      1

      zip
      0.734
      0.094
      4

      zipper
      0.481
      0.000
      1

      zirconium
      0.999
      0.000
      1

      zis
      0.625
      0.083
      2

      zo
      0.577
      0.013
      2

      zob
      0.994
      0.000
      1

      zone
      0.360
      0.129
      14

      zoning
      0.725
      0.133
      15

      zoo
      0.532
      0.113
      8

      zoology
      0.385
      0.000
      1

      zoom
      0.788
      0.000
      1

      zucchini
      0.983
      0.006
      2

      zum
      1.000
      0.000
      1

10912 rows × 3 columns

```python
modifier_sim.describe()
```

      mean_sim_as_modifier
      std_sim_as_modifier

      count
      9841.000
      9841.000

      mean
      0.558
      0.065

      std
      0.219
      0.075

      min
      0.167
      0.000

      25%
      0.386
      0.000

      50%
      0.491
      0.034

      75%
      0.707
      0.122

      max
      1.000
      0.504

```python
#modifier_sim.mean_sim_as_modifier[modifier_sim.mean_sim_as_modifier==0]
```

Same as with the modifiers, the head cosine values are now added to the constituents in the form of 3 feautures -

**mean_sim_as_head** - The average similarity measure of the constituent as a head

**sim_with_head** - The standard deviation of the similarity of the constituent as a head

**presence_as_head** - The number of times the constituent appeared as a head.

```python
head_sim=compounds.groupby(level=['head'])['sim_with_head'].agg(['mean','std','size'])
head_sim.columns=['mean_sim_as_head','std_sim_as_head','presence_as_head']

head_sim.presence_as_head[(head_sim.mean_sim_as_head==0) & (head_sim.std_sim_as_head==0)]=0
head_sim.std_sim_as_head.fillna(0,inplace=True)
head_sim.std_sim_as_head[head_sim.mean_sim_as_head.isnull()]=np.nan
head_sim['presence_as_head']=head_sim["presence_as_head"].astype('category')
head_sim
```

      mean_sim_as_head
      std_sim_as_head
      presence_as_head

      head

      a
      0.532
      0.000
      1

      a-settin
      0.680
      0.000
      1

      ab
      0.500
      0.000
      1

      abandonment
      0.266
      0.000
      1

      abatement
      0.468
      0.051
      4

      abbess
      0.608
      0.000
      1

      abbey
      0.348
      0.000
      2

      abbot
      0.325
      0.000
      2

      abc
      nan
      nan
      1

      abduction
      nan
      nan
      1

      ability
      0.401
      0.181
      21

      abner
      0.638
      0.000
      1

      abnormality
      nan
      nan
      1

      abolition
      0.258
      0.000
      2

      abolitionist
      nan
      nan
      1

      aborigine
      0.350
      0.000
      1

      abortion
      nan
      nan
      2

      abrams
      0.630
      0.000
      1

      abrasion
      0.519
      0.000
      1

      abscess
      0.504
      0.047
      3

      absence
      0.281
      0.000
      1

      absolutism
      nan
      nan
      1

      absorber
      0.884
      0.000
      1

      absorption
      0.346
      0.050
      9

      abundance
      0.253
      0.000
      4

      abuse
      0.340
      0.086
      17

      abuser
      0.908
      0.054
      4

      abutment
      0.387
      0.000
      2

      abyss
      0.362
      0.000
      1

      ac
      0.427
      0.060
      2

      ...
      ...
      ...
      ...

      youth
      0.313
      0.130
      20

      yuh
      0.513
      0.128
      12

      yule
      0.987
      0.000
      1

      yurt
      0.902
      0.000
      2

      zack
      0.602
      0.000
      1

      zaharias
      0.993
      0.000
      1

      zamor
      0.958
      0.000
      1

      zeal
      0.579
      0.110
      2

      zebra
      nan
      nan
      1

      zegri
      0.994
      0.000
      1

      zeil
      0.998
      0.002
      2

      zeisberger
      0.970
      0.000
      1

      zero
      0.707
      0.000
      2

      zest
      nan
      nan
      2

      zeta
      0.630
      0.000
      1

      ziani
      1.000
      0.000
      1

      ziel
      1.000
      0.000
      1

      zionist
      0.652
      0.000
      1

      zip
      0.704
      0.000
      2

      zis
      0.750
      0.000
      1

      zo
      0.621
      0.000
      1

      zodiac
      0.405
      0.000
      1

      zon
      0.997
      0.000
      1

      zone
      0.435
      0.154
      66

      zoning
      0.577
      0.000
      1

      zoo
      0.909
      0.000
      2

      zoon
      0.998
      0.000
      1

      zori
      0.993
      0.000
      1

      zum
      1.000
      0.000
      1

      zwilnik
      0.987
      0.000
      2

10016 rows × 3 columns

```python
head_sim.describe()
```

      mean_sim_as_head
      std_sim_as_head

      count
      9082.000
      9082.000

      mean
      0.527
      0.067

      std
      0.204
      0.076

      min
      0.139
      0.000

      25%
      0.380
      0.000

      50%
      0.458
      0.038

      75%
      0.624
      0.128

      max
      1.000
      0.464

The 6 new features are now added to the individual constituents.

```python
constituents=pd.concat([constituents,modifier_sim,head_sim],axis=1)
constituents
```

      1810s
      1820s
      1830s
      1840s
      1850s
      1860s
      1870s
      1880s
      1890s
      1900s
      ...
      1980s
      cf
      presence
      pattern
      mean_sim_as_modifier
      std_sim_as_modifier
      presence_as_modifier
      mean_sim_as_head
      std_sim_as_head
      presence_as_head

      a
      0.000
      0.000
      0.000
      0.000
      0.000
      2.190
      1.301
      2.354
      1.580
      1.505
      ...
      0.000
      2.687
      6
      1111101000000
      0.679
      0.024
      5.000
      0.532
      0.000
      1.000

      a'r
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      0.000
      0.000
      ...
      0.000
      1.146
      1
      10000000000
      nan
      nan
      nan
      nan
      nan
      nan

      a'ready
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.230
      0.000
      1.114
      ...
      0.000
      1.653
      3
      10110000000
      nan
      nan
      nan
      nan
      nan
      nan

      a'ter
      0.000
      0.000
      0.000
      0.000
      1.568
      1.255
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.892
      3
      11000001000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-bomb
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.204
      2.017
      4
      1111
      nan
      nan
      nan
      nan
      nan
      nan

      a-c
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.322
      1
      1000
      nan
      nan
      nan
      nan
      nan
      nan

      a-class
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.204
      1
      1000
      1.000
      0.000
      1.000
      nan
      nan
      nan

      a-crystals
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.398
      1
      10000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-fishing
      0.000
      0.000
      0.000
      0.000
      1.114
      1.000
      0.000
      0.000
      1.041
      1.041
      ...
      0.000
      1.653
      4
      11001100000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-frame
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.204
      1.204
      1
      1
      0.984
      0.000
      1.000
      nan
      nan
      nan

      a-gettin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.491
      2
      11000
      nan
      nan
      nan
      nan
      nan
      nan

      a-gittin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.176
      0.000
      ...
      0.000
      1.568
      3
      1001010000
      nan
      nan
      nan
      nan
      nan
      nan

      a-gonna
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.633
      2
      110000
      0.795
      0.000
      1.000
      nan
      nan
      nan

      a-gwine
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.301
      1.204
      0.000
      ...
      0.000
      1.556
      2
      11000000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-holdin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      0.000
      ...
      0.000
      1.041
      1
      1000000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-huntin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      ...
      0.000
      1.000
      1
      100000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-hunting
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.146
      0.000
      ...
      0.000
      1.146
      1
      1000000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-layin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      1.041
      0.000
      ...
      0.000
      1.322
      2
      11000000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-leavin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.041
      1
      10000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-lookin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.279
      0.000
      1.176
      1.041
      ...
      0.000
      1.869
      5
      101110010000
      nan
      nan
      nan
      nan
      nan
      nan

      a-particle
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.204
      1
      10000
      nan
      nan
      nan
      nan
      nan
      nan

      a-particles
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.643
      2
      10100
      nan
      nan
      nan
      nan
      nan
      nan

      a-ridin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.114
      1
      10000000
      nan
      nan
      nan
      nan
      nan
      nan

      a-runnin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.176
      1.000
      ...
      0.000
      1.851
      5
      1101011000
      nan
      nan
      nan
      nan
      nan
      nan

      a-sayin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.114
      0.000
      ...
      0.000
      1.602
      3
      1001010000
      nan
      nan
      nan
      nan
      nan
      nan

      a-settin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.322
      1.279
      1.041
      ...
      0.000
      1.881
      5
      11101010000
      nan
      nan
      nan
      0.680
      0.000
      1.000

      a-sittin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.041
      1
      1000
      nan
      nan
      nan
      nan
      nan
      nan

      a-tellin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.204
      1
      10000
      nan
      nan
      nan
      nan
      nan
      nan

      a-thinkin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      0.000
      0.000
      0.000
      ...
      0.000
      1.431
      2
      100000010000
      nan
      nan
      nan
      nan
      nan
      nan

      a-waitin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.176
      0.000
      1.000
      ...
      0.000
      1.398
      2
      10100000000
      nan
      nan
      nan
      nan
      nan
      nan

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      zoon
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.114
      0.000
      0.000
      ...
      0.000
      1.114
      1
      10000000000
      nan
      nan
      nan
      0.998
      0.000
      1.000

      zoophyte
      0.000
      0.000
      0.000
      1.279
      1.204
      0.000
      1.079
      0.000
      0.000
      0.000
      ...
      0.000
      1.672
      3
      110100000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zoorph
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.447
      1
      10000
      nan
      nan
      nan
      nan
      nan
      nan

      zopyrus
      0.000
      0.000
      1.857
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.857
      1
      1000000000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zoraida
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.380
      ...
      0.000
      1.380
      1
      100000000
      nan
      nan
      nan
      nan
      nan
      nan

      zoraya
      0.000
      0.000
      0.000
      1.978
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.978
      1
      100000000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zori
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.362
      1
      1000
      nan
      nan
      nan
      0.993
      0.000
      1.000

      zoro
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.000
      1
      100000
      nan
      nan
      nan
      nan
      nan
      nan

      zosephine
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.041
      0.000
      0.000
      ...
      0.000
      1.041
      1
      10000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zouaves
      0.000
      0.000
      0.000
      0.000
      0.000
      1.491
      0.000
      1.114
      0.000
      0.000
      ...
      0.000
      1.643
      2
      1010000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zoya
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.623
      1.623
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

      zuber
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.146
      1
      10000
      nan
      nan
      nan
      nan
      nan
      nan

      zubriski
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.505
      1
      100000
      nan
      nan
      nan
      nan
      nan
      nan

      zucchini
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.763
      1.763
      1
      1
      0.983
      0.006
      2.000
      nan
      nan
      nan

      zuckerman
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.146
      1.146
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

      zula
      0.000
      0.000
      1.881
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.881
      1
      1000000000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zuleika
      0.000
      0.000
      0.000
      1.204
      0.000
      0.000
      0.000
      1.544
      0.000
      0.000
      ...
      0.000
      1.708
      2
      100010000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zulime
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.778
      1
      1000000
      nan
      nan
      nan
      nan
      nan
      nan

      zulu
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.176
      0.000
      1.041
      1.342
      ...
      1.892
      2.364
      10
      101110111111
      nan
      nan
      nan
      nan
      nan
      nan

      zum
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.279
      1
      100
      1.000
      0.000
      1.000
      1.000
      0.000
      1.000

      zuni
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      1.000
      0.000
      0.000
      ...
      0.000
      1.000
      1
      10000000000
      nan
      nan
      nan
      nan
      nan
      nan

      zura
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.114
      1
      10000000
      nan
      nan
      nan
      nan
      nan
      nan

      zvuv
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.230
      1
      10000
      nan
      nan
      nan
      nan
      nan
      nan

      zwilnik
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.255
      1
      1000
      nan
      nan
      nan
      0.987
      0.000
      2.000

      zwingli
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      1.322
      1
      10
      nan
      nan
      nan
      nan
      nan
      nan

      zworykin
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.146
      1.146
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

      zygote
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.301
      1.633
      3
      111
      nan
      nan
      nan
      nan
      nan
      nan

      zylong
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.491
      1.491
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

      zylongi
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.820
      1.820
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

      zyunsinth
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      1.204
      1.204
      1
      1
      nan
      nan
      nan
      nan
      nan
      nan

38380 rows × 27 columns

```python
constituents.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Index: 38380 entries, a to zyunsinth
    Data columns (total 27 columns):
    1810s                   38380 non-null float64
    1820s                   38380 non-null float64
    1830s                   38380 non-null float64
    1840s                   38380 non-null float64
    1850s                   38380 non-null float64
    1860s                   38380 non-null float64
    1870s                   38380 non-null float64
    1880s                   38380 non-null float64
    1890s                   38380 non-null float64
    1900s                   38380 non-null float64
    1910s                   38380 non-null float64
    1920s                   38380 non-null float64
    1930s                   38380 non-null float64
    1940s                   38380 non-null float64
    1950s                   38380 non-null float64
    1960s                   38380 non-null float64
    1970s                   38380 non-null float64
    1980s                   38380 non-null float64
    cf                      38380 non-null float64
    presence                38380 non-null category
    pattern                 38380 non-null category
    mean_sim_as_modifier    9841 non-null float64
    std_sim_as_modifier     9841 non-null float64
    presence_as_modifier    10912 non-null category
    mean_sim_as_head        9082 non-null float64
    std_sim_as_head         9082 non-null float64
    presence_as_head        10016 non-null category
    dtypes: category(4), float64(23)
    memory usage: 7.5+ MB

```python
constituents.to_csv('constituents.csv')
```

```python
constituents.loc['zone']
```

    1810s                               0.000
    1820s                               1.447
    1830s                               2.009
    1840s                               2.340
    1850s                               2.021
    1860s                               2.134
    1870s                               1.908
    1880s                               2.188
    1890s                               2.173
    1900s                               2.486
    1910s                               2.655
    1920s                               2.579
    1930s                               2.710
    1940s                               2.945
    1950s                               2.930
    1960s                               2.823
    1970s                               2.752
    1980s                               2.901
    cf                                  3.805
    presence                               17
    pattern                 11111111111111111
    mean_sim_as_modifier                0.360
    std_sim_as_modifier                 0.129
    presence_as_modifier                   14
    mean_sim_as_head                    0.435
    std_sim_as_head                     0.154
    presence_as_head                       66
    Name: zone, dtype: object


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  This material largely follows the text "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there.

```python
%matplotlib inline
%precision 3
import numpy
import matplotlib.pyplot as plt
```

# Numerical Linear Algebra

Numerical methods for linear algebra problems lies at the heart of many numerical approaches and is something we will spend some time on.  Roughly we can break down problems that we would like to solve into two general problems, solving a system of equations

$$A \mathbf{x} = \mathbf{b}$$

and solving the eigenvalue problem

$$A \mathbf{v} = \lambda \mathbf{v}.$$

We examine each of these problems separately and will evaluate some of the fundamental properties and methods for solving these problems. We will be careful in deciding how to evaluate the results of our calculations and try to gain some understanding of when and how they fail.

## General Problem Specification

The number and power of the different tools made available from the study of linear algebra makes it an invaluable field of study. Before we dive in to numerical approximations we first consider some of the pivotal problems that numerical methods for linear algebra are used to address.

For this discussion we will be using the common notation $m \times n$ to denote the dimensions of a matrix $A$.  The $m$ refers to the number of rows and $n$ the number of columns.  If a matrix is square, i.e. $m = n$, then we will use the notation that $A$ is $m \times m$.

### Systems of Equations

The first type of problem is to find the solution to a linear system of equations.  If we have $m$ equations for $m$ unknowns it can be written in matrix/vector form,

$$A \mathbf{x} = \mathbf{b}.$$

For this example $A$ is an $m \times m$ matrix, denoted as being in $\mathbb{R}^{m\times m}$, and $\mathbf{x}$ and $\mathbf{b}$ are column vectors with $m$ entries, denoted as $\mathbb{R}^m$.

**Note**: row vectors denoted as $\mathbf{x}^\top$ or $\mathbf{b}^\top$

#### Example: Vandermonde Matrix

We have data $(x_i, y_i), ~~ i = 1, 2, \ldots, m$ that we want to fit a polynomial of order $m-1$.  Solving the linear system $A \mathbf{p} = \mathbf{y}$ does this for us where

$$A = \begin{bmatrix}
    1 & x_1 & x_1^2 & \cdots & x_1^{m-1} \\
    1 & x_2 & x_2^2 & \cdots & x_2^{m-1} \\
    \vdots & \vdots & \vdots & & \vdots \\
    1 & x_m & x_m^2 & \cdots & x_m^{m-1}
\end{bmatrix} \quad \quad \mathbf{y} = \begin{bmatrix}
y_1 \\ y_2 \\ \vdots \\ y_m
\end{bmatrix}$$

and $\mathbf{p}$ are the coefficients of the interpolating polynomial $\mathcal{P}_N(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_m x^{m-1}$. The solution to this system satisfies $\mathcal{P}_N(x_i)=y_i$ for $i=1, 2, \ldots, m$.

#### Example: Linear least squares 1

In a similar case as above, say we want to fit a particular function (could be a polynomial) to a given number of data points except in this case we have more data points than free parameters.  In the case of polynomials this could be the same as saying we have $m$ data points but only want to fit a $n - 1$ order polynomial through the data where $n - 1 \leq m$.  One of the common approaches to this problem is to minimize the "least-squares" error between the data and the resulting function:
$$
    E = \left( \sum^m_{i=1} |y_i - f(x_i)|^2 \right )^{1/2}.
$$

But how do we do this if our matrix $A$ is now $m \times n$ and looks like
$$
    A = \begin{bmatrix}
        1 & x_1 & x_1^2 & \cdots & x_1^{n-1} \\
        1 & x_2 & x_2^2 & \cdots & x_2^{n-1} \\
        \vdots & \vdots & \vdots & & \vdots \\
        1 & x_m & x_m^2 & \cdots & x_m^{n-1}
    \end{bmatrix}?
$$

Turns out if we solve the system

$$A^T A \mathbf{x} = A^T \mathbf{b}$$

we can guarantee that the error is minimized in the least-squares sense[1](#footnoteRegression). (Although we will also show that this is **not the most numerically stable way to solve this problem**)

**Note**: cannot solve if $A$ is singular.

#### Example:  Linear least squares 2

Fitting a line through data that has random noise added to it.

```python
# Linear Least Squares Problem

# First define the independent and dependent variables.
N = 20
x = numpy.linspace(-1.0, 1.0, N)
y = x + numpy.random.random((N))

# Define the Vandermonde matrix based on our x-values
A = numpy.array([ numpy.ones(x.shape), x]).T
A = numpy.array([ numpy.ones(x.shape), x, x**2]).T
A
```

    array([[ 1.   , -1.   ,  1.   ],
           [ 1.   , -0.895,  0.801],
           [ 1.   , -0.789,  0.623],
           [ 1.   , -0.684,  0.468],
           [ 1.   , -0.579,  0.335],
           [ 1.   , -0.474,  0.224],
           [ 1.   , -0.368,  0.136],
           [ 1.   , -0.263,  0.069],
           [ 1.   , -0.158,  0.025],
           [ 1.   , -0.053,  0.003],
           [ 1.   ,  0.053,  0.003],
           [ 1.   ,  0.158,  0.025],
           [ 1.   ,  0.263,  0.069],
           [ 1.   ,  0.368,  0.136],
           [ 1.   ,  0.474,  0.224],
           [ 1.   ,  0.579,  0.335],
           [ 1.   ,  0.684,  0.468],
           [ 1.   ,  0.789,  0.623],
           [ 1.   ,  0.895,  0.801],
           [ 1.   ,  1.   ,  1.   ]])

```python
# Determine the coefficients of the polynomial that will
# result in the smallest sum of the squares of the residual.
p = numpy.linalg.solve(numpy.dot(A.T, A), numpy.dot(A.T, y))
print("Error in slope = %s, y-intercept = %s" % (numpy.abs(p[1] - 1.0), numpy.abs(p[0] - 0.5)))
print(p)
```

    Error in slope = 0.18583474659498123, y-intercept = 0.006640778501272648
    [0.507 1.186 0.088]

```python
# Plot it out, cuz pictures are fun!
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

f = numpy.zeros(x.shape)
for i in range(len(p)):
    f += p[i] * x**i
axes.plot(x, y, 'ko')
axes.plot(x, f, 'r')
axes.set_title("Least Squares Fit to Data")
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Eigenproblems

Eigenproblems come up in a variety of contexts and often are integral to many problem of scientific and engineering interest. It is such a powerful idea that it is not uncommon for us to take a problem and convert it into an eigenproblem. Here we introduce the idea and give some examples.

As a review, if $A \in \mathbb{C}^{m\times m}$ (a square matrix with complex values), a non-zero vector $\mathbf{v}\in\mathbb{C}^m$ is an **eigenvector** of $A$ with a corresponding **eigenvalue** $\lambda \in \mathbb{C}$ if

$$A \mathbf{v} = \lambda \mathbf{v}.$$

One way to interpret the eigenproblem is that we are attempting to ascertain the "action" of the matrix $A$ on some subspace of $\mathbb{C}^m$ where this action acts like scalar multiplication.  This subspace is called an **eigenspace**.

**Note**: eigenproblems are nonlinear problems

### General idea of EigenProblems

Rewriting the standard Eigen problem $A\mathbf{v}=\lambda\mathbf{v}$ for $A \in \mathbb{C}^{m\times m}$, $\mathbf{v}\in\mathbb{C}^m$ as

$$
    (A - \lambda I)\mathbf{v} = 0
$$

it becomes clear that for $\mathbf{v}$ to be non-trivial (i.e. $\neq \mathbf{0}$), requires that the matrix $(A-\lambda I)$ be singular,

This is equivalent to finding all values of $\lambda$ such that $|A-\lambda I| = 0$ (the determinant of singular matrices is always zero).  However, it can also be shown that

$$
   | A-\lambda I| = P_m(\lambda)
$$

which is a $m$th order polynomial in $\lambda$.  Thus $P_m(\lambda)=0$ implies the eigenvalues are the $m$ roots of $P$, and the **eigenspace** corresponding to $\lambda_i$ is just $N(A-\lambda_i I)$

**Note**: if $A$ is invertible (NOT singular), $\mathbf{v}$ MUST be $0$.

Singular matrix: determinant is $0$

Determinant: alternating sum of products of all components from every row and column of $A$

Can't find root of $P_m(\lambda)$ if $m > 4$.

### Solving EigenProblems

The temptation (and what) we usually teach in introductory linear algebra is to simply find the roots of $P_m(\lambda)$.  However that would be **wrong**.  The best algorithms for finding Eigenvalues are completely unrelated to rootfinding as we shall see.

#### Example

Compute the eigenspace of the matrix
$$
    A = \begin{bmatrix}
        1 & 2 \\
        2 & 1
    \end{bmatrix}
$$

Recall that we can find the eigenvalues of a matrix by computing $\det(A - \lambda I) = 0$.

In this case we have
$$\begin{aligned}
    A - \lambda I &= \begin{bmatrix}
        1 & 2 \\
        2 & 1
    \end{bmatrix} - \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \lambda\\
    &= \begin{bmatrix}
        1 - \lambda & 2 \\
        2 & 1 - \lambda
    \end{bmatrix}.
\end{aligned}$$
The determinant of the matrix is
$$\begin{aligned}
\begin{vmatrix}
    1 - \lambda & 2 \\
    2 & 1 - \lambda
\end{vmatrix} &= (1 - \lambda) (1 - \lambda) - 2 \cdot 2 \\
&= 1 - 2 \lambda + \lambda^2 - 4 \\
&= \lambda^2 - 2 \lambda - 3.
\end{aligned}$$
This result is sometimes referred to as the characteristic equation of the matrix, $A$.

Setting the determinant equal to zero we can find the eigenvalues as
$$\begin{aligned}
    & \\
    \lambda &= \frac{2 \pm \sqrt{4 - 4 \cdot 1 \cdot (-3)}}{2} \\
    &= 1 \pm 2 \\
    &= -1 \mathrm{~and~} 3
\end{aligned}$$

The eigenvalues are used to determine the eigenvectors. The eigenvectors are found by going back to the equation $(A - \lambda I) \mathbf{v}_i = 0$ and solving for each vector.  A trick that works some of the time is to normalize each vector such that the first entry is 1 ($\mathbf{v}_1 = 1$):

$$
    \begin{bmatrix}
        1 - \lambda & 2 \\
        2 & 1 - \lambda
    \end{bmatrix} \begin{bmatrix} 1 \\ v_2 \end{bmatrix} = 0
$$

$$\begin{aligned}
    1 - \lambda + 2 v_2 &= 0 \\
    v_2 &= \frac{\lambda - 1}{2}
\end{aligned}$$

We can check this by
$$\begin{aligned}
    2 + \left(1- \lambda \frac{\lambda - 1}{2}\right) & = 0\\
    (\lambda - 1)^2 - 4 &=0
\end{aligned}$$

which by design is satisfied by our eigenvalues.  Another sometimes easier approach is to plug-in the eigenvalues to   find the Null space of $A-\lambda I$ where the eigenvectors will be a basis for $N(A-\lambda I)$.  The eigenvectors are therefore

$$\mathbf{v} = \begin{bmatrix}1 \\ -1 \end{bmatrix}, \begin{bmatrix}1 \\ 1 \end{bmatrix}.$$

Note that these are linearly independent (and because $A^T = A$, also orthogonal)

## Fundamentals

### Matrix-Vector Multiplication

One of the most basic operations we can perform with matrices is to multiply them be a vector.  This matrix-vector product $A \mathbf{x} = \mathbf{b}$ is defined as
$$
    b_i = \sum^n_{j=1} a_{ij} x_j \quad \text{where}\quad i = 1, \ldots, m
$$

### row picture
In addition to index form, we can consider matrix-vector as a sequence of inner products (dot-products between the rows of $A$ and the vector $\mathbf{x}$.
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
     &=
    \begin{bmatrix}  \mathbf{a}_1^T \mathbf{x} \\ \mathbf{a}_2^T \mathbf{x} \\ \vdots \\ \mathbf{a}_m^T \mathbf{x}\end{bmatrix}
\end{align}
where $\mathbf{a}_i^T$ is the $i$th **row** of $A$

#### Operation Counts
This view is convenient for calculating the **Operation counts** required for to compute $A\mathbf{x}$.  If $A\in\mathbb{C}^{m\times n}$ and $\mathbf{x}\in\mathbb{C}^n$.  Then just counting the number of multiplications involved to compute $A\mathbf{x}$ is $O(??)$

**Note**: For $A\in\mathbb{C}^{m\times n}$ and $\mathbf{x}\in\mathbb{C}^n$, $A\mathbf{x}$ is $O(mn)$.

### Column picture

An alternative (and entirely equivalent way) to write the matrix-vector product is as a linear combination of the columns of $A$ where each column's weighting is $x_j$.

$$
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n \\  &  &  &   \\  &  &  &  \end{bmatrix}
    \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}, \\
  &= x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + \cdots + x_n \mathbf{a}_n.
\end{align}
$$

This view will be useful later when we are trying to interpret various types of matrices.

One important property of the matrix-vector product is that is a **linear** operation, also known as a **linear operator**.  This means that the for any $\mathbf{x}, \mathbf{y} \in \mathbb{C}^n$ and any $c \in \mathbb{C}$ we know that

1. $A (\mathbf{x} + \mathbf{y}) = A\mathbf{x} + A\mathbf{y}$
1. $A\cdot (c\mathbf{x}) = c A \mathbf{x}$

#### Example:  Vandermonde Matrix

In the case where we have $m$ data points and want $m - 1$ order polynomial interpolant the matrix $A$ is a square, $m \times m$, matrix as before.  Using the above interpretation the polynomial coefficients $p$ are the weights for each of the monomials that give exactly the $y$ values of the data.

#### Example:  Numerical matrix-vector multiply

Write a matrix-vector multiply function and check it with the appropriate `numpy` routine.  Also verify the linearity of the matrix-vector multiply.

```python
#A x = b
#(m x n) (n x 1) = (m x 1)
def matrix_vector_product(A, x):
    # Should check that x and A have correct shape, but didn't
    m, n = A.shape
    b = numpy.zeros(m)
    for i in range(m):
        for j in range(n):
            b[i] += A[i, j] * x[j]
    return b

m = 4
n = 3
A = numpy.random.uniform(size=(m,n))
x = numpy.random.uniform(size=(n))
y = numpy.random.uniform(size=(n))
c = numpy.random.uniform()
b = matrix_vector_product(A, x)
print(numpy.allclose(b, numpy.dot(A, x)))
print(numpy.allclose(matrix_vector_product(A, (x + y)), matrix_vector_product(A, x) + matrix_vector_product(A, y)))
print(numpy.allclose(matrix_vector_product(A, c * x), c*matrix_vector_product(A, x)))
```

    True
    True
    True

**Note**: `numpy.dot` uses compiled C (more efficient than `for` loop)

### Matrix-Matrix Multiplication

The matrix product with another matrix $A B = C$ is defined as
$$
    c_{ij} = \sum^m_{k=1} a_{ik} b_{kj} = \mathbf{a}_i^T\mathbf{b}_j
$$

i.e. each component of $C$ is a dot-product between the $i$th row of $A$ and the $j$th column of $B$

As with matrix-vector multiplication, Matrix-matrix multiplication can be thought of multiple ways

* $m\times p$ dot products
* $A$ multiplying the columns of $B$
$$
    C = AB = \begin{bmatrix}
                A\mathbf{b}_1 & A\mathbf{b}_2 & \ldots & A\mathbf{b}_p\\
             \end{bmatrix}
$$
* Linear combinations of the rows of $B$
$$
C = AB = \begin{bmatrix}
                \mathbf{a}_1^T B \\ \mathbf{a}_2^T B \\ \vdots \\ \mathbf{a}_m^T B\\
             \end{bmatrix}
$$

**Note**: $O(mnp)$, for $A\in\mathbb{C}^{m\times n}$ and $B\in\mathbb{C}^{n\times p}$

### Questions
* What are the dimensions of $A$ and $B$ so that the multiplication works?
* What are the Operations Counts for Matrix-Matrix Multiplication?
* Comment on the product $\mathbf{c}=(AB)\mathbf{x}$ vs. $\mathbf{d} = A(B\mathbf{x})$

1. num columns of $A$ = num rows of $B$
2. $O(mnp)$
3. $\mathbf{c}=(AB)\mathbf{x}$ is $O(m^3 + m^2)$, $\mathbf{d} = A(B\mathbf{x})$ is $O(m^2 + m^2)$

#### Example:  Outer Product

The product of two vectors $\mathbf{u} \in \mathbb{C}^m$ and $\mathbf{v} \in \mathbb{C}^n$ is a $m \times n$ matrix where the columns are the vector $u$ multiplied by the corresponding value of $v$:
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  u_1 \\ u_2 \\ \vdots \\ u_n  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

It is useful to think of these as operations on the column vectors, and an equivalent way to express this relationship is
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  \\ \mathbf{u} \\ \\  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{u}v_1  & \mathbf{u} v_2  & \cdots & \mathbf{u} v_n  \\  &  &  & \\  &  &  &  \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

### rank 1 updates

We call any matrix of the form $\mathbf{u}\mathbf{v}^T$ a "rank one matrix"  ( because its rank r=?).  These sort of matrix operations are very common in numerical algorithms for orthogonalization, eigenvalues and the original page-rank algorithm for google.  Again, the order of operations is critical.

Comment on the difference in values and operation counts between

$$
    \mathbf{y} = (\mathbf{u}\mathbf{v}^T)\mathbf{x} \rightarrow O(mn)
$$

and
$$
    \mathbf{y}' = \mathbf{u}(\mathbf{v}^T\mathbf{x}) \rightarrow O(m)
$$
for $\mathbf{u}$, $\mathbf{v}$, $\mathbf{x}$, $\mathbf{y}$, $\mathbf{y}'\in\mathbb{R}^n$,

#### Example:  Upper Triangular Multiplication

Consider the multiplication of a matrix $A \in \mathbb{C}^{m\times n}$ and the **upper-triangular** matrix $R$ defined as the $n \times n$ matrix with entries $r_{ij} = 1$ for $i \leq j$ and $r_{ij} = 0$ for $i > j$.  The product can be written as
$$
    \begin{bmatrix}  \\  \\ \mathbf{b}_1 & \cdots & \mathbf{b}_n \\ \\ \\ \end{bmatrix} = \begin{bmatrix} \\ \\  \mathbf{a}_1 & \cdots & \mathbf{a}_n \\ \\ \\ \end{bmatrix} \begin{bmatrix} 1 & \cdots & 1 \\  & \ddots & \vdots \\  &  & 1 \end{bmatrix}.
$$

The columns of $B$ are then
$$
    \mathbf{b}_j = A \mathbf{r}_j = \sum^j_{k=1} \mathbf{a}_k
$$
so that $\mathbf{b}_j$ is the sum of the first $j$ columns of $A$.

#### Example: Write Matrix-Matrix Multiplication

Write a function that computes matrix-matrix multiplication and demonstrate the following properties:
1. $A (B + C) = AB + AC$ (for square matrices))
1. $A (cB) = c AB$ where $c \in \mathbb{C}$
1. $AB \neq BA$ in general

```python
def matrix_matrix_product(A, B):
    C = numpy.zeros((A.shape[0], B.shape[1]))
    for i in range(A.shape[0]):
        for j in range(B.shape[1]):
            for k in range(A.shape[1]):
                C[i, j] += A[i, k] * B[k, j]
    return C

m = 4
n = 4
p = 4
A = numpy.random.uniform(size=(m, n))
B = numpy.random.uniform(size=(n, p))
C = numpy.random.uniform(size=(m, p))
c = numpy.random.uniform()
print(numpy.allclose(matrix_matrix_product(A, B), numpy.dot(A, B)))
print(numpy.allclose(matrix_matrix_product(A, (B + C)), matrix_matrix_product(A, B) + matrix_matrix_product(A, C)))
print(numpy.allclose(matrix_matrix_product(A, c * B), c*matrix_matrix_product(A, B)))
print(numpy.allclose(matrix_matrix_product(A, B), matrix_matrix_product(B, A)))
```

    True
    True
    True
    False

#### NumPy Products

NumPy and SciPy contain routines that are optimized to perform matrix-vector and matrix-matrix multiplication.  Given two `ndarray`s you can take their product by using the `dot` function.

```python
n = 10
m = 5
# Matrix vector with identity
A = numpy.identity(n)
x = numpy.random.random(n)
print(numpy.allclose(x, numpy.dot(A, x)))
print(x-A.dot(x))
print(A*x)

# Matrix vector product
A = numpy.random.random((m, n))
print(numpy.dot(A, x))

# Matrix matrix product
B = numpy.random.random((n, m))
print(numpy.dot(A, B))

```

    True
    [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    [[0.769 0.    0.    0.    0.    0.    0.    0.    0.    0.   ]
     [0.    0.647 0.    0.    0.    0.    0.    0.    0.    0.   ]
     [0.    0.    0.604 0.    0.    0.    0.    0.    0.    0.   ]
     [0.    0.    0.    0.591 0.    0.    0.    0.    0.    0.   ]
     [0.    0.    0.    0.    0.517 0.    0.    0.    0.    0.   ]
     [0.    0.    0.    0.    0.    0.706 0.    0.    0.    0.   ]
     [0.    0.    0.    0.    0.    0.    0.481 0.    0.    0.   ]
     [0.    0.    0.    0.    0.    0.    0.    0.017 0.    0.   ]
     [0.    0.    0.    0.    0.    0.    0.    0.    0.005 0.   ]
     [0.    0.    0.    0.    0.    0.    0.    0.    0.    0.126]]
    [1.377 2.097 2.561 2.558 2.423]
    [[2.186 1.663 2.245 1.912 2.132]
     [2.185 1.532 2.212 1.773 1.912]
     [3.05  2.127 3.276 2.512 2.895]
     [2.889 1.946 3.251 1.932 2.625]
     [3.02  2.904 3.703 2.743 3.78 ]]

### Range and Null-Space

#### Range
- The **range** of a matrix $A \in \mathbb R^{m \times n}$ (similar to any function), denoted as $\text{range}(A)$, is the set of vectors that can be expressed as $A x$ for $x \in \mathbb R^n$.
- We can also then say that that $\text{range}(A)$ is the space **spanned** by the columns of $A$.  In other words the columns of $A$ provide a basis for $\text{range}(A)$, also called the **column space** of the matrix $A$.
- $C(A)$ controls the **existence** of solutions to $A\mathbf{x}=\mathbf{b}$

#### Null-Space
- Similarly the **null-space** of a matrix $A$, denoted $\text{null}(A)$ is the set of vectors $x$ that satisfy $A x = 0$.
- $N(A)$ controls the **uniqueness** of solutions to $A\mathbf{x}=\mathbf{b}$
- A similar concept is the **rank** of the matrix $A$, denoted as $\text{rank}(A)$, is the dimension of the column space.  A matrix $A$ is said to have **full-rank** if $\text{rank}(A) = \min(m, n)$.  This property also implies that the matrix mapping is **one-to-one**.

**Note**:
- Range determines whether $A\mathbf{x} = \mathbf{b}$ has a solution ($\mathbf{b}$ must be in $C(A)$).
- Nullspace controls uniqueness of solutions to $A\mathbf{x} = \mathbf{b}$.
- Other 2 fundamental spaces: column space and null space of $A^\top$

### Inverse

A **non-singular** or **invertible** matrix is characterized as a matrix with full-rank.  This is related to why we know that the matrix is one-to-one, we can use it to transform a vector $x$ and using the inverse, denoted $A^{-1}$, we can map it back to the original matrix.  The familiar definition of this is
\begin{align*}
    A \mathbf{x} &= \mathbf{b}, \\
    A^{-1} A \mathbf{x} & = A^{-1} \mathbf{b}, \\
    x &=A^{-1} \mathbf{b}.
\end{align*}
Since $A$ has full rank, its columns form a basis for $\mathbb{R}^m$ and the vector $\mathbf{b}$ must be in the column space of $A$.

There are a number of important **properties of a non-singular matrix A**.  Here we list them as the following **equivalent statements**
1. $A$ has an inverse $A^{-1}$
1. $\text{rank}(A) = m$
1. $\text{range}(A) = \mathbb{C}^m$
1. $\text{null}(A) = \mathbf{0}$
1. 0 is not an eigenvalue of $A$
1. $\text{det}(A) \neq 0$

#### Example:  Properties of invertible matrices

Show that given an invertible matrix that the rest of the properties hold.  Make sure to search the `numpy` packages for relevant functions.

```python
m = 3
# Loop through random matrices until hit 100 or reach invertible matrix
for n in range(100):
    A = numpy.random.uniform(size=(m, m))
    if numpy.linalg.det(A) != 0:
        break

print(numpy.dot(numpy.linalg.inv(A), A))
print(numpy.linalg.matrix_rank(A))
print("N(A)= {}".format(numpy.linalg.solve(A, numpy.zeros(m))))
print("Eigenvalues = {}".format(numpy.linalg.eigvals(A)))
```

    [[ 1.000e+00 -1.586e-16 -1.376e-16]
     [ 9.190e-17  1.000e+00  1.646e-16]
     [ 8.277e-17  5.459e-17  1.000e+00]]
    3
    N(A)= [0. 0. 0.]
    Eigenvalues = [ 1.97  -0.329  0.508]

### Orthogonal Vectors and Matrices

Orthogonality is a very important concept in linear algebra that forms the basis of many of the modern methods used in numerical computations.

Two vectors are said to be *orthogonal* if their **inner-product** or **dot-product** defined as
$$
    < \mathbf{x}, \mathbf{y} > \equiv (\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^T\mathbf{y} \equiv \mathbf{x} \cdot \mathbf{y} = \sum^m_{i=1} x_i y_i = 0
$$
Here we have shown the various notations you may run into (the inner-product is in-fact a general term for a similar operation for mathematical objects such as functions).

If $\langle \mathbf{x},\mathbf{y} \rangle = 0$ then we say $\mathbf{x}$ and $\mathbf{y}$ are orthogonal.  The reason we use this terminology is that the inner-product of two vectors can also be written in terms of the angle between them where
$$
    \cos \theta = \frac{\langle \mathbf{x}, \mathbf{y} \rangle}{||\mathbf{x}||_2~||\mathbf{y}||_2}
$$
and $||\mathbf{x}||_2$ is the Euclidean ($\ell^2$) norm of the vector $\mathbf{x}$.

We can write this in terms of the inner-product as well as
$$
    ||\mathbf{x}||_2^2 = \langle \mathbf{x}, \mathbf{x} \rangle = \mathbf{x}^T\mathbf{x} = \sum^m_{i=1} |x_i|^2.
$$

$$
    ||\mathbf{x}||_2 = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}
$$

The generalization of the inner-product to complex spaces is defined as
$$
    \langle x, y \rangle = \sum^m_{i=1} x_i^* y_i
$$
where $x_i^*$ is the complex-conjugate of the value $x_i$.

#### Orthonormality

Taking this idea one step further we can say a set of vectors $\mathbf{x} \in X$ are orthogonal to $\mathbf{y} \in Y$ if $\forall \mathbf{x},\mathbf{y}$ $< \mathbf{x}, \mathbf{y} > = 0$.  If $\forall \mathbf{x},\mathbf{y}$ $||\mathbf{x}|| = 1$ and $||\mathbf{y}|| = 1$ then they are also called orthonormal. Note that we dropped the 2 as a subscript to the notation for the norm of a vector. Later we will explore other ways to define a norm of a vector other than the Euclidean norm defined above.

Another concept that is related to orthogonality is linear-independence.  A set of vectors $\mathbf{x} \in X$ are **linearly independent** if $\forall \mathbf{x} \in X$ that each $\mathbf{x}$ cannot be written as a linear combination of the other vectors in the set $X$.

An equivalent statement is that given a set of $n$ vectors $\mathbf{x}_i$,  the only  set of scalars $c_i$ that satisfies
$$
    \sum_{i=1}^n c_i\mathbf{x}_i = \mathbf{0}
$$
is if $c_i=0$ for all $i\in[1,n]$

This can be related directly through the idea of projection.  If we have a set of vectors $\mathbf{x} \in X$ we can project another vector $\mathbf{v}$ onto the vectors in $X$ by using the inner-product.  This is especially powerful if we have a set of **orthogonal** vectors $X$, which are said to **span** a space (or provide a **basis** for a space), s.t. any vector in the space spanned by $X$ can be expressed as a linear combination of the basis vectors $X$
$$
    \mathbf{v} = \sum^n_{i=1} \, \langle \mathbf{v}, \mathbf{x}_i \rangle \, \mathbf{x}_i.
$$
Note if $\mathbf{v} \in X$ that
$$
    \langle \mathbf{v}, \mathbf{x}_i \rangle = 0 \quad \forall \mathbf{x}_i \in X \setminus \mathbf{v}.
$$

Looping back to matrices, the column space of a matrix is spanned by its linearly independent columns.  Any vector $v$ in the column space can therefore be expressed via the equation above.  A special class of matrices are called **unitary** matrices when complex-valued and **orthogonal** when purely real-valued if the columns of the matrix are orthonormal to each other.  Importantly this implies that for a unitary matrix $Q$ we know the following

1. $Q^* = Q^{-1}$
1. $Q^*Q = I$

where $Q^*$ is called the **adjoint** of $Q$.  The adjoint is defined as the transpose of the original matrix with the entries being the complex conjugate of each entry as the notation implies.

As an example if we have the matrix
$$
\begin{aligned}
    Q &= \begin{bmatrix} q_{11} & q_{12} \\ q_{21} & q_{22} \\ q_{31} & q_{32} \end{bmatrix} \quad \text{then} \\
    Q^* &= \begin{bmatrix} q^*_{11} & q^*_{21} & q^*_{31} \\ q^*_{12} & q^*_{22} & q^*_{32} \end{bmatrix}
\end{aligned}
$$
The important part of being an unitary matrix is that the projection onto the column space of the matrix $Q$ preserves geometry in an Euclidean sense, i.e. preserves the Cartesian distance.

### Vector Norms

Norms (and also measures) provide a means for measure the "size" or distance in a space.  In general a norm is a function, denoted by $||\cdot||$, that maps $\mathbb{C}^m \rightarrow \mathbb{R}$.  In other words we stick in a multi-valued object and get a single, real-valued number out the other end.  All norms satisfy the properties:

1. $~~~~||\mathbf{x}|| \geq 0$
1. $~~~~||\mathbf{x}|| = 0$ only if $\mathbf{x} = \mathbf{0}$
1. $~~~~||\mathbf{x} + \mathbf{y}||\leq ||\mathbf{x}|| + ||\mathbf{y}||$ **(triangle inequality)**
1. $~~~||c \mathbf{x}|| = |c| ~ ||\mathbf{x}||$ where $c \in \mathbb{C}$

There are a number of relevant norms that we can define beyond the Euclidean norm, also know as the 2-norm or $\ell_2$ norm:

1. $\ell_1$ norm:
$$
    ||\mathbf{x}||_1 = \sum^m_{i=1} |x_i|,
$$
1. $\ell_2$ norm:
$$
    ||\mathbf{x}||_2 = \left( \sum^m_{i=1} |x_i|^2 \right)^{1/2},
$$

3. $\ell_p$ norm:
$$
    ||\mathbf{x}||_p = \left( \sum^m_{i=1} |x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$
1. $\ell_\infty$ norm:
$$
    ||\mathbf{x}||_\infty = \max_{1\leq i \leq m} |x_i|,
$$
1. weighted $\ell_p$ norm:
$$
    ||\mathbf{x}||_{W_p} = \left( \sum^m_{i=1} |w_i x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$

These are also related to other norms denoted by capital letters ($L_2$ for instance).  In this case we use the lower-case notation to denote finite or discrete versions of the infinite dimensional counterparts.

#### Example:  Comparisons Between Norms

Compute the norms given some vector $\mathbf{x}$ and compare their values.  Verify the properties of the norm for one of the norms.

```python
def pnorm(x, p):
    """ return the vector p norm of a vector
    parameters:
    -----------
    x: numpy array
        vector
    p: float or numpy.inf
        value of p norm such that ||x||_p = (sum(|x_i|^p))^{1/p} for p< inf
        for infinity norm return max(abs(x))
    returns:
    --------
    pnorm: float
        pnorm of x
    """
    if p == numpy.inf:
        norm = numpy.max(numpy.abs(x))
    else:
        norm = numpy.sum(numpy.abs(x)**p)**(1./p)
    return norm
```

```python
m = 10
p = 4
x = numpy.random.uniform(size=m)
ell_1 = pnorm(x, 1)
ell_2 = pnorm(x, 2)
ell_p = pnorm(x, p)
ell_infty = pnorm(x, numpy.inf)

print('x = {}'.format(x))
print()
print("L_1   = {}\nL_2   = {}\nL_{}   = {}\nL_inf = {}".format(ell_1, ell_2, p, ell_p, ell_infty))

y = numpy.random.uniform(size=m)
print()
print("Properties of norms:")

print('y = {}\n'.format(y))
p = 2
print('||x+y||_{p}         = {nxy}\n||x||_{p} + ||y||_{p} = {nxny}'.format(
    p=p,nxy=pnorm(x+y, p), nxny=pnorm(x, p) + pnorm(y, p)))
c = 0.1
print('||c x||_{} = {}'.format(p,pnorm(c * x, p)))
print(' c||x||_{} = {}'.format(p,c * pnorm(x, p)))

```

    x = [0.94  0.401 0.666 0.365 0.6   0.276 0.466 0.742 0.723 0.013]

    L_1   = 5.1928301532440795
    L_2   = 1.830021918310124
    L_4   = 1.155148486209196
    L_inf = 0.9399887351870834

    Properties of norms:
    y = [0.81  0.688 0.45  0.708 0.687 0.841 0.466 0.125 0.336 0.264]

    ||x+y||_2         = 3.516926186641789
    ||x||_2 + ||y||_2 = 3.681343317712442
    ||c x||_2 = 0.18300219183101243
     c||x||_2 = 0.1830021918310124

### Matrix Norms

The most direct way to consider a matrix norm is those induced by a vector-norm.  Given a vector norm, we can define a matrix norm as the smallest number $C$ that satisfies the inequality
$$
    ||A \mathbf{x}||_{m} \leq C ||\mathbf{x}||_{n}.
$$
or as the supremum of the ratios so that
$$
    C = \sup_{\mathbf{x}\in\mathbb{C}^n ~ \mathbf{x}\neq\mathbf{0}} \frac{||A \mathbf{x}||_{m}}{||\mathbf{x}||_n}.
$$

Noting that $||A \mathbf{x}||$ lives in the column space and $||\mathbf{x}||$ on the domain we can think of the matrix norm as the "size" of the matrix that maps the domain to the range.  Also noting that if $||\mathbf{x}||_n = 1$ we also satisfy the condition we can write the induced matrix norm as
$$
    ||A||_{(m,n)} = \sup_{\mathbf{x} \in \mathbb{C}^n ~ ||\mathbf{x}||_{n} = 1} ||A \mathbf{x}||_{m}.
$$

This definition has a **geometric interpretation**.  The set of all $\mathbf{x}$ such that $||\mathbf{x}||_n = 1$ is the "unit sphere" in $\mathbb{C}^n$.  So the induced matrix norm is the largest vector in the deformed "sphere" and measures how much the matrix distorts the unit sphere.

#### Example: Induced Matrix Norms

Consider the matrix
$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$
Compute the induced-matrix norm of $A$ for the vector norms $\ell_2$ and $\ell_\infty$.

$\ell^2$: For both of the requested norms the unit-length vectors $[1, 0]$ and $[0, 1]$ can be used to give an idea of what the norm might be and provide a lower bound.

$$
    ||A||_2 = \sup_{x \in \mathbb{R}^n} \left( ||A \cdot [1, 0]^T||_2, ||A \cdot [0, 1]^T||_2 \right )
$$

computing each of the norms we have

$$\begin{aligned}
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} &= \begin{bmatrix} 1 \\ 0 \end{bmatrix} \\
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \end{bmatrix} &= \begin{bmatrix} 2 \\ 2 \end{bmatrix}
\end{aligned}$$

which translates into the norms $||A \cdot [1, 0]^T||_2 = 1$ and $||A \cdot [0, 1]^T||_2 = 2 \sqrt{2}$.  This implies that the $\ell_2$ induced matrix norm of $A$ is at least $||A||_{2} = 2 \sqrt{2} \approx 2.828427125$.

The exact value of $||A||_2$ can be computed using the spectral radius defined as
$$
    \rho(A) = \max_{i} |\lambda_i|,
$$
where $\lambda_i$ are the eigenvalues of $A$.  With this we can compute the $\ell_2$ norm of $A$ as
$$
    ||A||_2 = \sqrt{\rho(A^\ast A)}
$$

Computing the norm again here we find
$$
    A^\ast A = \begin{bmatrix} 1 & 0 \\ 2 & 2 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 2 & 8 \end{bmatrix}
$$
which has eigenvalues
$$
    \lambda = \frac{1}{2}\left(9 \pm \sqrt{65}\right )
$$
so $||A||_2 \approx 2.9208096$.

The actual  induced 2-norm of a matrix can be derived using the Singular Value Decomposition (SVD) and is simply the largest singular value $\sigma_1$.

**Proof**:
Given that every Matrix $A\in\mathbb{C}^{m\times n}$ can be factored into its SVD (see notebook 10.1):

$$
    A = U\Sigma V^*
$$

where $U\in\mathbb{C}^{m\times m}$ and $V\in\mathbb{C}^{n\times n}$ are unitary matrices with the property $U^*U=I$ and $V^*V=I$ (of their respective sizes) and $\Sigma$ is a real diagonal matrix of singular values $\sigma_1 \geq\sigma_2\geq...\sigma_n\geq 0$.

Then the 2-norm squared of a  square matrix is
$$
    ||A||^2_2 = \sup_{\mathbf{x} \in \mathbb{C}^n ~ ||\mathbf{x}||_2 = 1} ||A \mathbf{x}||_2^2 = \mathbf{x}^TA^*A\mathbf{x}
$$
but $A^*A = V\Sigma^2V^*$ so

\begin{align}
    ||A \mathbf{x}||_2^2 &= \mathbf{x}^*V\Sigma^2V^*\mathbf{x} \\
                         &= \mathbf{y}^*\Sigma^2\mathbf{y} \quad\mathrm{where}\quad \mathbf{y}=V^*\mathbf{x}\\
                         &= \sum_{i=1}^n \sigma_i^2|y_i|^2\\
                         &\leq \sigma_1^2\sum_{i=1}^n |y_i|^2 = \sigma_i^2||\mathbf{y}||_2\\
\end{align}

but if $||\mathbf{x}||_2 = 1$ (i.e. is a unit vector), then so is $\mathbf{y}$ because unitary matrices don't change the length of vectors.  So it follows that
$$
    ||A||_2 = \sigma_1
$$

```python
A = numpy.array([[1, 2], [0, 2]])

#calculate the SVD(A)
U, S, Vt = numpy.linalg.svd(A)

print('Singular_values = {}'.format(S))
print('||A||_2 = {}'.format(S.max()))
print('||A||_2 = {}'.format(numpy.linalg.norm(A, ord=2)))

# more fun facts about the SVD
#print(U.T.dot(U))
#print(Vt.T.dot(Vt))
#print(A - numpy.dot(U,numpy.dot(numpy.diag(S),Vt)))

```

    Singular_values = [2.921 0.685]
    ||A||_2 = 2.9208096264818897
    ||A||_2 = 2.9208096264818897

####  Other useful norms of a  Matrix

The 2-norm of a matrix can be expensive to compute,  however there are other norms that are equivalent that can be directly computed from  the components of $A$.  For example

* The induced 1-norm is simply max of the 1-norm of the **columns** of $A$

$$
||A \mathbf{x}||_1 = || \sum^n_{j=1} x_j \mathbf{a}_j ||_1 \leq \sum^n_{j=1} |x_j| ||\mathbf{a}_j||_1 \leq \max_{1\leq j\leq n} ||\mathbf{a}_j||_1 ||\mathbf{x}||_1 =  \max_{1\leq j\leq n} ||\mathbf{a}_j||_1
$$

* The induce $\infty$-norm is simply the max of the 1-norm of **rows** of $A$

$$
||A \mathbf{x}||_\infty = \max_{1 \leq i \leq m} | \mathbf{a}^*_i \mathbf{x} | \leq \max_{1 \leq i \leq m} ||\mathbf{a}^*_i||_1
$$
because the **largest unit vector on the unit sphere in the $\infty$ norm** is a **vector of 1's**.

####  Example:

$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$

$$ ||A||_1 = 4, \quad ||A||_\infty = 3$$

```python
# Calculate the 1-norm of A
normA_1 = numpy.max(numpy.sum(numpy.abs(A), axis=0))
print('||A||_1 = {}'.format(normA_1))
print('||A||_1 = {}'.format(numpy.linalg.norm(A, ord=1)))
# calculate the infinity norm of A
normA_inf = numpy.max(numpy.sum(numpy.abs(A), axis=1))
print('||A||_inf = {}'.format(normA_inf))
print('||A||_inf = {}'.format(numpy.linalg.norm(A, ord=numpy.inf)))
```

    ||A||_1 = 4
    ||A||_1 = 4.0
    ||A||_inf = 3
    ||A||_inf = 3.0

One of the most useful ways to think about matrix norms is as a transformation of a unit-ball to an ellipse.  Depending on the norm in question, the norm will be some combination of the resulting ellipse.

```python
A = numpy.array([[1, 2], [0, 2]])

U, S, Vt = numpy.linalg.svd(A)

print("")
print('||A||_2 = {}'.format(S.max()))
print('||A||_2 = {}'.format(numpy.linalg.norm(A, ord=2)))

```

    ||A||_2 = 2.9208096264818897
    ||A||_2 = 2.9208096264818897

#### 2-Norm

```python
# ============
# 2-norm

import matplotlib.patches as patches

def draw_unit_vectors(axes, A, head_width=0.1):
    head_length = 1.5 * head_width
    image_e = numpy.empty(A.shape)
    angle = numpy.empty(A.shape[0])
    image_e[:, 0] = numpy.dot(A, numpy.array((1.0, 0.0)))
    image_e[:, 1] = numpy.dot(A, numpy.array((0.0, 1.0)))
    for i in range(A.shape[0]):
        angle[i] = numpy.arccos(image_e[0, i] / numpy.linalg.norm(image_e[:, i], ord=2))
        axes.arrow(0.0, 0.0, image_e[0, i] - head_length * numpy.cos(angle[i]),
                             image_e[1, i] - head_length * numpy.sin(angle[i]),
                             head_width=head_width, color='b', alpha=0.5)

# Unit-ball
fig = plt.figure()
fig.suptitle("2-Norm: $||A||_2 = ${:3.4f}".format(numpy.linalg.norm(A,ord=2)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.add_artist(plt.Circle((0.0, 0.0), 1.0, edgecolor='r', facecolor='none'))
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Compute some geometry
u, s, v = numpy.linalg.svd(A)
theta = numpy.empty(A.shape[0])
ellipse_axes = numpy.empty(A.shape)
theta[0] = numpy.arccos(u[0][0]) / numpy.linalg.norm(u[0], ord=2)
theta[1] = theta[0] - numpy.pi / 2.0
for i in range(theta.shape[0]):
    ellipse_axes[0, i] = s[i] * numpy.cos(theta[i])
    ellipse_axes[1, i] = s[i] * numpy.sin(theta[i])

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.add_artist(patches.Ellipse((0.0, 0.0), 2 * s[0], 2 * s[1], theta[0] * 180.0 / numpy.pi,
                                edgecolor='r', facecolor='none'))
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, ellipse_axes[0, i] - head_length * numpy.cos(theta[i]),
                         ellipse_axes[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')
draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-s[0] + 0.1, s[0] + 0.1))
axes.set_ylim((-s[0] + 0.1, s[0] + 0.1))
axes.grid(True)

plt.show()
```

#### 1-Norm

```python
# Note: that this code is a bit fragile to angles that go beyond pi
# due to the use of arccos.

import matplotlib.patches as patches

def draw_unit_vectors(axes, A, head_width=0.1):
    head_length = 1.5 * head_width
    image_e = numpy.empty(A.shape)
    angle = numpy.empty(A.shape[0])
    image_e[:, 0] = numpy.dot(A, numpy.array((1.0, 0.0)))
    image_e[:, 1] = numpy.dot(A, numpy.array((0.0, 1.0)))
    for i in range(A.shape[0]):
        angle[i] = numpy.arccos(image_e[0, i] / numpy.linalg.norm(image_e[:, i], ord=2))
        axes.arrow(0.0, 0.0, image_e[0, i] - head_length * numpy.cos(angle[i]),
                             image_e[1, i] - head_length * numpy.sin(angle[i]),
                             head_width=head_width, color='b', alpha=0.5)

head_width = 0.2
head_length = 1.5 * head_width
# ============
# 1-norm
# Unit-ball
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.suptitle("1-Norm: $||A||_1 = {}$".format(numpy.linalg.norm(A,ord=1)), fontsize=16)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, 0.0, -1.0, 0.0, 1.0), (0.0, 1.0, 0.0, -1.0, 0.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((1.0, 2.0, -1.0, -2.0, 1.0), (0.0, 2.0, 0.0, -2.0, 0.0), 'r')
draw_unit_vectors(axes, A, head_width=0.2)

axes.set_title("Images Under A")
axes.grid(True)

plt.show()
```

#### $\infty$-Norm

```python
# ============
# infty-norm
# Unit-ball
fig = plt.figure()
fig.suptitle("$\infty$-Norm: $||A||_\infty = {}$".format(numpy.linalg.norm(A,ord=numpy.inf)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, -1.0, -1.0, 1.0, 1.0), (1.0, 1.0, -1.0, -1.0, 1.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Geometry - Corners are A * ((1, 1), (1, -1), (-1, 1), (-1, -1))
# Symmetry implies we only need two.  Here we just plot two
u = numpy.empty(A.shape)
u[:, 0] = numpy.dot(A, numpy.array((1.0, 1.0)))
u[:, 1] = numpy.dot(A, numpy.array((-1.0, 1.0)))
theta[0] = numpy.arccos(u[0, 0] / numpy.linalg.norm(u[:, 0], ord=2))
theta[1] = numpy.arccos(u[0, 1] / numpy.linalg.norm(u[:, 1], ord=2))

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((3, 1, -3, -1, 3), (2, 2, -2, -2, 2), 'r')
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, u[0, i] - head_length * numpy.cos(theta[i]),
                         u[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')

draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-4.1, 4.1))
axes.set_ylim((-3.1, 3.1))
axes.grid(True)

plt.show()
```

#### Cauchy-Schwarz and Hölder Inequalities

Computing matrix norms where $p \neq 1$ or $\infty$ is more difficult unfortunately.  We have a couple of tools that can be useful however.

 - **Cauchy-Schwarz Inequality**:  For the special case where $p=q=2$, for any vectors $\mathbf{x}$ and $\mathbf{y}$
$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_2 ||\mathbf{y}||_2
$$
 - **Hölder's Inequality**:  Turns out this holds in general if given a $p$ and $q$ that satisfy $1/p + 1/q = 1$ with $1 \leq p, q \leq \infty$

$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_p ||\mathbf{y}||_q.
$$

**Note**: this is essentially what we used in the proof of the $\infty-$norm with $p=1$ and $q=\infty$

#### General Matrix Norms (induced and  non-induced)

In general matrix-norms have the following properties whether they are induced from a vector-norm or not:
1. $||A|| \geq 0$ and $||A|| = 0$ only if $A = 0$
1. $||A + B|| \leq ||A|| + ||B||$ (Triangle Inequality)
1. $||c A|| = |c| ||A||$

The most widely used matrix norm not induced by a vector norm is the **Frobenius norm** defined by
$$
    ||A||_F = \left( \sum^m_{i=1} \sum^n_{j=1} |A_{ij}|^2 \right)^{1/2}.
$$

#### Invariance under unitary multiplication

One important property of the matrix 2-norm (and Frobenius norm) is that multiplication by a unitary matrix does not change the product (kind of like multiplication by 1).  In general for any $A \in \mathbb{C}^{m\times n}$ and unitary matrix $Q \in \mathbb{C}^{m \times m}$ we have
\begin{align*}
    ||Q A||_2 &= ||A||_2 \\ ||Q A||_F &= ||A||_F.
\end{align*}

1 http://www.utstat.toronto.edu/~brunner/books/LinearModelsInStatistics.pdf