```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 5: Managing Data with Pandas

Pandas is a Python library for managing datasets. Documentation and examples are available on the website for Pandas: http://pandas.pydata.org/.

In this Notebook, we'll make use of a dataset containing long-run averages of inflation, money growth, and real GDP. The dataset is available here: https://www.briancjenkins.com/data/csv/qtyTheoryData.csv (Python code to generate the dataset: https://github.com/letsgoexploring/economic-data). Recall that the quantity theory of money implies the following linear relationship between the long-run rate of money growth, the long-run rate of inflation, and the long-run rate of real GDP growth in a country:

\begin{align}
\text{inflation} & = \text{money growth} - \text{real GDP growth},
\end{align}

Generally, we treat real GDP growth and money supply growth as exogenous so this is a theory about the determination of inflation.

### Import Pandas

```python
import pandas as pd
```

### Import data from a csv file

Pandas has a function called `read_csv()` for reading data from a csv file into a Pandas `DataFrame` object.

```python
# Import quantity theory data into a Pandas DataFrame called 'df' with country names as the index.

# Directly from internet
df = pd.read_csv('https://www.briancjenkins.com/data/csv/qtyTheoryData.csv')

# From current working directory
# df = pd.read_csv('qtyTheoryData.csv')
```

```python
# Print the first 5 rows
print(df.head())
```

                   country iso code  observations  inflation  money growth  \
    0              Albania      ALB            21    0.05186       0.12230
    1              Algeria      DZA            51    0.10688       0.16024
    2               Angola      AGO            20    0.82783       1.00940
    3  Antigua and Barbuda      ATG            38    0.03943       0.09630
    4            Argentina      ARG            54    0.73168       0.77245

       gdp growth
    0     0.04594
    1     0.03818
    2     0.08146
    3     0.03575
    4     0.02422

```python
# Print the last 10 rows
print(df.tail(10))
```

                      country iso code  observations  inflation  money growth  \
    168               Ukraine      UKR            23    0.58513       0.55392
    169  United Arab Emirates      ARE            40    0.03566       0.13659
    170         United States      USA            54    0.03377       0.05644
    171               Uruguay      URY            54    0.36846       0.40479
    172               Vanuatu      VUT            36    0.03509       0.08102
    173         Venezuela, RB      VEN            53    0.20298       0.26996
    174    West Bank and Gaza      PSE            17    0.03632       0.11749
    175           Yemen, Rep.      YEM            24    0.14113       0.13238
    176                Zambia      ZMB            27    0.21917       0.23465
    177              Zimbabwe      ZWE            32   -0.00124      -0.12919

         gdp growth
    168    -0.01023
    169     0.04688
    170     0.03081
    171     0.02251
    172     0.02889
    173     0.02740
    174     0.03186
    175     0.03838
    176     0.01163
    177     0.00813

```python
# Print the type of variable 'df'
print(type(df))
```

    <class 'pandas.core.frame.DataFrame'>

### Properties of `DataFrame` objects

Like entries in a spreadsheet file, elements in a `DataFrame` object have row (or *index*) and column coordinates. Column names are always strings. Index elements can be integers, strings, or dates.

```python
# Print the columns of df
print(df.columns)
```

    Index(['country', 'iso code', 'observations', 'inflation', 'money growth',
           'gdp growth'],
          dtype='object')

```python
# Create a new variable called money equal to the 'money growth' column and print
money = df['money growth']
print(money)
```

    0      0.12230
    1      0.16024
    2      1.00940
    3      0.09630
    4      0.77245
    5      0.46411
    6      0.08128
    7      0.10181
    8      0.07491
    9      0.50149
    10     0.06965
    11     0.08263
    12     0.13523
    13     0.10833
    14     0.61733
    15     0.06306
    16     0.11148
    17     0.10734
    18     0.18748
    19     0.42180
    20     0.19611
    21     0.15783
    22     0.89717
    23     0.02883
    24     0.34614
    25     0.10509
    26     0.13430
    27     0.09847
    28     0.21306
    29     0.09689
            ...
    148    0.12114
    149    0.14659
    150    0.09518
    151    0.09639
    152    0.29333
    153    0.29243
    154    0.14436
    155    0.10587
    156    0.06059
    157    0.15539
    158    0.30609
    159    0.20918
    160    0.09645
    161    0.18530
    162    0.09973
    163    0.09616
    164    0.11841
    165    0.10958
    166    0.37305
    167    0.36831
    168    0.55392
    169    0.13659
    170    0.05644
    171    0.40479
    172    0.08102
    173    0.26996
    174    0.11749
    175    0.13238
    176    0.23465
    177   -0.12919
    Name: money growth, Length: 178, dtype: float64

```python
# Print the type of the variable money
print(type(money))
```

    <class 'pandas.core.series.Series'>

A Pandas `Series` stores one column of data. Like a `DataFrame`, a `Series` object has an index. Note that `money` has the same index as `df`. Instead of having a column, the `Series` has a `name` attribute.

```python
# Print the name of the 'money' variable
print(money.name)
```

    money growth

Select multiple columns of a `DataFrame` by puting the desired column names in a set a of square brackets (i.e., in a `list`).

```python
# Print the first 5 rows of just the inflation, money growth, and gdp growth columns
print(df[['inflation','money growth','gdp growth']].head())
```

       inflation  money growth  gdp growth
    0    0.05186       0.12230     0.04594
    1    0.10688       0.16024     0.03818
    2    0.82783       1.00940     0.08146
    3    0.03943       0.09630     0.03575
    4    0.73168       0.77245     0.02422

As mentioned, the set of row coordinates is the index. Unless specified otherwise, Pandas automatically assigns an integer index starting at 0 to rows of the `DataFrame`.

```python
# Print the index of 'df'
print(df.index)
```

    RangeIndex(start=0, stop=178, step=1)

Note that in the index of the `df` is the numbers 0 through 177. We could have specified a different index when we imported the data using `read_csv()`. For example, suppose we want to the country names to be the index of `df`. Since country names are in the first column of the data file, we can pass the argument `index_col=0` to `read_csv()`

```python
# Import quantity theory data into a Pandas DataFrame called 'df' with country names as the index.
df = pd.read_csv('https://www.briancjenkins.com/data/csv/qtyTheoryData.csv',index_col=0)

# Print first 5 rows of df
print(df.head())
```

                        iso code  observations  inflation  money growth  \
    country
    Albania                  ALB            21    0.05186       0.12230
    Algeria                  DZA            51    0.10688       0.16024
    Angola                   AGO            20    0.82783       1.00940
    Antigua and Barbuda      ATG            38    0.03943       0.09630
    Argentina                ARG            54    0.73168       0.77245

                         gdp growth
    country
    Albania                 0.04594
    Algeria                 0.03818
    Angola                  0.08146
    Antigua and Barbuda     0.03575
    Argentina               0.02422

Use the `loc` attribute to select rows of the `DataFrame` by index *values*.

```python
# Create a new variable called 'df_usa' equal to the 'United States' row and print
df_usa = df.loc['United States']
print(df_usa)
```

    iso code            USA
    observations         54
    inflation       0.03377
    money growth    0.05644
    gdp growth      0.03081
    Name: United States, dtype: object

Use `iloc` attribute to select row based on integer location (starting from 0).

```python
# Create a new variable called 'df_third' equal to the third row in the DataFrame and print
df_first = df.iloc[2]
print(df_first)
```

    iso code            AGO
    observations         20
    inflation       0.82783
    money growth     1.0094
    gdp growth      0.08146
    Name: Angola, dtype: object

There are several ways to return a single element of a Pandas `DataFrame`. For example, here are three that we want to return the value of inflation for the United States from the DataFrame `df`:

1. `df.loc['United States','inflation']`
2. `df.loc['United States']['inflation']`
3. `df['inflation'].loc['United States']`

The first method points directly to the element in the `df` while the second and third methods return *copies* of the element. That means that you can modify the value of inflation for the United States by running:

    df.loc['United States','inflation'] = new_value

But running either:

    df.loc['United States']['inflation'] = new_value

or:

    df['inflation'].loc['United States'] = new_value

will return an error.

```python
# Print the inflation rate of the United States  (By index and column together)
print('Long-run average inflation in US: ',df.loc['United States','inflation'])
```

    Long-run average inflation in US:  0.03377

```python
# Print the inflation rate of the United States (first by index, then by column)
print('Long-run average inflation in US: ',df.loc['United States']['inflation'])
```

    Long-run average inflation in US:  0.03377

```python
# Print the inflation rate of the United States  (first by column, then by index)
print('Long-run average inflation in US: ',df['inflation'].loc['United States'])
```

    Long-run average inflation in US:  0.03377

New columns are easily created as functions of existing columns.

```python
# Create a new column called 'difference' equal to the money growth column minus
# the inflation column and print the modified DataFrame
df['difference'] = df['money growth'] - df['inflation']
print(df['difference'])
```

    country
    Albania                           0.07044
    Algeria                           0.05336
    Angola                            0.18157
    Antigua and Barbuda               0.05687
    Argentina                         0.04077
    Armenia                           0.02154
    Aruba                             0.04959
    Australia                         0.05106
    Austria                           0.05845
    Azerbaijan                        0.07387
    Bahamas, The                      0.02115
    Bahrain                           0.05318
    Bangladesh                        0.05657
    Barbados                          0.04723
    Belarus                           0.02929
    Belgium                           0.04654
    Belize                            0.08926
    Benin                             0.05905
    Bhutan                            0.12045
    Bolivia                           0.08569
    Bosnia and Herzegovina            0.15565
    Botswana                          0.06462
    Brazil                            0.03037
    Brunei Darussalam                -0.02624
    Bulgaria                          0.06415
    Burkina Faso                      0.06647
    Burundi                           0.04095
    Cabo Verde                        0.06715
    Cambodia                          0.15604
    Cameroon                          0.04300
                                       ...
    Sri Lanka                         0.02800
    St. Kitts and Nevis               0.11041
    St. Lucia                         0.05407
    St. Vincent and the Grenadines    0.04987
    Sudan                             0.04654
    Suriname                          0.03204
    Swaziland                         0.05194
    Sweden                            0.03408
    Switzerland                       0.04355
    Syrian Arab Republic              0.06709
    Tajikistan                        0.13993
    Tanzania                          0.06030
    Thailand                          0.05509
    Timor-Leste                       0.09552
    Togo                              0.05267
    Tonga                             0.04220
    Trinidad and Tobago               0.04894
    Tunisia                           0.05190
    Turkey                            0.06002
    Uganda                            0.09962
    Ukraine                          -0.03121
    United Arab Emirates              0.10093
    United States                     0.02267
    Uruguay                           0.03633
    Vanuatu                           0.04593
    Venezuela, RB                     0.06698
    West Bank and Gaza                0.08117
    Yemen, Rep.                      -0.00875
    Zambia                            0.01548
    Zimbabwe                         -0.12795
    Name: difference, Length: 178, dtype: float64

```python
# Print the average difference between money growth and inflation
print(df.difference.mean())
```

    0.06082084269662919

```python
# Remove the following columns from the DataFrame: 'iso code','observations','difference'
df = df.drop(['observations','difference'],axis=1)

# Print the modified DataFrame
print(df)
```

                                   iso code  inflation  money growth  gdp growth
    country
    Albania                             ALB    0.05186       0.12230     0.04594
    Algeria                             DZA    0.10688       0.16024     0.03818
    Angola                              AGO    0.82783       1.00940     0.08146
    Antigua and Barbuda                 ATG    0.03943       0.09630     0.03575
    Argentina                           ARG    0.73168       0.77245     0.02422
    Armenia                             ARM    0.44257       0.46411     0.05471
    Aruba                               ABW    0.03169       0.08128     0.00825
    Australia                           AUS    0.05075       0.10181     0.03461
    Austria                             AUT    0.01646       0.07491     0.01446
    Azerbaijan                          AZE    0.42762       0.50149     0.05468
    Bahamas, The                        BHS    0.04850       0.06965     0.01829
    Bahrain                             BHR    0.02945       0.08263     0.03984
    Bangladesh                          BGD    0.07866       0.13523     0.04486
    Barbados                            BRB    0.06110       0.10833     0.02450
    Belarus                             BLR    0.58804       0.61733     0.04919
    Belgium                             BEL    0.01652       0.06306     0.01456
    Belize                              BLZ    0.02222       0.11148     0.04995
    Benin                               BEN    0.04829       0.10734     0.03684
    Bhutan                              BTN    0.06703       0.18748     0.07094
    Bolivia                             BOL    0.33611       0.42180     0.03016
    Bosnia and Herzegovina              BIH    0.04046       0.19611     0.04219
    Botswana                            BWA    0.09321       0.15783     0.06992
    Brazil                              BRA    0.86680       0.89717     0.04129
    Brunei Darussalam                   BRN    0.05507       0.02883     0.01031
    Bulgaria                            BGR    0.28199       0.34614     0.01943
    Burkina Faso                        BFA    0.03862       0.10509     0.04196
    Burundi                             BDI    0.09335       0.13430     0.02690
    Cabo Verde                          CPV    0.03132       0.09847     0.06541
    Cambodia                            KHM    0.05702       0.21306     0.04995
    Cameroon                            CMR    0.05389       0.09689     0.03480
    ...                                 ...        ...           ...         ...
    Sri Lanka                           LKA    0.09314       0.12114     0.04756
    St. Kitts and Nevis                 KNA    0.03618       0.14659     0.03959
    St. Lucia                           LCA    0.04111       0.09518     0.03606
    St. Vincent and the Grenadines      VCT    0.04652       0.09639     0.03776
    Sudan                               SDN    0.24679       0.29333     0.03785
    Suriname                            SUR    0.26039       0.29243     0.01445
    Swaziland                           SWZ    0.09242       0.14436     0.04881
    Sweden                              SWE    0.07179       0.10587     0.02789
    Switzerland                         CHE    0.01704       0.06059     0.01700
    Syrian Arab Republic                SYR    0.08830       0.15539     0.05270
    Tajikistan                          TJK    0.16616       0.30609     0.07200
    Tanzania                            TZA    0.14888       0.20918     0.04978
    Thailand                            THA    0.04136       0.09645     0.05997
    Timor-Leste                         TLS    0.08978       0.18530     0.07596
    Togo                                TGO    0.04706       0.09973     0.03432
    Tonga                               TON    0.05396       0.09616     0.01657
    Trinidad and Tobago                 TTO    0.06947       0.11841     0.02876
    Tunisia                             TUN    0.05768       0.10958     0.04585
    Turkey                              TUR    0.31303       0.37305     0.04647
    Uganda                              UGA    0.26869       0.36831     0.05673
    Ukraine                             UKR    0.58513       0.55392    -0.01023
    United Arab Emirates                ARE    0.03566       0.13659     0.04688
    United States                       USA    0.03377       0.05644     0.03081
    Uruguay                             URY    0.36846       0.40479     0.02251
    Vanuatu                             VUT    0.03509       0.08102     0.02889
    Venezuela, RB                       VEN    0.20298       0.26996     0.02740
    West Bank and Gaza                  PSE    0.03632       0.11749     0.03186
    Yemen, Rep.                         YEM    0.14113       0.13238     0.03838
    Zambia                              ZMB    0.21917       0.23465     0.01163
    Zimbabwe                            ZWE   -0.00124      -0.12919     0.00813

    [178 rows x 4 columns]

### Methods

A Pandas `DataFrame` has a bunch of useful methods defined for it. `describe()` returns some summary statistics.

```python
# Print the summary statistics for 'df'
print(df.describe())
```

            inflation  money growth  gdp growth
    count  178.000000    178.000000  178.000000
    mean     0.123952      0.184773    0.036382
    std      0.165354      0.163258    0.021763
    min     -0.001240     -0.129190   -0.023680
    25%      0.041170      0.099833    0.023447
    50%      0.068250      0.137390    0.036875
    75%      0.125903      0.198862    0.046857
    max      1.220970      1.171190    0.159540

The `corr()` method returns a `DataFrame` containing the correlation coefficients of the specified `DataFrame`.

```python
# Create a variable called 'correlations' containg the correlation coefficients for columns in 'df'
correlations = df.corr()

# Print the correlation coefficients
print(correlations)
```

                  inflation  money growth  gdp growth
    inflation      1.000000      0.972160   -0.043356
    money growth   0.972160      1.000000    0.084328
    gdp growth    -0.043356      0.084328    1.000000

```python
# Print the correlation coefficient for inflation and money growth
print('corr of inflation and money growth: ',round(correlations.loc['inflation','money growth'],4))

# Print the correlation coefficient for inflation and real GDP growth
print('corr of inflation and gdp growth:   ',round(correlations.loc['inflation','gdp growth'],4))

# Print the correlation coefficient for money growth and real GDP growth
print('corr of money growth and gdp growth:',round(correlations.loc['money growth','gdp growth'],4))
```

    corr of inflation and money growth:  0.9722
    corr of inflation and gdp growth:    -0.0434
    corr of money growth and gdp growth: 0.0843

`sort_values()` returns a copy of the original `DataFrame` sorted along the given column. The optional argument `ascending` is set to `True` by default, but can be changed to `False` if you want to print the lowest first.

```python
# Print rows for the countries with the 10 lowest inflation rates
print(df.sort_values('inflation').head(10))
```

                         iso code  inflation  money growth  gdp growth
    country
    Zimbabwe                  ZWE   -0.00124      -0.12919     0.00813
    Djibouti                  DJI    0.00033       0.07611     0.01616
    Germany                   DEU    0.01049       0.07344     0.01155
    Ireland                   IRL    0.01375       0.12348     0.03326
    Hong Kong SAR, China      HKG    0.01391       0.09846     0.03618
    France                    FRA    0.01461       0.06584     0.01233
    Kosovo                    KSV    0.01540       0.08659     0.03359
    Greece                    GRC    0.01611       0.01423    -0.00452
    Austria                   AUT    0.01646       0.07491     0.01446
    Finland                   FIN    0.01647       0.06528     0.01387

```python
# Print rows for the countries with the 10 highest inflation rates
print(df.sort_values('inflation',ascending=False).head(10))
```

                     iso code  inflation  money growth  gdp growth
    country
    Congo, Dem. Rep.      COD    1.22097       1.17119    -0.00232
    Brazil                BRA    0.86680       0.89717     0.04129
    Angola                AGO    0.82783       1.00940     0.08146
    Argentina             ARG    0.73168       0.77245     0.02422
    Nicaragua             NIC    0.61301       0.66175     0.02475
    Belarus               BLR    0.58804       0.61733     0.04919
    Ukraine               UKR    0.58513       0.55392    -0.01023
    Peru                  PER    0.47231       0.53655     0.03444
    Armenia               ARM    0.44257       0.46411     0.05471
    Azerbaijan            AZE    0.42762       0.50149     0.05468

Note that `sort_values` and `sort_index` return *copies* of the original `DataFrame`. If, in the previous example, we had wanted to actually modify `df`, we would have need to explicitly overwrite it:

    df = df.sort_index(ascending=False)

```python
# Print first 10 rows with the index sorted in descending alphabetical order
print(df.sort_index(ascending=False).head(10))
```

                         iso code  inflation  money growth  gdp growth
    country
    Zimbabwe                  ZWE   -0.00124      -0.12919     0.00813
    Zambia                    ZMB    0.21917       0.23465     0.01163
    Yemen, Rep.               YEM    0.14113       0.13238     0.03838
    West Bank and Gaza        PSE    0.03632       0.11749     0.03186
    Venezuela, RB             VEN    0.20298       0.26996     0.02740
    Vanuatu                   VUT    0.03509       0.08102     0.02889
    Uruguay                   URY    0.36846       0.40479     0.02251
    United States             USA    0.03377       0.05644     0.03081
    United Arab Emirates      ARE    0.03566       0.13659     0.04688
    Ukraine                   UKR    0.58513       0.55392    -0.01023

### Quick plotting example

Construct a graph that visually confirms the quantity theory of money by making a scatter plot with average money growth on the horizontal axis and average inflation on the vertical axis. Set the marker size `s` to 50 and opacity (`alpha`) 0.25. Add a 45 degree line, axis labels, and a title. Lower and upper limits for the horizontal and vertical axes should be -0.2 and 1.2.

```python
# Create data for 45 degree line
x45 = [-0.2,1.2]
y45 = [-0.2,1.2]

# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot 45 degree line and create legend in lower right corner
ax.plot(x45,y45,'-r',label = '$45^{\circ}$')
ax.legend(loc='lower right')

# Scatter plot of data inflation against money growth
ax.scatter(df['money growth'],df['inflation'],s=50,alpha = 0.25)
ax.set_xlim([-0.2,1.2])
ax.set_ylim([-0.2,1.2])
ax.set_xlabel('money growth')
ax.set_ylabel('inflation')
ax.set_title('Average inflation against average money growth \nfor '+str(len(df.index))+' countries.')
ax.grid()
```

### Exporting a `DataFrame` to csv

Use the DataFrame method `to_csv()` to export DataFrame to a csv file.

```python
# Export the DataFrame 'df' to a csv file called 'modified_data.csv'.
df.to_csv('modified_data.csv')
```


# N-Gram Language Models

#### Preliminaries: Install KenLM

[KenLM](https://kheafield.com/code/kenlm/) is used in the latter part of this lab.

To install it, run the following commands **in the jupyter terminal** (see screenshot):

    sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev

    cd /home/jupyter
    wget -O - https://kheafield.com/code/kenlm.tar.gz |tar xz
    mkdir kenlm/build
    cd kenlm/build
    cmake ..
    make -j 4

    cd /home/jupyter/kenlm
    python setup.py install

```python
!pip install jsonlines
```

    Collecting jsonlines
      Downloading https://files.pythonhosted.org/packages/4f/9a/ab96291470e305504aa4b7a2e0ec132e930da89eb3ca7a82fbe03167c131/jsonlines-1.2.0-py2.py3-none-any.whl
    Requirement already satisfied: six in /home/aims/anaconda3/envs/aims/lib/python3.7/site-packages (from jsonlines) (1.12.0)
    Installing collected packages: jsonlines
    Successfully installed jsonlines-1.2.0

```python
%pylab inline
import os, sys, glob, json, math
import numpy as np
import pandas as pd
from tqdm import tqdm
from pprint import pprint
from collections import defaultdict
pd.set_option('display.max_colwidth', -1)
```

    Populating the interactive namespace from numpy and matplotlib

### N-gram Language Modeling

In **language modeling** we want to model the probability of variable length sequences, $$\large p(w_1,\ldots,w_T)=\prod_{t=1}^T p(w_t|w_{<t}).$$

An **n-gram language model** assumes that each word $w_t$ only depends on the preceding $n-1$ words, $$\large p(w_1,\ldots,w_T)=\prod_{t=1}^T p(w_t|w_{t-n+1},\ldots,w_{t-1}).$$

#### Example
For instance, when modeling the sentence $$\texttt{the cat sat on the mat .}$$ a 3-gram language model assumes that $$p(\texttt{mat}|\texttt{the cat sat on the}) \approx p(\texttt{mat}|\texttt{on the}).$$

The sub-sequence $(\texttt{on the mat})$ is a *3-gram* or *trigram*.

### Count-based Estimation

Given some dataset $D$ of sequences, we can estimate an n-gram model through counting, derived as follows:

\begin{align}
p(w_t|w_{t-n+1},\ldots,w_{t-1}) &= \frac{p(w_{t-n+1},\ldots,w_t)}{p(w_{t-n+1},\ldots,w_{t-1})} & \text{definition of conditional probability}\\
                       &= \frac{p(w_{t-n+1},\ldots,w_t)}{\sum_{w_{t'}}p(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &\approx \frac{\frac{1}{N}\text{count}(w_{t-n+1},\ldots,w_t)}{\frac{1}{N}\sum_{w_{t'}}\text{count}(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &= \frac{\text{count}(w_{t-n+1},\ldots,w_t)}{\sum_{w_{t'}}\text{count}(w_{t-n+1},\ldots,w_{t-1},w_{t'})}\\
                       &= \frac{\text{count}(w_{t-n+1},\ldots,w_t)}{\text{count}(w_{t-n+1},\ldots,w_{t-1})},
\end{align}

where $N$ is the number of $n$-grams in the dataset.

In Python, we can collect these counts into a dictionary mapping a prefix to a dictionary of counts:

        count[(w_n+1,...,w_t-1)] = {wt1: count of (w_n+1,...,w_t1),
                                    wt2: count of (w_n+1,...,w_t2),
                                    ...
                                   }

and for the denominator, maintain a dictionary of totals:

        total[(w_n+1,...,w_t-1)] = count of w_n+1,...,w_t-1

### Simplified Language

To get intuition, lets start by modeling a simple language called `ABC`. A word in this language is one of three tokens, $$w\in \{\texttt{A, B, C}\},$$
and we'll denote a sentence as $\textbf{w}=(w_1,\ldots,w_{|\textbf{w}|})$.

Suppose we are given the following dataset, and want to estimate a **bigram model**: $$p(\textbf{w})\approx\prod_{t=1}^{|\textbf{w}|}p(w_t|w_{t-1})\quad\quad(*)$$

```python
data_raw = [['A', 'A', 'B', 'B'],
            ['A', 'A', 'B'],
            ['A', 'A', 'B', 'C'],
            ['A', 'A', 'A'],
            ['A', 'A', 'A', 'A']]
```

Since our model is a probability distribution, the total probability of all possible strings in the language must sum to 1, i.e.: $$\sum_{\textbf{w}}p(\textbf{w})=1.$$

In order to satisfy this criterion it turns out that we need an additional **beginning token**, `<bos>`, and **end token**, `<eos>`:

```python
data = [['<bos>'] + d + ['<eos>'] for d in data_raw]
data
```

    [['<bos>', 'A', 'A', 'B', 'B', '<eos>'],
     ['<bos>', 'A', 'A', 'B', '<eos>'],
     ['<bos>', 'A', 'A', 'B', 'C', '<eos>'],
     ['<bos>', 'A', 'A', 'A', '<eos>'],
     ['<bos>', 'A', 'A', 'A', 'A', '<eos>']]

Now let's estimate a bigram model:

\begin{align}
p(w_t|w_{t-1}) &= \frac{\text{count}(w_{t-1}w_{t})}{\sum_{w_{t'}}\text{count}(w_{t-1}w_{t'})}\\
               &= \texttt{count[prefix][wt] / totals[prefix]}
\end{align}

where $\texttt{prefix}$ is $w_{t-1}$ in this case.

```python
count = defaultdict(lambda: defaultdict(float))
total = defaultdict(float)

n = 2
for sequence in data:
    for i in range(len(sequence)-n+1):         # for each ngram
        ngram = tuple(sequence[i:i+n])
        prefix, word = ngram[:-1], ngram[-1]
        count[prefix][word] += 1               # count(w_{t-n+1}...w_t)
        total[prefix] += 1                     # count(w_{t-n+1}...w_{t-1})
```

Let's see if the counts and totals make sense:

- How many times did (A, B) occur? What about (B, B)?
- How many times did (A) occur? What about (C)?

```python
print("Counts:")
pprint(dict(count))
print("\nTotals:")
pprint(dict(total))
```

    Counts:
    {('<bos>',): defaultdict(<class 'float'>, {'A': 5.0}),
     ('A',): defaultdict(<class 'float'>, {'A': 8.0, 'B': 3.0, '<eos>': 2.0}),
     ('B',): defaultdict(<class 'float'>, {'B': 1.0, '<eos>': 2.0, 'C': 1.0}),
     ('C',): defaultdict(<class 'float'>, {'<eos>': 1.0})}

    Totals:
    {('<bos>',): 5.0, ('A',): 13.0, ('B',): 4.0, ('C',): 1.0}

#### Conditional probability queries

We can now query a conditional probability:

\begin{align}
\texttt{p(word|prefix)} =&\ \texttt{count[prefix][word] / totals[prefix]}
\end{align}

```python
queries = [('<bos>', 'A'),
           ('B', 'C')]

for query in queries:
    prefix, word = query[:-1], query[-1]
    p = count[prefix][word] / total[prefix]  # We'll discuss the case when `total[prefix] = 0` below.
    print("p( %s | %s) = \t%.5f" % (word, ', '.join(prefix), p))
```

    p( A | <bos>) = 	1.00000
    p( C | B) = 	0.25000

**Exercise**: Look at the training set and convince yourself that these conditional probabilities are correct according to the count-based estimation procedure.

#### Sequence Probability

We can compute the probability of a sequence using the conditional probabilities along with the chain rule of probability:

\begin{align}
p(w_1,\ldots,w_T)&\approx\prod_{t=1}^T p(w_t|w_{t-1})
\end{align}

(Here $w_0$ is `<bos>` and $w_T$ is `<eos>`)

```python
sequence = ['<bos>', 'A', 'A', 'B', '<eos>']

def sequence_p(sequence, log=False):
    total_p = 1

    for i in range(len(sequence)-n+1):
        ngram = tuple(sequence[i:i+n])
        prefix = ngram[:-1]
        word = ngram[-1]
        p = count[prefix][word] / max(total[prefix], 1)
        if log:
            print("p(%s | %s) =\t%.3f" % (word, ', '.join(prefix), p))

        total_p *= p
    return total_p

print("\nProduct: p(%s) = %.3f" % (''.join(sequence[1:-1]), sequence_p(sequence, log=True)))
```

    p(A | <bos>) =	1.000
    p(A | A) =	0.615
    p(B | A) =	0.231
    p(<eos> | B) =	0.500

    Product: p(AAB) = 0.071

### Real Example: Dialogue Utterances

Now lets use the same ideas on a more realistic text corpus.

We will use utterances from a dialogue dataset called **Persona-Chat**. This dataset is relatively small and centers on a single domain, but it is simple and interpretable for our purposes here.

We'll also use an off-the-shelf ngram modeling package called `KenLM`.

#### Loading the dataset

```python
import kenlm
```

```python
train = []
valid = []
import jsonlines

for ds, name in [(train, 'train'), (valid, 'valid')]:
    for line in jsonlines.Reader(open('data/personachat/personachat_all_sentences_%s.jsonl' % name, 'r')):
        ds.append(line['tokens'])

vocab = list(set([t for ts in train for t in ts]))
print("Number of train examples: %d" % (len(train)))
print("Number of valid examples: %d" % (len(valid)))
print("Vocab size: %d" % (len(vocab)))

print("\nExamples:")
pprint(train[:3])
```

### KenLM

KenLM estimates n-gram language models using **modified Kneser-Ney smoothing**, and has a fast and memory-efficient implementation.
- While we won't go into details here, **smoothing** is a technique used to account for ngrams that do not occur in the training corpus.
- Normally, these ngrams would receive zero-probability mass. Smoothing ensures every ngram receives some probability.

Please see page 48 of the [lecture note](https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf) for an overview of Kneser-Ney smoothing, and [[Chen & Goodman 1998]](https://dash.harvard.edu/bitstream/handle/1/25104739/tr-10-98.pdf?sequence=1) for further details.

```python
KENLM_DIR='/home/jupyter/kenlm'
```

#### Tokenize data

```python
if not os.path.exists('data/tokenized'):
    os.makedirs('data/tokenized')
with open('data/tokenized/pchat_train', 'w') as f:
    for line in tqdm(train):
        f.write(' '.join(line))
        f.write('\n')
```

    100%|██████████| 133176/133176 [00:00<00:00, 795531.18it/s]

#### Build kenlm n-gram models

This uses the `kenlm` commands `lmplz` to estimate the language model, then `build_binary` to convert it to an efficient format. We load the resulting model using the `kenlm` python wrapper.

We do this for n-gram models of order `2,3,4`:

```python
import kenlm

data_prefix = 'pchat'
dataset = 'pchat_train'
if not os.path.exists('models'):
    os.makedirs('models')

models = {}
for n in [2,3,4]:
    model_temp = 'models/%s_%d.arpa' % (data_prefix, n)
    model_name = 'models/%s_%d.klm' % (data_prefix, n)
    ! cat ./data/tokenized/$dataset | $KENLM_DIR/build/bin/lmplz -o $n > $model_temp
    ! $KENLM_DIR/build/bin/build_binary $model_temp $model_name
    models[n] = kenlm.LanguageModel(model_name)
```

    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:10928379904
    Statistics:
    1 19156 D1=0.588857 D2=1.0348 D3+=1.3388
    2 231267 D1=0.708637 D2=1.06132 D3+=1.37629
    Memory estimate for binary LM:
    type      kB
    probing 4551 assuming -p 1.5
    probing 4626 assuming -r models -p 1.5
    trie    1747 without quantization
    trie    1099 assuming -q 8 -b 8 quantization
    trie    1747 assuming -a 22 array pointer compression
    trie    1099 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
    === 3/5 Calculating and sorting initial probabilities ===
    Chain sizes: 1:229872 2:3700272
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 4/5 Calculating and writing order-interpolated probabilities ===
    Chain sizes: 1:229872 2:3700272
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 5/5 Writing ARPA model ===
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    Name:lmplz	VmPeak:10874216 kB	VmRSS:10720 kB	RSSMax:2938548 kB	user:0.848	sys:1.288	CPU:2.13913	real:2.13823
    Reading models/pchat_2.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    SUCCESS
    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:3801175808 3:7127204352
    Statistics:
    1 19156 D1=0.588857 D2=1.0348 D3+=1.3388
    2 231267 D1=0.722265 D2=1.09097 D3+=1.44737
    3 617624 D1=0.798924 D2=1.0703 D3+=1.33013
    Memory estimate for binary LM:
    type       kB
    probing 16763 assuming -p 1.5
    probing 18193 assuming -r models -p 1.5
    trie     6683 without quantization
    trie     3625 assuming -q 8 -b 8 quantization
    trie     6354 assuming -a 22 array pointer compression
    trie     3296 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
    === 3/5 Calculating and sorting initial probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 4/5 Calculating and writing order-interpolated probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 5/5 Writing ARPA model ===
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    Name:lmplz	VmPeak:10866020 kB	VmRSS:22604 kB	RSSMax:2500264 kB	user:1.408	sys:1.228	CPU:2.63778	real:2.52355
    Reading models/pchat_3.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    SUCCESS
    === 1/5 Counting and sorting n-grams ===
    File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
    Unigram tokens 1602042 types 19156
    === 2/5 Calculating and sorting adjusted counts ===
    Chain sizes: 1:229872 2:1860149760 3:3487780608 4:5580449280
    Statistics:
    1 19156 D1=0.588857 D2=1.0348 D3+=1.3388
    2 231267 D1=0.722265 D2=1.09097 D3+=1.44737
    3 617624 D1=0.821472 D2=1.14168 D3+=1.40217
    4 932153 D1=0.863174 D2=1.12888 D3+=1.29333
    Memory estimate for binary LM:
    type       kB
    probing 36767 assuming -p 1.5
    probing 41816 assuming -r models -p 1.5
    trie    15838 without quantization
    trie     8356 assuming -q 8 -b 8 quantization
    trie    14567 assuming -a 22 array pointer compression
    trie     7085 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
    === 3/5 Calculating and sorting initial probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480 4:22371672
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 4/5 Calculating and writing order-interpolated probabilities ===
    Chain sizes: 1:229872 2:3700272 3:12352480 4:22371672
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ####################################################################################################
    === 5/5 Writing ARPA model ===
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    Name:lmplz	VmPeak:10865048 kB	VmRSS:44524 kB	RSSMax:2182940 kB	user:2.32	sys:1.384	CPU:3.7062	real:3.30219
    Reading models/pchat_4.arpa
    ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
    ****************************************************************************************************
    SUCCESS

## Evaluation

### Perplexity

Intuitively, a good model should assign high probabilities to sequences from the 'true' distribution that it is modeling.

A common way of quantifying this is with **perplexity**, a metric inversely-proportional to the probability that the model assigns to a set of sequences, e.g. a 'test set':

\begin{align}
\large \text{ppl}(p, D) &\large\ = 2^{-\frac{1}{N_{total}}\log_2 p(D)}
\end{align}

where $D=\{(w_1,\ldots,w_{N_i})_i\}_{i=1}^M$ is a dataset of $M$ sequences with total length $N_{\text{total}}=\sum_{i}N_i$.

Perplexity is defined on $[1,\infty)$, with 1 being a perfect model (assigning probability 1 to $D$), and a 'worse' model as perplexity increases.

Intuitively, _perplexity measures the average rank of the true next-token, when tokens are ordered by the model's conditional probabilities_.

#### Evaluate Perplexity

`kenlm` outputs log probabilities in **log base 10**. For the standard definition of perplexity we need **log base 2**. See the code below:

```python
def perplexity_kenlm(model, sequences):
    n_total = 0
    logp_total = 0
    for sequence in sequences:
        text = ' '.join(sequence)
        logp_total += model.score(text)
        n_total += len(sequence) + 1  # add 1 for </s>

    # Convert log10 to log2
    log2p_total = logp_total / np.log10(2)
    ppl = 2 ** (- (1.0 / n_total) * log2p_total)
    return ppl
```

```python
print("=== Train ===")
df = pd.DataFrame([(n, perplexity_kenlm(models[n], train)) for n in [2, 3, 4]], columns=['n', 'ppl'])
df.style.hide_index()
```

    === Train ===

        n	        ppl

                                2
                        39.9248

                                3
                        15.462

                                4
                        8.91961

```python
print("=== Valid ===")
df = pd.DataFrame([(n, perplexity_kenlm(models[n], valid)) for n in [2, 3, 4]], columns=['n', 'ppl'])
df.style.hide_index()
```

    === Valid ===

        n	        ppl

                                2
                        59.188

                                3
                        44.9771

                                4
                        43.2188

#### Sequence probabilities:
\begin{align}
p(w_1,\ldots,w_T)&\approx\prod_{t=1}^T p(w_t|w_{t-2},w_{t-1})\\
&=\sum_{t=1}^T \log p(w_t|w_{t-2},w_{t-1}).
\end{align}

where we use log probabilities in practice to avoid a product of many small numbers.

```python
sentences = [
    'i like my pet dog .',
    'i like my pet zebra .',
    'i like my pet lion .',
    'i live in the united states .',
    'i live in the united states of america .'
]

for sentence in sentences:
    print(sentence)
    for n in [2, 3, 4]:
        log10p = models[n].score(sentence)
        log2p = log10p / np.log10(2)
        print("n: %d\t logp: %.3f" % (n, log2p))
    print()
```

    i like my pet dog .
    n: 2	 logp: -29.216
    n: 3	 logp: -28.843
    n: 4	 logp: -29.219

    i like my pet zebra .
    n: 2	 logp: -32.303
    n: 3	 logp: -31.157
    n: 4	 logp: -32.101

    i like my pet lion .
    n: 2	 logp: -38.352
    n: 3	 logp: -41.017
    n: 4	 logp: -42.320

    i live in the united states .
    n: 2	 logp: -22.599
    n: 3	 logp: -20.095
    n: 4	 logp: -17.854

    i live in the united states of america .
    n: 2	 logp: -40.958
    n: 3	 logp: -26.627
    n: 4	 logp: -24.311


# Further information

## Why do we use `@` for matrix multiplication and not `*`?

With `sympy` it is in fact possible to use the `*` operator for matrix
multiplication:

```python
import sympy as sym

matrix = sym.Matrix([[sym.S(1) / 5, 1], [1, 1]])
other_matrix = sym.Matrix([[sym.S(4) / 5, 0], [0, 0]])
matrix * other_matrix
```

$\displaystyle \left[\begin{matrix}\frac{4}{25} & 0\\\frac{4}{5} & 0\end{matrix}\right]$

However there are other libraries that can be used for linear algebra and in
those libraries the `*` does not do matrix multiplication, it does element wise
multiplication instead. So for clarity it is preferred to use `@` throughout.

## I have read that `numpy` is a library for linear algebra?

`numpy` is one of the most popular and important libraries in the Python
ecosystem. It is in fact the best library to use when doing linear algebra as it
is computationally efficient, **however** it cannot handle symbolic variables
which is why we are seeing how to use `Sympy` here. {ref}`numpy` gives an
introduction to `numpy`.


```python
%matplotlib inline
```

Hooks for autograd saved tensors
=======================

PyTorch typically computes gradients using backpropagation. However,
certain operations require intermediary results to be saved in order to
perform backpropagation. This tutorial walks through how these tensors
are saved/retrieved and how you can define hooks to control the
packing/unpacking process.

This tutorial assumes you are familiar with how backpropagation works in
theory. If not, read this first:
https://colab.research.google.com/drive/1aWNdmYt7RcHMbUk-Xz2Cv5-cGFSWPXe0#scrollTo=AHcEJ6nXUb7W

Saved tensors
-------------

Training a model usually consumes more memory than running it for
inference. Broadly speaking, one can say that it is because “PyTorch
needs to save the computation graph, which is needed to call
``backward``”, hence the additional memory usage. One goal of this
tutorial is to finetune this understanding.

In fact, the graph in itself sometimes does not consume much more memory
as it never copies any tensors. However, the graph can keep *references*
to tensors that would otherwise have gone out of scope: those are
referred to as **saved tensors**.

Why does training a model (typically) requires more memory than evaluating it?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We start with a simple example: :math: `y = a \mapsto \cdot b` , for which
we know the gradients of $y$ with respect to $a$ and
$b$:

\begin{align}\frac{\partial y}{\partial a} = b\end{align}

\begin{align}\frac{\partial y}{\partial b} = a\end{align}

```python
import torch

a = torch.randn(5, requires_grad=True)
b = torch.ones(5, requires_grad=True)
y = a * b
```

Using a torchviz, we can visualize the computation graph

 .. figure:: https://user-images.githubusercontent.com/8019486/130124513-72e016a3-c36f-42b9-88e2-53baf3e016c5.png
   :width: 300
   :align: center

In this example, PyTorch saves intermediary values $a$ and
$b$ in order to compute the gradient during the backward.

 .. figure:: https://user-images.githubusercontent.com/8019486/130124538-3da50977-6f0b-46d0-8909-5456ade9b598.png
   :width: 300
   :align: center

Those intermediary values (in orange above) can be accessed (for
debugging purposes) by looking for attributes of the ``grad_fn`` of
``y`` which start with the prefix ``_saved``:

```python
print(y.grad_fn._saved_self)
print(y.grad_fn._saved_other)
```

As the computation graph grows in depth, it will store more *saved
tensors*. Meanwhile, those tensors would have gone out of scope if not
for the graph.

```python
def f(x):
    return x * x

x = torch.randn(5, requires_grad=True)
y = f(f(f(x)))
```

.. figure:: https://user-images.githubusercontent.com/8019486/130124570-f1074098-1bb3-459e-bf5a-03bf6f65b403.png
  :width: 500
  :align: center

In the example above, executing without grad would only have kept ``x``
and ``y`` in the scope, But the graph additionnally stores ``f(x)`` and
``f(f(x)``. Hence, running a forward pass during training will be more
costly in memory usage than during evaluation (more precisely, when
autograd is not required).

The concept of packing / unpacking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Going back to the first example: ``y.grad_fn._saved_self`` and
``y.grad_fn._saved_other`` point to the original tensor object,
respectively ``a`` and ``b``.

```python
a = torch.randn(5, requires_grad=True)
b = torch.ones(5, requires_grad=True)
y = a * b

print(y.grad_fn._saved_self is a)   # True
print(y.grad_fn._saved_other is b)  # True
```

However, that may not always be the case.

```python
a = torch.randn(5, requires_grad=True)
y = torch.exp(a)
print(y.grad_fn._saved_result.equal(y))  # True
print(y.grad_fn._saved_result is y)      # False
```

Under the hood, PyTorch has **packed** and **unpacked** the tensor
``y`` to prevent reference cycles.

As a rule of thumb, you should *not* rely on the fact that accessing
the tensor saved for backward will yield the same tensor object as the
original tensor. They will however share the same *storage*.

Saved tensors hooks
-------------------

PyTorch provides an API to control how saved tensors should be packed /
unpacked.

```python
def pack_hook(x):
    print("Packing", x)
    return x

def unpack_hook(x):
    print("Unpacking", x)
    return x
a = torch.ones(5, requires_grad=True)
b = torch.ones(5, requires_grad=True) * 2

with torch.autograd.graph.saved_tensors_hooks(pack_hook, unpack_hook):
    y = a * b

y.sum().backward()
```

The ``pack_hook`` function will be called everytime an operation saves
a tensor for backward.
The output of ``pack_hook`` is then stored in the computation graph
instead of the original tensor.
The ``unpack_hook`` uses that return value to compute a new tensor,
which is the one actually used during the backward pass.
In general, you want ``unpack_hook(pack_hook(t))`` to be equal to
``t``.

```python
x = torch.randn(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(lambda x: x * 4, lambda x: x / 4):
    y = torch.pow(x, 2)
y.sum().backward()
assert(x.grad.equal(2 * x))
```

One thing to note is that the output of ``pack_hook`` can be *any Python
object*, as long as ``unpack_hook`` can derive a tensor with the correct
value from it.

Some unconventional examples
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First, some silly examples to illustrate what is possible but you
probably don’t ever want to do it.

**Returning and int**

```python
# Returning the index of a Python list
# Relatively harmless but with debatable usefulness

storage = []

def pack(x):
    storage.append(x)
    return len(storage) - 1

def unpack(x):
    return storage[x]

x = torch.randn(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(pack, unpack):
    y = x * x
y.sum().backward()

assert(x.grad.equal(2 * x))
```

**Returning a tuple**

```python
# Returning some tensor and a function how to unpack it
# Quite unlikely to be useful in its current form

def pack(x):
    delta = torch.randn(*x.size())
    return x - delta, lambda x: x + delta

def unpack(packed):
    x, f = packed
    return f(x)

x = torch.randn(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(pack, unpack):
    y = x * x
y.sum().backward()

assert(torch.allclose(x.grad, 2 * x))
```

**Returning a str**

```python
# Returning the __repr__ of the tensor
# Probably never do this

x = torch.randn(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(lambda x: repr(x), lambda x: eval("torch." + x)):
    y = x * x
y.sum().backward()
assert(torch.all(x.grad - 2 * x <= 1e-4))
```

Although those examples will not be useful in practice, they
illustrate that the output of ``pack_hook`` can really be any Python
object as long as it contains enough information to retrieve the
content of the original tensor.
In the next sections, we focus on more useful applications.

Saving tensors to CPU
~~~~~~~~~~~~~~~~~~~~~

Very often, the tensors involved in the computation graph live on GPU.
Keeping a reference to those tensors in the graph is what causes most
models to run out of GPU memory during training while they would have
done fine during evaluation.

Hooks provide a very simple way to implement that.

```python
def pack_hook(x):
    return (x.device, x.cpu())

def unpack_hook(packed):
    device, tensor = packed
    return tensor.to(device)

x = torch.randn(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(pack, unpack):
    y = x * x
y.sum().backward()

torch.allclose(x.grad, (2 * x))
```

In fact, PyTorch provides an API to conveniently use those hooks (as
well as the ability to use pinned memory).

```python
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.w = nn.Parameter(torch.randn(5))

    def forward(self, x):
        with torch.autograd.graph.save_on_cpu(pin_memory=True):
            # some computation
            return self.w * x

x = torch.randn(5)
model = Model()
loss = model(x).sum()
loss.backward()
```

In practice, on a A100 GPU, for a resnet-152 with batch size 256, this
corresponds to a GPU memory usage reduction from 48GB to 5GB, at the
cost of a 6x slowdown.

Of course, you can modulate the tradeoff by only saving to CPU certain
parts of the network.

For instance, you could define a special ``nn.Module`` that wraps any
module and saves its tensors to CPU.

```python
class SaveToCpu(nn.Module):
    def __init__(self, module):
        super().__init__()
        self.module = module

    def forward(self, *args, **kwargs):
        with torch.autograd.graph.save_on_cpu(pin_memory=True):
            return self.module(*args, **kwargs)

model = nn.Sequential(
    nn.Linear(10, 100),
    SaveToCpu(nn.Linear(100, 100)),
    nn.Linear(100, 10),
)

x = torch.randn(10)
loss = model(x).sum()
loss.backward()
```

Saving tensors to disk
~~~~~~~~~~~~~~~~~~~~~~

Similarly, you may want to save those tensors to disk. Again, this is
achievable with those hooks.

A naive version would look like this.

```python
# Naive version - HINT: Don't do this

import uuid
tmp_dir = "temp"

def pack_hook(tensor):
    name = os.path.join(tmp_dir, str(uuid.uuid4()))
    torch.save(tensor, name)
    return name

def unpack_hook(name):
    return torch.load(name)
```

The reason the above code is bad is that we are leaking files on the
disk and they are never cleared. Fixing this is not as trivial as it
seems.

```python
# Incorrect version - HINT: Don't do this

import uuid
import os
import tempfile
tmp_dir_obj = tempfile.TemporaryDirectory()
tmp_dir = tmp_dir_obj.name

def pack_hook(tensor):
    name = os.path.join(tmp_dir, str(uuid.uuid4()))
    torch.save(tensor, name)
    return name

def unpack_hook(name):
    tensor = torch.load(name)
    os.remove(name)
    return tensor
```

The reason the above code doesn’t work is that ``unpack_hook`` can be
called multiple times. If we delete the file during unpacking the first
time, it will not be available when the saved tensor is accessed a
second time, which will raise an error.

```python
x = torch.ones(5, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(pack_hook, unpack_hook):
    y = x.pow(2)
print(y.grad_fn._saved_self)
try:
    print(y.grad_fn._saved_self)
    print("Double access succeeded!")
except:
    print("Double access failed!")
```

To fix this, we can write a version of those hooks that takes advantage
of the fact that PyTorch automatically releases (deletes) the saved data
when it is no longer needed.

```python
class SelfDeletingTempFile():
    def __init__(self):
        self.name = os.path.join(tmp_dir, str(uuid.uuid4()))

    def __del__(self):
        os.remove(self.name)

def pack_hook(tensor):
    temp_file = SelfDeletingTempFile()
    torch.save(tensor, temp_file.name)
    return temp_file

def unpack_hook(temp_file):
    return torch.load(temp_file.name)
```

When we call ``backward``, the output of ``pack_hook`` will be deleted,
which causes the file to be removed, so we’re no longer leaking the
files.

This can then be used in your model, in the following way:

```python
# Only save on disk tensors that have size >= 1000
SAVE_ON_DISK_THRESHOLD = 1000

def pack_hook(x):
    if x.numel() < SAVE_ON_DISK_THRESHOLD:
        return x
    temp_file = SelfDeletingTempFile()
    torch.save(tensor, temp_file.name)
    return temp_file

def unpack_hook(tensor_or_sctf):
    if isinstance(tensor_or_sctf, torch.Tensor):
        return tensor_or_sctf
    return torch.load(tensor_or_sctf.name)

class SaveToDisk(nn.Module):
    def __init__(self, module):
        super().__init__()
        self.module = module

    def forward(self, *args, **kwargs):
        with torch.autograd.graph.saved_tensors_hooks(pack_hook, unpack_hook):
            return self.module(*args, **kwargs)

net = nn.DataParallel(SaveToDisk(Model()))
```

In this last example, we also demonstrate how to filter which tensors
should be saved (here, those whose number of elements is greater than
1000) and how to combine this feature with ``nn.DataParallel``.

If you’ve made it this far, congratulations! You now know how to use
saved tensor hooks and how they can be useful in a few scenarios to
tradeoff memory for compute.


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, sqrt, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

+ This chapter starts with explanations using sympy
+ A proper method using numpy and sympy is described in the example problem at the end

# Singular value decomposition (SVD)

## Derivation

* This is the final form of matrix factorization
* The factors are an orthogonal matrix A, a diagonal matrix Σ, and an orthogonal matrix V
$$ {A}={U}{\Sigma}{V}^{T} $$

+ In case the matrix A is symmetric positive definite, the decomposition is akin to the following
$$  {A}={Q}{\Lambda}{Q}^{T} $$

* Consider a vector *v*1 in ℝn row space, transformed into a vector *u*1 in ℝm column space by the matrix A
$$ {u}_{1}={A}{v}_{1} $$

* What we are looking for is an orthogonal basis in ℝn row space, transformed into an orthogonal basis in ℝm column space
$$ {u}_{1}={A}{v}_{1} \\ { v }_{ 1 }\bot { v }_{ 2 };{ u }_{ 1 }\bot { u }_{ 2 }$$

* It's easy to calculate an orthogonal basis in the row space using Gram-Schmidt
* Now, though, we need something special in A that would ensure that the basis *u*i in in ℝm column space is also orthogonal (and at the same time make it orthonormal, so that *v*i ends up as σi*u*i)
* The two nullspaces are not required
* So, we are looking for the following
$$ A\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ { v }_{ 1 } & { v }_{ 2 } & \cdots  & { v }_{ r } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}=\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ { u }_{ 1 } & u_{ 2 } & \cdots  & { u }_{ r } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}\begin{bmatrix} { \sigma  }_{ 1 } & \quad  & \quad  & \quad  & \quad  \\ \quad  & { \sigma  }_{ 2 } & \quad  & \quad  & \quad  \\ \quad  & \quad  & \quad \ddots  & \quad  & \quad  \\ \quad  & \quad  & \quad  & { \sigma  }_{ r }\quad  & \quad  \\ \quad  & \quad  & \quad  & \quad  & \left( 0 \right)  \end{bmatrix} \\ {A}{V}={U}{\Sigma}$$

+ In the case that we are not changing spaces V and U would be the same matrix Q (and then Q-1)

### Example problem explaining the derivation

+ Look at the next matrix A that is square and invertible (i.e. rank 2)

```python
A = Matrix([[4, 4], [-3, 3]])
A
```

$$\left[\begin{matrix}4 & 4\\-3 & 3\end{matrix}\right]$$

* We are looking for *v*1 and *v*2 in the ℝ2 rowspace and *u*1 and *u*2 in the ℝ2 columnspace, as well as the scaling factors *σ*1>0 and *σ*2>0

+ Just to be complete, we extend V until *v*n with zero columns and U with zero columns until *u*m, as well as zeros for Σ to include the nullspaces

+ Now A is not symmetric so that their eigenvectors are not orthogonal (Q), so we can't go that route

+ From above we have the following and because V is square and orthogonal we have
$$ {A}={U}{\Sigma}{V}^{-1} \\ {A}={U}{\Sigma}{V}^{T} $$

+ Multiplying both sides by AT we will have a left-hand side that is square and definte (semi)definte
$$ A=U\Sigma { V }^{ T }\\ { A }^{ T }A=V{ \Sigma  }^{ T }{ U }^{ T }U\Sigma { V }^{ T }\\ \because \quad { U }^{ T }U=I\\ \because \quad { \Sigma  }^{ T }\Sigma =\dots { \sigma  }_{ i }^{ 2 }\dots \\ { A }^{ T }A=V{ \Sigma  }^{ T }\Sigma { V }^{ T } $$

+ Because ATA is now definite (semi)positive, we have a perfect situation akin to being able to use QΛQT
+ The eigenvalues are the squares of the σi values
+ To get U we use AAT and use its eigenvalues and eigenvectors

* All of this is easy to accomplish with the mpmath submodule of sympy

```python
from sympy.mpmath import svd
```

```python
U, S, V = svd(A)
```

```python
U # The numbers round to zero!!! Please see it as zero
```

$$\left[\begin{matrix}1.0 & 3.33066907387547 \cdot 10^{-16}\\1.11022302462516 \cdot 10^{-16} & -1.0\end{matrix}\right]$$

```python
S # Not the final Sigma matrix
```

    matrix(
    [['5.65685424949238'],
     ['4.24264068711928']])

```python
V
```

    matrix(
    [['0.707106781186547', '0.707106781186548'],
     ['0.707106781186548', '-0.707106781186547']])

+ There are square roots, so the values are given instead of symbols

* Now let's do it step-by-step

```python
A.transpose() * A
```

$$\left[\begin{matrix}25 & 7\\7 & 25\end{matrix}\right]$$

```python
(A.transpose() * A).eigenvals()
```

$$\begin{Bmatrix}18 : 1, & 32 : 1\end{Bmatrix}$$

```python
(A.transpose() * A).eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}18, & 1, & \begin{bmatrix}\left[\begin{matrix}-1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}32, & 1, & \begin{bmatrix}\left[\begin{matrix}1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

* These are not normalized, though
* Also remember to take the square roots of the eigenvalues
* ... and to add zeros to incorporate the correct size for *m* and *n*
* ... and to take the transpose

* Now let's tackle U

```python
A * A.transpose()
```

$$\left[\begin{matrix}32 & 0\\0 & 18\end{matrix}\right]$$

```python
(A * A.transpose()).eigenvals()
```

$$\begin{Bmatrix}18 : 1, & 32 : 1\end{Bmatrix}$$

* The eigenvalues are always the same

```python
(A * A.transpose()).eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}18, & 1, & \begin{bmatrix}\left[\begin{matrix}0\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}32, & 1, & \begin{bmatrix}\left[\begin{matrix}1\\0\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

* Also remember to normalize (see example problem below)

* We now have U, Σ and VT (although Σ must still be constructed; see below)

### Example problem to explain the derivation for dependent rows, columns

+ Let's consider this rank=1, 2×2 singular matrix
+ The rowspace is just a line (the second row is a constant multiple of the first)
+ The nullspace of this row picture is a line perpendicular to this
+ The columnspace is also on a line, with the nullspace of AT being a line perpendicular to this

```python
A = Matrix([[4, 3], [8, 6]])
A
```

$$\left[\begin{matrix}4 & 3\\8 & 6\end{matrix}\right]$$

+ Let's use *svd*() first

```python
U, S, V = svd(A, full_matrices = True, compute_uv = True)
```

* This is likely to be different to the value you calculate for U
* We are talking unit basis vectors, though, which can be in a different direction depending on your choice

```python
U
```

$$\left[\begin{matrix}-0.447213595499958 & -0.894427190999916\\-0.894427190999916 & 0.447213595499958\end{matrix}\right]$$

```python
S
```

    matrix(
    [['11.1803398874989'],
     ['0.0']])

+ Note that the size of our Σ matrix is wrong
+ It has to be 2×2 and we have to create it from this info
+ Since A has rank = 1 and all off-diagonal entries must be zero, we will only have a value in the first row, first column position
+ Below I show you how to correct this

```python
V
```

    matrix(
    [['-0.8', '-0.6'],
     ['-0.6', '0.8']])

```python
A.transpose() * A # Which will be symmetric positive definite and of rank = 1
```

$$\left[\begin{matrix}80 & 60\\60 & 45\end{matrix}\right]$$

```python
(A.transpose() * A).eigenvals() # One eigenvalue will be zero and the other must then be the trace
```

$$\begin{Bmatrix}0 : 1, & 125 : 1\end{Bmatrix}$$

```python
(A.transpose() * A).eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}- \frac{3}{4}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}125, & 1, & \begin{bmatrix}\left[\begin{matrix}\frac{4}{3}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

```python
A * A.transpose()
```

$$\left[\begin{matrix}25 & 50\\50 & 100\end{matrix}\right]$$

```python
(A * A.transpose()).eigenvals()
```

$$\begin{Bmatrix}0 : 1, & 125 : 1\end{Bmatrix}$$

```python
(A * A.transpose()).eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}-2\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}125, & 1, & \begin{bmatrix}\left[\begin{matrix}\frac{1}{2}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

+ Inserted below is the three resultant matrices from our calculations above (normalized, etc)

```python
Matrix([[1 / sqrt(5), 2 / sqrt(5)], [2 / sqrt(5), 1 / sqrt(5)]]), Matrix([[sqrt(125), 0], [0, 0]]), Matrix([[0.8, 0.6], [0.6, -0.8]])
```

$$\begin{pmatrix}\left[\begin{matrix}\frac{\sqrt{5}}{5} & \frac{2 \sqrt{5}}{5}\\\frac{2 \sqrt{5}}{5} & \frac{\sqrt{5}}{5}\end{matrix}\right], & \left[\begin{matrix}5 \sqrt{5} & 0\\0 & 0\end{matrix}\right], & \left[\begin{matrix}0.8 & 0.6\\0.6 & -0.8\end{matrix}\right]\end{pmatrix}$$

+ Now let me show you how to correct the *svd*() solutions

```python
U
```

$$\left[\begin{matrix}-0.447213595499958 & -0.894427190999916\\-0.894427190999916 & 0.447213595499958\end{matrix}\right]$$

```python
S
```

    matrix(
    [['11.1803398874989'],
     ['0.0']])

```python
S = Matrix([[11.1803398874989, 0], [0, 0]])
S # Composed by hand (proper method further below)
```

$$\left[\begin{matrix}11.1803398874989 & 0\\0 & 0\end{matrix}\right]$$

```python
V
```

    matrix(
    [['-0.8', '-0.6'],
     ['-0.6', '0.8']])

```python
V = Matrix([[-0.8, -0.6], [-0.6, 0.8]])
V # Remember that this is actually V transpose
```

$$\left[\begin{matrix}-0.8 & -0.6\\-0.6 & 0.8\end{matrix}\right]$$

+ Let's calculate AΣVT

```python
U * S * V
```

$$\left[\begin{matrix}3.99999999999998 & 2.99999999999999\\7.99999999999996 & 5.99999999999997\end{matrix}\right]$$

+ Compensating for rounding, this is the original matrix A

## Summary

+ The orthonormal basis for the rowspace is
$$ {v}_{1},{v}_{2},\dots,{v}_{r} $$
+ The orthonormal basis for the columnspace is
$$ {u}_{1},{u}_{2},\dots,{u}_{r} $$
+ The orthonormal basis for the nullspace is
$$ {v}_{r+1},{v}_{r+2},\dots,{v}_{n} $$
+ The orthonormal basis for the nullspace of AT
$$ {u}_{r+1},{u}_{r+2},\dots,{u}_{m} $$

## Example problem

### Example problem 1

* Find the singular value decomposition of the matrix
$$ \begin{bmatrix}5&5\\-1&7\end{bmatrix} $$

#### Solution

+ First off, I'll show you how to make proper use of numpy and scipy (as opposed to sympy) to solve singular value decomposition problems

```python
from numpy import matrix, transpose # Importing the matrix object and the
# transpose object from numerical python (numpy)
from numpy.linalg import svd, det # Importing the svd and determinant
# methods from the linalg submodule
from scipy.linalg import diagsvd
```

```python
type(transpose) # Type tells us what 'something' is (sometimes)
```

    function

```python
C = matrix([[5, 5], [-1, 7]]) # Using the numpy matrix object
C
```

    matrix([[ 5,  5],
            [-1,  7]])

* We can see from the determinant that the rows and columns are independent

```python
det(C) # Notice the difference in syntax
```

$$40.0$$

+ Let's calculate U by looking at ATA

```python
transpose(C) *C # Notice the difference in synmtax
```

    matrix([[26, 18],
            [18, 74]])

+ This is symmetric, positive definite
+ One eigenvalue will be 0 and the other, the trace (since they (the eigenvalues) must sum to the trace)
+ Remember that the eigenvalues are the squares of the *σ*i values

+ Now let's put numpy and sympy to good use

```python
U, S, VT = svd(C) # I use the computer variable VT to remind us that
# this is the transpose of V
```

+ S will only indicate the eigenvalues and must be converted to the correct sized matrix

```python
M, N = C.shape # Shape returns a tuple (two values), indicating
# row and column size
M, N
```

$$\begin{pmatrix}2, & 2\end{pmatrix}$$

```python
Sig = diagsvd(S, M, N) # Creating a m times n matrix from S
Sig
```

    array([[ 8.94427191,  0.        ],
           [ 0.        ,  4.47213595]])

```python
VT
```

    matrix([[ 0.31622777,  0.9486833 ],
            [ 0.9486833 , -0.31622777]])

+ Let's check if it worked!

```python
U * Sig * VT
```

    matrix([[ 5.,  5.],
            [-1.,  7.]])

Now, let's use good old sympy

```python
C = Matrix([[5, 5], [-1, 7]])
C
```

$$\left[\begin{matrix}5 & 5\\-1 & 7\end{matrix}\right]$$

+ We need to work with a positive (semi)definite matrix

```python
CTC = C.transpose() * C # Using the computer variable CTC to remind that
# it is C transpose times C
CTC, CTC.det()
```

$$\begin{pmatrix}\left[\begin{matrix}26 & 18\\18 & 74\end{matrix}\right], & 1600\end{pmatrix}$$

+ Let's look at the eigenvalues

```python
CTC.eigenvals()
```

$$\begin{Bmatrix}20 : 1, & 80 : 1\end{Bmatrix}$$

+ Σ will contain along its main diagonal the square root of these eigenvalues

```python
Sig = Matrix([[sqrt(20), 0], [0, sqrt(80)]])
Sig
```

$$\left[\begin{matrix}2 \sqrt{5} & 0\\0 & 4 \sqrt{5}\end{matrix}\right]$$

+ For V we require the eigenvectors of CTC
+ We need to remember to normalize each vector (dividing each component by the length (norm) of that vector

```python
CTC.eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}20, & 1, & \begin{bmatrix}\left[\begin{matrix}-3\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}80, & 1, & \begin{bmatrix}\left[\begin{matrix}\frac{1}{3}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

+ Let's normalize each *v*i by calculating the length (norm) of each

```python
v1 = Matrix([-3, 1])
v1.norm()
```

$$\sqrt{10}$$

```python
v2 = Matrix([Rational(1, 3), 1])
v2.norm()
```

$$\frac{\sqrt{10}}{3}$$

+ We'll get each element of V by dividing by these norms

```python
-3 / v1.norm(), 1 / v1.norm()
```

$$\begin{pmatrix}- \frac{3 \sqrt{10}}{10}, & \frac{\sqrt{10}}{10}\end{pmatrix}$$

```python
Rational(1, 3) / v2.norm(), 1 / v2.norm()
```

$$\begin{pmatrix}\frac{\sqrt{10}}{10}, & \frac{3 \sqrt{10}}{10}\end{pmatrix}$$

```python
V = Matrix([[-3 / sqrt(10), 1 / sqrt(10)], [1 / sqrt(10), 3 / sqrt(10)]])
# Just remember to put the elements of V in the correct place
V
```

$$\left[\begin{matrix}- \frac{3 \sqrt{10}}{10} & \frac{\sqrt{10}}{10}\\\frac{\sqrt{10}}{10} & \frac{3 \sqrt{10}}{10}\end{matrix}\right]$$

+ Remember that it is equal to the transpose of V

```python
V == V.transpose()
```

    True

+ Now for U using CCT

```python
CCT = C * C.transpose() # Using the computer variable CCT
CCT
```

$$\left[\begin{matrix}50 & 30\\30 & 50\end{matrix}\right]$$

+ The eigenvalues will be the same

```python
CCT.eigenvals()
```

$$\begin{Bmatrix}20 : 1, & 80 : 1\end{Bmatrix}$$

```python
CCT.eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}20, & 1, & \begin{bmatrix}\left[\begin{matrix}-1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}80, & 1, & \begin{bmatrix}\left[\begin{matrix}1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

```python
u1 = Matrix([-1, 1])
u2 = Matrix([1, 1])
```

```python
-1 / u1.norm(), 1 / u1.norm()
```

$$\begin{pmatrix}- \frac{\sqrt{2}}{2}, & \frac{\sqrt{2}}{2}\end{pmatrix}$$

```python
1 / u2.norm(), 1 / u2.norm()
```

$$\begin{pmatrix}\frac{\sqrt{2}}{2}, & \frac{\sqrt{2}}{2}\end{pmatrix}$$

```python
U = Matrix([[-sqrt(2) / 2, sqrt(2) / 2], [sqrt(2) / 2, sqrt(2) / 2]])
# Just remember to put the elements of U in the correct place
U
```

$$\left[\begin{matrix}- \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\\\frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\end{matrix}\right]$$

+ Let's see if it worked!

```python
U * Sig * V
```

$$\left[\begin{matrix}5 & 5\\-1 & 7\end{matrix}\right]$$

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_29_Singular_value_decomposition.ipynb