+ This notebook is part of lecture 31 *Change of basis and image compression* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, sqrt, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Image compression and change of basis

## Lossy image compression

+ Consider a 29 × 29 monochrome image
+ Every pixel in this 512×512 image can take a value of 0 ≤ *x*i < 255 (this is 8-bit)
+ This make **x** a vector in ℝn, with *n* = 5122 (for color images this would be 3×n)

```python
# Just look at what 512 square is
512 ** 2
```

$$262144$$

+ This is a very large, unwieldy basis
+ Consider the standard basis
$$ \begin{bmatrix} 1 \\ 0 \\ 0 \\ \vdots  \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 1 \\ 0 \\ \vdots  \\ 0 \end{bmatrix},\cdots ,\begin{bmatrix} 0 \\ 0 \\ 0 \\ \vdots  \\ 1 \end{bmatrix} $$
+ Consider now the better basis
$$  \begin{bmatrix} 1 \\ 1 \\ 1 \\ \vdots  \\ 1 \end{bmatrix},\begin{bmatrix} 1 \\ \vdots  \\ 1 \\ -1 \\ \vdots  \\ -1 \end{bmatrix},\begin{bmatrix} 1 \\ -1 \\ 1 \\ -1 \\ \vdots  \end{bmatrix},\cdots  $$
+ Indeed, there are many options

+ JPEG uses an 8 × 8 Fourier basis
    + This means that an image is broken up into 64 × 64 pixel blocks
    + See the lectures on the Fourier basis
    $$ \begin{bmatrix} 1 \\ 1 \\ 1 \\ \vdots  \\ 1 \end{bmatrix},\begin{bmatrix} 1 \\ W \\ { W }^{ 2 } \\ \vdots  \\ { W }^{ n-1 } \end{bmatrix},\cdots  $$
    + This gives us a vector **x** in ℝ64 (i.e. with 64 coefficients)
    + Up until this point the compression is lossless
    + Now comes the compression (of which there are many such as thresholding)
    + Thresholding
        + Get rid of values more or less than set values (now there a less coefficients)
        $$ \hat{x}=\sum{\hat{c}_{i}{v}_{i}} $$

* Video is a sequence of images that are highly correlated (not big changes from one image to the next) and you can predict future changes from previous changes

+ There are newer basis such as *wavelets*
+ Here is an example
$$  \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{bmatrix},\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\ -1 \\ -1 \\ -1 \\ -1 \end{bmatrix},\begin{bmatrix} 1 \\ 1 \\ -1 \\ -1 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \\ 1 \\ 1 \\ 1 \\ 1 \end{bmatrix},\begin{bmatrix} 1 \\ -1 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 0 \\ 1 \\ -1 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \\ 1 \\ -1 \\ 0 \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 1 \\ -1 \end{bmatrix} $$
+ Every vector in ℝ8 is a linear combination of these 8 basis vectors

+ Let's do some linear algebra
+ Consider only a top row of 8 pixels
    + The standard vector of the values will be as follows (with 0 ≤ *p*i < 255)
    $$ \begin{bmatrix} { p }_{ 1 } \\ { p }_{ 2 } \\ { p }_{ 3 } \\ { p }_{ 4 } \\ { p }_{ 5 } \\ { p }_{ 6 } \\ { p }_{ 7 } \\ { p }_{ 8 } \end{bmatrix} $$
+ We have to write this as a linear combination of the wavelet basis vectors *w*i (the lossless step)
$$ {P}={c}_{1}{w}_{1}+{c}_{2}{w}_{2}+\dots+{c}_{8}{w}_{8} $$
+ In vector form we have the following
$$ P=\begin{bmatrix} \vdots  & \cdots  & \vdots  \\ { w }_{ 1 } & \cdots  & { w }_{ 8 } \\ \vdots  & \cdots  & \vdots  \end{bmatrix}\begin{bmatrix} { c }_{ 1 } \\ \vdots  \\ { c }_{ 8 } \end{bmatrix} \\ P=Wc \\ c={W}^{-1}{P}$$

+ Let's bring some reality to this
    + For fast computation, W must be as easy to invert as possible
    + There is great competition to come up with *better* compression matrices
    + A *good* matrix must have the following
        + Be fast, i.e. the fast Fourier transform (FFT)
            + The wavelet basis above is fast
                + The basis vectors are orthogonal (and can be made orthonormal)
                + **If they are orthonormal then the inverse is equal to the transpose**
        + Good compression
            + If we threw away some of the *p*i values, we would just have a dark image
            + We we threw away, say the last two *c*i values (last two basis vectors) that won't lose us so much quality

## Change of basis

+ Let's look at this change in basis
+ Above, we had the following
$$ x=Wc $$
+ Here W is the matrix that takes us from the vector **x** in the old basis to the vector **c** in the new basis

+ Consider any transformation T (such as a rotation transformation)
    + With respect to *v*1,...,*v*8 it has a matrix A
    + With respect to *w*1,...,*w*8 it has a matrix B

+ Turns out that matrices A and B are similar
$$ B={M}^{-1}AM $$
+ Here M is the matrix that transforms the basis

+ What is A then, using the basis *v*1,...,*v*8?
+ We know T completely from T(*v*i)...
    + ... because if every **x**=Σ*c*i*v*i
    + ... then T(**x**)=Σ*c*iT(*v*i)

+ Constructing A
    + Write down all the transformations
    $$ T\left( { v }_{ 1 } \right) ={ a }_{ 11 }{ v }_{ 1 }+{ a }_{ 21 }{ v }_{ 2 }+\dots +{ a }_{ 81 }{ v }_{ 8 }\\ T\left( { v }_{ 2 } \right) ={ a }_{ 12 }{ v }_{ 1 }+{ a }_{ 22 }{ v }_{ 2 }+\dots +{ a }_{ 82 }{ v }_{ 8 }\\ \vdots \\ T\left( { v }_{ 8 } \right) ={ a }_{ 18 }{ v }_{ 1 }+{ a }_{ 28 }{ v }_{ 2 }+\dots +{ a }_{ 88 }{ v }_{ 8 } $$
    + Now we know A
    $$ A=\begin{bmatrix} { a }_{ 11 } & \cdots  & { a }_{ 18 } \\ \vdots  & \cdots  & \vdots  \\ { a }_{ 81 } & \cdots  & { a }_{ 88 } \end{bmatrix} $$

+ Let's consider the linear transformation T(*v*i)=λi
+ This makes A the following
$$ A=\begin{bmatrix} { \lambda  }_{ 1 } & 0 & \cdots  & \cdots  & 0 \\ 0 & { \lambda  }_{ 2 } & 0 & \cdots  & \vdots  \\ \vdots  & 0 & \ddots  & \cdots  & \vdots  \\ \vdots  & \vdots  & \vdots  & \ddots  & 0 \\ 0 & \cdots  & \cdots  & 0 & { \lambda  }_{ 8 } \end{bmatrix} $$

## Example problems

### Example problem 1

+ The vector space of all polynomials in *x* (of degree ≤ 2) has the basis 1, *x*, *x*2
+ Consider a different basis *w*1, *w*2, *w*3 whose values at *x* = -1, 0, and 1 are given by the following
$$ x=-1\rightarrow 1{ w }_{ 1 }+{ 0w }_{ 2 }+{ 0w }_{ 3 }\\ x=0\rightarrow 0{ w }_{ 1 }+1{ w }_{ 2 }+{ 0w }_{ 3 }\\ x=1\rightarrow 0{ w }_{ 1 }+{ 0w }_{ 2 }+{ 1w }_{ 3 } $$
+ Express *y*(*x*)=-*x*+5 in the new basis
+ Find the change of basis matrices
+ Find the matrix of taking derivatives in both of the basis

#### Solution

$$ y\left( x \right) =5-x\\ y\left( x \right) =\alpha { w }_{ 1 }+\beta { w }_{ 2 }+\gamma { w }_{ 3 } \\ y\left( -1 \right) =6 \\ y\left( 0 \right) =5\\ y\left( 1 \right) =4\\ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} \alpha  \\ \beta  \\ \gamma  \end{bmatrix}=\begin{bmatrix} 6 \\ 5 \\ 4 \end{bmatrix} \\ \alpha =6,\beta =5,\gamma =4 \\ y=6{w}_{1}+5{w}_{2}+4{w}_{3} $$

+ For the second part let's look at what happens at *x* for the various values at 1 (which is *x*0), *x*, and *x*2
    + For -1 we have 1, -1, and 1
    + For 0 we have 1, 0, and 0
    + For 1 we have 1, 1, and 1
+ From this we can conclude the following
$$ 1={w}_{1}+{w}_{2}+{w}_{3} \\ x=-{w}_{1}+{w}_{3} \\ {x}^{2}={w}_{1}+{w}_{3} $$
+ Now we have the following matrix
$$ A=\begin{bmatrix}1&-1&1\\1&0&0\\1&1&1\end{bmatrix} $$
+ This converts the first basis to the second
+ To convert the second basis to the original we just need A-1

```python
A = Matrix([[1, -1, 1], [1, 0, 0], [1, 1, 1]])
A.inv()
```

$$\left[\begin{matrix}0 & 1 & 0\\- \frac{1}{2} & 0 & \frac{1}{2}\\\frac{1}{2} & -1 & \frac{1}{2}\end{matrix}\right]$$

+ Now for derivative matrices
    + For the original basis, this is easy
    $$ {D}_{x}=\begin{bmatrix}0&1&0\\0&0&2\\0&0&0\end{bmatrix} $$
    + For the second basis we need the following
    $$ {D}_{w}=AD{A}^{-1} $$

```python
Dx = Matrix([[0, 1, 0], [0, 0, 2], [0, 0, 0]])
Dw = A * Dx * A.inv()
Dw
```

$$\left[\begin{matrix}- \frac{3}{2} & 2 & - \frac{1}{2}\\- \frac{1}{2} & 0 & \frac{1}{2}\\\frac{1}{2} & -2 & \frac{3}{2}\end{matrix}\right]$$

+ Just to conclude we can write the values for *w*i from the inverse of A (the columns)
$$ {w}_{1}=\frac{-1}{2}{x}+\frac{1}{2}{x}^{2} \\ {w}_{2}=1-{x}^{2} \\ {w}_{3}=\frac{1}{2}x+\frac{1}{2}{x}^{2} $$

```python

```


#  CS109B Data Science 2: Advanced Topics in Data Science
## Lecture 3 - Coding Environment Setup and review of `statsmodels`
###  Notebook B

**Harvard University**

**Spring 2021**

**Instructors:** Mark Glickman, Pavlos Protopapas, and Chris Tanner

**Additional Instructor:** Eleni Kaxiras

*Content:* Eleni Kaxiras and Will Claybaugh

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import numpy as np
from scipy.interpolate import interp1d
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline
```

## Learning Goals

By the end of this lab, you should be able to:
* use `np.linalg.vander`
* use the weird R-style formulas in `statsmodels`
* practice least-squares regression in `statsmodels`

### Basis Functions

In our models we can use various types of functions as basis functions. Strictly speaking, in linear algebra where a basis for a subspace S of $\mathbb{R}^n$ is a set of vectors that spans S and is linearly independent. As a reminder, a set of vectors $\textbf{v}_1, \textbf{v}_2, ..., \textbf{v}_k$ are considered linearly independent if they cannot be written as a linear combination of each other, such that, if: $c_1\textbf{v}_1+c_2\textbf{v}_2+ ...+ c_k\textbf{v}_k = \textbf{0}$ then $c_1,c_2,...,c_k$ are all zero.

In data science where we have lots of imperfect data (with errors), as well as imperfect computers (with round-off errors), when we substitute their values into the matrices we almost always get column degeneracy, meaning, some of our columns become linear combinations of each other. Especially so if we use the monomial basis and go beyond ~5,6 degree of the polynomial.

Examples are:

- Monomials such as $x,x^2,x^4,x^5$ - don't usually go higher than this due to correlation
- Sigmoid/ReLU functions (neural networks)
- Fourier functions
- Wavelets
- Splines

The matrix produced when we substitute the values of our data into the basis functions is called the *design matrix*.

### Linear/Polynomial Regression

We will use the `diabetes` dataset.

Variables are:
- subject:   subject ID number
- age:       age diagnosed with diabetes
- acidity:   a measure of acidity called base deficit
Response:
- y:         natural log of serum C-peptide concentration

*Original source is Sockett et al. (1987) mentioned in Hastie and Tibshirani's book
"Generalized Additive Models".*

Reading the data in Pandas:

```python
diab = pd.read_csv("data/diabetes.csv")
diab.head()
```

      subject
      age
      acidity
      y

      0
      1
      5.2
      -8.1
      4.8

      1
      2
      8.8
      -16.1
      4.1

      2
      3
      10.5
      -0.9
      5.2

      3
      4
      10.6
      -7.8
      5.5

      4
      5
      10.4
      -29.0
      5.0

### Create the design matrix for a fictitious dataset
Let's keep just the `age` feature and create some columns of our own. Let's see how good this matrix is before we create the design matrix.

```python
diab_age = diab[['age']].copy()
diab_age['age2'] = diab_age.apply(lambda row: row['age']**2, axis=1)
diab_age['random'] = np.random.normal(0,1,len(diab_age))
diab_age['same'] = diab_age['age']
diab_age.head()
```

      age
      age2
      random
      same

      0
      5.2
      27.04
      -0.260697
      5.2

      1
      8.8
      77.44
      -0.979202
      8.8

      2
      10.5
      110.25
      0.898251
      10.5

      3
      10.6
      112.36
      -0.005460
      10.6

      4
      10.4
      108.16
      1.300648
      10.4

```python
A = diab_age.to_numpy(copy=True)
A[:5]
```

    array([[ 5.20000000e+00,  2.70400000e+01, -2.60696714e-01,
             5.20000000e+00],
           [ 8.80000000e+00,  7.74400000e+01, -9.79201665e-01,
             8.80000000e+00],
           [ 1.05000000e+01,  1.10250000e+02,  8.98250869e-01,
             1.05000000e+01],
           [ 1.06000000e+01,  1.12360000e+02, -5.46032145e-03,
             1.06000000e+01],
           [ 1.04000000e+01,  1.08160000e+02,  1.30064782e+00,
             1.04000000e+01]])

Let's check if the columns of A are linearly independent by using some linear algebra methods from `numpy.linalg` and `sympy`.

```python
from numpy.linalg import matrix_rank
matrix_rank(A)
```

    3

```python
# check out which rows are linearly independent
import sympy
_, inds = sympy.Matrix(A).T.rref()
inds
```

    (0, 1, 2)

_condition number_: how well the system of equations is solved with the matrix. Want a lower value.

```python
np.linalg.cond(A)
```

    7.81775159289351e+16

### Create the design matrix for `age` using a polynomial basis
Let's keep just the `age` feature again and create the design matrix using a polynomial of degree `n`. First we will use the basic `numpy` formula `vander()`.

```python
vand = np.vander(diab_age.age, 2, increasing=True)
vand[:3], vand.shape
```

    (array([[ 1. ,  5.2],
            [ 1. ,  8.8],
            [ 1. , 10.5]]),
     (43, 2))

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
## find the condition number of this matrix. usually bad if you go beyond degree 5-6
np.linalg.cond(vand), matrix_rank(vand)
```

    (24.709088478926237, 2)

Exercise: Vandermonde matrix

Change the degree of the polynomial and comment on what happens to the condition and rank of the matrix.

```python
vand = np.vander(diab_age.age, 8, increasing=True)
vand[:3], vand.shape
```

    (array([[1.00000000e+00, 5.20000000e+00, 2.70400000e+01, 1.40608000e+02,
             7.31161600e+02, 3.80204032e+03, 1.97706097e+04, 1.02807170e+05],
            [1.00000000e+00, 8.80000000e+00, 7.74400000e+01, 6.81472000e+02,
             5.99695360e+03, 5.27731917e+04, 4.64404087e+05, 4.08675596e+06],
            [1.00000000e+00, 1.05000000e+01, 1.10250000e+02, 1.15762500e+03,
             1.21550625e+04, 1.27628156e+05, 1.34009564e+06, 1.40710042e+07]]),
     (43, 8))

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
## find the condition number of this matrix
np.linalg.cond(vand), matrix_rank(vand)
```

    (4207821284.7228966, 8)

### Linear/Polynomial regression with statsmodels.

As you remember from 109a, we have two tools for Linear Regression:
- `statsmodels` [https://www.statsmodels.org/stable/regression.html](https://www.statsmodels.org/stable/regression.html), and
- `sklearn`[https://scikit-learn.org/stable/index.html](https://scikit-learn.org/stable/index.html)

Previously, in this notebook, we worked from a vector of target values and a design matrix we built ourself. In 109a we used e.g. `sklearn`'s PolynomialFeatures to build the matrix. Now we will look at `statsmodels` which allows users to fit statistical models using R-style **formulas**. They build the target value and design matrix for you.

**Note:** Categorical features (e.g. let's say we had a categorical feature called Region, are designated by `C(Region)`), polynomial features (e.g. age) are entered as `np.power(age, n)` where `n` is the degree of the polynomial **OR** `np.vander(age, n, increasing=True)`.

```
# Example: if our target variable is 'Lottery', while 'Region' is a categorical predictor and all the others are numerical:
df = dta.data[['Lottery', 'Literacy', 'Wealth', 'Region']]

formula='Lottery ~ Literacy + Wealth + C(Region) + Literacy * Wealth'
```

For more on these formulas see:

- https://www.statsmodels.org/stable/examples/notebooks/generated/formulas.html
- https://patsy.readthedocs.io/en/latest/overview.html

```python
import statsmodels.formula.api as smf

model1 = smf.ols('y ~ age', data=diab)
fit1_lm = model1.fit()
```

Let's build a dataframe to predict values on (sometimes this is just the test or validation set). Very useful for making pretty plots of the model predictions - predict for TONS of values, not just whatever's in the training set.

```python
x_pred = np.linspace(0.5,20,100)

predict_df = pd.DataFrame(data={"age":x_pred})
predict_df.head()
```

      age

      0
      0.500000

      1
      0.696970

      2
      0.893939

      3
      1.090909

      4
      1.287879

Use `get_prediction(<data>).summary_frame()` to get the model's prediction (and error bars!)

```python
prediction_output = fit1_lm.get_prediction(predict_df).summary_frame()
prediction_output.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      4.037574
      0.233301
      3.566412
      4.508736
      2.650278
      5.424870

      1
      4.053940
      0.228886
      3.591696
      4.516184
      2.669647
      5.438232

      2
      4.070305
      0.224489
      3.616940
      4.523670
      2.688952
      5.451658

      3
      4.086671
      0.220113
      3.642143
      4.531198
      2.708193
      5.465149

      4
      4.103036
      0.215759
      3.667302
      4.538770
      2.727369
      5.478703

Plot the data, the fitted model, the **confidence intervals**, and the *prediction intervals*. For more on how `statsmodels` calculates these intervals see: https://www.statsmodels.org/stable/_modules/statsmodels/regression/_prediction.html

```python
ax1 = diab.plot.scatter(x='age',y='y',c='brown',title="Diabetes data with least-squares linear fit")
ax1.set_xlabel("Age at Diagnosis")
ax1.set_ylabel("Log C-Peptide Concentration")

ax1.plot(predict_df.age, prediction_output['mean'],color="green")
ax1.plot(predict_df.age, prediction_output['mean_ci_lower'], color="blue",linestyle="dashed")
ax1.plot(predict_df.age, prediction_output['mean_ci_upper'], color="blue",linestyle="dashed")
ax1.plot(predict_df.age, prediction_output['obs_ci_lower'], color="green",linestyle="dashdot")
ax1.plot(predict_df.age, prediction_output['obs_ci_upper'], color="green",linestyle="dashdot");
```

Breakout Room Exercise

- Fit a 3rd degree polynomial model to predict `y` using only `age` and
- Plot the model and its confidence intervals.
- Change the degree of your polynomial and see what happens to the fitted curve.
- Does our model have an intercept? *Note*: we can discover the existence or not of an intercept in our model by running:
```
model_name.params
```

```python
# your answer here
fit2_lm = smf.ols(formula="y ~ age + np.power(age, 2) + np.power(age, 3)", data=diab).fit()

poly_predictions = fit2_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.057129
      0.405478
      2.236972
      3.877285
      1.599098
      4.515159

      1
      3.173396
      0.370403
      2.424185
      3.922608
      1.754061
      4.592732

      2
      3.285013
      0.338413
      2.600508
      3.969517
      1.898743
      4.671282

      3
      3.392068
      0.309541
      2.765962
      4.018174
      2.033685
      4.750451

      4
      3.494653
      0.283823
      2.920567
      4.068739
      2.159449
      4.829857

```python
# solution
fit2_lm = smf.ols(formula="y ~ age + np.power(age, 2) + np.power(age, 3)", data=diab).fit()

poly_predictions = fit2_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.057129
      0.405478
      2.236972
      3.877285
      1.599098
      4.515159

      1
      3.173396
      0.370403
      2.424185
      3.922608
      1.754061
      4.592732

      2
      3.285013
      0.338413
      2.600508
      3.969517
      1.898743
      4.671282

      3
      3.392068
      0.309541
      2.765962
      4.018174
      2.033685
      4.750451

      4
      3.494653
      0.283823
      2.920567
      4.068739
      2.159449
      4.829857

Vandermonde matrix in formulas

It's easier to build higher order polynomials using `np.vandrer()`.

```python
formula = "y ~ np.vander(age, 6, increasing=True) -1" # -1: don't give the intercept
# instead, the intercept is the 0th index of the following
fit3_lm = smf.ols(formula=formula, data=diab).fit()
```

```python
fit3_lm.params
```

    np.vander(age, 6, increasing=True)[0]    3.147753
    np.vander(age, 6, increasing=True)[1]    0.130248
    np.vander(age, 6, increasing=True)[2]    0.136034
    np.vander(age, 6, increasing=True)[3]   -0.028749
    np.vander(age, 6, increasing=True)[4]    0.002064
    np.vander(age, 6, increasing=True)[5]   -0.000050
    dtype: float64

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
# find the condition number of this matrix
np.linalg.cond(np.vander(predict_df.age, 6, increasing=True))
```

    3703184.6695084195

```python
# solution
poly_predictions = fit3_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.243420
      0.734767
      1.754641
      4.732198
      1.308481
      5.178358

      1
      3.281393
      0.607441
      2.050600
      4.512185
      1.537152
      5.025633

      2
      3.323028
      0.500327
      2.309268
      4.336788
      1.724518
      4.921538

      3
      3.367891
      0.413960
      2.529128
      4.206654
      1.874221
      4.861560

      4
      3.415567
      0.349163
      2.708094
      4.123039
      1.991473
      4.839660

```python
# solution
x_pred = np.linspace(0.5,15,100)
predict_df = pd.DataFrame(data={"age":x_pred})

ax2 = diab.plot.scatter(x='age',y='y',c='Red',title="Diabetes data with least-squares cubic fit")
ax2.set_xlabel("Age at Diagnosis")
ax2.set_ylabel("Log C-Peptide Concentration")

ax2.plot(predict_df.age, poly_predictions['mean'],color="green")
ax2.plot(predict_df.age, poly_predictions['mean_ci_lower'], color="blue",linestyle="dashed", label='confidence interval')
ax2.plot(predict_df.age, poly_predictions['mean_ci_upper'], color="blue",linestyle="dashed");
ax2.legend();
```

Discussion

#### QR decomposition (**Beyond the scope of this class**)

As you know, to find the parameters of our model, we may try to solve the so-called *normal equations*, which, written in matrix form, are:

\begin{equation}
    (\boldsymbol{A^T}\cdot \boldsymbol{A}) \cdot \boldsymbol{b} = \boldsymbol{A} \cdot \boldsymbol{y}
\end{equation}

The direct solution is $\hat{\boldsymbol{b}}=(\boldsymbol{A}^T\cdot \boldsymbol{A})^{-1}\cdot \boldsymbol{A}^T \cdot \boldsymbol{y}$

Solving the least-squares problem directly via the normal equations is susceptible to roundoff error when the condition of the matrix $\boldsymbol{A}$ is large. An alternative technique involves QR decomposition (details in any good linear algebra book). `statsmodels` lets you use this technique via a parameter in the `.fit`:
```
   .fit(method='qr')
```

#### Let's try with QR now

```python
formula = "y ~ np.vander(age, 6, increasing=True) -1"
fit3_lm = smf.ols(formula=formula, data=diab).fit(method='qr')
```

```python
fit3_lm.params
```

    np.vander(age, 6, increasing=True)[0]    3.147753
    np.vander(age, 6, increasing=True)[1]    0.130248
    np.vander(age, 6, increasing=True)[2]    0.136034
    np.vander(age, 6, increasing=True)[3]   -0.028749
    np.vander(age, 6, increasing=True)[4]    0.002064
    np.vander(age, 6, increasing=True)[5]   -0.000050
    dtype: float64

```python
# solution
poly_predictions = fit3_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.243420
      0.734767
      1.754641
      4.732198
      1.308481
      5.178358

      1
      3.281393
      0.607441
      2.050600
      4.512185
      1.537152
      5.025633

      2
      3.323028
      0.500327
      2.309268
      4.336788
      1.724518
      4.921538

      3
      3.367891
      0.413960
      2.529128
      4.206654
      1.874221
      4.861560

      4
      3.415567
      0.349163
      2.708094
      4.123039
      1.991473
      4.839660

```python
# solution
ax2 = diab.plot.scatter(x='age',y='y',c='Red',title="Diabetes data with least-squares cubic fit")
ax2.set_xlabel("Age at Diagnosis")
ax2.set_ylabel("Log C-Peptide Concentration")

ax2.plot(predict_df.age, poly_predictions['mean'],color="green")
ax2.plot(predict_df.age, poly_predictions['mean_ci_lower'], color="blue",linestyle="dashed", label='confidence interval')
ax2.plot(predict_df.age, poly_predictions['mean_ci_upper'], color="blue",linestyle="dashed");
ax2.legend();
```


# Surfinpy

#### Tutorial 3 - Pressure

In the previous tutorials we went through the process of generating a simple phase diagram for bulk phases and introducing temperature dependence for gaseous species.  This useful however, sometimes it can be more beneficial to convert the chemical potenials (eVs) to partial presure (bar).

Chemical potential can be converted to pressure values using

\begin{align}
P & = \frac{\mu_O}{k_B T} ,
\end{align}

where P is the pressure, $\mu$ is the chemical potential of oxygen, $k_B$ is the Boltzmnann constant and T is the temperature.

```python
import matplotlib.pyplot as plt

from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data

colors = ['#5B9BD5', '#4472C4', '#A5A5A5', '#772C24', '#ED7D31', '#FFC000', '#70AD47']
```

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

Additionally, SurfinPy has the functionality to allow you to choose which colours are used for each phase.  Specify within the DataSet class color.

```python
MgO = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, color=colors[0], label = "Periclase")
Mag = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, color=colors[1], label = "Magnesite")
Bru = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, color=colors[2], label = "Brucite")
Hyd = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, color=colors[3], label = "Hydromagnesite")
Art = data.DataSet(cation = 10, x = 5, y = 20, energy = -470.0, color=colors[4], label = "Artinite")
Nes = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, color=colors[5], label = "Nesquehonite")
Lan = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, color=colors[6], label = "Lansfordite")

```

```python
data = [MgO, Art, Bru, Nes,  Mag, Lan, Hyd]
```

```python
x_energy=-20.53412969
y_energy=-12.83725889
```

```python
CO2_exp = ut.fit_nist("CO2.txt")[298]
Water_exp = ut.fit_nist("H2O.txt")[298]

CO2_corrected = x_energy + CO2_exp
Water_corrected = y_energy + Water_exp

deltaX = {'Range': [ -1, 0.6],  'Label': 'CO_2'}
deltaY = {'Range': [ -1, 0.6], 'Label': 'H_2O'}
```

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy=CO2_corrected, y_energy=Water_corrected)

ax = system.plot_phase(figsize=(6, 4))

plt.show()
```

To convert chemical potential to pressure use the plot_pressure command and the temperature at which the pressure is calculated.  For this example we have used 298 K.

```python
import surfinpy
ax = system.plot_pressure(temperature=298, figsize=(5, 4), cbar_title="MgO System")
plt.show()
```

```python
fig = plt.figure()
ax1 = plt.subplot()
fig.set_size_inches(3.5, 2.75)

CM1 = ax1.contourf(system.x, system.y, system.z, levels=system.levels, cmap=system.cmap)
ax1.contour(system.x, system.y, system.z, levels=system.levels, colors="black", linewidths=2)

ax1.set_ylabel("$\Delta \mu_{\mathrm{" + "H_2O" + "}}$" + " (eV)", fontsize=9)
ax1.set_xlabel("$\Delta \mu_{\mathrm{" + "CO_2" + "}}$" + " (eV)", fontsize=9)

cbar = fig.colorbar(CM1, ax=ax1, ticks=system.ticky)
cbar.ax.set_yticklabels(system.labels, fontsize=9)
cbar.ax.set_title("MgO System", y=0.9, x=5.5, fontsize=9)
ax1.tick_params(labelsize=9)

plt.tight_layout()
plt.savefig("../../../paper/surfinpy.pdf", format="pdf")
plt.show()
```

```python

```


```python
from autoreduce import *
import numpy as np
from sympy import symbols
```

```python
n = 8 # Number of states
x_init = np.zeros(n)
x_init[6] = 100
x_init[7] = 500
timepoints_ode = np.linspace(0, 40, 100)
error_tol = 1000
nstates_tol = 5
#      x = 0, T1, 1, A1, 2, S1, 3, S2, 4, T2, 5, A2, 6, C1, 7, C2
#      P = 0, beta_S1, 1, l_S1, 2, K_S1, 3, kb, 4, beta_S2, 5, l_S2, 6,
#      K_S2, 7, beta_lac, 8, l_lac, 9, K_lac, 10, beta_tet, 11, l_tet, 12,
#      K_tet, 13, kc, 14, C_max, 15, dc, 16, 17, I, 18, atc, 20,K_tox
P = np.zeros(22)
P[0] = 6
P[1] = 2e-3
P[2] = 430
P[3] = 30
P[4] = 6
P[5] = 2e-3
P[6] = 190
P[7] = 19.8e-3
P[8] = 1.5e-3
P[9] = 1.4e5
P[10] = 14.4e-3
P[11] = 2.1e-4
P[12] = 13
P[13] = 0.6
P[14] = 5500
P[15] = 0.8
P[16] = 1e6 #17 -> 16
P[17] = 324 # 19 -> 17
P[18] = 1 #20 -> 18
P[19] = 0.1 #21 -> 19
P[20] = 1.5 #22 -> 20
P[21] = 0.5 #23 ->21
params_values = P.copy()

params = P
n = 8
x, f, P = system.load_ODE_model(n, len(params_values))

beta_S1 = symbols('beta_S1')
l_S1 = symbols('l_S1')
K_S1 = symbols('K_S1')
kb = symbols('kb')
beta_S2 = symbols('beta_S2')
l_S2 = symbols('l_S2')
K_S2 = symbols('K_S2')
beta_lac = symbols('beta_lac')
l_lac = symbols('l_lac')
K_lac = symbols('K_lac')
beta_tet = symbols('beta_tet')
l_tet = symbols('l_tet')
K_tet = symbols('K_tet')
kc = symbols('kc')
C_max = symbols('C_max')
dc = symbols('dc')
I = symbols('I')
atc = symbols('atc')
K_tox = symbols('K_tox')
d = symbols('d')
d_T = symbols('d_T')
d_S = symbols('d_S')
P = [beta_S1, l_S1, K_S1, kb, beta_S2, l_S2, K_S2, beta_lac, l_lac,
    K_lac, beta_tet, l_tet, K_tet, kc, C_max, dc, I,
    atc, K_tox, d, d_T, d_S]
y0 = symbols('y0')
y1 = symbols('y1')

# T1 and A1
f[0] = P[0]*(P[1] + x[2]**2/(P[2]+x[2]**2)) - P[3]*x[0]*x[1] - P[20] * x[0]
f[1] = 5*P[4]*(P[5] + x[3]**2/(P[6]+x[3]**2)) - P[20] * x[1] - P[3]*x[0]*x[1]

# f[0] = P[0]*(x[2]**2/(P[2]+x[2]**2)) - P[3]*x[0]*x[1]
# f[1] = P[4]*(x[3]**2/(P[6]+x[3]**2)) - P[3]*x[0]*x[1]

#  S1 and S2 (scaled with cell count)
f[2] = P[7]*(P[8] + P[16]**2/(P[9]+P[16]**2))*x[6] - P[21] * x[2]
f[3] = P[10]*(P[11] + P[17]**2/(P[12]+P[17]**2))*x[7] - P[21] * x[3]

# f[2] = P[7]*(P[16]**2/(P[9]+P[16]**2))*x[6] - P[21] * x[2]
# f[3] = P[10]*(P[17]**2/(P[12]+P[17]**2))*x[7] - P[21] * x[3]

#  T2 and A2
f[4] = P[4]*(P[5] + x[3]**2/(P[6]+x[3]**2)) - P[3]*x[4]*x[5] - P[20] * x[4]
f[5] = 5*P[0]*(P[1] + x[2]**2/(P[2]+x[2]**2)) - P[20] * x[5]-P[3]*x[4]*x[5]

# f[4] = P[4]*(x[3]**2/(P[6]+x[3]**2)) - P[3]*x[4]*x[5] - P[20] * x[4]
# f[5] = P[0]*(x[2]**2/(P[2]+x[2]**2)) - P[20] * x[5]-P[3]*x[4]*x[5]

#  Cell 1 and Cell 2
f[6] = P[13]*(1 - (x[6] + x[7])/P[14])*x[6] - P[15]*x[6]*(x[0]/(P[18] + x[0])) - P[19] * x[6]
f[7] = P[13]*(1 - (x[6] + x[7])/P[14])*x[7] - P[15]*x[7]*(x[4]/(P[18] + x[4])) - P[19] * x[7]

C = np.zeros((2,len(x)), dtype=int)
C[0][6] = 1
C[1][7] = 1
C = C.tolist()

sys = System(x, f, params = P, params_values = params_values, C = C, x_init = x_init)
```

```python
from autoreduce.utils import get_ODE
sys_ode = get_ODE(sys, timepoints_ode)
sol = sys_ode.solve_system().T
try:
    import matplotlib.pyplot as plt
    plt.plot(timepoints_ode, np.transpose(np.array(C)@sol))
    plt.xlabel('Time')
    plt.ylabel('[Product]')
    plt.show()
except:
    print('Plotting libraries missing.')
```

```python
from autoreduce.utils import get_SSM
timepoints_ssm = np.linspace(0,100,5)
sys_ssm = get_SSM(sys, timepoints_ssm)
# Uncomment to run:
# Ss = sys_ssm.compute_SSM() # len(timepoints) x len(params) x len(states)

```

```python
# nouts = 2
# out_Ss = []
# for i in range(len(params)):
#     out_Ss.append((np.array(C)@(Ss[:,i,:].T)))
# out_Ss = np.reshape(np.array(out_Ss), (len(timepoints_ssm), len(params), nouts))
```

```python
# try:
    # import seaborn as sn
    # for j in range(nouts):
    #     sn.heatmap(out_Ss[:,:,j].T)
    #     plt.xlabel('Time')
    #     plt.ylabel('Parameters')
    #     plt.title('Sensitivity of output[{0}] with respect to all parameters'.format(j))
    #     plt.show()
# except:
#     print('Plotting libraries missing.')
```

```python
from autoreduce.utils import get_reducible
timepoints_ssm = np.linspace(0,100,10)
timepoints_ode = np.linspace(0, 100, 100)
sys_reduce = get_reducible(sys, timepoints_ode, timepoints_ssm)
sys_reduce.nstates_tol = 4
sys_reduce.nstates_tol_min = 2
# results = sys_reduce.reduce_simple(skip_numerical_computations = True)
```

```python
reduced_sys, collapsed_sys = sys_reduce.solve_timescale_separation([x[1], x[5], x[6], x[7]], debug = False)
```

    Successful time-scale separation solution obtained with states: [x1, x5, x6, x7]!

```python

```


# Evaluation for Skewed Dataset

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
from IPython import display
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# produce vector inline graphics
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
from weka.core import dataset
import weka.core.jvm as jvm
from weka.core.converters import Loader
from weka.classifiers import Classifier, Evaluation
from weka.core.classes import Random
from weka.core.classes import complete_classname
```

## Class imbalance problem

In this notebook, we will analyze a skewed dataset for detecting microcalcifications in mammograms. The goal is to build a classifier to identify whether a bright spot in a mammogram is a micro-calcification, which is an early sign of breast cancer as shown below.

The dataset can be downloaded from
[OpenML](https://www.openml.org/d/310) in [ARFF format](https://www.openml.org/data/download/52214/phpn1jVwe). The following loads the data using `python-weka-wrapper`.

```python
jvm.start()
loader = Loader(classname="weka.core.converters.ArffLoader")
data = loader.load_url('https://www.openml.org/data/download/52214/phpn1jVwe')
data.class_is_last()
print(data.summary(data))
```

There are 7 attributes and over 11 thousand instances. To understand the dataset, refer to Section 4 of the original paper:

- Woods, Kevin S., et al. "Comparative evaluation of pattern recognition techniques for detection of microcalcifications." Biomedical Image Processing and Biomedical Visualization. Vol. 1905. International Society for Optics and Photonics, 1993. [(Available via CityU VPN.)](https://www.worldscientific.com/doi/abs/10.1142/9789812797834_0011)

A set of 24 mammograms were segmented to locate small bright spots, which are the candidates for the classification of malignant clusters of micro-calcifications. The classification problem is based on attributes of the image segments:

- Area (number of pixels)
- Average grey level
- Gradient strength (of perimeter pixels)
- Root mean square noise (fluctuation of the pixel values)
- Root mean square noise of local background
- Contrast (average grey level minus average of a 2-pixel wide border)
- (Low order moment-based) Shape descriptor

To compute the 10-fold cross-validation accuracy for `J48`:

```python
clf = Classifier(classname="weka.classifiers.trees.J48")
evl = Evaluation(data)
evl.crossvalidate_model(clf, data, 10, Random(1))

print(f'Accuracy: {evl.percent_correct:.3g}%')
```

You should see that the accuracy is close to 100%. To show the detailed confusion matrix:

```python
confusion_matrix = pd.DataFrame(evl.confusion_matrix, dtype = int,
    columns=[ f'predicted class "{v}"' for v in data.class_attribute.values],
    index=[f'class "{v}"' for v in data.class_attribute.values])
confusion_matrix
```

Each row of the confusion matrix corresponds to a class value (1: malignant, -1: benign) and each column corresponds to a predicted class. Each entry is a count of instances belonging to certain class and having certain predicted class.

**Exercise** Assign to `percent_of_malignant_detected` the percentage of instances of class 1 predicted as class 1:

```python
# YOUR CODE HERE
raise NotImplementedError()
print(
    f'Percentage of malignant detected: {percent_of_malignant_detected:.3g}%'
)
```

```python
# tests
```

## Different Performance Metrics

Since a high accuracy can be misleading for a skewed dataset, we should use other performance metrics to evaluate the performance of a classification algorithm:

```python
pos_class = 1  # specify the postive class value
performance = {
    'precision': evl.precision(pos_class),
    'recall': evl.recall(pos_class),
    'specificity': evl.true_negative_rate(pos_class)
}
performance
```

Although specificity is close to 100%, precision and recall are below 80% and 60% respectively. These metrics can be computed from the numbers of true/false positives/negatives:

\begin{align}
\text{precision} &= \frac{\text{TP}}{\text{TP} + \text{FP}}\\
\text{recall} &= \frac{\text{TP}}{\text{TP} + \text{FN}}\\
\text{specificity} &= \frac{\text{TN}}{\text{TN} + \text{FP}}
\end{align}

```python
TP = evl.num_true_positives(pos_class)
FN = evl.num_false_negatives(pos_class)
FP = evl.num_false_positives(pos_class)
TN = evl.num_true_negatives(pos_class)

assert np.isclose(performance['precision'], TP / (TP + FP))
assert np.isclose(performance['recall'], TP / (TP + FN))
assert np.isclose(performance['specificity'], TN / (TN + FP))

TFPN = pd.DataFrame(
    [[TP, FN],
     [FP, TP]],
    dtype = int,
    columns=['predicted +ve', 'predicted -ve'],
    index=['+ve', '-ve'])
TFPN
```

The above table is not the same as a confusion matrix since a confusion matrix

- does not specify a positive class, and
- can have more than two rows/columns in multi-class classification problems.

**Exercise** Modify `performance` to include the negative predictive value. You can add the value using:
```Python
performance['NPV'] = ...
```

```python
# YOUR CODE HERE
raise NotImplementedError()
print(f"negative predictive value (NPV): {performance['NPV']:.3g}")
```

```python
# tests
```

$F_{\beta}$ score is another measure that captures the performance in both precision and recall:

\begin{align}
F_{\beta} &:= \left( \frac{\text{precision}^{-1} + \beta^2 \cdot \text{recall}^{-1}}{\beta^2 + 1}\right)^{-1} = \frac{(\beta^2+1)\cdot \text{precision}\cdot \text{recall} }{\beta^2\text{precision} + \text{recall}}\\
F := F_1 &= \left( \frac{\text{precision}^{-1} + \text{recall}^{-1}}{2}\right)^{-1} = \frac{2\cdot \text{precision}\cdot \text{recall} }{\text{precision} + \text{recall}},
\end{align}

which is the harmonic mean of precision and recall. This is useful in training a classifier to maximize both precision and recall.

```python
performance['F'] = evl.f_measure(pos_class)
print(f"F-score: {performance['F']:.3g}")
```

**Exercise** For the $F_{\beta}$ score, we can increase $\beta$ to put more weight on recall. Modify `performance` to include the $F_2$ score. You can set the value using:
```Python
performance['F_2'] = ...
```

```python
# YOUR CODE HERE
raise NotImplementedError()
print(f"F_2 score: {performance['F_2']:.3g}")
```

```python
# tests
```

**Exercise** Using `ZeroR` as the classifier, assign to `ZeroR_performance` a dictionary of precision, recall, and specificity. You can create the dictionary as follows:
```Python
ZeroR_performance = {
    'precision': ___,
    'recall': ___,
    'specificity': ___
}
```

Use 10-fold cross-validation with a random seed of `1`. If the value is not a number, you may enter it as `np.nan`.

```python
# YOUR CODE HERE
raise NotImplementedError()
ZeroR_performance
```

```python
# tests
```

**Exercise** Is ZeroR a good baseline classifier?

[*Hint: Is the accuracy misleading? Can a random decision maker do better than zeroR?*]

YOUR ANSWER HERE

## Operating Curves for Probabilistic Classifier

For a probabilistic classifier that returns probabilities of different classes, we can obtain a trade-off between precision and recall by changing a threshold $\gamma$ for positive prediction, i.e., predict positive if and only if the probability estimate for positive class is larger than $\gamma$.

The following plots the [precision-recall curve](https://waikato.github.io/weka-wiki/roc_curves/) and prints the area under the curve.

```python
import weka.plot.classifiers as plcls

plcls.plot_prc(evl, class_index=[1])
performance['PRC'] = evl.area_under_prc(pos_class)
print(f"area under precision-recall curve (PRC): {performance['PRC']:.3g}")
```

**Exercise** One can operate the classifier at any point on the curve by an appropriate choice of $\gamma$. Is it a good idea to operate at the initial part of the curve where the slope is strictly positive?

YOUR ANSWER HERE

**Exercise** The above curve shows that the classifier can achieve 100% recall but not 100% precision. If you can choose any classifier, is it always possible to achieve 100% recall for any given data set? How about 100% precision?

YOUR ANSWER HERE

We can also plot the ROC (receiver operator characteristics) curve to show the trade-off between recall (true positive rate) and false positive rate:

```python
plcls.plot_roc(evl, class_index=[1])
performance['AUC'] = evl.area_under_roc(pos_class)
print(f"area under ROC curve (AUC): {performance['AUC']:.3g}")
```

**Exercise** The above curve shows that the classifier can achieve 0% false positive rate. If you can choose the classifier, is it always possible to achieve 0% false positive rate for any given dataset?

YOUR ANSWER HERE

Tutorial5/Evaluation for Skewed Dataset.ipynb


# Lab 8 - Deep Learning 3

The goal of this week's lab is to learn to use another widely-used neural network module: recurrent neural networks (RNNs). We can use it to learn features from sequences such as time series and text.

How should we extract features from sequences, which might be of variable length? Recurrent Neural Networks (RNNs) provide a solution by summarizing the entire sequence into a fixed-size vector representation. This week we will walk through how to use RNNs for sequence processing.

* **Review**: Convolutional Neural Networks (CNNs)
* **Unit A**: Time Series Classification and Recurrent Neural Networks (RNNs)
* **Unit B**: Recurrent Neural Networks for Text Classification

## Review

Last time we learned the basics of convolutional neural networks (CNNs) and used them for image classification.

```python
# For Tables
import pandas as pd
# For Visualization
import altair as alt
# For Scikit-Learn
import sklearn
from keras.wrappers.scikit_learn import KerasClassifier
# For Neural Networks
import tensorflow as tf
import keras
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPool2D, Flatten, Reshape
```

We will also turn off warnings.

```python
import warnings
warnings.filterwarnings('ignore')
```

We saw in last week how to store images and their labels in Pandas dataframes.

```python
df_train = pd.read_csv('https://srush.github.io/BT-AI/notebooks/mnist_train.csv.gz', compression='gzip')
df_test = pd.read_csv('https://srush.github.io/BT-AI/notebooks/mnist_test.csv.gz', compression='gzip')
```

The column `class` stores the class of each image, which is a number between 0 and 9.

```python
df_train[:100]["class"].unique()
```

    array([5, 0, 4, 1, 9, 2, 3, 6, 7, 8])

The rest of columns store the features, where we have 784 features since our images are 28x28. Each feature stores the intensity at each pixel : for instance, the column "3x4" stores the pixel value at the 3rd row and the 4th column. Since the size of each image is 28x28, there are 28 rows and 28 columns.

To make later processing easier, we store the names of pixel value columns in a list `features`.

```python
features = []
for i in range(1, 29):
    for j in range(1, 29):
        features.append(str(i) + "x" + str(j))
len(features)
```

    784

We used the below utility functions for visualizing the images.

Convert feature to x, y, and value.

```python
def position(row):
    y, x = row["index"].split("x")
    return {"x":int(x),
            "y":int(y),
            "val":row["val"]}
def draw_image(i, shuffle=False):
    t = df_train[i:i+1].T.reset_index().rename(columns={i: "val"})
    out = t.loc[t["index"] != "class"].apply(position, axis=1, result_type="expand")

    label = df_train.loc[i]["class"]
    title = "Image of a " + str(label)
    if shuffle:
        out["val"] = sklearn.utils.shuffle(out["val"], random_state=1234).reset_index()["val"]
        title = "Shuffled Image of a " + str(label)

    return (alt.Chart(out)
            .mark_rect()
            .properties(title=title)
            .encode(
                x="x:O",
                y="y:O",
                fill="val:Q",
                color="val:Q",
                tooltip=("x", "y", "val")
            ))
```

We can visualize an example image.

```python
im = draw_image(0)
im
```

The task is to classify the label given an image. To do that, we first need to define a function that creates our model.

Here is what a CNN model looks like. It contains two convolution layers and two max pooling layers.

```python
def create_cnn_model():
    # create model
    input_shape = (28, 28, 1)
    model = Sequential()
    model.add(Reshape(input_shape))
    model.add(Conv2D(32, kernel_size=(3, 3), activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, kernel_size=(3, 3), activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Flatten())
    model.add(Dense(10, activation="softmax")) # output a vector of size 10
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                   optimizer="adam",
                   metrics=["accuracy"])
    return model
```

Then we create the model and fit it on training data.

```python
model = KerasClassifier(build_fn=create_cnn_model,
                         epochs=2,
                         batch_size=20,
                         verbose=1)
# fit model
model.fit(x=df_train[features].astype(float),
          y=df_train["class"])
```

    Epoch 1/2
    3000/3000 [==============================] - 61s 20ms/step - loss: 1.1024 - accuracy: 0.8967
    Epoch 2/2
    3000/3000 [==============================] - 58s 19ms/step - loss: 0.0767 - accuracy: 0.9782

    <keras.callbacks.History at 0x7faa554a2a10>

With a trained model, we can apply it to the test dataset and measure the test accuracy.

```python
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
```

    500/500 [==============================] - 3s 6ms/step
    accuracy:  0.98

### Review Exercise

Change the model above to have the kernel size of convolution layers to be (1, 1). How does this affect the performance? Why?

```python
#📝📝📝📝 FILLME
pass
#SOLUTION
def create_cnn_model():
    # create model
    input_shape = (28, 28, 1)
    model = Sequential()
    model.add(Reshape(input_shape))
    model.add(Conv2D(32, kernel_size=(1, 1), activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, kernel_size=(1, 1), activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Flatten())
    model.add(Dense(10, activation="softmax")) # output a vector of size 10
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                   optimizer="adam",
                   metrics=["accuracy"])
    return model
model = KerasClassifier(build_fn=create_cnn_model,
                         epochs=2,
                         batch_size=20,
                         verbose=1)
# fit model
model.fit(x=df_train[features].astype(float),
          y=df_train["class"])
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
```

    Epoch 1/2
    3000/3000 [==============================] - 40s 13ms/step - loss: 1.3524 - accuracy: 0.7538
    Epoch 2/2
    3000/3000 [==============================] - 38s 13ms/step - loss: 0.4753 - accuracy: 0.8502
    500/500 [==============================] - 2s 4ms/step
    accuracy:  0.8671

Change the model above to have the kernel size of the first convolution layer to be (28, 28), and remove other convolution and pooling layers. How does this affect the performance? Why?

```python
#📝📝📝📝 FILLME
pass
#📝📝📝📝 FILLME
pass
#SOLUTION
def create_cnn_model():
    # create model
    input_shape = (28, 28, 1)
    model = Sequential()
    model.add(Reshape(input_shape))
    model.add(Conv2D(32, kernel_size=(28, 28), activation="relu"))
    model.add(Flatten())
    model.add(Dense(10, activation="softmax")) # output a vector of size 10
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                   optimizer="adam",
                   metrics=["accuracy"])
    return model
model = KerasClassifier(build_fn=create_cnn_model,
                         epochs=2,
                         batch_size=20,
                         verbose=1)
# fit model
model.fit(x=df_train[features].astype(float),
          y=df_train["class"])
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
```

    Epoch 1/2
    3000/3000 [==============================] - 6s 2ms/step - loss: 1.2770 - accuracy: 0.6850
    Epoch 2/2
    3000/3000 [==============================] - 5s 2ms/step - loss: 0.4928 - accuracy: 0.8810
    500/500 [==============================] - 1s 1ms/step
    accuracy:  0.9156

## Unit A

### Time Series Classification and Recurrent Neural Networks (RNNs)

Just like CNNs are suitable for the processing of 2-D images (and 1-D sequences since they can be viewed as a special case of 2-D images), recurrent neural networks (RNNs) are suitable for 1-D sequence modeling.

Let's start from a concrete sequence classification example, where we want to classify a curve into one of three possible classes: rectangle, triangle, and ellipse:

```python
df_train = pd.read_csv('https://raw.githubusercontent.com/srush/BT-AI/main/notebooks/shape_classification_train.csv')
df_test = pd.read_csv('https://raw.githubusercontent.com/srush/BT-AI/main/notebooks/shape_classification_train.csv')
df_train
```

      class
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54

      0
      square
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      1
      square
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      0.0
      0
      0
      0
      0
      0
      0

      2
      triangle
      0
      0
      0
      0
      0
      0.0
      0.0
      0.08
      0.16
      0.24
      0.32
      0.400000
      0.480000
      0.560000
      0.640000
      0.720000
      0.800000
      0.880000
      1.040000
      0.960000
      0.880000
      0.800000
      0.720000
      0.640000
      0.560000
      0.480000
      0.400000
      0.320000
      0.240000
      0.160000
      0.080000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      3
      triangle
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.125000
      0.250000
      0.375000
      0.500000
      0.625000
      0.750000
      0.875000
      1.000000
      0.875000
      0.750000
      0.625000
      0.500000
      0.375000
      0.250000
      0.125000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      4
      square
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1195
      ellipse
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.359011
      0.498888
      0.600000
      0.679869
      0.745356
      0.800000
      0.845905
      0.884433
      0.916515
      0.942809
      0.963789
      0.979796
      0.991071
      0.997775
      1.000000
      0.997775
      0.991071
      0.979796
      0.963789
      0.942809
      0.916515
      0.884433
      0.845905
      0.800000
      0.745356
      0.679869
      0.600000
      0.498888
      0.359011
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      1196
      ellipse
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.320145
      0.446594
      0.539313
      0.613784
      0.676065
      0.729285
      0.775312
      0.815365
      0.850289
      0.880695
      0.907036
      0.929659
      0.948829
      0.964753
      0.977588
      0.987456
      0.994444
      0.998614
      1.000000
      0.998614
      0.994444
      0.987456
      0.977588
      0.964753
      0.948829
      0.929659
      0.907036
      0.880695
      0.850289
      0.815365
      0.775312
      0.729285
      0.676065
      0.613784
      0.539313
      0.446594
      0.320145
      0.0
      0
      0
      0
      0
      0
      0

      1197
      square
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      1198
      ellipse
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      0.515079
      0.699854
      0.820652
      0.903508
      0.958315
      0.989743
      1.000000
      0.989743
      0.958315
      0.903508
      0.820652
      0.699854
      0.515079
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0
      0
      0
      0
      0
      0

      1199
      triangle
      0
      0
      0
      0
      0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.071429
      0.142857
      0.214286
      0.285714
      0.357143
      0.428571
      0.500000
      0.571429
      0.642857
      0.714286
      0.785714
      0.857143
      0.928571
      1.000000
      0.928571
      0.857143
      0.785714
      0.714286
      0.642857
      0.571429
      0.500000
      0.428571
      0.357143
      0.285714
      0.214286
      0.142857
      0.071429
      0.0
      0
      0
      0
      0
      0
      0

1200 rows × 56 columns

In this example, the curve is stored as a vector with 55 entries. In the dataframe, the i-th entry of this vector is stored in a column named $i$.
As before, we store the names of feature columns in a list `features`.

```python
input_length = 55
features = []
for i in range(input_length):
    features.append(str(i))
```

We can visualize some examples using the following function.

```python
import matplotlib.pyplot as plt
def draw_curve(i):
    t = df_train[features].iloc[i]
    c = df_train['class'].iloc[i]
    plt.plot(list(t))
    plt.title(f'class {c}')
    plt.show()
draw_curve(0)
draw_curve(3)
draw_curve(5)
```

First, let's try to apply an MLP classifier to this problem. Note that we need to set the size of the last layer to be 3 since there are three output classes.

```python
def create_mlp_model():
    # create model
    model = Sequential()
    model.add(Dense(64, activation='relu'))
    model.add(Dense(3, activation='softmax')) # output a vector of size 3
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer="adam",
                  metrics=["accuracy"])
    return model
```

```python
# create model
model = KerasClassifier(build_fn=create_mlp_model,
                        epochs=10,
                        batch_size=20,
                        verbose=1)
# fit model
model.fit(x=df_train[features], y=df_train["class"])
# print summary
print (model.model.summary())
# predict on test set
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
```

    Epoch 1/10
    60/60 [==============================] - 1s 2ms/step - loss: 1.0707 - accuracy: 0.3499
    Epoch 2/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.9794 - accuracy: 0.4853
    Epoch 3/10
    60/60 [==============================] - 0s 1ms/step - loss: 0.9556 - accuracy: 0.4800
    Epoch 4/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.9145 - accuracy: 0.5051
    Epoch 5/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.9016 - accuracy: 0.5360
    Epoch 6/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.8772 - accuracy: 0.5470
    Epoch 7/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.8426 - accuracy: 0.5903
    Epoch 8/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.8201 - accuracy: 0.6059
    Epoch 9/10
    60/60 [==============================] - 0s 2ms/step - loss: 0.7947 - accuracy: 0.6644
    Epoch 10/10
    60/60 [==============================] - 0s 1ms/step - loss: 0.7720 - accuracy: 0.6689
    Model: "sequential_6"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_6 (Dense)              (20, 64)                  3584
    _________________________________________________________________
    dense_7 (Dense)              (20, 3)                   195
    =================================================================
    Total params: 3,779
    Trainable params: 3,779
    Non-trainable params: 0
    _________________________________________________________________
    None
    60/60 [==============================] - 0s 1ms/step
    accuracy:  0.6933333333333334

The accuracy of the MLP classifier is quite low considering the simplicity of this task. The major challenge of this task is that even for the same shapes, the positions where they appear in the sequence, and their sizes may vary. For example, let's look at some rectangles.

```python
draw_curve(0)
draw_curve(1)
draw_curve(4)
```

👩🎓**Student question: Do you think MLPs are suitable for this task? Why or why not?**

```python
#📝📝📝📝 FILLME
pass
```

### Recurrent Neural Networks (RNNs)

Recurrent Neural Networks (RNNs) work by iteratively applying the same base operation to each element in the input sequence. To keep track of what it has seen so far, it also maintains an internal state. We call the base operation an RNN cell. Throughout this lab, we will use a special kind of RNN cell, a Long short-term memory (LSTM) cell due to its empirical successes.

Let's assume that we have a sequence of inputs $x_1, \ldots, x_T$. (We're notating the input elements as if they are scalars, but you should keep in mind that they might well be vectors themselves.) Let's consider a single update operation at step $t$, where the current internal state is denoted by $h_t$:

$h_{t+1} = {\text{LSTM}}_{\phi} (h_t, x_t)$,

where the LSTM cell has two inputs and one output: it uses both the input element $x_t$ and the old memory $h_t$ to compute the updated memory $h_{t+1}$. $\phi$ denotes the parameters of the LSTM cell, which we can adjust during training. For simplicy, we omit these parameters througout the rest of this lab. The internal computations of the LSTM cell are beyond the scope of this course, but for anyone interested in knowing further details, [this blog post](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) might be a good starting point.

Now that we have defined a single update step, we can chain them together to produce a summary of $x_1, \ldots, x_T$, starting from $h_0=0$:

\begin{align}
h_0 &= 0 \\
h_1 &= \text{LSTM} (h_0, x_1) \\
h_2 &= \text{LSTM} (h_1, x_2) \\
h_3 &= \text{LSTM} (h_2, x_3) \\
\vdots\\
h_T &= \text{LSTM} (h_{T-1}, x_T) \\
\end{align}"

$h_T$ can be used as a feature representation for the entire input sequence $x_1, \ldots, x_T$.

In `Keras`, the API for an LSTM cell is
```
LSTM(
    units,
    activation="tanh",
    recurrent_activation="sigmoid",
    use_bias=True,
    kernel_initializer="glorot_uniform",
    recurrent_initializer="orthogonal",
    bias_initializer="zeros",
    unit_forget_bias=True,
    kernel_regularizer=None,
    recurrent_regularizer=None,
    bias_regularizer=None,
    activity_regularizer=None,
    kernel_constraint=None,
    recurrent_constraint=None,
    bias_constraint=None,
    dropout=0.0,
    recurrent_dropout=0.0,
    return_sequences=False,
    return_state=False,
    go_backwards=False,
    stateful=False,
    time_major=False,
    unroll=False,
    **kwargs
)
```
It appears intimidating, but we only need to set `units` in this lab. In a nutshell, it controls the size of the hidden states in the LSTM cell: the larger the size, the more powerful the model will be, but the more likely the model will overfit to training data (overfitting means memorizing the training data without being able to generlize to the test data).

```python
from keras.layers import LSTM
hidden_size = 32
lstm_layer = LSTM(hidden_size)
```

This layer can be applied to a sequence of inputs $x_1, \ldots, x_T$, and the output will be the final hidden state $h_T$. Below shows an example of how to use this layer. Note that we need to use a `Reshape` layer to add one additional dimension to the input sequence since the expected input shape of the LSTM layer is `sequence_length x input size`, and the `input_size` in the case is 1 since $x_t$'s are scalars is 1.

```python
input_shape = (input_length, 1)
model = Sequential()
model.add(Reshape(input_shape))
model.add(lstm_layer)
#
# take the first example as input
# input shape: num samples x input_length
# output shape: num samples x hidden size
inputs = tf.convert_to_tensor(df_train[features].iloc[:1])
output = model(inputs)
print (inputs.shape)
print (output.shape)
```

    (1, 55)
    (1, 32)

# Group Exercise A

## Question 0

Icebreakers

Who are other members of your group today?

Karina, Nadiia and Esther

📝📝📝📝 FILLME

* What's their favorite place?

Own room and the park

📝📝📝📝 FILLME

* What are their goals by the end of the decade?

To find an internship/job and start own company

📝📝📝📝 FILLME

## Question 1

Look at this figure again. Can you figure out where are $h_t$'s and $x_t$'s? (Don't worry if you not understand the entire diagram does for now, we will elaborate on it later)**

```python
#📝📝📝📝 FILLME
# The ht's xt's are in the long-short term memory networks and sequence of neural networks.
```

Why can LSTMs process variable length inputs?

```python
#📝📝📝📝 FILLME
#The LSTMs process variable length inputs step by step.
```

## Question 2

Modify the MLP code to use LSTM instead. We recommend using a hidden size of 32 or 64. Train the model and report the test accuracy. You should expect to see at least 90% accuracy. Hint: don't forget the reshape layer before the LSTM!

```python
#📝📝📝📝 FILLME
pass
def create_rnn_model():
    tf.random.set_seed(1234)
    # create model
    input_shape = (input_length, 1)
    model = Sequential()
    model.add(Reshape(input_shape))
    model.add(LSTM(32))
    model.add(Dense(3, activation='softmax')) # output a vector of size 3
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                   optimizer="adam",
                   metrics=["accuracy"])
    return model
#
# create model
model = KerasClassifier(build_fn=create_rnn_model,
                         epochs=10,
                         batch_size=20,
                         verbose=1)
# fit model
model.fit(x=df_train[features], y=df_train["class"])
# print summary
print (model.model.summary())
# predict on test set
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
```

    Epoch 1/10
    60/60 [==============================] - 3s 14ms/step - loss: 1.0924 - accuracy: 0.3909
    Epoch 2/10
    60/60 [==============================] - 1s 14ms/step - loss: 0.8978 - accuracy: 0.5822
    Epoch 3/10
    60/60 [==============================] - 1s 14ms/step - loss: 0.7772 - accuracy: 0.6318
    Epoch 4/10
    60/60 [==============================] - 1s 14ms/step - loss: 0.6576 - accuracy: 0.7066
    Epoch 5/10
    60/60 [==============================] - 1s 15ms/step - loss: 0.4706 - accuracy: 0.8408
    Epoch 6/10
    60/60 [==============================] - 1s 16ms/step - loss: 1.0184 - accuracy: 0.5086
    Epoch 7/10
    60/60 [==============================] - 1s 15ms/step - loss: 0.5751 - accuracy: 0.7174
    Epoch 8/10
    60/60 [==============================] - 1s 15ms/step - loss: 0.4743 - accuracy: 0.8543
    Epoch 9/10
    60/60 [==============================] - 1s 15ms/step - loss: 0.3043 - accuracy: 0.9343
    Epoch 10/10
    60/60 [==============================] - 1s 15ms/step - loss: 0.1980 - accuracy: 0.9595
    Model: "sequential_11"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    reshape_10 (Reshape)         (20, 55, 1)               0
    _________________________________________________________________
    lstm_4 (LSTM)                (20, 32)                  4352
    _________________________________________________________________
    dense_10 (Dense)             (20, 3)                   99
    =================================================================
    Total params: 4,451
    Trainable params: 4,451
    Non-trainable params: 0
    _________________________________________________________________
    None
    60/60 [==============================] - 1s 6ms/step
    accuracy:  0.9283333333333333

## Unit B

### Recurrent Neural Networks for Text Classification

Now let's go to a real application: text classification. Text classification raises new challenges, as the inputs are strings instead of the numeric values we have been familiar with in this course. In this unit, we will first find a suitable feature representation for the text input, and then we will apply an LSTM-based model to this task.

The text classification task we will be working with is sentiment analysis,  where the goal is to classify the sentiment of a text sequence. In particular, we will use the Stanford Sentiment Treebank v2 (SST-2) dataset, where we want to predict the sentiment (positive or negative) for a movie review.

```python
df_train = pd.read_csv('https://raw.githubusercontent.com/srush/BT-AI/main/notebooks/sst_movie_reviews_processed_train.csv.gz', compression='gzip')
df_test = pd.read_csv('https://raw.githubusercontent.com/srush/BT-AI/main/notebooks/sst_movie_reviews_processed_test.csv.gz', compression='gzip')
df_train[:10]
```

      class
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54

      0
      positive
      an
      ambitious
      '
      what
      if
      ?
      '
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      1
      positive
      a
      great
      cast
      and
      a
      wonderful
      but
      sometimes
      confusing
      OOV
      movie
      about
      growing
      up
      in
      a
      dysfunctional
      family
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      2
      positive
      remarkable
      for
      its
      intelligence
      and
      intensity
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      3
      positive
      steven
      soderbergh
      '
      s
      digital
      video
      experiment
      is
      a
      clever
      and
      cutting
      ,
      quick
      and
      dirty
      look
      at
      modern
      living
      and
      movie
      life
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      4
      positive
      a
      whole
      lot
      of
      fun
      and
      funny
      in
      the
      middle
      ,
      though
      somewhat
      less
      OOV
      at
      the
      start
      and
      finish
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      5
      positive
      constantly
      touching
      ,
      surprisingly
      funny
      ,
      OOV
      exploration
      of
      the
      creative
      act
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      6
      negative
      it
      '
      s
      as
      if
      allen
      ,
      at
      OOV
      ,
      has
      OOV
      challenging
      himself
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      7
      negative
      check
      your
      brain
      and
      your
      secret
      agent
      OOV
      ring
      at
      the
      door
      because
      you
      don
      '
      t
      want
      to
      think
      too
      much
      about
      what
      '
      s
      going
      on
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      8
      negative
      it
      '
      s
      just
      hard
      to
      believe
      that
      a
      life
      like
      this
      can
      sound
      so
      dull
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

      9
      negative
      the
      rollerball
      sequences
      feel
      OOV
      and
      OOV
      .
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD
      PAD

The column `class` stores the sentiment of each review, which is either "positive" or "negative".

```python
df_train[:100]["class"].unique()
```

    array(['positive', 'negative'], dtype=object)

The other columns store the words, where the i-th word is stored in a feature column i (counting from 0). For example, the first word of each movie review is stored in column 0, the second word is stored in column 1, and so on. As before, we store all feature column names in a list. The maximum length of sentences is 55 on this dataset, so we have 55 feature columns.

```python
input_length = 55
features = []
for i in range(input_length):
    features.append(str(i))
```

Notice that some tokens towards the end are `PAD`. They are actually placeholders to pad every sentence into the same length such that we can store them in a table.

```python
data = df_train[features].values
labels = df_train['class'].values
print (labels[1], data[1])
print (labels[8], data[8])
```

    positive ['a' 'great' 'cast' 'and' 'a' 'wonderful' 'but' 'sometimes' 'confusing'
     'OOV' 'movie' 'about' 'growing' 'up' 'in' 'a' 'dysfunctional' 'family'
     '.' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD']
    negative ['it' "'" 's' 'just' 'hard' 'to' 'believe' 'that' 'a' 'life' 'like' 'this'
     'can' 'sound' 'so' 'dull' '.' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD'
     'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD' 'PAD']

### Input Representation

Different from all examples we've seen so far, the input in text classification cannot be directly fed into a neural network, since they are strings but not numeric values. A natural idea is to associate each word type with an integer id, such that we can use those integer ids to represent words.

First, we need to build the mapping from word types to integer ids.

```python
# build vocabulary
word2id = {}
id2word = {}
unassigned_id = 0
for review in data:
    for token in review:
        if token not in word2id:
            word2id[token] = unassigned_id
            id2word[unassigned_id] = token
            unassigned_id += 1
vocab_size = len(word2id)
print ('Vocab size: ', vocab_size)
```

    Vocab size:  4823

With `word2id`, we can map a word to its associated id:

```python
print (word2id['the'])
```

    51

With `id2word`, we can map an integer id to the corresponding word:

```python
print (id2word[51])
```

    the

👩🎓**Student question: Convert the sentence "a great cast" to a sequence of integer ids using `word2id`.**

```python
#📝📝📝📝 FILLME
pass
print (word2id['a'], word2id['great'], word2id['cast'])
```

    7 8 9

👩🎓**Student question: Convert a sequence of integer ids `[7, 8, 9]` to the original sentence.**

```python
#📝📝📝📝 FILLME
pass
print (id2word[7], id2word[8], id2word[9])
```

    a great cast

Now we can convert all the strings into integer ids using those mappings.

```python
def word_to_id(word):
    return word2id[word]
df_train[features] = df_train[features].applymap(word_to_id)
df_test[features] = df_test[features].applymap(word_to_id)
df_train[:10]
```

      class
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54

      0
      positive
      0
      1
      2
      3
      4
      5
      2
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      1
      positive
      7
      8
      9
      10
      7
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      7
      21
      22
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      2
      positive
      24
      25
      26
      27
      10
      28
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      3
      positive
      29
      30
      2
      31
      32
      33
      34
      35
      7
      36
      10
      37
      38
      39
      10
      40
      41
      42
      43
      44
      10
      16
      45
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      4
      positive
      7
      46
      47
      48
      49
      10
      50
      20
      51
      52
      38
      53
      54
      55
      15
      42
      51
      56
      10
      57
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      5
      positive
      58
      59
      38
      60
      50
      38
      15
      61
      48
      51
      62
      63
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      6
      negative
      64
      2
      31
      65
      4
      66
      38
      42
      15
      38
      67
      15
      68
      69
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      7
      negative
      70
      71
      72
      10
      71
      73
      74
      15
      75
      42
      51
      76
      77
      78
      79
      2
      80
      81
      82
      83
      84
      85
      17
      3
      2
      31
      86
      87
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      8
      negative
      64
      2
      31
      88
      89
      82
      90
      91
      7
      45
      92
      93
      94
      95
      96
      97
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

      9
      negative
      51
      98
      99
      100
      15
      10
      15
      23
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6
      6

### Word Embeddings

Now that we can convert the original text (a sequence of strings) into a sequence of integer ids, can we directly feed that into the LSTM layer as we did for the shape classification problem?

If we directly use those integer ids in the neural network, we are implicitly assuming that the word with id `1001` is closer to the word with id `1002` than it is to the word with id `10`. However, the way we constructed the mappings between word types and ids does not provide this property.
Instead of directly using those word ids, for each word id, we maintain a different vector (usually termed an embedding), which can be stored in a matrix $E$ of size `vocab_size x embedding_size`. To get the word embedding for word id i, we can simply take the i-th row in the matrix $E_i$.

In `Keras`, this embedding matrix is maintained in an `Embedding` layer.
```
Embedding(
   input_dim,
   output_dim,
   embeddings_initializer="uniform",
   embeddings_regularizer=None,
   activity_regularizer=None,
   embeddings_constraint=None,
   mask_zero=False,
   input_length=None,
   **kwargs
)
```
Again, we don't need to use all arguments. The only two arguments that we need to understand are: `input_dim`, which is the size of the vocabulary `vocab_size`, and `output_dim`, which is the size of the word embeddings `embedding_size`.

```python
from keras.layers import Embedding
model = Sequential()
embedding_size = 32
model.add(Embedding(vocab_size, embedding_size))
# The model will take as input an integer matrix of size (num_samples, input_length).
# The output_shape is (num_samples, input_length, embedding_size)
#
# take the first example as input
inputs = tf.convert_to_tensor(df_train[features].iloc[:1])
outputs = model(inputs)
print (inputs.shape)
print (outputs.shape)
```

    (1, 55)
    (1, 55, 32)

So now we have converted words to their word ids to their embeddings (token strings -> integer word ids -> vector word embeddings). (You might notice that the intermediate word id step is not necessary and we can directly map each word type to a word embedding: we used this intermediate word id step since tensors are easier to work with than strings, and we only need to do this conversion once for the dataset.)

👩🎓**Student question: By representing words as word embeddings, are we still making implicit assumptions that the 1001-st word is closer to the 1002-nd word than it is to the 10-th word?**

```python
#📝📝📝📝 FILLME
# No implicit assumptions
```

### Putting Everything Together

Now we can put everything together and assemble a model for text classification: we have converted the token strings into word ids. The model first uses an embedding layer to convert those word ids into word embeddings, then the LSTM runs on top of those word embeddings, and we use a final projection layer to project to the output shape.

# Group Exercise B

## Question 1

Take another look at this model diagram. Can you explain what's happening in this diagram? What are the modules used? What are the inputs and outputs of each module?

```python
#📝📝📝📝 FILLME
# Long-short term memory networks and sequence of neural networks are being used to make predictions of the outcome.
```

## Question 2

Finish the `TODO`s in the `create_rnn_model` function, train the network and report the test accuracy.

```python
#📝📝📝📝 FILLME
def create_rnn_model():
    tf.random.set_seed(1234)
    # create model
    model = Sequential()
    # TODO: add embedding layer with embedding_size 32
    pass
    # TODO: add LSTM layer with hidden_size 32
    pass
    model.add(Dense(2, activation='softmax'))
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer="adam",
                  metrics=["accuracy"])
    return model
#
# create model
model = KerasClassifier(build_fn=create_rnn_model,
                        epochs=6,
                        batch_size=150,
                        verbose=1)
# fit model
model.fit(x=df_train[features], y=df_train["class"])
# print summary
print (model.model.summary())
# predict on test set
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
pass

def create_rnn_model():
    tf.random.set_seed(1234)
    # create model
    model = Sequential()
    # TODO: add embedding layer
    model.add(Embedding(vocab_size, 32)) # output size: length, 32
    # TODO: add LSTM layer
    model.add(LSTM(32))
    model.add(Dense(2, activation='softmax'))
    # Compile model
    model.compile(loss="sparse_categorical_crossentropy",
                  optimizer="adam",
                  metrics=["accuracy"])
    return model
#
# create model
model = KerasClassifier(build_fn=create_rnn_model,
                        epochs=6,
                        batch_size=150,
                        verbose=1)
# fit model
model.fit(x=df_train[features], y=df_train["class"])
# print summary
print (model.model.summary())
# predict on test set
df_test["predict"] = model.predict(df_test[features])
correct = (df_test["predict"] == df_test["class"])
accuracy = correct.sum() / correct.size
print ("accuracy: ", accuracy)
pass
```

    Epoch 1/6
    47/47 [==============================] - 1s 2ms/step - loss: 470.6189 - accuracy: 0.4892
    Epoch 2/6
    47/47 [==============================] - 0s 2ms/step - loss: 318.7600 - accuracy: 0.5008
    Epoch 3/6
    47/47 [==============================] - 0s 2ms/step - loss: 231.9498 - accuracy: 0.4877
    Epoch 4/6
    47/47 [==============================] - 0s 2ms/step - loss: 154.8466 - accuracy: 0.4855
    Epoch 5/6
    47/47 [==============================] - 0s 1ms/step - loss: 80.7914 - accuracy: 0.5069
    Epoch 6/6
    47/47 [==============================] - 0s 2ms/step - loss: 38.9167 - accuracy: 0.4904
    Model: "sequential_14"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_11 (Dense)             (None, 2)                 112
    =================================================================
    Total params: 112
    Trainable params: 112
    Non-trainable params: 0
    _________________________________________________________________
    None
    13/13 [==============================] - 0s 2ms/step
    accuracy:  0.49093904448105435
    Epoch 1/6
    47/47 [==============================] - 4s 45ms/step - loss: 0.6928 - accuracy: 0.5062
    Epoch 2/6
    47/47 [==============================] - 2s 45ms/step - loss: 0.6925 - accuracy: 0.5193
    Epoch 3/6
    47/47 [==============================] - 2s 44ms/step - loss: 0.6804 - accuracy: 0.5501
    Epoch 4/6
    47/47 [==============================] - 2s 44ms/step - loss: 0.5050 - accuracy: 0.7672
    Epoch 5/6
    47/47 [==============================] - 2s 47ms/step - loss: 0.3202 - accuracy: 0.8787
    Epoch 6/6
    47/47 [==============================] - 2s 47ms/step - loss: 0.2308 - accuracy: 0.9160
    Model: "sequential_15"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    embedding_2 (Embedding)      (None, None, 32)          154336
    _________________________________________________________________
    lstm_5 (LSTM)                (None, 32)                8320
    _________________________________________________________________
    dense_12 (Dense)             (None, 2)                 66
    =================================================================
    Total params: 162,722
    Trainable params: 162,722
    Non-trainable params: 0
    _________________________________________________________________
    None
    13/13 [==============================] - 1s 14ms/step
    accuracy:  0.8034047226798462

## Question 3

Word embeddings might sound like a very abstract concept: we are associating each word with a vector, but what do these vectors mean? What properties do they possess? In this question, we will use the [Tensorflow Embedding Projector](https://projector.tensorflow.org/) to explore some pretrained word embeddings. (We can also take our trained model and visualize the embeddings from the embedding layer, but we usually need to train on very large datasets to see meaningful visualizations)

[Embedding Projector](https://projector.tensorflow.org/)

This visualization tool visualizes word embeddings in a 3-D space, but keep in mind that those embeddings are actually of much higher dimensionality (`embedding_size` is 200 in the default setting), and their neighbors are found in the original (200-D) space, not the 3-D space, which might lead to some seemingly nearby points not being shown as nearest neighbors.

* Use the search tool in the right-side panel and search "smith". The point cloud in the middle pannel will show this word as well as its nearest neighbors. What did you observe about the neighbors of the word "smith"?

```python
#📝📝📝📝 FILLME
# The neighbors of smith displayed the nearest points of words consisting of other names near the original space.
```

* Let's try another word "apple" and find its nearest neighbors. What's your observation? Is "apple" considered a fruit or a company here? Do you consider this an issue for using word embeddings to represent text?

```python
#📝📝📝📝 FILLME
# Apple is considered to be a company since there are word embeddings such as mac, intel, microsoft, windows and computers.
```

## Question 4

Word embeddings are numeric representations of word types, hence they support algebraic operations. For example, we cannot compute `water + bird - air` in the string space, but we can compute `embedding_of_water + embedding_of_bird - embedding_of_air`. Then we can convert the resulting vector back to word by finding its nearest neighbors like we did in the previous question.

Let's use this demo to perform word algebra.

[Word Algebra](https://turbomaze.github.io/word2vecjson/)

* Use the tool under section "Word Algebra" to find the nearest neighbors of `water + bird -air`. What do you get?

```python
#📝📝📝📝 FILLME
# The neighbors were birds, water, fish, turtle, frog, animals, snake and owl.
```

* Can you find some other interesting examples?

```python
#📝📝📝📝 FILLME
# books + computers -magazine
# The neighbors were library, machines, phones, toys, files and items
```