# Curse of Dimensionality

* The number of points needed to densely populate a space grows quickly with dimensionality. If you add informative, discriminating features - more features can be helpful - but you will need more and more points to fully understand the space.

* Things do not always behave as you would expect in high dimensions - this is part of the Curse of Dimensionality.

## Unit Porcupine:  The unit hyper-sphere inscribed within the unit hyper-cube.

* Consider a sphere with radius $r$ in $D$ dimensions
\begin{equation}
S = \left\{ \mathbf{x} \left| \sum_{i=1}^D x_i^2 \le r^2 \right. \right\} \nonumber
\end{equation}
It's volume is:
\begin{equation}
v_D(r) = \frac{r^D \pi^{\frac{D}{2}}}{\Gamma(\frac{D}{2} + 1)} \nonumber
\end{equation}
where $\Gamma(n) = \int_0^\infty e^{-x}x^{n-1}dx$.

  So, for $D = 1$: $v_1(r) = \frac{r \pi^{1/2}}{\Gamma(1/2 + 1)} = 2r$

  $D = 2$: $v_2(r) = \frac{r^2 \pi}{\Gamma(2)} = \pi r^2$

   $D = 3$: $v_3(r) = \frac{r^2 \pi^{3/2}}{\Gamma(3/2 + 1)} = \frac{4}{3}\pi r^3$

* Consider a hypercube with radius $r$.  It's volume is $(2r)^D$.

So, for $D = 1$: $v_{1,c} = 2r$

$D = 2$: $v_{2,c} = 4r^2$

$D = 3$: $v_{3,c} = 8r^3$
* Take the case where the hyper-sphere is inscribed within the unit hyper-cube.  What happens to the relative volume of the sphere and cube as $D$ increases?

\begin{eqnarray}
\frac{Vol(Sphere)}{Vol(Cube)} &=& \frac{r^D\pi^{\frac{D}{2}}}{\Gamma(\frac{D}{2} +1)(2r)^D}\\ \nonumber
&=& \frac{\pi^{\frac{D}{2}}}{2^D\Gamma(\frac{D}{2} + 1)}
\end{eqnarray}
Note: The $r$ dropped out, relative volume depends only on dimension.

```python
#Import needed python libraries
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
import math

#The Unit Porcupine Plot
#The unit hyper-sphere inscribed within the unit hyper-cube.

fig = plt.figure()
d = range(1,100)
V = [math.pi**(i/2)/(2**i*math.gamma(i/2 + 1)) for i in d]
ax = fig.add_subplot(*[1,2,1])
ax.plot(d, V)
ax.set_title('Ratio of Volume')

dCorner = [math.sqrt(d) for d in range(1,10000)]
ax = fig.add_subplot(*[1,2,2])
ax.plot(range(1,10000), dCorner)
ax.set_title('Distance to Corner')

plt.show();
```

## Volume of space between two spheres with slightly different radii in high dimensions

* $Vol_{crust} = Vol_{S_1} - Vol_{S_2}$ where radius of $S_1$ is greater than the radius of $S_2$
\begin{eqnarray}
Vol_{crust} &=& Vol_{S_1} - Vol_{S_2} \nonumber\\
&=& \left[ 1 - \frac{Vol_{S_2}}{Vol_{S_1}}\right]Vol_{S_1} \nonumber \\
&=& \left[ 1 - \frac{\frac{(a - \epsilon)^D \pi^{\frac{D}{2}}}{\Gamma(\frac{D}{2}+1)}}{\frac{a^D \pi^{\frac{D}{2}}}{\Gamma(\frac{D}{2}+1)}}\right]Vol_{S_1} \nonumber \\
&=& \left[ 1 - \frac{a^D ( 1 - \frac{\epsilon}{a})^D}{a^D}\right]Vol_{S_1}   \nonumber \\
&=&  \left[ 1 - \left( 1 - \frac{\epsilon}{a}\right)^D \right]Vol_{S_1}\nonumber
\end{eqnarray}

* What happens as $D$ increases?

```python
#Crust volume between spheres with epsilon different radii and increasing dimensionality
a = 1
eps = 0.001
D = np.arange(1,10000)
RatioVol = [1-(1-eps/a)**d for d in D]
fig = plt.figure()
plt.plot(D, RatioVol)
plt.title('Ratio of Volume of Crust to Bigger Sphere')
plt.show()
```

## Radius needed to cover the same percentage volume with growing dimensionality:

* Volume of unit line, square, cube, hyper-cube:  $s^D = 1^D$
* Side of a cube covering some percentage of the area: say, 10% would be $r^D = 1/10$, $r = (1/10)^{(1/D)}$
* What happens as D increases?

```python
'''Illustrate that average distance between uniform random points increases when dimensionality
increases and number of data points held constant. Illustrates that you need more and more
points to characterize the unit cube as you go up in dimensionality.'''

from mpl_toolkits.mplot3d import Axes3D
import scipy.spatial.distance as sc
import textwrap

N = 100

x1 = np.random.uniform(0,1,(N,1)) #generate N uniform random data points in the unit interval
x2 = np.random.uniform(0,1,(N,2)) #generate N uniform random data points in the unit square
x3 = np.random.uniform(0,1,(N,3)) #generate N uniform random data points in the unit cube
y1 = np.mean(sc.pdist(x1, 'euclidean')) #compute avg euclidean distance between points
y2 = np.mean(sc.pdist(x2, 'euclidean'))
y3 = np.mean(sc.pdist(x3, 'euclidean'))
z1 = np.mean(sc.pdist(x1, 'cityblock')) #compute l1 distance between points
z2 = np.mean(sc.pdist(x2, 'cityblock'))
z3 = np.mean(sc.pdist(x3, 'cityblock'))

#plot results
fig = plt.figure()
ax = fig.add_subplot(*[1,3,1])
ax.scatter(x1, np.zeros((N,1)))
myTitle = 'Avg. L2 Dist.: ' + str("%.2f"%y1) + ' Avg. L1 Dist.: ' + str("%.2f"%z1);
ax.set_title("\n".join(textwrap.wrap(myTitle, 20)))

ax = fig.add_subplot(*[1,3,2])
ax.scatter(x2[:,0], x2[:,1])
myTitle = 'Avg. L2 Dist.: ' + str("%.2f"%y2) + ' Avg. L1 Dist.: ' + str("%.2f"%z2);
ax.set_title("\n".join(textwrap.wrap(myTitle, 20)))

ax = fig.add_subplot(*[1,3,3], projection='3d')
ax.scatter(x3[:,0], x3[:,1], x3[:,2])
myTitle = 'Avg. L2 Dist.: ' + str("%.2f"%y3) + ' Avg. L1 Dist.: ' + str("%.2f"%z3);
ax.set_title("\n".join(textwrap.wrap(myTitle, 20)))
plt.show();
```

```python
#Percentage Volume Illustration (p=.1)

p = 0.1
fig = plt.figure()
x = np.arange(0,1,.05)

# 1D
ax = fig.add_subplot(*[1,3,1])
ax.plot(x, np.zeros(x.size))
z = np.arange(0,p,.01)
ax.plot(z, np.zeros(z.size), 'r')

# 2D
ax = fig.add_subplot(*[1,3,2])
[xx,yy] = np.meshgrid(x,x)
z = np.arange(0,math.sqrt(p),0.01)
[zx,zy] = np.meshgrid(z,z)
ax.scatter(xx, yy)
ax.scatter(zx, zy, color='red')

# 3D
ax = fig.add_subplot(*[1,3,3], projection='3d')
[xx,yy,zz] = np.meshgrid(x,x,x)
z = np.arange(0,math.sqrt(p),0.01)
[zx,zy,zz2] = np.meshgrid(z,z,z)
ax.scatter(xx, yy, zz)
ax.scatter(zx, zy, zz2, color='red')
plt.show()
```

# Principal Components Analysis

* So, a major take away from the Curse of Dimensionality discussion is that when we are in high dimensional spaces, much of the space is empty and the data lives at the surface.  Given this, it makes sense to use a lower-dimensional *manifold* representation of the data.

* A very common approach (and one of the simplest approaches) to dimensionality reduction is  Principal Components Analysis (PCA).  PCA  takes data from sensor coordinates to data centric coordinates using linear projections (i.e., it is assuming that the informative components of the data lies on a linear manifold.)

* PCA uses a linear transformation to minimize the redundancy of the resulting transformed data (by ending up with data that is uncorrelated).

* PCA finds the directions of maximum variance in high-dimensional data and projects it onto a new subspace with equal or fewer dimensions than the original one.

* Without loss of generality, let's assume the input data has zero mean.
\begin{equation}
\mathbf{y} = \mathbf{A}\mathbf{x}
\end{equation}

The correlation matrix of $\mathbf{y}$ is:
\begin{eqnarray}
R_y &=& E[\mathbf{y}\mathbf{y}^T]\\
&=& E[\mathbf{A}\mathbf{x}\mathbf{x}^T\mathbf{A}^T]\\
&=&\mathbf{A}R_x\mathbf{A}^T
\end{eqnarray}

 If we are given $N$ data vectors, $\{ \mathbf{x}_1, \ldots, \mathbf{x}_N \}$, we can estimate $R_x$ as
 \begin{equation}
 R_x \approx \frac{1}{n} \sum_{k=1}^n \mathbf{x}_k \mathbf{x}_k^T
 \end{equation}
 * This is a symmetric matrix, so, it's eigenvectors are mutually orthogonal.
 * So, if we choose $\mathbf{A}$ to have columns equal to the orthonormal eigenvectors of $R_x$, the $R_y$ is diagonal.
 \begin{eqnarray}
 \mathbf{R}_y = \left[ \begin{array}{c} \mathbf{e_1}^T \\ \mathbf{e_2}^T \\ \vdots \\ \mathbf{e_D}^T\end{array}\right]\mathbf{R}_x\left[ \mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_D \right] & = & \left[ \begin{array}{c c c c}\mathbf{e}_1^T\mathbf{R}_x\mathbf{e}_1 & \mathbf{e}_1^T\mathbf{R}_x\mathbf{e}_2 & \ldots & \mathbf{e}_1^T\mathbf{R}_x\mathbf{e}_D \\ \mathbf{e}_2^T\mathbf{R}_x\mathbf{e}_1 & \mathbf{e}_2^T\mathbf{R}_x\mathbf{e}_2 & \ldots & \mathbf{e}_2^T\mathbf{R}_x\mathbf{e}_D \\ \vdots & & \ddots & \vdots \\ \mathbf{e}_D^T\mathbf{R}_x\mathbf{e}_1 & \mathbf{e}_D^T\mathbf{R}_x\mathbf{e}_D & \ldots & \mathbf{e}_D^T\mathbf{R}_x\mathbf{e}_D \end{array}\right]\\
 & = & \left[ \begin{array}{c c c c} \lambda_1 & 0 & \cdots & 0 \\ 0 & \lambda_2 & \cdots & 0 \\ \vdots & & \ddots & \\ 0 & 0 & \cdots & \lambda_D \end{array}\right]
\end{eqnarray}
where $\mathbf{e}_i \in \mathbb{R}^{D\times1}$ and $\mathbf{R}_x \in \mathbb{R}^{D\times D}$.

* Note: Given that $\mathbf{e}_i$ is an eigenvector of $\mathbf{R}_x$, we know that $\lambda_i\mathbf{e}_i = \mathbf{R}_x\mathbf{e}_i$.  So, $\mathbf{e}_i^T\mathbf{R}_x\mathbf{e}_i = \mathbf{e}_i^T\left( \lambda_i \mathbf{e}_i \right) = \lambda_i$ using the fact that $\mathbf{e}_i$ is normalized (i.e., $\left\|\mathbf{e}_i\right\|_2^2$ = 1)

* Similarly, Given that $\mathbf{e}_i$ and $\mathbf{e}_j$ are orthogonal eigenvectors of $\mathbf{R}_x$, we know that $\lambda_i\mathbf{e}_i = \mathbf{R}_x\mathbf{e}_i$.  So, $\mathbf{e}_j^T\mathbf{R}_x\mathbf{e}_i = \mathbf{e}_j^T\left( \lambda_i \mathbf{e}_i \right) = 0$

## Given a symmetric matrix, the eigenvectors of distinct eigenvalues are orthogonal:

Let $\mathbf{A}$ be an $l\times l$ symmetric matrix, $\mathbf{A}^T = \mathbf{A}$.  Then the eigenvectors corresponding to distinct eigenvalues are orthogonal.  Let $\lambda_i \ne \lambda_j$ be two such eigenvalues.  The definitions we have:
 \begin{eqnarray}
 \mathbf{A} \mathbf{v}_i = \lambda_i\mathbf{v}_i \\
 \mathbf{A} \mathbf{v}_j = \lambda_j\mathbf{v}_j
 \end{eqnarray}

 By multiplying the first eigenvalue equation on the left by $\mathbf{v}_j^T$ and by the transpose of the second equation on the right by $\mathbf{v}_i$, we get:
\begin{eqnarray}
  \mathbf{v}_j^T \mathbf{A} \mathbf{v}_i =  \mathbf{v}_j^T \lambda_i\mathbf{v}_i \\
 \left(\mathbf{A} \mathbf{v}_j\right)^T\mathbf{v}_i = \left(\lambda_j\mathbf{v}_j\right)^T\mathbf{v}_i\\
   \mathbf{v}_j^T\mathbf{A}\mathbf{v}_i = \lambda_j\mathbf{v}_j^T\mathbf{v}_i\\
   \mathbf{v}_j^T \mathbf{A} \mathbf{v}_i - \mathbf{v}_j^T \mathbf{A} \mathbf{v}_i = 0 = \left(\lambda_i - \lambda_j\right)\mathbf{v}_j^T\mathbf{v}_i
\end{eqnarray}

Thus, $\mathbf{v}_j^T\mathbf{v}_i = 0$.

* The eigenvectors can be interpreted as an orthogonal axis defined by the data.
* Since all data is noisy, we can concentrate on the axes corresponding to the largest eigenvectors if you are interested in preserving the variance of the data.
* However, when using PCA within a classification problem, it is much more difficult because we are interested in discriminability (not necessarily variance).

## Principal Components Analysis - Maximal Variance Formulation

* PCA is a linear transformation
* PCA minimizes the redundancy of the resulting transformed data (by ending up data that is uncorrelated), minimizes the mean squared error between original and transformed/reduced data, and maximizes the retained variance of the data.

* Consider a data set of observations $\left\{ \mathbf{x}_n \right\}_{n=1}^N$ and $\mathbf{x}_n \in \mathbb{R}^D$. We want to maximize the variance of the projected data.

* Let us first consider reducing dimensionality to $M = 1$.  Let us define the projection as a vector $\mathbf{u}_1$ where $\mathbf{u}_1^T\mathbf{u}_1 = 1$.  Then, each projected data point into 1-D would be $y_n = \mathbf{u}_1^T\mathbf{x}_n$

* The mean of the sample data is $\bar{\mathbf{x}} = \frac{1}{N}\sum_{n=1}^N\mathbf{x}_n$ and the mean of the projected data is $\mathbf{u}_1^T\bar{\mathbf{x}}$

* The variance of projected data is:
\begin{eqnarray}
\frac{1}{N} \sum_{n=1}^N \left\{ \mathbf{u}_1^T\mathbf{x}_n - \mathbf{u}_1^T\bar{\mathbf{x}} \right\}^2 & = & \frac{1}{N} \sum_{n=1}^N  \left( \mathbf{u}_1^T\mathbf{x}_n - \mathbf{u}_1^T\bar{\mathbf{x}} \right)\left( \mathbf{u}_1^T\mathbf{x}_n - \mathbf{u}_1^T\bar{\mathbf{x}} \right)^T\\
& = & \frac{1}{N} \sum_{n=1}^N  \left( \mathbf{u}_1^T\mathbf{x}_n - \mathbf{u}_1^T\bar{\mathbf{x}} \right) \left( \mathbf{x}_n^T\mathbf{u}_1 - \bar{\mathbf{x}}^T\mathbf{u}_1 \right)\\
& = & \frac{1}{N} \sum_{n=1}^N  \mathbf{u}_1^T\mathbf{x}_n\mathbf{x}_n^T\mathbf{u}_1 - \mathbf{u}_1^T\mathbf{x}_n\bar{\mathbf{x}}^T\mathbf{u}_1 - \mathbf{u}_1^T\bar{\mathbf{x}}\mathbf{x}_n^T\mathbf{u}_1 + \mathbf{u}_1^T\bar{\mathbf{x}}\bar{\mathbf{x}}^T\mathbf{u}_1\\
& = & \mathbf{u}_1^T \left( \frac{1}{N} \sum_{n=1}^N \mathbf{x}_n\mathbf{x}_n^T - \mathbf{x}_n\bar{\mathbf{x}}^T - \bar{\mathbf{x}}\mathbf{x}_n^T+ \bar{\mathbf{x}}\bar{\mathbf{x}}^T\right)\mathbf{u}_1\\
& = & \mathbf{u}_1^T  \left( \frac{1}{N} \sum_{n=1}^N  \left(\mathbf{x}_n - \bar{\mathbf{x}}\right)\left(\mathbf{x}_n - \bar{\mathbf{x}}\right)^T \right) \mathbf{u}_1\\
& = & \mathbf{u}_1^T\mathbf{S}\mathbf{u_1}
\end{eqnarray}

* Now, we can maximize the projected variance with respect to $\mathbf{u}_1$ while constraining $\mathbf{u}_1^T\mathbf{u}_1 = 1$.  We will do this using a Lagrange multiplier:
\begin{equation}
L = \mathbf{u}_1^T\mathbf{S}\mathbf{u_1} + \lambda_1\left(1 - \mathbf{u}_1^T\mathbf{u}_1\right)
\end{equation}

* By taking the derivative of the Lagrangian and setting it equal to zero, we get:
\begin{equation}
\mathbf{S}\mathbf{u}_1 = \lambda_1\mathbf{u_1}
\end{equation}

* We can multiply the left side by $\mathbf{u}_1^T$ and get:
	\begin{equation}
	\mathbf{u_1}^T\mathbf{S}\mathbf{u_1} = \lambda_1
	\end{equation}

* So the variance of the projected data is equal to the eigenvalue of the covariance matrix of the sample data along the direction of the eigenvector used for dimensionality reduction.
* We can incrementally add new eigenvector directions (ordered by maximal eigenvalue/variance) to project into an $M$ dimensional space where $1\leq M \leq D$

## PCA for Minimization of Mean Squared Error

* We can also look at PCA as a minimization of mean squared error.
* Consider $\mathbf{x}\in R^n$ and an orthogonal basis $\mathbf{a}$:

 \begin{equation}
 \hat{\mathbf{x}} = \sum_{i=1}^m y_i\mathbf{a}_i
 \end{equation}
 where $m < n$.
 \begin{equation}
 y_j = \mathbf{x}^T\mathbf{a}_j
 \end{equation} where $\mathbf{A}^T\mathbf{A}=\mathbf{I}$

We want to minimize the residual error:
\begin{equation}
\epsilon = \mathbf{x} - \hat{\mathbf{x}} = \sum_{j=m+1}^n y_j \mathbf{a}_j
\end{equation}
The objective we will use is the mean square residual:
\begin{eqnarray}
J &=& E\{ \|\epsilon\|^2_2\}\\
&=& E\left\{\left( \sum_{i=m+1}^n y_i \mathbf{a}_i^T\right)\left( \sum_{i=m+1}^n y_i \mathbf{a}_i\right) \right\}\\
&=&\sum_{j=m+1}^n E \{y_j^2\}\\
&=&\sum_{j=m+1}^n E \{(\mathbf{a}_j^T\mathbf{x})(\mathbf{x}^T\mathbf{a}_j)\}\\
&=& \sum_{j=m+1}^n \mathbf{a}_j^T E\{\mathbf{x}\mathbf{x}^T\}\mathbf{a}_j\\
&=& \sum_{j=m+1}^n \mathbf{a}_j^T R_x\mathbf{a}_j
\end{eqnarray}
Minimize the error and incorporate Lagrange parameters for $\mathbf{A}^T\mathbf{A}=\mathbf{I}$:
\begin{eqnarray}
\frac{\partial J}{\partial \mathbf{a}_j} &=& 2(R_x\mathbf{a}_j - \lambda_j\mathbf{a}_j) = 0 \text{ for }j = m+1 \ldots n\\
R_x\mathbf{a}_j &=& \lambda_j\mathbf{a}_j
\end{eqnarray}
So, the sum of the error is the sum of the eigenvalues of the unused eigenvectors.  So, we want to select the eigenvectors with the $m$ largest values.

## Reading Assignment

* Section 1.4
* Section 12.1
* Appendix C
* (and Appendix E for Lagrange Optimization discussed in the previous lecture)

```python

```

14_CurseDim/Curse of Dimensionality and Principal Components Analysis.ipynb


# Big Assignment 4

### A number of organizations and governments are considering policies to supply students with access to free computers, in an effort to boost learning outcomes. Suppose that you want to learn about the effect of having a free computer on test scores. For a sample of high school students, you collect data on test scores (`test` ), parent’s income (`inc`), and whether they have a computer at home (`comp`).

### Part A:
#### Suppose that a local organization held a lottery, and offered computers at random to a number of students, indicated by freecomp. Suppose that among the population who were not offered a free computer, 60% have a computer, whereas after the lottery, 90% of people offered a computer had a computer. Compare the specifications

\begin{align}
test=\beta_0+\beta_1comp+u
\end{align}

#### or

\begin{align}
test=\beta_0+\beta_1freecomp+u
\end{align}

#### Both tell you something different about the effect of computers on test scores. Describe the limitations of each approach.

Click here to type you answer to part A.

### Part B:
#### You are concerned that household income may be correlated with test scores for a variety of reasons. Suggest an alternate specification that holds income constant, and discuss how you anticipate it changing inference using each estimated $\beta_1$ above.

Click here to type you answer to part B.

### Part C:
#### Propose a “Treatment on the treated” estimator for the free computer lottery.

Click here to type you answer to part C.

### Part D:
#### You think computers might have two different effects on test scores. On the one hand, they are a productive tool which can help learning. On the other hand, they open up recreational opportunities (like social networking) which may not be very useful for learning. As a result, you wonder what the effects of a free computer would be if you could hold recreational screentime constant. That is, you would like to estimate
\begin{align}
test=\beta_0+\beta_1freecomp+\beta_2screentime+u
\end{align}

#### However, in practice you cannot observe screentime. You have two options:

#### - either you can just use whether they have a smart phone as a proxy variable for screentime (variable `Smartphone`),

#### -or you can ask the students to report how much screentime they spend per day (`repscreentime`).
#### Describe algebraically how each approach will affect estimation of $\beta_1$ and $\beta_2$ in the equation for part D, and the analogue to MLR4 which is necessary to grant unbiased estimates. Discuss different potential forms for any measurement error, and the likely consequences of that measurement error.

Click here to type you answer to part D.

### Part E:

#### Suppose instead that the organization offered free computers to the students whose parent’s income was in the bottom 20% of the school. Write down a regression equation which will implement a regression-discontinuity estimator to learn the impact of receiving a free computer. Discuss why results from that estimator may be different from the results from the randomized lottery in part A.

Click here to type you answer to part E.


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
import sympy as sym
from approx1D import least_squares_orth, comparison_plot
import matplotlib.pyplot as plt
x = sym.Symbol('x')

def efficient(f, B, s, Omega, N=10, basis='a'):
    u = B
    for i in range(N+1):
        if basis == 'a':
            psi = [sym.sin((i+1)*x)]
        elif basis == 'b':
            psi = [sym.sin((2*i+1)*x)]
        elif basis == 'c':
            psi = [sym.sin(2*(i+1)*x)]
        next_term, c = least_squares_orth(f-B, psi, Omega, False)
        u = u + next_term
        # Make only plot for i even
        if i % 2 == 0:
            comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                            legend_loc='upper left', show=False,
                            plot_title='s=%g, i=%d' % (s, i))

if __name__ == '__main__':
    s = 20  # steepness
    f = sym.tanh(s*(x-sym.pi))
    from math import pi
    Omega = [0, 2*pi]  # sym.pi did not work here

    # sin((i+1)*x) basis
    xL = Omega[0]
    xR = Omega[1]
    B = ((xR-x)*f.subs(x, xL) + (x-xL)*f.subs(x, xR))/(xR-xL)
    for exercise in 'a', 'b', 'c':
        efficient(f, B, s, Omega, N=16, basis=exercise)
        # Make movie
        cmd = 'convert -delay 200 tmp_sin*.png '
        cmd += 'tanh_sines_boundary_term_%s.gif' % exercise
        os.system(cmd)
        # Make static plots, 3 figures on 2 lines
        for ext in 'pdf', 'png':
            cmd = 'doconce combine_images %s -3 ' % ext
            cmd += 'tmp_sin00x tmp_sin02x tmp_sin04x tmp_sin08x '
            cmd += 'tmp_sin12x tmp_sin16x '
            cmd += 'tanh_sines_boundary_term_%s' % exercise
            os.system(cmd)

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/26_TANH_SINES_BOUNDARY_TERM.ipynb


# 微積分の計算について　　　N0.4　定積分の内容－1

###  学籍番号[＿＿＿＿＿＿＿＿＿]クラス[＿＿＿＿＿]　クラス番号[＿＿＿＿＿] 名前[＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿]

##### 積分の手法

$$ \int_a^b f(x)dx =\left[ F(x) \right]_{a}^{b} = F(b)-F(a)$$ 　　　　F(x)はf(x)の積分

$$ \int_a^b f(x)dx =-\int_b^a f(x)dx  $$ ,$$   \int_a^a f(x)dx =0  $$

$$ \int_a^c f(x)dx = \int_a^b f(x)dx + \int_b^c f(x)dx $$

f(x)が偶関数なら　$$   \int_{-a}^{a} f(x)dx = 2 \int_0^{a} f(x)dx$$
奇関数なら$$   \int_{-a}^{a} f(x)dx = 0 $$

```python
from sympy import *
x, n , y, a = symbols('x n y a')
init_printing()
m ='3//5'
i =0
```

 ### 例題ー1

```python
expr = x**2
itg = Integral(expr,(x,-1,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**3
itg = Integral(expr,(x,-1,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = sin(x)
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = cos(x)+sin(x)
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = 1/(x+1)
itg = Integral(expr,(x,2,4))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = sqrt(x)
itg = Integral(expr,(x,2,3))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = exp(2*x)
itg = Integral(expr,(x,2,3))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr =sqrt(1+(4/9)*x**(-2/3))
itg = Integral(expr,(x,0,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python

```


```python
import sympy as sm
```

```python
par = {}
par["alpha"] = 5.76
par["h"] = 0.5
par["b"] = 0.5
par["phi"] = 0
par["gamma"]=0.075
alpha = par["alpha"]
h = par["h"]
b = par["b"]
phi = par["phi"]
gamma = par["gamma"]
```

```python

```

```python
sm.init_printing(use_unicode=True)
pi_t = sm.symbols('pi_t')
pi_t1 = sm.symbols('pi_t-1')

v_t = sm.symbols('v_t')
y_t = sm.symbols('y_t')
y_t1 = sm.symbols('y_t-1')
s_t = sm.symbols('s_t')
s_t1 = sm.symbols('s_t-1')
ad = 1/(h*alpha)*(v_t-(1+b*alpha)*y_t)
sras = pi_t1 + gamma*y_t - phi*gamma*y_t1 + s_t - phi*s_t1
```

```python
AD = sm.Eq(ad, pi_t)
SRAS = sm.Eq(sras, pi_t)
```

```python
SRAS
```

```python
ADSRAS=sm.Eq(sras,ad)

```

```python
y_star = sm.solve(ADSRAS, y_t)
```

```python
y_star
```

```python

pi_star = 1/(h * alpha) * (v_t - (1 + b * alpha) * y_star[0])
pi_star
```

```python

```


## 유클리드 유사도 vs 코사인 유사도
- 유클리드 유사도 구하는 함수 구현하기
- 코사인 유사도 구하는 함수 구현하기
- 코드설명
- 두 결과 비교
- 응용분야 예시

#### Similarity
```The similairt measure is the measure of how much alike
two data objects are.

Similarity measure in a data mining context is a distance with
dimensions representing features of the objects.

If this distance is small, it will be the high degree of similarity where large distance will be the low degree of similarity.```

##### 1.  Euclidean distance
- literally distance b/w two vectors

```python
import sympy
sympy.init_printing(use_latex='mathjax')
```

```python
from math import*

def e_d(x,y) :
    "e_d stands for Euclidean distance"
    c = [(a-b)**2 for a,b in zip(x,y)]
    return sqrt(sum(c))

print(e_d([1,2], [4,1]))
```

    3.1622776601683795

```python
a = np.array([1,2])
b = np.array([4,1])
c = a-b

plt.plot(a[0],a[1], 'ro')
plt.plot(b[0],b[1], 'ro')

plt.text(0.6, 1.6 , "$a$", fontdict={"size" : 18})
plt.text(3.6, 0.6 , "$b$", fontdict={"size" : 18})

plt.annotate('', xy=a, xytext=(0,0), arrowprops=dict(facecolor='gray', ls="dashed"))
plt.annotate('', xy=b, xytext=(0,0), arrowprops=dict(facecolor='gray', ls="dashed"))
plt.arrow(4,1,-3,1, width=0.008)

plt.xticks(range(6))
plt.yticks(range(6))
plt.show()
```

```python
e_d([1,2], [4,1]), sqrt(10)
```

$$\left ( 3.1622776601683795, \quad 3.1622776601683795\right )$$

##### 2. Cosine similarity
- Cosine similarty metric finds the nomalized dot product of two attributes.

```python
def norm(x) :
    return sqrt(sum([ a**2 for a in x]))

def dot_product(x,y):
    return sum(a*b for a,b in zip(x,y))

def c_s(x,y):
    """c_s stands for consine similarity"""
    denominator = norm(x)*norm(y)
    numerator = dot_product(x,y)
    return numerator / denominator

def c_d(*arg):
    "c_d stands for consine distance"
    return 1 - c_s(x,y)

print(c_s([1,2],[4,1]))
print(c_d([1,2],[4,1]))
```

    0.6507913734559685
    0.3492086265440315

```python

```

#### Example

```python
index = ["D1","D2","D3","D4"]
columns = ["team","coach","hockey","baseball","soccer","penalty","score","win","loss","season"]
datas = {
    "team" : [5,3,0,0],
    "coach" : [0,0,7,1],
    "hockey" : [3,2,0,0],
    "baseball" : [0,0,2,0],
    "soccer" : [2,1,1,1],
    "penalty" : [0,1,0,2],
    "score" : [0,0,0,2],
    "win" : [2,1,3,0],
    "loss" : [0,0,0,3],
    "season" : [0,1,0,0]
}
```

```python
df = pd.DataFrame(index=index, columns=columns, data=datas)
df
```

      team
      coach
      hockey
      baseball
      soccer
      penalty
      score
      win
      loss
      season

      D1
      5
      0
      3
      0
      2
      0
      0
      2
      0
      0

      D2
      3
      0
      2
      0
      1
      1
      0
      1
      0
      1

      D3
      0
      7
      0
      2
      1
      0
      0
      3
      0
      0

      D4
      0
      1
      0
      0
      1
      2
      2
      0
      3
      0

```python
#cosine similarity
```

```python
def norm(x) :
    return sqrt(sum([ a**2 for a in x]))

def dot_product(x,y):
    return sum(a*b for a,b in zip(x,y))

def c_s(x,y):
    """c_s stands for consine similarity"""
    denominator = norm(x)*norm(y)
    numerator = dot_product(x,y)
    return numerator / denominator

def c_d(*arg):
    "c_d stands for consine distance"
    return 1 - c_s(x,y)

print(c_s([1,2],[4,1]))
print(c_d([1,2],[4,1]))
```

    0.6507913734559685
    0.3492086265440315

```python
C_D1_D2 = c_s(df.loc["D1"],df.loc["D2"])
C_D1_D2
```

$$0.9356014857063997$$

```python
C_D1_D3 = c_s(df.loc["D1"],df.loc["D3"])
C_D1_D3
```

$$0.1555231582719478$$

```python
C_D1_D4 = c_s(df.loc["D1"],df.loc["D4"])
C_D1_D4
```

$$0.07079923254047886$$

```python
C_D2_D3 = c_s(df.loc["D2"],df.loc["D3"])
C_D2_D3
```

$$0.12222646627042817$$

```python
C_D2_D4 = c_s(df.loc["D2"],df.loc["D4"])
C_D2_D4
```

$$0.16692446522239712$$

```python
C_D3_D4 = c_s(df.loc["D3"],df.loc["D4"])
C_D3_D4
```

$$0.23122932520643197$$

```python
#Euclidean similarty
```

```python
from math import*

def e_d(x,y) :
    "e_d stands for Euclidean distance"
    c = [(a-b)**2 for a,b in zip(x,y)]
    return sqrt(sum(c))

print(e_d([1,2], [4,1]))
```

    3.1622776601683795

```python
E_D1_D2 = e_d(df.loc["D1"],df.loc["D2"])
E_D1_D2
```

$$3.0$$

```python
E_D1_D3 = e_d(df.loc["D1"],df.loc["D3"])
E_D1_D3
```

$$9.433981132056603$$

```python
E_D1_D4 = e_d(df.loc["D1"],df.loc["D4"])
E_D1_D4
E_D1_D4
```

$$7.54983443527075$$

```python
E_D2_D3 = e_d(df.loc["D2"],df.loc["D3"])
E_D2_D3
```

$$8.48528137423857$$

```python
E_D2_D4 = e_d(df.loc["D2"],df.loc["D4"])
E_D2_D4
```

$$5.477225575051661$$

```python
E_D3_D4 = e_d(df.loc["D3"],df.loc["D4"])
E_D3_D4
```

$$8.12403840463596$$

```python
data1 = {"D1_D2" : [C_D1_D2, E_D1_D2] ,
         "D1_D3" : [C_D1_D3, E_D1_D3],
         "D1_D4" : [C_D1_D4, E_D1_D4],
         "D2_D3" : [C_D2_D3, E_D2_D3],
         "D2_D4" : [C_D2_D4, E_D2_D4],
         "D3_D4" : [C_D3_D4, E_D3_D4] }
```

```python
df2 = pd.DataFrame(columns=["D1_D2","D1_D3","D1_D4","D2_D3","D2_D4","D3_D4"], index=["Cosine_similarity","Euclidean_similarty"], data=data1)
df2
```

      D1_D2
      D1_D3
      D1_D4
      D2_D3
      D2_D4
      D3_D4

      Cosine_similarity
      0.935601
      0.155523
      0.070799
      0.122226
      0.166924
      0.231229

      Euclidean_similarty
      3.000000
      9.433981
      7.549834
      8.485281
      5.477226
      8.124038

```python

```

```python

```

```python
data1 = {"D1_D2" : [C_D1_D2, E_D1_D2**2] ,
         "D1_D3" : [C_D1_D3, E_D1_D3**2],
         "D1_D4" : [C_D1_D4, E_D1_D4**2],
         "D2_D3" : [C_D2_D3, E_D2_D3**2],
         "D2_D4" : [C_D2_D4, E_D2_D4**2],
         "D3_D4" : [C_D3_D4, E_D3_D4**2] }
```

```python
df2 = pd.DataFrame(columns=["D1_D2","D1_D3","D1_D4","D2_D3","D2_D4","D3_D4"], index=["Cosine_similarity","Euclidean_similarty"], data=data1)
df2
```

      D1_D2
      D1_D3
      D1_D4
      D2_D3
      D2_D4
      D3_D4

      Cosine_similarity
      0.935601
      0.155523
      0.070799
      0.122226
      0.166924
      0.231229

      Euclidean_similarty
      9.000000
      89.000000
      57.000000
      72.000000
      30.000000
      66.000000

```python

```

```python

```

```python

```

```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import euclidean_distances
```

```python
corpus =[
    "All my cats in a row",
    "When my cat sits down, she looks like a Furby toy!",
    "The cat from outer space",
    "Sunshine loves to sit like this for some reason."
]
```

```python
vectorizer = CountVectorizer()
features = vectorizer.fit_transform(corpus).todense()
print(vectorizer.vocabulary_ )
```

    {'all': 0, 'my': 11, 'cats': 2, 'in': 7, 'row': 14, 'when': 25, 'cat': 1, 'sits': 17, 'down': 3, 'she': 15, 'looks': 9, 'like': 8, 'furby': 6, 'toy': 24, 'the': 21, 'from': 5, 'outer': 12, 'space': 19, 'sunshine': 20, 'loves': 10, 'to': 23, 'sit': 16, 'this': 22, 'for': 4, 'some': 18, 'reason': 13}

```python
for i in features:
    print(i)
```

    [[1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0]]
    [[0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1]]
    [[0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0]]
    [[0 0 0 0 1 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 1 1 0 0]]

```python
for i in features :
    print(euclidean_distances(features[0], i))
```

    [[0.]]
    [[3.60555128]]
    [[3.16227766]]
    [[3.74165739]]

```python
for i in features:
    print(cosine_similarity(features[0], i))
```

01_linearalgebra/01_Euclidean Distance & Cosine distance.ipynb


# CNTK 207: Sampled Softmax

For classification and prediction problems a typical criterion function is cross-entropy with softmax. If the number of output classes is high the computation of this criterion and the corresponding gradients could be quite costly. Sampled Softmax is a heuristic to speed up training in these cases. (see: [Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model](http://www.iro.umontreal.ca/~lisa/pointeurs/importance_samplingIEEEtnn.pdf), [Exploring the Limits of Language Modeling](https://arxiv.org/pdf/1602.02410v1.pdf), [What is Candidate Sampling](https://www.tensorflow.org/extras/candidate_sampling.pdf))

#### Select the notebook runtime environment devices / settings

Before we dive into the details we run some setup that is required for automated testing of this notebook.

```python
from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
from __future__ import division

import os
import cntk as C
```

In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU).

```python
# Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        C.device.try_set_default_device(C.device.cpu())
    else:
        C.device.try_set_default_device(C.device.gpu(0))
```

## Basics

The softmax function is used in neural networks if we want to interpret the network output as a probability distribution over a set of classes $C$ with $|C|=N_C$.

Softmax maps an $N_C$-dimensional vector $z$, which has unrestricted values, to an $N_C$ dimensional vector $p$ with non-negative values that sum up to 1 so that they can be interpreted as probabilities. More precisely:

$$
\begin{align}
p_i &= softmax(z, i)\\
    &= \frac{exp(z_i)}{\sum_{k\in C} exp(z_k)}\\
\end{align}
$$

In what follows we assume that the input $z$ to the softmax is computed from some hidden vector $h$ of dimension $N_h$  in a specific way, namely:

$$ z = W h + b $$

where $W$ is a learnable weight matrix of dimension $(N_c, N_h)$ and $b$ is a learnable bias vector.
We restrict ourselves to this specific choice of $z$ because it helps in implementing an efficient sampled softmax.

In a typical use-case like for example a recurrent language model, the hidden vector $h$ would be the output of the recurrent layers and $C$ would be the set of words to predict.

As a training criterion, we use cross-entropy which is a function of the expected (true) class $t\in C$ and the probability predicted for it:

$$cross\_entropy := -log(p_t)$$

## Sampled Softmax from the outside

For the normal softmax the CNTK Python-api provides the function [cross_entropy_with_softmax](https://cntk.ai/pythondocs/cntk.ops.html?highlight=softmax#cntk.ops.cross_entropy_with_softmax). This takes as input the $N_C$-dimensional vector $z$. As mentioned for our sampled softmax implementation we assume that this z is computed by $ z = W h + b $. In sampled softmax this has to be part of the whole implementation of the criterion.

Below we show the code for `cross_entropy_with_sampled_softmax_and_embedding`. Let’s look at the signature first.

One fundamental difference to the corresponding function in the Python-api (`cross_entropy_with_softmax`) is that in the Python api function the input corresponds to $z$ and must have the same dimension as the target vector, while in cross_entropy_with_full_softmax the input corresponds to our hidden vector $h$ can have any dimension (hidden_dim).
Actually, hidden_dim will be typically much lower than the dimension of the target vector.

We also have some additional parameters `num_samples, sampling_weights, allow_duplicates` that control the random sampling.
Another difference to the api function is that we return a triple (z, cross_entropy_on_samples, error_on_samples).

We will come back to the details of the implementation below.

```python

# Creates a subgraph computing cross-entropy with sampled softmax.
def cross_entropy_with_sampled_softmax_and_embedding(
    hidden_vector,            # Node providing hidden input
    target_vector,            # Node providing the expected labels (as sparse vectors)
    num_classes,              # Number of classes
    hidden_dim,               # Dimension of the hidden vector
    num_samples,              # Number of samples to use for sampled softmax
    sampling_weights,         # Node providing weights to be used for the weighted sampling
    allow_duplicates = True,  # Boolean flag to control whether to use sampling with replacemement
                              # (allow_duplicates == True) or without replacement.
    ):
    # define the parameters learnable parameters
    b = C.Parameter(shape = (num_classes, 1), init = 0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())

    # Define the node that generates a set of random samples per minibatch
    # Sparse matrix (num_samples * num_classes)
    sample_selector = C.random_sample(sampling_weights, num_samples, allow_duplicates)

    # For each of the samples we also need the probablity that it in the sampled set.
    inclusion_probs = C.random_sample_inclusion_frequency(sampling_weights, num_samples, allow_duplicates) # dense row [1 * vocab_size]
    log_prior = C.log(inclusion_probs) # dense row [1 * num_classes]

    # Create a submatrix wS of 'weights
    W_sampled = C.times(sample_selector, W) # [num_samples * hidden_dim]
    z_sampled = C.times_transpose(W_sampled, hidden_vector) + C.times(sample_selector, b) - C.times_transpose (sample_selector, log_prior)# [num_samples]

    # Getting the weight vector for the true label. Dimension hidden_dim
    W_target = C.times(target_vector, W) # [1 * hidden_dim]
    z_target = C.times_transpose(W_target, hidden_vector) + C.times(target_vector, b) - C.times_transpose(target_vector, log_prior) # [1]

    z_reduced = C.reduce_log_sum_exp(z_sampled)

    # Compute the cross entropy that is used for training.
    # We don't check whether any of the classes in the random samples conincides with the true label, so it might
    # happen that the true class is counted
    # twice in the normalising demnominator of sampled softmax.
    cross_entropy_on_samples = C.log_add_exp(z_target, z_reduced) - z_target

    # For applying the model we also output a node providing the input for the full softmax
    z = C.times_transpose(W, hidden_vector) + b
    z = C.reshape(z, shape = (num_classes))

    zSMax = C.reduce_max(z_sampled)
    error_on_samples = C.less(z_target, zSMax)
    return (z, cross_entropy_on_samples, error_on_samples)

```

To give a better idea of what the inputs and outputs are and how this all differs from the normal softmax we give below a corresponding function using normal softmax:

```python
# Creates subgraph computing cross-entropy with (full) softmax.
def cross_entropy_with_softmax_and_embedding(
    hidden_vector,  # Node providing hidden input
    target_vector,  # Node providing the expected labels (as sparse vectors)
    num_classes,    # Number of classes
    hidden_dim      # Dimension of the hidden vector
    ):
    # Setup bias and weights
    b = C.Parameter(shape = (num_classes, 1), init = 0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())

    z = C.reshape( C.times_transpose(W, hidden_vector) + b, (1, num_classes))

    # Use cross_entropy_with_softmax
    cross_entropy = C.cross_entropy_with_softmax(z, target_vector)

    zMax = C.reduce_max(z)
    zT = C.times_transpose(z, target_vector)
    error_on_samples = C.less(zT, zMax)

    return (z, cross_entropy, error_on_samples)
```

As you can see the main differences to the api function `cross_entropy_with_softmax` are:
* We include the mapping $ z = W h + b $ into the function.
* We return a triple (z, cross_entropy, error_on_samples) instead of just returning the cross entropy.

## A toy example

To explain how to integrate sampled softmax let us look at a toy example. In this toy example we first transform one-hot input vectors via some random projection into a lower dimensional vector $h$. The modeling task is to reverse this mapping using (sampled) softmax. Well, as already said this is a toy example.

```python
import numpy as np
from math import log, exp, sqrt
from cntk.logging import ProgressPrinter
import timeit

# A class with all parameters
class Param:
    # Learning parameters
    learning_rate = 0.03
    minibatch_size = 100
    num_minbatches = 100
    test_set_size = 1000
    momentum_time_constant = 5 * minibatch_size
    reporting_interval = 10
    allow_duplicates = False

    # Parameters for sampled softmax
    use_sampled_softmax = True
    use_sparse = True
    softmax_sample_size = 10

    # Details of data and model
    num_classes = 50
    hidden_dim = 10

data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

# Creates random one-hot vectors of dimension 'num_classes'.
# Returns a tuple with a list of one-hot vectors, and list with the indices they encode.
def get_random_one_hot_data(num_vectors):
    indices = np.random.choice(
        range(Param.num_classes),
        size=num_vectors,
        p = data_sampling_distribution()).reshape((num_vectors, 1))
    list_of_vectors = C.Value.one_hot(indices, Param.num_classes)
    return (list_of_vectors, indices.flatten())

# Create a network that:
# * Transforms the input one hot-vectors with a constant random embedding
# * Applies a linear decoding with parameters we want to learn
def create_model(labels):
    # random projection matrix
    random_data = np.random.normal(scale = sqrt(1.0/Param.hidden_dim), size=(Param.num_classes, Param.hidden_dim)).astype(np.float32)
    random_matrix = C.constant(shape = (Param.num_classes, Param.hidden_dim), value = random_data)

    h = C.times(labels, random_matrix)

    # Connect the latent output to (sampled/full) softmax.
    if Param.use_sampled_softmax:
        sampling_weights = np.asarray(softmax_sampling_weights(), dtype=np.float32)
        sampling_weights.reshape((1, Param.num_classes))
        softmax_input, ce, errs = cross_entropy_with_sampled_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim,
            Param.softmax_sample_size,
            softmax_sampling_weights(),
            Param.allow_duplicates)
    else:
        softmax_input, ce, errs = cross_entropy_with_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim)

    return softmax_input, ce, errs

def train(do_print_progress):
    labels = C.input_variable(shape = Param.num_classes, is_sparse = Param.use_sparse)
    z, cross_entropy, errs = create_model(labels)

    # Setup the trainer
    learning_rate_schedule = C.learning_rate_schedule(Param.learning_rate, C.UnitType.sample)
    momentum_schedule = C.momentum_as_time_constant_schedule(Param.momentum_time_constant)
    learner = C.momentum_sgd(z.parameters, learning_rate_schedule, momentum_schedule, True)
    progress_writers = None
    if do_print_progress:
        progress_writers = [ProgressPrinter(freq=Param.reporting_interval, tag='Training')]
    trainer = C.Trainer(z, (cross_entropy, errs), learner, progress_writers)

    minbatch = 0
    average_cross_entropy = compute_average_cross_entropy(z)
    minbatch_data = [0] # store minibatch values
    cross_entropy_data = [average_cross_entropy] # store cross_entropy values

    # Run training
    t_total= 0

    # Run training
    for minbatch in range(1,Param.num_minbatches):
        # Specify the mapping of input variables in the model to actual minibatch data to be trained with
        label_data, indices = get_random_one_hot_data(Param.minibatch_size)
        arguments = ({labels : label_data})

        # If do_print_progress is True, this will automatically print the progress using ProgressPrinter
        # The printed loss numbers are computed using the sampled softmax criterion
        t_start = timeit.default_timer()
        trainer.train_minibatch(arguments)
        t_end = timeit.default_timer()

        t_delta = t_end - t_start
        samples_per_second = Param.minibatch_size / t_delta

        # We ignore the time measurements of the first two minibatches
        if minbatch > 2:
            t_total += t_delta

        # For comparison also print result using the full criterion
        if minbatch % Param.reporting_interval == int(Param.reporting_interval/2):
            # memorize the progress data for plotting
            average_cross_entropy = compute_average_cross_entropy(z)
            minbatch_data.append(minbatch)
            cross_entropy_data.append(average_cross_entropy)

            if do_print_progress:
                print("\nMinbatch=%d Cross-entropy from full softmax = %.3f perplexity = %.3f samples/s = %.1f"
                    % (minbatch, average_cross_entropy, exp(average_cross_entropy), samples_per_second))

    # Number of samples we measured. First two minbatches were ignored
    samples_measured = Param.minibatch_size * (Param.num_minbatches - 2)
    overall_samples_per_second = samples_measured / t_total
    return (minbatch_data, cross_entropy_data, overall_samples_per_second)

def compute_average_cross_entropy(softmax_input):
    vectors, indices = get_random_one_hot_data(Param.test_set_size)
    total_cross_entropy = 0.0
    arguments = (vectors)
    z = softmax_input.eval(arguments).reshape(Param.test_set_size, Param.num_classes)

    for i in range(len(indices)):
        log_p = log_softmax(z[i], indices[i])
        total_cross_entropy -= log_p

    return total_cross_entropy / len(indices)

# Computes log(softmax(z,index)) for a one-dimensional numpy array z in an numerically stable way.
def log_softmax(z,    # numpy array
                index # index into the array
            ):
    max_z = np.max(z)
    return z[index] - max_z - log(np.sum(np.exp(z - max_z)))

np.random.seed(1)

print("start...")
train(do_print_progress = True)
print("done.")
```

    start...
    Learning rate per sample: 0.03
    Momentum per sample: 0.9980019986673331

    Minbatch=5 Cross-entropy from full softmax = 3.888 perplexity = 48.807 samples/s = 7652.9
     Minibatch[   1-  10]: loss = 2.389856 * 1000, metric = 89.10% * 1000;

    Minbatch=15 Cross-entropy from full softmax = 3.528 perplexity = 34.042 samples/s = 8891.7
     Minibatch[  11-  20]: loss = 2.062891 * 1000, metric = 54.30% * 1000;

    Minbatch=25 Cross-entropy from full softmax = 3.103 perplexity = 22.271 samples/s = 8691.4
     Minibatch[  21-  30]: loss = 1.720161 * 1000, metric = 37.20% * 1000;

    Minbatch=35 Cross-entropy from full softmax = 2.823 perplexity = 16.823 samples/s = 8585.7
     Minibatch[  31-  40]: loss = 1.488539 * 1000, metric = 27.60% * 1000;

    Minbatch=45 Cross-entropy from full softmax = 2.492 perplexity = 12.085 samples/s = 8477.3
     Minibatch[  41-  50]: loss = 1.219984 * 1000, metric = 10.80% * 1000;

    Minbatch=55 Cross-entropy from full softmax = 2.292 perplexity = 9.892 samples/s = 7757.4
     Minibatch[  51-  60]: loss = 1.068673 * 1000, metric = 12.00% * 1000;

    Minbatch=65 Cross-entropy from full softmax = 2.092 perplexity = 8.100 samples/s = 4728.7
     Minibatch[  61-  70]: loss = 0.971144 * 1000, metric = 7.00% * 1000;

    Minbatch=75 Cross-entropy from full softmax = 1.929 perplexity = 6.884 samples/s = 9467.1
     Minibatch[  71-  80]: loss = 0.905958 * 1000, metric = 6.80% * 1000;

    Minbatch=85 Cross-entropy from full softmax = 1.775 perplexity = 5.898 samples/s = 10616.4
     Minibatch[  81-  90]: loss = 0.794641 * 1000, metric = 6.20% * 1000;

    Minbatch=95 Cross-entropy from full softmax = 1.665 perplexity = 5.286 samples/s = 9264.1
    done.

In the above code we use two different methods to report training progress:
1. Using a function that computes the average cross entropy on full softmax.
2. Using the built-in ProgressPrinter

ProgressPrinter reports how the value of the training criterion changes over time.
In our case the training criterion is cross-entropy from **sampled** softmax.
The same is true for the error rate computed by progress printer, this is computed only for true-class vs sampled-classes and will therefore underestimate the true error rate.

Therefore while ProgressPrinter already gives us some idea how training goes on, if we want to compare the behavior for different sampling strategies (sample size, sampling weights, ...) we should not rely on numbers that are computed only using the sampled subset of classes.

## Importance sampling

Often the we don't have uniform distribution for the classes on the output side. The typical example is when we have words as output classes. A typical example are words where e.g. 'the' will be much more frequent than most others.

In such cases one often uses a non uniform distribution for drawing the samples in sampled softmax but instead increases the sampling weight for the frequent classes. This is also called importane sampling.
In our example the sampling distribution is controlled by the weight array `softmax_sampling_weights`.

As an example let's look at the case where the classes are distrubted according to zipf-distrubtion like:
$$
p[i] \propto \frac{1}{i+5},
$$
actually we use this distribution already in our example.

How does training behavior change if we switch uniform sampling to sampling with the zipfian distribution in sampled softmax?

```python
# We want to lot the data
import matplotlib.pyplot as plt
%matplotlib inline

# Define weights of zipfian distributuion
def zipf(index):
    return 1.0 / (index + 5)

# Use zipifian distribution for the classes
def zipf_sampling_weights():
    return np.asarray([ zipf(i) for i in range(Param.num_classes)], dtype=np.float32)

data_sampling_distribution = lambda: zipf_sampling_weights() / np.sum(zipf_sampling_weights())

print("start...")

# Train using uniform sampling (like before)
np.random.seed(1)
softmax_sampling_weights = lambda: np.repeat(1.0/Param.num_classes, Param.num_classes)
minibatch_data, cross_entropy_data, _ = train(do_print_progress = False)

# Train using importance sampling
np.random.seed(1)
softmax_sampling_weights = zipf_sampling_weights
minibatch_data2, cross_entropy_data2, _ = train(do_print_progress = False)

plt.plot(minibatch_data, cross_entropy_data, 'r--',minibatch_data, cross_entropy_data2, 'b--')
plt.xlabel('number of mini-batches')
plt.ylabel('cross entropy')
plt.show()

```

In the example above we compare uniform sampling (red) vs sampling with the same distribution the classes have (blue).
You will need to experiment to find the best settings for all the softmax parameters.

## What speedups to expect?

The speed difference between full softmax and sampled softmax in terms of training instances depends strongly on the concrete settings, namely
* Number of classes. Typically the speed-up will increase the more output classes you have.
* Number of samples used in sampled softmax
* Dimension of hiddlen layer input
* Minibatch size
* Hardware

Also you need to test how much you can reduce sample size without degradation of the result.

```python
print("start...")

# Reset parameters
class Param:
    # Learning parameters
    learning_rate = 0.03
    minibatch_size = 8
    num_minbatches = 100
    test_set_size = 1 # we are only interrested in speed
    momentum_time_constant = 5 * minibatch_size
    reporting_interval = 1000000 # Switch off reporting to speed up
    allow_duplicates = False

    # Parameters for sampled softmax
    use_sampled_softmax = True
    use_sparse = True
    softmax_sample_size = 10

    # Details of data and model
    num_classes = 50000
    hidden_dim = 10

data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)
softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

sample_sizes = [5, 10, 100, 1000]
speed_with_sampled_softmax = []

# Get the speed with sampled softmax for different sizes
for sample_size in sample_sizes:
    print("Measuring speed of sampled softmax for sample size %d ..." % (sample_size))
    Param.use_sampled_softmax = True
    Param.softmax_sample_size = sample_size
    _, _,  samples_per_second = train(do_print_progress = False)
    speed_with_sampled_softmax.append(samples_per_second)

# Get the speed with full softmax
Param.use_sampled_softmax = False
print("Measuring speed of full softmax ...")
_, _,  samples_per_second = train(do_print_progress = False)
speed_without_sampled_softmax = np.repeat(samples_per_second, len(sample_sizes))

# Plot the speed of sampled softmax (blue) as a function of sample sizes
# and compare it to the speed with full softmax (red).
plt.plot(sample_sizes, speed_without_sampled_softmax, 'r--',sample_sizes, speed_with_sampled_softmax, 'b--')
plt.xlabel('softmax sample size')
plt.ylabel('speed: instances / second')
plt.title("Speed 'sampled softmax' (blue) vs. 'full softmax' (red)")
plt.ylim(ymin=0)
plt.show()

```


# Multi-Layer Networks

Now that we have tried our hand at some single-layer nets, let's see how they _stack up_ compared to multi-layer nets.

We will be exploring the basic concepts of learning non-linear functions using the classic XOR problem, and then explore the basics of "deep learning" technology on this problem as well. After that, we will see if a multi-layer approach can perform better than single-layer approach on some data sets that we have seen before. In particular, we will try to explore the wider vs. deeper issue, and see if ReLU really helps.

For now, let's look into the XOR problem, and see what insights we can gain when comparing single- and multi-layer network approaches.

```python
# NN-Tools
import numpy as np
import keras

# Visualization
from IPython.display import SVG
from IPython.display import display
from keras.utils.vis_utils import model_to_dot

# Printing
from sympy import *
init_printing(use_latex=True)

# Plotting
import matplotlib.pyplot as plt
%matplotlib inline
```

    Using TensorFlow backend.

Let's create the XOR data set. As we mentioned in class, _avoiding zeros_ in the **input** patterns is a good general approach for stable learning behavior.

```python
# XOR data set
X = np.array([[-1,-1],[-1,1],[1,-1],[1,1]])
display(Matrix(X))
Y = np.array([0,1,1,0])
display(Matrix(Y))
```

We are performing a classification task since there are **two discrete targets**, so using **binary cross-entropy** error at the output layer makes sense. Also, binary cross-entropy suggest the **sigmoid** activation function for the **output** unit since it is comparing only two classes. Given that the hyperbolic tangent (tanh) activation function is a decent choice for two-layer networks, let's create a network with a hidden layer consisting of units which use this activation function. Also, we will keep it rather small: just two units.

```python
# Multi-layer net with tanh hidden layer
model = keras.models.Sequential()
model.add(keras.layers.Dense(2,input_dim=2,activation='tanh'))
model.add(keras.layers.Dense(1,activation='sigmoid'))
model.compile(loss='binary_crossentropy',optimizer='adam')
print(model.summary())
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_3 (Dense)              (None, 2)                 6
    _________________________________________________________________
    dense_4 (Dense)              (None, 1)                 3
    =================================================================
    Total params: 9
    Trainable params: 9
    Non-trainable params: 0
    _________________________________________________________________
    None

Since the input vectors are of length 2, and the hidden layer is of size 2, there will be a 2x2=4 element weight matrix connecting the input layer to the hidden layer. Also, each hidden unit will have a bias weight (6 weights total so far). The hidden to output layer weight matrix will be 2x1=2, and the output unit has its own bias weight as well (total of 9, altogether).

```python
for i in model.get_weights():
    display(Matrix(i))
```

You can see that for tanh units, the bias weight is initialized to zero. We know that zero-element weights can be problematic in some cases, but remember that the bias is **always** used and therefore has access to the full delta for the units. They will only differentiate when required by the task at-hand.

If we want a graph-like representation of our network, we can use some functions from `keras.utils.viz_utils` and the `SVD()` function from `IPython.display` to create such a representation:

```python
SVG(model_to_dot(model).create(prog='dot', format='svg'))
```

Not so interesting really for this simple network, but will be more informative for more complex models that we make later in the semester.

Before we get to training, let's see if we can set up a visualization tool that will help us understand how the network is behaving. Simple examples can be very insightful since they can often be investigated more thoroughly than a more complex example, given that the simple example contains the interesting elements that we are looking for.

First, we will create a **meshgrid** of data points (i.e. feature vectors) which _cover_ the space of input vectors for the XOR task thoroughly. This isn't easy to do in a high-dimensional space, since the space requirements grow exponentially with the dimensionality of the input vectors. However, just two dimensions can be visualized and covered with a reasonable number of points.

The `linspace()` function will be used again to create a set of points, now ranging from -2 to 2 (20 distinct values in that range). Next, we use the `meshgrid()` function to create all possible combinations of those values (20x20=400 points!). The `xgrid` and `ygrid` data structures are each 20x20 matrices, where each point we want to visualize will be a combination of corresponding values from each matrix: $\left[ xgrid_{ij}, ygrid_{ij} \right]$ for all i,j. However, this means we need to flatten each of those matrices and combine their corresponding elements into these 2-dimensional vectors. We use a combination of `ravel()` (to perform the flattening) and `vstack()` (to perform the concatenation). The result will actually be the transpose of the final data matrix that we want, so don't forget the `T` at the end.

Now, we will plot all of those data points in green just to show the grid we have made. However, we will also then plot the XOR input vectors (X) colored by their corresponding class labels (0,1), and also a little larger to make them stand out. In this case, red is zero, and blue is one.

```python
# Sample plot of classification space
xpoints = np.linspace(-2,2,20)
ypoints = np.linspace(-2,2,20)
xgrid, ygrid = np.meshgrid(xpoints,ypoints)
positions = np.vstack([xgrid.ravel(),ygrid.ravel()]).T

# Green grid points where we will evaluate the network
# outputs.
plt.plot(positions[:,0],positions[:,1],'go')

# Color the zero-expected outputs red
plt.plot(X[[0,3],0],X[[0,3],1],'ro',markersize=10)
# Color the one-expected outputs blue
plt.plot(X[[1,2],0],X[[1,2],1],'bo',markersize=10)
plt.show()
```

Now, let's use this data to visualize how the XOR network behaves. That is, we can use the visualization to examine the current decision boundary that the network is using. You may ask, how could there be a boundary before training? Well, we initialize the weights in the network randomly, so while probably not very useful, the boundary is already there.

For this operation, we need to use the `predict_classes()` function from the model. This is similar to "testing" the neural network, but instead of calculating a loss or accuracy value, we prefer to just see what class label (`predictions`) was assigned to each input vector. We provide all of the grid `positions` calculated above and each one gets assigned to a class of 0 or 1. We can then separate the data into two sets: those input vectors classified as a zero and those input vectors classified as a one. We perform a slice using a boolean operator this time, and then select the matching rows from the `positions` matrix and create two non-overlapping matrices of vectors: `zeros` and `ones`.

Now, rather than coloring them green, we can color them according to the class label that the network _predicted_ for each vector in the grid. By making the XOR vectors a little larger on the plot, it's easy to separate them from the others in the grid as well.

```python
# Let's color those points by the classification labels...
predictions = model.predict_classes(positions)[:,0]
zeros = positions[predictions==0,:]
ones = positions[predictions==1,:]

# Color predictions by class
plt.plot(zeros[:,0],zeros[:,1],'ro')
plt.plot(ones[:,0],ones[:,1],'bo')

# Color the zero-expected outputs red
plt.plot(X[[0,3],0],X[[0,3],1],'ro',markersize=10)
# Color the one-expected outputs blue
plt.plot(X[[1,2],0],X[[1,2],1],'bo',markersize=10)

plt.show()
```

The grid point covering let's us visualize the decision boundary determined by the network, since the grid points will change from red to blue or vice-versa.

As you can see, the network starts with a suboptimal decision boundary for the task (as expected). Now, let's see if the network can learn to solve the problem.

```python
# Train it!
history = model.fit(X, Y,
    batch_size=1,
    epochs=1000,
    verbose=0)
```

```python
# summarize history for loss
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()
```

Now that we have trained the network for a little while, let's see what the network has learned. We can perform the same steps as above, but using predictions from the network _after training_.

```python
# Let's color those points by the classification labels...
predictions = model.predict_classes(positions)[:,0]
zeros = positions[predictions==0,:]
ones = positions[predictions==1,:]

# Color predictions by class
plt.plot(zeros[:,0],zeros[:,1],'ro')
plt.plot(ones[:,0],ones[:,1],'bo')

# Color the zero-expected outputs red
plt.plot(X[[0,3],0],X[[0,3],1],'ro',markersize=10)
# Color the one-expected outputs blue
plt.plot(X[[1,2],0],X[[1,2],1],'bo',markersize=10)

plt.show()
```

So, the tanh hidden units were able to provide good boundaries that the output unit could use to decipher the class structure in the data, even in a non-linear manner. If you reinitialized the network weights, you might find that the learned boundary is not always the same. Some problems can be solved in different ways, and therefore different networks may utilize different ways of partitioning the feature space to solve them even if being trained on _the same data_.

## Enter the ReLU

Let's try to see what happens if we utilize our new activation function, ReLU, which we now know has some nice properties for solving complex problems using deeper networks.

The same approach will be used here as was used above, so we will only replace the tanh function with the ReLU activation function. Remember that it has been suggested in the deep learning literature, that initializing the bias on ReLU units to a small positive value for the bias. We will use that technique here to demonstrate how this might work, but it _may not always be a good idea_. Keep that in mind during the exercises below...

```python
# Multi-layer net with ReLU hidden layer
model = keras.models.Sequential()
model.add(keras.layers.Dense(2,input_dim=2,activation='relu',
                             bias_initializer=keras.initializers.Constant(0.1)))
model.add(keras.layers.Dense(1,activation='sigmoid'))
model.compile(loss='binary_crossentropy',optimizer='adam')
print(model.summary())
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_21 (Dense)             (None, 2)                 6
    _________________________________________________________________
    dense_22 (Dense)             (None, 1)                 3
    =================================================================
    Total params: 9
    Trainable params: 9
    Non-trainable params: 0
    _________________________________________________________________
    None

```python
for i in model.get_weights():
    display(Matrix(i))
```

Let's see how this net stacks up to the tanh net using our visualization method above. Remember, we are still _pre-training_ here for this network.

```python
# Let's color those points by the classification labels...
predictions = model.predict_classes(positions)[:,0]
zeros = positions[predictions==0,:]
ones = positions[predictions==1,:]

# Color predictions by class
plt.plot(zeros[:,0],zeros[:,1],'ro')
plt.plot(ones[:,0],ones[:,1],'bo')

# Color the zero-expected outputs red
plt.plot(X[[0,3],0],X[[0,3],1],'ro',markersize=10)
# Color the one-expected outputs blue
plt.plot(X[[1,2],0],X[[1,2],1],'bo',markersize=10)

plt.show()
```

Sometimes, we will see a boundary, but with other weights we may not. Let's see if some training pushes that boundary into the visible space of input vectors...

```python
# Train it!
history = model.fit(X, Y,
    batch_size=1,
    epochs=1000,
    verbose=0)
```

```python
# summarize history for loss
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()
```

```python
# Let's color those points by the classification labels...
predictions = model.predict_classes(positions)[:,0]
zeros = positions[predictions==0,:]
ones = positions[predictions==1,:]

# Color predictions by class
plt.plot(zeros[:,0],zeros[:,1],'ro')
plt.plot(ones[:,0],ones[:,1],'bo')

# Color the zero-expected outputs red
plt.plot(X[[0,3],0],X[[0,3],1],'ro',markersize=10)
# Color the one-expected outputs blue
plt.plot(X[[1,2],0],X[[1,2],1],'bo',markersize=10)

plt.show()
```

Well, that's not very promising is it? Is ReLU that useful after-all? What might be the problem here?

We've seen how to train a multilayer network using both tanh and ReLU at least, even if the results are not very spectacular. However, you should have some intuition about how this problem might be rectified using some hints on the slides and what we have talked about in class...

## Classifying MNIST

The MNIST data set consists of 60,000 images of handwritten digits for training and 10,000 images of handwritten digits for testing. We looked at this data in our last homework assignment some, but now we will try to train up a network for classifying these images.

First, let's load the data and prepare it for presentation to a network.

```python
from keras.datasets import mnist

# Digits are zero through nine, so 10 classes
num_classes = 10

# input image dimensions
img_rows, img_cols = 28, 28

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(x_train.shape[0], img_rows * img_cols)
x_test = x_test.reshape(x_test.shape[0], img_rows * img_cols)

# Data normalization (0-255 is encoded as 0-1 instead)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255.0
x_test /= 255.0

# Convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# Shape information
print('x_train shape:', x_train.shape)
print('y_train shape:', y_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
```

    x_train shape: (60000, 784)
    y_train shape: (60000, 10)
    60000 train samples
    10000 test samples

You can see that we have flattened each image (28x28 pixels) into a 728-element vector using the `reshape()` function. The intensity values in the original data are in the range 0-255, but we divide them all by 255 in order to scale the intensity between 0 and 1. This just keeps us from starting at extreme values in the weight space (where tanh and sigmoid get stuck easily). Finally, we convert the integer class labels (originally just 0,1,2,...,9) into the categorical representation that we need for `categorical cross-entropy`.

OK, so we need to train a network to recognize these digits. We will fully utilize the training/validation data alone when tuning parameters, and run a final accuracy check on the _test_ data which was _never seen_ during the training process. Let's see what a single-layer network can do to start with...

```python
model = keras.models.Sequential()

# Linear
model.add(keras.layers.Dense(num_classes, activation='softmax',input_shape=[x_train.shape[1]]))
model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adam(),
              metrics=['accuracy'])
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_20 (Dense)             (None, 10)                7850
    =================================================================
    Total params: 7,850
    Trainable params: 7,850
    Non-trainable params: 0
    _________________________________________________________________

```python
# Train it!
history = model.fit(x_train, y_train,
          batch_size=128,
          epochs=30,
          verbose=1,
          validation_split = 0.2)
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Train on 48000 samples, validate on 12000 samples
    Epoch 1/30
    48000/48000 [==============================] - 2s 41us/step - loss: 0.7545 - acc: 0.8146 - val_loss: 0.4094 - val_acc: 0.8968
    Epoch 2/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.3902 - acc: 0.8962 - val_loss: 0.3356 - val_acc: 0.9103
    Epoch 3/30
    48000/48000 [==============================] - 2s 35us/step - loss: 0.3385 - acc: 0.9079 - val_loss: 0.3070 - val_acc: 0.9160
    Epoch 4/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.3148 - acc: 0.9134 - val_loss: 0.2934 - val_acc: 0.9190
    Epoch 5/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.3005 - acc: 0.9168 - val_loss: 0.2843 - val_acc: 0.9223
    Epoch 6/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.2909 - acc: 0.9184 - val_loss: 0.2791 - val_acc: 0.9226
    Epoch 7/30
    48000/48000 [==============================] - 2s 35us/step - loss: 0.2835 - acc: 0.9208 - val_loss: 0.2743 - val_acc: 0.9245
    Epoch 8/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2782 - acc: 0.9222 - val_loss: 0.2714 - val_acc: 0.9263
    Epoch 9/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.2737 - acc: 0.9233 - val_loss: 0.2684 - val_acc: 0.9272
    Epoch 10/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2698 - acc: 0.9238 - val_loss: 0.2668 - val_acc: 0.9273
    Epoch 11/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2669 - acc: 0.9255 - val_loss: 0.2640 - val_acc: 0.9275
    Epoch 12/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2642 - acc: 0.9263 - val_loss: 0.2636 - val_acc: 0.9283
    Epoch 13/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2617 - acc: 0.9270 - val_loss: 0.2626 - val_acc: 0.9284
    Epoch 14/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2598 - acc: 0.9270 - val_loss: 0.2620 - val_acc: 0.9295
    Epoch 15/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2578 - acc: 0.9280 - val_loss: 0.2619 - val_acc: 0.9287
    Epoch 16/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.2557 - acc: 0.9282 - val_loss: 0.2609 - val_acc: 0.9293
    Epoch 17/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2545 - acc: 0.9290 - val_loss: 0.2596 - val_acc: 0.9300
    Epoch 18/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2527 - acc: 0.9296 - val_loss: 0.2607 - val_acc: 0.9288
    Epoch 19/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2513 - acc: 0.9295 - val_loss: 0.2595 - val_acc: 0.9301
    Epoch 20/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2501 - acc: 0.9303 - val_loss: 0.2586 - val_acc: 0.9308
    Epoch 21/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2492 - acc: 0.9311 - val_loss: 0.2598 - val_acc: 0.9303
    Epoch 22/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2480 - acc: 0.9299 - val_loss: 0.2588 - val_acc: 0.9294
    Epoch 23/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2470 - acc: 0.9319 - val_loss: 0.2591 - val_acc: 0.9300
    Epoch 24/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2461 - acc: 0.9317 - val_loss: 0.2591 - val_acc: 0.9308
    Epoch 25/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2449 - acc: 0.9315 - val_loss: 0.2588 - val_acc: 0.9298
    Epoch 26/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.2441 - acc: 0.9317 - val_loss: 0.2594 - val_acc: 0.9307
    Epoch 27/30
    48000/48000 [==============================] - 2s 34us/step - loss: 0.2432 - acc: 0.9327 - val_loss: 0.2577 - val_acc: 0.9317
    Epoch 28/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2425 - acc: 0.9321 - val_loss: 0.2624 - val_acc: 0.9302
    Epoch 29/30
    48000/48000 [==============================] - 2s 33us/step - loss: 0.2419 - acc: 0.9325 - val_loss: 0.2618 - val_acc: 0.9302
    Epoch 30/30
    48000/48000 [==============================] - 2s 32us/step - loss: 0.2411 - acc: 0.9333 - val_loss: 0.2591 - val_acc: 0.9307
    Test loss: 0.264308974689
    Test accuracy: 0.9271

```python
plt.figure()

# summarize history for accuracy

plt.subplot(211)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

# summarize history for loss

plt.subplot(212)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

plt.tight_layout()
plt.show()
```

Excellent! Let's take one more look at that accuracy...

```python
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Test loss: 0.264308974689
    Test accuracy: 0.9271

Just a single layer network is capable of performing above 92% accuracy on examples that it has never even seen during training. So, the gauntlet has been thown: can a deeper net do better? It will take some modifications to the architecture (tuning the *inductive bias*) in order to get there...

```python

```

Feed-forward Neural Networks/Multilayer Networks.ipynb