# Taylor Method

This notebook implements the 3rd order Taylor method for three different population intial value problems.
The Taylor method is derived from the Taylor expansion as depicted by Monica Alexander in the figure below:

# 3rd Order Taylor
The general 3rd Order Taylor method for to the first order differential equation
\begin{equation} y^{'} = f(t,y), \end{equation}
numerical approximates $y$ the at time point $t_i$ as $w_i$
with the  formula:
\begin{equation}
 w_{i+1}=w_i+h\big[f(t_i,w_i)+\frac{h}{2}f'(t_i,w_i)+\frac{h^2}{6}f''(t_i,w_i)\big],\end{equation}
where $h$ is the stepsize.
for $i=0,...,N-1$.
With the local truncation error of
\begin{equation}
\tau=\frac{h^3}{24}y^{''''}(\xi_i),\end{equation}
where $\xi \in [t_i,t_{i+1}]$.
To illustrate the method we will apply it to three intial value problems:
## 1. Linear
Consider the linear population Differential Equation
\begin{equation}
 y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}
y(2000)=6.\end{equation}

## 2. Non-Linear Population Equation
Consider the non-linear population Differential Equation
\begin{equation}
 y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}
y(2000)=6.
\end{equation}

## 3. Non-Linear Population Equation with an oscillation
Consider the non-linear population Differential Equation with an oscillation
\begin{equation}
 y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020),
 \end{equation}
with the initial condition,
\begin{equation}
y(2000)=6.\end{equation}

## Read in Libraries

```python
## Library
import numpy as np
import math
import pandas as pd

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation} h=\frac{b-a}{N}.\end{equation}
Here the interval is $2000\leq t \leq 2020,$
\begin{equation} h=\frac{2020-2000}{200}=0.1.\end{equation}
This gives the 201 discrete points:
\begin{equation} t_0=2000, \ t_1=2000.1, \ ... t_{200}=2020. \end{equation}
This is generalised to
\begin{equation} t_i=2000+i0.1, \ \ \ i=0,1,...,200.\end{equation}
The plot below shows the discrete time steps:

```python
N=200
t_end=2020.0
t_start=2000.0
h=((t_end-t_start)/N)
t=np.arange(t_start,t_end+h/2,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.show()
```

# 1. Linear Population Equation
## Exact Solution
The linear population equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
has a known exact (analytic) solution
\begin{equation} y=6e^{0.1(t-2000)}. \end{equation}

## Specific 3rd Order Taylor for the Linear Population Equation
To write the specific 3rd Order Taylor difference equation for the intial value problem we need to calculate the first derivative of
\begin{equation}f(t,y)=0.1y,\end{equation}

```python
def linfun(t,w):
    ftw=0.1*w
    return ftw
```

with respect to $t$,
\begin{equation} f'(t,y)=0.1y'=0.1(0.1y)=0.01y, \end{equation}

```python
def linfun_d(t,w):
    ftw=0.01*w
    return ftw
```

and the second derivative of $f$ with respect to $t$,
\begin{equation}f'(t,y)=0.01y'=0.01(0.1y)=0.001y.\end{equation}

```python
def linfun_dd(t,w):
    ftw=0.001*w
    return ftw
```

### Linear Population 3rd Order Taylor Difference equation
Substituting the derivatives of the linear population equation into the 3rd order Taylor equation gives the difference equation
\begin{equation} w_{i+1}= w_{i} + h\big[0.1 w_i+\frac{h}{2}(0.01 w_i)+\frac{h^2}{6}(0.001w_i)\big],\end{equation}
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with the initial condition
\begin{equation}w_0=6.\end{equation}

## Method

```python
w=np.zeros(N+1)
w[0]=6.0
for i in range (0,N):
    w[i+1]=w[i]+h*(linfun(t[i],w[i])+h/2*linfun_d(t[i],w[i])+h*h/6*linfun_dd(t[i],w[i]))
```

## Results
The plot below shows the exact solution, $y$ (squares) and the 3rd order numberical approximation, $w$ (circles) for the linear population equation:

```python
y=6*np.exp(0.1*(t-2000))
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.plot(t,y,'s:',color='black',label='Exact')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time, the 3rd order numerical approximation, $w$,  the exact solution, $y$, and the exact error $|y(t_i)-w_i|$ for the linear population equation:

```python

d = {'time t_i': t[0:10],    'Taylor ':w[0:10],'Exact (y)':y[0:10],'Exact Error':np.abs(np.round(y[0:10]-w[0:10],10))}
df = pd.DataFrame(data=d)
df

```

      time t_i
      Taylor
      Exact (y)
      Exact Error

      0
      2000.0
      6.000000
      6.000000
      0.000000e+00

      1
      2000.1
      6.060301
      6.060301
      2.500000e-09

      2
      2000.2
      6.121208
      6.121208
      5.100000e-09

      3
      2000.3
      6.182727
      6.182727
      7.700000e-09

      4
      2000.4
      6.244865
      6.244865
      1.030000e-08

      5
      2000.5
      6.307627
      6.307627
      1.300000e-08

      6
      2000.6
      6.371019
      6.371019
      1.580000e-08

      7
      2000.7
      6.435049
      6.435049
      1.860000e-08

      8
      2000.8
      6.499722
      6.499722
      2.150000e-08

      9
      2000.9
      6.565046
      6.565046
      2.440000e-08

## 2. Non-Linear Population Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
## Specific 3rd Order Taylor for the Non-Linear Population Equation
To write the specific 3rd Order Taylor difference equation for the intial value problem we need to calculate the first derivative of
\begin{equation}f(t,y)=0.2y-0.01y^2,\end{equation}

```python
def nonlinfun(t,w):
    ftw=0.2*w-0.01*w*w
    return ftw
```

with respect to $t$
\begin{equation} f'(t,y)=0.2y'-0.02y'y=0.2(0.2y-0.01y^2)-0.02(0.2y-0.01y^2)y,\end{equation}
\begin{equation}=(0.2-0.02y)(0.2y-0.01y^2)=(0.2-0.02y)f(t,y), \end{equation}

```python
def nonlinfun_d(t,w):
    ftw=0.2*nonlinfun(t,w)-0.02*nonlinfun(t,w)*w
    return ftw
```

and the second derivative with respect to $t$
\begin{equation} f''(t,y)=-0.02y'(0.2y-0.01y^2)+(0.2-0.02y)(0.2y'-0.02y'y),\end{equation}
\begin{equation}=-0.02(0.2y-0.01y^2)^2+(0.2-0.02y)^2(0.2y-0.01y^2), \end{equation}

```python
def nonlinfun_dd(t,w):
    ftw=-0.02*nonlinfun(t,w)*nonlinfun(t,w)+(0.2-0.02*w)*nonlinfun_d(t,w)
    return ftw
```

###  Non-Linear Population 3rd Order Taylor Difference equation
Substituting the derivatives of the non-linear population equation into the 3rd order Taylor equation gives the difference equation
\begin{equation} w_{i+1}= w_{i} + h\big[\big(0.2w_i-0.01w_i^2\big)+\frac{h}{2}\big((0.2-0.02w_i)(0.2w_i-0.01w_i^2)\big)+\end{equation}
\begin{equation}
\frac{h^2}{6}\big(-0.02(0.2w_i-0.01w_i^2)^2+(0.2-0.02w_i)^2(0.2w_i-0.01w_i^2)\big)\big], \end{equation}
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with the initial condition
$$w_0=6.$$

```python
w=np.zeros(N+1)
w[0]=6.0
for i in range (0,N):
    w[i+1]=w[i]+h*(nonlinfun(t[i],w[i])+h/2*nonlinfun_d(t[i],w[i])+h*h/6*nonlinfun_dd(t[i],w[i]))
```

## Results
The plot below shows the 3rd order numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the 3rd order numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t[0:10],    'Taylor ':w[0:10]}
df = pd.DataFrame(data=d)
df

```

      time t_i
      Taylor

      0
      2000.0
      6.000000

      1
      2000.1
      6.084335

      2
      2000.2
      6.169332

      3
      2000.3
      6.254983

      4
      2000.4
      6.341279

      5
      2000.5
      6.428207

      6
      2000.6
      6.515760

      7
      2000.7
      6.603924

      8
      2000.8
      6.692689

      9
      2000.9
      6.782043

## 3. Non-Linear Population Equation with an oscilation
$$ y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), $$
with the initial condition,
$$y(2000)=6.$$

## Specific 3rd Order Taylor for the Non-Linear Population Equation with an oscilation
To write the specific 3rd Order Taylor difference equation for the intial value problem we need calculate the first derivative of
$$f(t,y)=0.2y-0.01y^2+\sin(2\pi t),$$

```python
def nonlin_oscfun(t,w):
    ftw=0.2*w-0.01*w*w+np.sin(2*np.math.pi*t)
    return ftw
```

with respect to $t$
$$ f'(t,y)=0.2y'-0.02y'y+2\pi\cos(2\pi t)$$
$$=(0.2-0.02y)\big(0.2y-0.01y^2+\sin(2\pi t)\big)+2\pi\cos(2\pi t) $$

```python
def nonlin_oscfun_d(t,w):
    ftw=0.2*nonlinfun(t,w)-0.02*nonlinfun(t,w)*w+2*np.math.pi*np.cos(2*np.math.pi*t)
    return ftw
```

and the second derivative with respect to $t$
$$ f''(t,y)=-0.02y'(0.2y-0.01y^2+\sin(2\pi t))$$
$$+(0.2-0.02y)\big(0.2y'-0.02y'y+2\pi\cos(2\pi t)\big)-(2\pi)^2\sin(2\pi t)$$
$$=-0.02(0.2y-0.01y^2+2\pi\sin(2\pi t))^2$$
$$+(0.2-0.02y)\big((0.2-0.02y)(0.2y-0.01y^2+\sin(2\pi t))+ 2\pi\cos(2\pi t) \big)-(2\pi)^2\sin(2\pi t) $$

```python
def nonlin_oscfun_dd(t,w):
    ftw=-0.02*nonlin_oscfun(t,w)*nonlin_oscfun(t,w)+(0.2-0.02*w)*((0.2-0.02*w)*nonlin_oscfun(t,w)+2*np.math.pi*np.cos(2*np.math.pi*t))#-2*np.math.pi*2*np.math.pi*np.sin(2*np.math.pi*t)
    return ftw
```

###  Non-Linear Population with oscilation 3rd Order Taylor Difference equation
Substituting the derivatives of the non-linear population equation with oscilation into the 3rd order Taylor equation gives the difference equation
$$ w_{i+1}= w_{i} + h\big[\big(0.2w_i-0.01w_i^2+\sin(2\pi t_i)\big)$$
$$+\frac{h}{2}\big((0.2-0.02w_i)\big(0.2w_i-0.01w_i^2+\sin(2\pi t_i)\big)+2\pi\cos(2\pi t_i)\big)+$$
$$\frac{h^2}{6}\big(-0.02(0.2w_i-0.01w_i^2+2\pi\sin(2\pi t_i))^2$$
$$+(0.2-0.02w_i)\big((0.2-0.02w_i)[0.2w_i-0.01w_i^2+\sin(2\pi t_i)]$$
$$+ 2\pi\cos(2\pi t_i) \big)-(2\pi)^2\sin(2\pi t_i) \big)\big], $$
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with the initial condition
$$w_0=6.$$

```python
w=np.zeros(N+1)
w[0]=6.0
for i in range (0,N):
    w[i+1]=w[i]+h*(nonlin_oscfun(t[i],w[i])+h/2*nonlin_oscfun_d(t[i],w[i])+h*h/6*nonlin_oscfun_dd(t[i],w[i]))
```

## Results
The plot below shows the 3rd order numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the 3rd order numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t[0:10],    'Taylor ':w[0:10]}
df = pd.DataFrame(data=d)
df

```

      time t_i
      Taylor

      0
      2000.0
      6.000000

      1
      2000.1
      6.115834

      2
      2000.2
      6.285332

      3
      2000.3
      6.476682

      4
      2000.4
      6.649942

      5
      2000.5
      6.772316

      6
      2000.6
      6.830702

      7
      2000.7
      6.836694

      8
      2000.8
      6.822138

      9
      2000.9
      6.826948

```python

```

Chapter 02 - Higher Order Methods/201_3rd Order Taylor_Population_growth.ipynb


# Getting started with TensorFlow

**Learning Objectives**
  1. Practice defining and performing basic operations on constant Tensors
  1. Use Tensorflow's automatic differentiation capability
  1. Learn how to train a linear regression from scratch with TensorFLow

In this notebook, we will start by reviewing the main operations on Tensors in TensorFlow and understand how to manipulate TensorFlow Variables. We explain how these are compatible with python built-in list and numpy arrays.

Then we will jump to the problem of training a linear regression from scratch with gradient descent. The first order of business will be to understand how to compute the gradients of a function (the loss here) with respect to some of its arguments (the model weights here). The TensorFlow construct allowing us to do that is `tf.GradientTape`, which we will describe.

At last we will create a simple training loop to learn the weights of a 1-dim linear regression using synthetic data generated from a linear model.

As a bonus exercise, we will do the same for data generated from a non linear model, forcing us to manual engineer non-linear features to improve our linear model performance.

```python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
```

```python
print(tf.__version__)
```

    2.1.1

## Operations on Tensors

### Variables and Constants

Tensors in TensorFlow are either contant (`tf.constant`) or variables (`tf.Variable`).
Constant values can not be changed, while variables values can be.

The main difference is that instances of `tf.Variable` have methods allowing us to change
their values while tensors constructed with `tf.constant` don't have these methods, and
therefore their values can not be changed. When you want to change the value of a `tf.Variable`
`x` use one of the following method:

* `x.assign(new_value)`
* `x.assign_add(value_to_be_added)`
* `x.assign_sub(value_to_be_subtracted`

```python
x = tf.constant([2, 3, 4])
x
```

    <tf.Tensor: shape=(3,), dtype=int32, numpy=array([2, 3, 4], dtype=int32)>

```python
x = tf.Variable(2.0, dtype=tf.float32, name='my_variable')
x
```

    <tf.Variable 'my_variable:0' shape=() dtype=float32, numpy=2.0>

**Exercise.** Use the `assign(..)` method to assign a new value to the variable `x` you created above. After each, print `x` to verify how the value changes.

```python
x.assign_add(4) # TODO: Your code goes here.
x
```

    <tf.Variable 'my_variable:0' shape=() dtype=float32, numpy=6.0>

```python
x.assign(1) # TODO: Your code goes here.
x
```

    <tf.Variable 'my_variable:0' shape=() dtype=float32, numpy=1.0>

```python
x.assign_sub(3) # TODO: Your code goes here.
x
```

    <tf.Variable 'my_variable:0' shape=() dtype=float32, numpy=-2.0>

### Point-wise operations

Tensorflow offers similar point-wise tensor operations as numpy does:

* `tf.add` allows to add the components of a tensor
* `tf.multiply` allows us to multiply the components of a tensor
* `tf.subtract` allow us to substract the components of a tensor
* `tf.math.*` contains the usual math operations to be applied on the components of a tensor
* and many more...

Most of the standard aritmetic operations (`tf.add`, `tf.substrac`, etc.) are overloaded by the usual corresponding arithmetic symbols (`+`, `-`, etc.)

**Exercise.** Create two tensorflow constants `a = [5, 3, 8]` and `b = [3, -1, 2]`. Then, compute
1. the sum of the constants `a` and `b` below using `tf.add` and `+` and verify both operations produce the same values.
2. the product of the constants `a` and `b` below using `tf.multiply` and `*` and verify both operations produce the same values.
3. the exponential of the constant `a` using `tf.math.exp`. Note, you'll need to specify the type for this operation.

```python
a = tf.constant([5, 3, 8]) # TODO: Your code goes here.
b = tf.constant([3, -1, 2]) # TODO: Your code goes here.
c = tf.add(a, b) # TODO: Your code goes here.
d = a + b # TODO: Your code goes here.

print("c:", c)
print("d:", d)
```

    c: tf.Tensor([ 8  2 10], shape=(3,), dtype=int32)
    d: tf.Tensor([ 8  2 10], shape=(3,), dtype=int32)

```python
a = tf.constant([5, 3, 8]) # TODO: Your code goes here.
b = tf.constant([3, -1, 2]) # TODO: Your code goes here.
c = tf.multiply(a, b) # TODO: Your code goes here.
d = a * b # TODO: Your code goes here.

print("c:", c)
print("d:", d)
```

    c: tf.Tensor([15 -3 16], shape=(3,), dtype=int32)
    d: tf.Tensor([15 -3 16], shape=(3,), dtype=int32)

```python
# tf.math.exp expects floats so we need to explicitly give the type
a = tf.constant([5, 3, 8], dtype=tf.float32) # TODO: Your code goes here.
b = tf.math.exp(a) # TODO: Your code goes here.

print("b:", b)
```

    b: tf.Tensor([ 148.41316    20.085537 2980.958   ], shape=(3,), dtype=float32)

### NumPy Interoperability

In addition to native TF tensors, tensorflow operations can take native python types and NumPy arrays as operands.

```python
# native python list
a_py = [1, 2]
b_py = [3, 4]
```

**Exercise.** Use `tf.add` to compute the sum of the native python arrays `a` and `b`.

```python
#TODO: Your code goes here.
c_py = tf.add(a_py, b_py)
c_py
```

    <tf.Tensor: shape=(2,), dtype=int32, numpy=array([4, 6], dtype=int32)>

```python
# numpy arrays
a_np = np.array([1, 2])
b_np = np.array([3, 4])
```

**Exercise.** Use `tf.add` to compute the sum of the NumPy arrays `a` and `b`.

```python
#TODO: Your code goes here.
c_np = tf.add(a_np, b_np)
c_np
```

    <tf.Tensor: shape=(2,), dtype=int64, numpy=array([4, 6])>

```python
# native TF tensor
a_tf = tf.constant([1, 2])
b_tf = tf.constant([3, 4])
```

**Exercise.** Use `tf.add` to compute the sum of the Tensorflow constants `a` and `b`.

```python
c_tf = tf.add(a_tf, b_tf)
c_tf
```

    <tf.Tensor: shape=(2,), dtype=int32, numpy=array([4, 6], dtype=int32)>

You can convert a native TF tensor to a NumPy array using .numpy()

```python
a_tf.numpy()
```

    array([1, 2], dtype=int32)

## Linear Regression

Now let's use low level tensorflow operations to implement linear regression.

Later in the course you'll see abstracted ways to do this using high level TensorFlow.

### Toy Dataset

We'll model the following function:

\begin{equation}
y= 2x + 10
\end{equation}

```python
X = tf.constant(range(10), dtype=tf.float32)
Y = 2 * X + 10

print("X:{}".format(X))
print("Y:{}".format(Y))
```

    X:[0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
    Y:[10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]

Let's also create a test dataset to evaluate our models:

```python
X_test = tf.constant(range(10, 20), dtype=tf.float32)
Y_test = 2 * X_test + 10

print("X_test:{}".format(X_test))
print("Y_test:{}".format(Y_test))
```

    X_test:[10. 11. 12. 13. 14. 15. 16. 17. 18. 19.]
    Y_test:[30. 32. 34. 36. 38. 40. 42. 44. 46. 48.]

#### Loss Function

The simplest model we can build is a model that for each value of x returns the sample mean of the training set:

```python
y_mean = Y.numpy().mean()

def predict_mean(X):
    y_hat = [y_mean] * len(X)
    return y_hat

Y_hat = predict_mean(X_test)
```

Using mean squared error, our loss is:
\begin{equation}
MSE = \frac{1}{m}\sum_{i=1}^{m}(\hat{Y}_i-Y_i)^2
\end{equation}

For this simple model the loss is then:

```python
errors = (Y_hat - Y)**2
loss = tf.reduce_mean(errors)
loss.numpy()
```

    33.0

This values for the MSE loss above will give us a baseline to compare how a more complex model is doing.

Now, if $\hat{Y}$ represents the vector containing our model's predictions when we use a linear regression model
\begin{equation}
\hat{Y} = w_0X + w_1
\end{equation}

we can write a loss function taking as arguments the coefficients of the model:

```python
def loss_mse(X, Y, w0, w1):
    Y_hat = w0 * X + w1
    errors = (Y_hat - Y)**2
    return tf.reduce_mean(errors)
```

### Gradient Function

To use gradient descent we need to take the partial derivatives of the loss function with respect to each of the weights. We could manually compute the derivatives, but with Tensorflow's automatic differentiation capabilities we don't have to!

During gradient descent we think of the loss as a function of the parameters $w_0$ and $w_1$. Thus, we want to compute the partial derivative with respect to these variables.

For that we need to wrap our loss computation within the context of `tf.GradientTape` instance which will reccord gradient information:

```python
with tf.GradientTape() as tape:
    loss = # computation
```

This will allow us to later compute the gradients of any tensor computed within the `tf.GradientTape` context with respect to instances of `tf.Variable`:

```python
gradients = tape.gradient(loss, [w0, w1])
```

We illustrate this procedure with by computing the loss gradients with respect to the model weights:

**Lab Task #6:** Complete the function below to compute the loss gradients with respect to the model weights `w0` and `w1`.

```python
def compute_gradients(X, Y, w0, w1):
    # TODO: Your code goes here.
    with tf.GradientTape() as tape:
        loss = loss_mse(X, Y, w0, w1)
    return (tape.gradient(loss, [w0, w1]))
```

```python
w0 = tf.Variable(0.0)
w1 = tf.Variable(0.0)

dw0, dw1 = compute_gradients(X, Y, w0, w1)
```

```python
print("dw0:", dw0.numpy())
```

    dw0: -204.0

```python
print("dw1", dw1.numpy())
```

    dw1 -38.0

### Training Loop

Here we have a very simple training loop that converges. Note we are ignoring best practices like batching, creating a separate test set, and random weight initialization for the sake of simplicity.

**Lab Task #7:** Complete the `for` loop below to train a linear regression.
1. Use `compute_gradients` to compute `dw0` and `dw1`.
2. Then, re-assign the value of `w0` and `w1` using the `.assign_sub(...)` method with the computed gradient values and the `LEARNING_RATE`.
3. Finally, for every 100th step , we'll compute and print the `loss`. Use the `loss_mse` function we created above to compute the `loss`.

```python
STEPS = 1000
LEARNING_RATE = .02
MSG = "STEP {step} - loss: {loss}, w0: {w0}, w1: {w1}\n"

w0 = tf.Variable(0.0)
w1 = tf.Variable(0.0)

for step in range(0, STEPS + 1):

    dw0, dw1 = compute_gradients(X, Y, w0, w1) #TODO: Your code goes here.
    w0.assign_sub(dw0 * LEARNING_RATE) #TODO: Your code goes here.
    w1.assign_sub(dw1 * LEARNING_RATE) #TODO: Your code goes here.

    if step % 100 == 0:
        loss = loss_mse(X, Y, w0, w1) #TODO: Your code goes here.
        print(MSG.format(step=step, loss=loss, w0=w0.numpy(), w1=w1.numpy()))

```

    STEP 0 - loss: 35.70719528198242, w0: 4.079999923706055, w1: 0.7599999904632568

    STEP 100 - loss: 2.6017532348632812, w0: 2.4780430793762207, w1: 7.002389907836914

    STEP 200 - loss: 0.26831889152526855, w0: 2.153517961502075, w1: 9.037351608276367

    STEP 300 - loss: 0.027671903371810913, w0: 2.0493006706237793, w1: 9.690855979919434

    STEP 400 - loss: 0.0028539239428937435, w0: 2.0158326625823975, w1: 9.90071964263916

    STEP 500 - loss: 0.0002943490108009428, w0: 2.005084753036499, w1: 9.96811580657959

    STEP 600 - loss: 3.0356444767676294e-05, w0: 2.0016329288482666, w1: 9.989760398864746

    STEP 700 - loss: 3.1322738323069643e-06, w0: 2.0005245208740234, w1: 9.996710777282715

    STEP 800 - loss: 3.2238213520940917e-07, w0: 2.0001683235168457, w1: 9.998944282531738

    STEP 900 - loss: 3.369950718479231e-08, w0: 2.000054359436035, w1: 9.999658584594727

    STEP 1000 - loss: 3.6101481803996194e-09, w0: 2.0000178813934326, w1: 9.99988842010498

Now let's compare the test loss for this linear regression to the test loss from the baseline model that outputs always the mean of the training set:

```python
loss = loss_mse(X_test, Y_test, w0, w1)
loss.numpy()
```

    2.4563633e-08

This is indeed much better!

## Bonus

Try modelling a non-linear function such as: $y=xe^{-x^2}$

```python
X = tf.constant(np.linspace(0, 2, 1000), dtype=tf.float32)
Y = X * tf.exp(-X**2)
```

```python
%matplotlib inline

plt.plot(X, Y)
```

```python
def make_features(X):
    f1 = tf.ones_like(X)  # Bias. JOSE: all ones with the same shape as X
    f2 = X
#    f3 = tf.square(X)
    f3 = tf.math.pow(X, 1.5)
    f4 = tf.sqrt(X)
    f5 = tf.exp(X)
    return tf.stack([f1, f2, f3, f4, f5], axis=1) #JOSE: Create input tensor with features
```

```python
def predict(X, W):
    return tf.squeeze(X @ W, -1)
```

```python
def loss_mse(X, Y, W):
    Y_hat = predict(X, W)
    errors = (Y_hat - Y)**2
    return tf.reduce_mean(errors)
```

```python
def compute_gradients(X, Y, W):
    with tf.GradientTape() as tape:
        loss = loss_mse(Xf, Y, W)
    return tape.gradient(loss, W)
```

```python
STEPS = 2000
LEARNING_RATE = .02

Xf = make_features(X)
n_weights = Xf.shape[1]

W = tf.Variable(np.zeros((n_weights, 1)), dtype=tf.float32)

# For plotting
steps, losses = [], []
plt.figure()

for step in range(1, STEPS + 1):

    dW = compute_gradients(X, Y, W)
    W.assign_sub(dW * LEARNING_RATE)

    if step % 100 == 0:
        loss = loss_mse(Xf, Y, W)
        steps.append(step)
        losses.append(loss)
        plt.clf()
        plt.plot(steps, losses)

print("STEP: {} MSE: {}".format(STEPS, loss_mse(Xf, Y, W)))

plt.figure()
plt.plot(X, Y, label='actual')
plt.plot(X, predict(Xf, W), label='predicted')
plt.legend()
```

Copyright 2021 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License


```python
from sympy import *
init_printing(use_unicode=True)
```

```python
m = Matrix([1, 2, 3])
```

```python
m = Matrix(["c"])
```

```python
m
```

$$\left[\begin{matrix}c\end{matrix}\right]$$

```python
m*m
```

$$\left[\begin{matrix}c^{2}\end{matrix}\right]$$

```python
from sympy.abc import a,b,c
```

```python
m
```

$$\left[\begin{matrix}c\end{matrix}\right]$$

```python
m = ImmutableMatrix([c,c**2])
```

```python
m
```

$$\left[\begin{matrix}c\\c^{2}\end{matrix}\right]$$

```python
m*m.T
```

$$\left[\begin{matrix}c^{2} & c^{3}\\c^{3} & c^{4}\end{matrix}\right]$$

```python
(m*m.T)**2
```

$$\left[\begin{matrix}c^{6} + c^{4} & c^{7} + c^{5}\\c^{7} + c^{5} & c^{8} + c^{6}\end{matrix}\right]$$

```python
m[0,0]
```

```python
q = Matrix([[1,2],[3,4]])
```

```python
q^2
```

```python
q^-1
```

```python
q**-1
```

```python

```

```python

import numpy as np
from bqplot import *
from bqplot.marks import Graph
from ipywidgets import Layout
from relmath import *
type(M)

fig_layout = Layout(width='960px', height='500px')
def disp(expr):

    print(type(expr))
    if type(expr) is MatMul:
        binop = expr

        left =  binop.left.simp()
        right = binop.right.simp()

        if type(left) is not Rel:
            raise ValueError("Can't simplify left operand to a Rel ! Found %s " % left)

        if type(right) is not Rel:
            raise ValueError("Can't simplify right operand to a Rel ! Found %s " % right)

        print(left)
        print(right)

        node_data =  left.dom + left.cod + right.cod
        print(node_data)

        link_data = []
        n = len(left.dom)
        m = len(left.cod)
        w = len(right.cod)
        for i in range(n):
            for j in range(m):
                link_data.append({'source': i, 'target': n+j, 'value': left.g[i][j].val})

        for i in range(m):
            for j in range(w):
                link_data.append({'source': n+i, 'target': n+m+j, 'value': right.g[i][j].val})

        print(link_data)

        xs = LinearScale()
        ys = LinearScale()
        lcs = ColorScale(scheme='Greens')
        x = ([100] * n) + ([200]*m) + ([300]*w)
        y = list(range(n)) + list(range(m)) + list(range(w))
        print(x)
        print(y)
        graph = Graph(node_data=node_data, link_data=link_data, link_type='line',
                      colors=['orange'], directed=False,
                      scales={'x': xs, 'y': ys, 'link_color': lcs},
                      x=x, y=y,  color=np.random.rand(len(node_data)))
        return Figure(marks=[graph], layout=fig_layout)

    elif type(expr) is Rel:
        node_data = expr.dom() + expr.cod()
        #print(node_data)

        link_data = []
        for i in range(len(expr.dom)):
            for j in range(len(expr.cod)):
                link_data.append({'source': i, 'target': i+j, 'value': expr.g[i][j].val})

        #print(link_data)

        xs = LinearScale()
        ys = LinearScale()
        lcs = ColorScale(scheme='Greens')
        x = ([100] * len(expr.dom)) + ([200]*len(expr.cod))
        y = list(range(len(expr.dom))) + list(range(len(expr.cod)))
        #print(x)
        #print(y)
        graph = Graph(node_data=node_data, link_data=link_data, link_type='line',
                      colors=['orange'], directed=False,
                      scales={'x': xs, 'y': ys, 'link_color': lcs},
                      x=x, y=y,  color=np.random.rand(len(node_data)))
        return Figure(marks=[graph], layout=fig_layout)
    else:
        raise ValueError("not supported type: %s" % type(expr) )

disp(E)
```

    M =
    [[9, 0, 6], [0, 5, 7]]
    ~M =
    [[9, 0], [0, 5], [6, 7]]
    -M =
    [[-9, 0, -6], [0, -5, -7]]
    M@~M=
    [[36, 42], [42, 49]]
    MatMul(M, T(M))
    [[9, 0, 6], [0, 5, 7]]@~[[9, 0, 6], [0, 5, 7]]
    MatMul(M, T(M)).simp()
    [[36, 42], [42, 49]]
    <class 'relmath.MatMul'>
    [[9, 0, 6], [0, 5, 7]]
    [[9, 0], [0, 5], [6, 7]]
    ['a', 'b', 'x', 'y', 'z', 'a', 'b']
    [{'source': 0, 'target': 2, 'value': 9}, {'source': 0, 'target': 3, 'value': 0}, {'source': 0, 'target': 4, 'value': 6}, {'source': 1, 'target': 2, 'value': 0}, {'source': 1, 'target': 3, 'value': 5}, {'source': 1, 'target': 4, 'value': 7}, {'source': 2, 'target': 5, 'value': 9}, {'source': 2, 'target': 6, 'value': 0}, {'source': 3, 'target': 5, 'value': 0}, {'source': 3, 'target': 6, 'value': 5}, {'source': 4, 'target': 5, 'value': 6}, {'source': 4, 'target': 6, 'value': 7}]
    [100, 100, 200, 200, 200, 300, 300]
    [0, 1, 0, 1, 2, 0, 1]

    Figure(fig_margin={'right': 60, 'top': 60, 'left': 60, 'bottom': 60}, layout=Layout(height='500px', width='960…

```python

```

```python
X = MatrixSymbol('X', 3, 4)
```

```python
X
```

```python
X[0,0]
```

```python
m
```

$$\left[\begin{matrix}c\\c^{2}\end{matrix}\right]$$

```python
srepr(m*m)
```

    "ImmutableDenseMatrix([[Pow(Symbol('c'), Integer(2)), Pow(Symbol('c'), Integer(3))], [Pow(Symbol('c'), Integer(3)), Pow(Symbol('c'), Integer(4))]])"

```python
sympify("m * m.T", locals={'m':m}, evaluate=False)
```

$$\left[\begin{matrix}c\\c^{2}\end{matrix}\right] \left[\begin{matrix}c & c^{2}\end{matrix}\right]$$

```python
help(sympify)
```

    Help on function sympify in module sympy.core.sympify:

    sympify(a, locals=None, convert_xor=True, strict=False, rational=False, evaluate=None)
        Converts an arbitrary expression to a type that can be used inside SymPy.

        For example, it will convert Python ints into instances of sympy.Integer,
        floats into instances of sympy.Float, etc. It is also able to coerce symbolic
        expressions which inherit from Basic. This can be useful in cooperation
        with SAGE.

        It currently accepts as arguments:
           - any object defined in sympy
           - standard numeric python types: int, long, float, Decimal
           - strings (like "0.09" or "2e-19")
           - booleans, including ``None`` (will leave ``None`` unchanged)
           - lists, sets or tuples containing any of the above

        .. warning::
            Note that this function uses ``eval``, and thus shouldn't be used on
            unsanitized input.

        If the argument is already a type that SymPy understands, it will do
        nothing but return that value. This can be used at the beginning of a
        function to ensure you are working with the correct type.

        >>> from sympy import sympify

        >>> sympify(2).is_integer
        True
        >>> sympify(2).is_real
        True

        >>> sympify(2.0).is_real
        True
        >>> sympify("2.0").is_real
        True
        >>> sympify("2e-45").is_real
        True

        If the expression could not be converted, a SympifyError is raised.

        >>> sympify("x***2")
        Traceback (most recent call last):
        ...
        SympifyError: SympifyError: "could not parse u'x***2'"

        Locals
        ------

        The sympification happens with access to everything that is loaded
        by ``from sympy import *``; anything used in a string that is not
        defined by that import will be converted to a symbol. In the following,
        the ``bitcount`` function is treated as a symbol and the ``O`` is
        interpreted as the Order object (used with series) and it raises
        an error when used improperly:

        >>> s = 'bitcount(42)'
        >>> sympify(s)
        bitcount(42)
        >>> sympify("O(x)")
        O(x)
        >>> sympify("O + 1")
        Traceback (most recent call last):
        ...
        TypeError: unbound method...

        In order to have ``bitcount`` be recognized it can be imported into a
        namespace dictionary and passed as locals:

        >>> from sympy.core.compatibility import exec_
        >>> ns = {}
        >>> exec_('from sympy.core.evalf import bitcount', ns)
        >>> sympify(s, locals=ns)
        6

        In order to have the ``O`` interpreted as a Symbol, identify it as such
        in the namespace dictionary. This can be done in a variety of ways; all
        three of the following are possibilities:

        >>> from sympy import Symbol
        >>> ns["O"] = Symbol("O")  # method 1
        >>> exec_('from sympy.abc import O', ns)  # method 2
        >>> ns.update(dict(O=Symbol("O")))  # method 3
        >>> sympify("O + 1", locals=ns)
        O + 1

        If you want *all* single-letter and Greek-letter variables to be symbols
        then you can use the clashing-symbols dictionaries that have been defined
        there as private variables: _clash1 (single-letter variables), _clash2
        (the multi-letter Greek names) or _clash (both single and multi-letter
        names that are defined in abc).

        >>> from sympy.abc import _clash1
        >>> _clash1
        {'C': C, 'E': E, 'I': I, 'N': N, 'O': O, 'Q': Q, 'S': S}
        >>> sympify('I & Q', _clash1)
        I & Q

        Strict
        ------

        If the option ``strict`` is set to ``True``, only the types for which an
        explicit conversion has been defined are converted. In the other
        cases, a SympifyError is raised.

        >>> print(sympify(None))
        None
        >>> sympify(None, strict=True)
        Traceback (most recent call last):
        ...
        SympifyError: SympifyError: None

        Evaluation
        ----------

        If the option ``evaluate`` is set to ``False``, then arithmetic and
        operators will be converted into their SymPy equivalents and the
        ``evaluate=False`` option will be added. Nested ``Add`` or ``Mul`` will
        be denested first. This is done via an AST transformation that replaces
        operators with their SymPy equivalents, so if an operand redefines any
        of those operations, the redefined operators will not be used.

        >>> sympify('2**2 / 3 + 5')
        19/3
        >>> sympify('2**2 / 3 + 5', evaluate=False)
        2**2/3 + 5

        Extending
        ---------

        To extend ``sympify`` to convert custom objects (not derived from ``Basic``),
        just define a ``_sympy_`` method to your class. You can do that even to
        classes that you do not own by subclassing or adding the method at runtime.

        >>> from sympy import Matrix
        >>> class MyList1(object):
        ...     def __iter__(self):
        ...         yield 1
        ...         yield 2
        ...         return
        ...     def __getitem__(self, i): return list(self)[i]
        ...     def _sympy_(self): return Matrix(self)
        >>> sympify(MyList1())
        Matrix([
        [1],
        [2]])

        If you do not have control over the class definition you could also use the
        ``converter`` global dictionary. The key is the class and the value is a
        function that takes a single argument and returns the desired SymPy
        object, e.g. ``converter[MyList] = lambda x: Matrix(x)``.

        >>> class MyList2(object):   # XXX Do not do this if you control the class!
        ...     def __iter__(self):  #     Use _sympy_!
        ...         yield 1
        ...         yield 2
        ...         return
        ...     def __getitem__(self, i): return list(self)[i]
        >>> from sympy.core.sympify import converter
        >>> converter[MyList2] = lambda x: Matrix(x)
        >>> sympify(MyList2())
        Matrix([
        [1],
        [2]])

        Notes
        =====

        Sometimes autosimplification during sympification results in expressions
        that are very different in structure than what was entered. Until such
        autosimplification is no longer done, the ``kernS`` function might be of
        some use. In the example below you can see how an expression reduces to
        -1 by autosimplification, but does not do so when ``kernS`` is used.

        >>> from sympy.core.sympify import kernS
        >>> from sympy.abc import x
        >>> -2*(-(-x + 1/x)/(x*(x - 1/x)**2) - 1/(x*(x - 1/x))) - 1
        -1
        >>> s = '-2*(-(-x + 1/x)/(x*(x - 1/x)**2) - 1/(x*(x - 1/x))) - 1'
        >>> sympify(s)
        -1
        >>> kernS(s)
        -2*(-(-x + 1/x)/(x*(x - 1/x)**2) - 1/(x*(x - 1/x))) - 1

```python
Matrix(['s',3])
```

$$\left[\begin{matrix}s\\3\end{matrix}\right]$$

```python
p
```

$$\left[\begin{matrix}s\\3\end{matrix}\right]$$

```python
Matrix(['s',{'a':'b'}])
```

$$\left[\begin{matrix}s\\\left \{ a : b\right \}\end{matrix}\right]$$

```python
p = Matrix(['s',['a']])
p
```

$$\left[\begin{matrix}s\\\left [ a\right ]\end{matrix}\right]$$

```python
p += Matrix(["c","d"])
```

```python
p = Matrix([[['s'],'r'],['a','e']])
```

$$\left[\begin{matrix}\left [ s\right ] & r\\a & e\end{matrix}\right]$$

```python
p.c = 2
```

```python
p
```

$$\left[\begin{matrix}s\\\left [ a\right ]\end{matrix}\right]$$

```python
p.c
```

```python
p.afdsf = 2
```

```python
m = ImmutableMatrix([2,3])
```

```python
isinstance(4, float)
```

    False

```python
from sympy.core.singleton import S

class Dioid(AtomicExpr):

    is_number = True

    @cacheit
    def __new__(cls,*args):
        tmp = []
        for a in args:
            a = sympify(a,strict=True)
            if type(a) is cls:
                tmp.extend(a.args)
            else:
                tmp.append(a)
        return super().__new__(cls,*tmp)

    def __mul__(self, other):
        return self._eval_mul(other)

    @decorators._sympifyit('other', NotImplemented)
    def __add__(self, other):
        return self._eval_add(other)

    def __neg__(self):
        return self._eval_neg()

    def s(self):
        """ The set of discourse """
        raise NotImplementedError("IMPLEMENT ME!")

    def zero(self):
        raise NotImplementedError("IMPLEMENT ME!")

    def one(self):
        raise NotImplementedError("IMPLEMENT ME!")

    def __str__(self):

        return "%s(%s)" % (self.__class__.__name__, str(self.args[0]))

    def __repr__(self):
        return self.__str__()

    def _eval_neg(self, other):
        """ NOTE: in a dioid, negation is _not_ mandatory. See page 7 of Graphs, Dioids and Semirings book
        """
        raise NotImplemented("Not available for this dioid !")

        raise NotImplementedError("")

    def _eval_add(self, other):
        raise NotImplementedError("")

    def _eval_mul(self, other):
        raise NotImplementedError("")

    def __radd__(self, a):
        """Implementation of reverse add method."""
        return a.__add__(self)

class RD(Dioid):
    def __init__(self, val):
        self.val = sympify(val)
        super().__init__()

    def zero(self):
        return S(0)

    def one(self):
        return S(1)

    def s(self):
        return "TODO R^ set "

    def _eval_neg(self):
        """ NOTE: in a dioid, negation is _not_ mandatory. See page 7 of Graphs, Dioids and Semirings book
        """
        return self.__class__(- self.args[0])

    def _eval_add(self, d2):

        if isinstance(d2,int) or isinstance(d2,float):
            return self.__class__(self.args[0] * d2)
        elif isinstance(d2, RD):
            return self.__class__(self.args[0] + d2.args[0])

    def _eval_mul(self, d2):
        if isinstance(d2,int) or isinstance(d2,float):
            return self.__class__(self.args[0] * d2)
        elif isinstance(d2, RD):
            return self.__class__(self.args[0] * d2.args[0])
        elif return NotImplemented

    #@classmethod
    #def eval(cls, arg):
        # Phi(x) + Phi(-x) == 1
        #if arg.could_extract_minus_sign():
        #    return 1-cls(-arg)

class Rel(ImmutableMatrix):
    def __init__(self, par, dom=[], cod=[]):
        super().__init__()
        self.dom=dom
        self.cod=cod

    def _eval_matrix_mul(self, other):
        r = super()._eval_matrix_mul(other)
        r.dom = self.dom
        r.cod = other.cod
        return r

    def _eval_transpose(self):
        r = super()._eval_transpose()
        r.dom = self.cod
        r.cod = self.dom
        return r

    T = property(transpose, None, None, "Matrix transposition.")
```

```python
(Add(RD(3), RD(7))).evalf()
```

```python
simplify(Add(RD(3), RD(7)))
```

```python
RD(5)
```

```python
RD(5) == RD(5)
```

    True

```python
-RD(5)
```

```python
RD(3) + RD(7)
```

```python
w = Rel([RD(3)])
```

```python
sympify(Add(RD(3), RD(7)))
```

```python
Add(RD(3), RD(7)).doit()
```

```python
simplify(Add(RD(3), RD(7)))
```

```python
(Add(Integer(3), Integer(7))).evalf()
```

```python
simplify(Add(Integer(3), Integer(7)))
```

```python
w = Rel([[RD(2),RD(5)]], dom=['a','b'], cod=['x','y','z'])
```

```python
w
```

$$\left[\begin{matrix}RD(2) & RD(5)\end{matrix}\right]$$

```python
x = w*w.T
```

```python
x
```

$$\left[\begin{matrix}RD(4) + RD(25)\end{matrix}\right]$$

```python
simplify(x[0])
```

```python
r = Rel([[2,5]], dom=['a','b'], cod=['x','y','z'])
```

```python
p = r*r.T
```

```python
p
```

$$\left[\begin{matrix}29\end{matrix}\right]$$

```python
type(p)
```

    __main__.Rel

```python
p.dom
```

    ['a', 'b']

```python
p.cod
```

    ['a', 'b']

```python
r
```

$$\left[\begin{matrix}2 & 5\end{matrix}\right]$$

```python
r.dom
```

    ['a', 'b']

```python
Rel([2,5])
```

$$\left[\begin{matrix}2\\5\end{matrix}\right]$$

```python
s = sympify("r * r.T", locals={'r':r}, evaluate=False)
```

```python
s
```

$$\left[\begin{matrix}2\\5\end{matrix}\right] \left[\begin{matrix}2 & 5\end{matrix}\right]$$

```python
s.args[0].dom
```

    ['a', 'b']

```python
s.args[0].cod
```

    ['x', 'y', 'z']

```python
s.simplify
```

```python
m
```

$$\left[\begin{matrix}2\\3\end{matrix}\right]$$

```python
m.prova = ['a','b']
```

```python
m.prova
```

    ['a', 'b']

```python
s = sympify("m * m.T", locals={'m':m}, evaluate=False)
```

```python
s
```

$$\left[\begin{matrix}2\\3\end{matrix}\right] \left[\begin{matrix}2 & 3\end{matrix}\right]$$

```python
s.args[0]
```

$$\left[\begin{matrix}2\\3\end{matrix}\right]$$

```python
s.args[0].prova
```

```python

```


# A simple demonstration of coregionalization

This notebook shows how to construct a multi-output GP model using GPflow. We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X_1, f_1), \dots, (X_P, f_P)$, that is, we do not necessarily observe all the outputs for a particular input location (for cases where there are fully observed outputs for each input, see [Multi-output Gaussian processes in GPflow](./multioutput.ipynb) for a more efficient implementation). We allow each $f_i$ to have a different noise distribution by assigning a different likelihood to each.

For this problem, we model $f$ as a *coregionalized* Gaussian process, which assumes a kernel of the form:

\begin{equation}
\textrm{cov}(f_i(X), f_j(X^\prime)) = k(X, X^\prime) \cdot B[i, j].
\end{equation}

The covariance of the $i$th function at $X$ and the $j$th function at $X^\prime$ is a kernel applied at $X$ and $X^\prime$, times the $(i, j)$th entry of a positive definite $P \times P$ matrix $B$. This is known as the **intrinsic model of coregionalization (ICM)** (Bonilla and Williams, 2008).

To make sure that B is positive-definite, we parameterize it as:

\begin{equation}
B = W W^\top + \textrm{diag}(\kappa).
\end{equation}

To build such a model in GPflow, we need to perform the following two steps:

 * Create the kernel function defined previously, using the `Coregion` kernel class.
 * Augment the training data X with an extra column that contains an integer index to indicate which output an observation is associated with. This is essential to make the data work with the `Coregion` kernel.
 * Create a likelihood for each output using the `SwitchedLikelihood` class, which is a container for other likelihoods.
 * Augment the training data Y with an extra column that contains an integer index to indicate which likelihood an observation is associated with.

```python
import gpflow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

from gpflow.ci_utils import ci_niter

plt.rcParams["figure.figsize"] = (12, 6)
np.random.seed(123)
```

## Data preparation
We start by generating some training data to fit the model with. For this example, we choose the following two correlated functions for our outputs:

\begin{align}
y_1 &= \sin(6x) + \epsilon_1, \qquad \epsilon_1 \sim \mathcal{N}(0, 0.009) \\
y_2 &= \sin(6x + 0.7) + \epsilon_2, \qquad \epsilon_2 \sim \mathcal{N}(0, 0.01)
\end{align}

```python
# make a dataset with two outputs, correlated, heavy-tail noise. One has more noise than the other.
X1 = np.random.rand(100, 1)  # Observed locations for first output
X2 = np.random.rand(50, 1) * 0.5  # Observed locations for second output

Y1 = np.sin(6 * X1) + np.random.randn(*X1.shape) * 0.03
Y2 = np.sin(6 * X2 + 0.7) + np.random.randn(*X2.shape) * 0.1

plt.figure(figsize=(8, 4))
plt.plot(X1, Y1, "x", mew=2)
_ = plt.plot(X2, Y2, "x", mew=2)
```

## Data formatting for the coregionalized model
We add an extra column to our training dataset that contains an index that specifies which output is observed.

```python
# Augment the input with ones or zeros to indicate the required output dimension
X_augmented = np.vstack((np.hstack((X1, np.zeros_like(X1))), np.hstack((X2, np.ones_like(X2)))))

# Augment the Y data with ones or zeros that specify a likelihood from the list of likelihoods
Y_augmented = np.vstack((np.hstack((Y1, np.zeros_like(Y1))), np.hstack((Y2, np.ones_like(Y2)))))
```

## Building the coregionalization kernel
We build a coregionalization kernel with the Matern 3/2 kernel as the base kernel. This acts on the leading ([0]) data dimension of the augmented X values. The `Coregion` kernel indexes the outputs, and acts on the last ([1]) data dimension (indices) of the augmented X values. To specify these dimensions, we use the built-in `active_dims` argument in the kernel constructor. To construct the full multi-output kernel, we take the product of the two kernels (for a more in-depth tutorial on kernel combination, see [Manipulating kernels](./kernels.ipynb)).

```python
output_dim = 2  # Number of outputs
rank = 1  # Rank of W

# Base kernel
k = gpflow.kernels.Matern32(active_dims=[0])

# Coregion kernel
coreg = gpflow.kernels.Coregion(output_dim=output_dim, rank=rank, active_dims=[1])

kern = k * coreg
```

**Note:** W = 0 is a saddle point in the objective, which would result in the value of `W` not being optimized to fit the data.
Hence, by default, the `W` matrix is initialized with 0.1. Alternatively, you could re-initialize the matrix to random entries.

## Constructing the model
The final element in building the model is to specify the likelihood for each output dimension. To do this, use the `SwitchedLikelihood` object in GPflow.

```python
# This likelihood switches between Gaussian noise with different variances for each f_i:
lik = gpflow.likelihoods.SwitchedLikelihood(
    [gpflow.likelihoods.Gaussian(), gpflow.likelihoods.Gaussian()]
)

# now build the GP model as normal
m = gpflow.models.VGP((X_augmented, Y_augmented), kernel=kern, likelihood=lik)

# fit the covariance function parameters
maxiter = ci_niter(10000)
gpflow.optimizers.Scipy().minimize(
    m.training_loss, m.trainable_variables, options=dict(maxiter=maxiter), method="L-BFGS-B",
)
```

          fun: -225.66185790128998
     hess_inv: <11483x11483 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 9.80972157e-03,  2.32377024e-03,  1.01201782e-02, ...,
           -1.20376378e-10,  4.51611141e-11,  3.58283369e-03])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 1443
          nit: 1317
       status: 0
      success: True
            x: array([ 7.68640462e-02, -6.25721849e-01, -1.28675184e+00, ...,
           -3.86079029e-12, -1.01535023e-12,  4.49399991e-01])

That's it: the model is trained. Let's plot the model fit to see what's happened.

```python
def plot_gp(x, mu, var, color, label):
    plt.plot(x, mu, color=color, lw=2, label=label)
    plt.fill_between(
        x[:, 0],
        (mu - 2 * np.sqrt(var))[:, 0],
        (mu + 2 * np.sqrt(var))[:, 0],
        color=color,
        alpha=0.4,
    )

def plot(m):
    plt.figure(figsize=(8, 4))
    Xtest = np.linspace(0, 1, 100)[:, None]
    (line,) = plt.plot(X1, Y1, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.zeros_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y1")

    (line,) = plt.plot(X2, Y2, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.ones_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y2")

    plt.legend()

plot(m)
```

From the plots, we can see:

 - The first function (blue) has low posterior variance everywhere because there are so many observations, and the noise variance is small.
 - The second function (orange) has higher posterior variance near the data, because the data are more noisy, and very high posterior variance where there are no observations (x > 0.5).
 - The model has done a reasonable job of estimating the noise variance and lengthscale.
 - The model recognises the correlation between the two functions and is able to suggest (with uncertainty) that because x > 0.5 the orange curve should follow the blue curve (which we know to be the truth from the data-generating procedure).

The covariance matrix between outputs is as follows:

```python
B = coreg.output_covariance().numpy()
print("B =", B)
_ = plt.imshow(B)
```

## References

Bonilla, Edwin V., Kian M. Chai, and Christopher Williams. "Multi-task Gaussian process prediction." _Advances in neural information processing systems_. 2008.


# Introduction to Deep Learning with PyTorch

In this notebook, you'll get introduced to [PyTorch](http://pytorch.org/), a framework for building and training neural networks. PyTorch in a lot of ways behaves like the arrays you love from Numpy. These Numpy arrays, after all, are just tensors. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with Python and the Numpy/Scipy stack compared to TensorFlow and other frameworks.

## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

### Stack them up!

We can assemble these unit neurons into layers and stacks, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.

We can express this mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.

With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
import torch
```

First, let's see how we work with PyTorch tensors. These are the fundamental data structures of neural networks and PyTorch, so it's imporatant to understand how these work.

```python
x = torch.rand(3, 2)
x.size()
```

    torch.Size([3, 2])

```python
y = torch.ones(x.size())
y
```

    tensor([[1., 1.],
            [1., 1.],
            [1., 1.]])

```python
z = x + y
z
```

    tensor([[1.0227, 1.0487],
            [1.4433, 1.3408],
            [1.1525, 1.4355]])

In general PyTorch tensors behave similar to Numpy arrays. They are zero indexed and support slicing.

```python
z[0]
```

    tensor([1.0227, 1.0487])

```python
z[:, 1:]
```

    tensor([[1.0487],
            [1.3408],
            [1.4355]])

Tensors typically have two forms of methods, one method that returns another tensor and another method that performs the operation in place. That is, the values in memory for that tensor are changed without creating a new tensor. In-place functions are always followed by an underscore, for example `z.add()` and `z.add_()`.

```python
# Return a new tensor z + 1
z.add(1)
```

    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])

```python
# z tensor is unchanged
z
```

    tensor([[1.0227, 1.0487],
            [1.4433, 1.3408],
            [1.1525, 1.4355]])

```python
# Add 1 and update z tensor in-place
z.add_(1)
```

    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])

```python
# z has been updated
z
```

    tensor([[2.0227, 2.0487],
            [2.4433, 2.3408],
            [2.1525, 2.4355]])

### Reshaping

Reshaping tensors is a really common operation. First to get the size and shape of a tensor use `.size()`. Then, to reshape a tensor, use `.resize_()`. Notice the underscore, reshaping is an in-place operation.

```python
z.size()
```

    torch.Size([3, 2])

```python
z.resize_(2, 3)
```

    tensor([[2.0227, 2.0487, 2.4433],
            [2.3408, 2.1525, 2.4355]])

```python
z
```

    tensor([[2.0227, 2.0487, 2.4433],
            [2.3408, 2.1525, 2.4355]])

## Numpy to Torch and back

Converting between Numpy arrays and Torch tensors is super simple and useful. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.

```python
a = np.random.rand(4,3)
a
```

    array([[0.71561694, 0.49365764, 0.42654007],
           [0.38610867, 0.73036579, 0.40426613],
           [0.30673351, 0.72503498, 0.343875  ],
           [0.48630183, 0.74443254, 0.40740172]])

```python
b = torch.from_numpy(a)
b
```

    tensor([[0.7156, 0.4937, 0.4265],
            [0.3861, 0.7304, 0.4043],
            [0.3067, 0.7250, 0.3439],
            [0.4863, 0.7444, 0.4074]], dtype=torch.float64)

```python
b.numpy()
```

    array([[0.71561694, 0.49365764, 0.42654007],
           [0.38610867, 0.73036579, 0.40426613],
           [0.30673351, 0.72503498, 0.343875  ],
           [0.48630183, 0.74443254, 0.40740172]])

The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

    tensor([[1.4312, 0.9873, 0.8531],
            [0.7722, 1.4607, 0.8085],
            [0.6135, 1.4501, 0.6877],
            [0.9726, 1.4889, 0.8148]], dtype=torch.float64)

```python
# Numpy array matches new values from Tensor
a
```

    array([[1.43123389, 0.98731528, 0.85308015],
           [0.77221734, 1.46073158, 0.80853226],
           [0.61346702, 1.45006997, 0.68775   ],
           [0.97260366, 1.48886509, 0.81480343]])

## Feedforward neural networks with PyTorch

Next I'll show you how to use this pattern to build a neural network with PyTorch. First up, we need to get our dataset. This is provided through the `torchvision` package. The code below will download the MNIST dataset, then create training and test datasets for us.

```python
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
```

```python
# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                             ])
# Download and load the training data
trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

# Download and load the test data
testset = datasets.MNIST('MNIST_data/', download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to MNIST_data/MNIST/raw/train-images-idx3-ubyte.gz

      0%|          | 0/9912422 [00:00<?, ?it/s]

    Extracting MNIST_data/MNIST/raw/train-images-idx3-ubyte.gz to MNIST_data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw/train-labels-idx1-ubyte.gz

      0%|          | 0/28881 [00:00<?, ?it/s]

    Extracting MNIST_data/MNIST/raw/train-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to MNIST_data/MNIST/raw/t10k-images-idx3-ubyte.gz

      0%|          | 0/1648877 [00:00<?, ?it/s]

    Extracting MNIST_data/MNIST/raw/t10k-images-idx3-ubyte.gz to MNIST_data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw/t10k-labels-idx1-ubyte.gz

      0%|          | 0/4542 [00:00<?, ?it/s]

    Extracting MNIST_data/MNIST/raw/t10k-labels-idx1-ubyte.gz to MNIST_data/MNIST/raw

```python
dataiter = iter(trainloader)
images, labels = dataiter.next()
```

We have the training data loaded into `trainloader` and we make that an iterator with `iter(trainloader)`. We'd use this to loop through the dataset for training, but here I'm just grabbing the first batch so we can check out the data. We can see below that `images` is just a tensor with size (64, 1, 28, 28). So, 64 images per batch, 1 color channel, and 28x28 images.

```python
images.size()
```

    torch.Size([64, 1, 28, 28])

```python
plt.imshow(images[1].numpy().squeeze(), cmap='Greys_r')
```

## Building networks with PyTorch

Here I'll use PyTorch to build a simple feedfoward network to classify the MNIST images. That is, the network will receive a digit image as input and predict the digit in the image.

To build a neural network with PyTorch, you use the `torch.nn` module. The network itself is a class inheriting from `torch.nn.Module`. You define each of the operations separately, like `nn.Linear(784, 128)` for a fully connected linear layer with 784 inputs and 128 units.

The class needs to include a `forward` method that implements the forward pass through the network. In this method, you pass some input tensor `x` through each of the operations you defined earlier. The `torch.nn` module also has functional equivalents for things like ReLUs in `torch.nn.functional`. This module is usually imported as `F`. Then to use a ReLU activation on some layer (which is just a tensor), you'd do `F.relu(x)`. Below are a few different commonly used activation functions.

So, for this network, I'll build it with three fully connected layers, then a softmax output for predicting classes. The softmax function is similar to the sigmoid in that it squashes inputs between 0 and 1, but it's also normalized so that all the values sum to one like a proper probability distribution.

```python
from torch import nn
from torch import optim
import torch.nn.functional as F
```

```python
class Network(nn.Module):
    def __init__(self):
        super().__init__()
        # Defining the layers, 128, 64, 10 units each
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 64)
        # Output layer, 10 units - one for each digit
        self.fc3 = nn.Linear(64, 10)

    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)

        return x

    def predict(self, x):
        ''' This function for predicts classes by calculating the softmax '''
        logits = self.forward(x)
        return F.softmax(logits)

net = Network()
net
```

    Network(
      (fc1): Linear(in_features=784, out_features=128, bias=True)
      (fc2): Linear(in_features=128, out_features=64, bias=True)
      (fc3): Linear(in_features=64, out_features=10, bias=True)
    )

### Initializing weights and biases

The weights and such are automatically initialized for you, but it's possible to customize how they are initialized. The weights and biases are tensors attached to the layer you defined, you can get them with `net.fc1.weight` for instance.

```python
print(net.fc1.weight)
print(net.fc1.bias)
```

    Parameter containing:
    tensor([[-0.0057, -0.0222, -0.0297,  ...,  0.0106,  0.0173, -0.0225],
            [-0.0339,  0.0224, -0.0163,  ..., -0.0079,  0.0323,  0.0258],
            [ 0.0109, -0.0319, -0.0306,  ..., -0.0210, -0.0067,  0.0306],
            ...,
            [-0.0046, -0.0081,  0.0092,  ...,  0.0042, -0.0266,  0.0322],
            [-0.0012,  0.0152, -0.0134,  ...,  0.0292, -0.0341,  0.0243],
            [ 0.0046, -0.0309, -0.0150,  ..., -0.0297, -0.0310,  0.0136]],
           requires_grad=True)
    Parameter containing:
    tensor([-0.0004, -0.0107,  0.0340,  0.0038, -0.0255,  0.0248, -0.0322,  0.0329,
             0.0173, -0.0025,  0.0151,  0.0064,  0.0170, -0.0010,  0.0085,  0.0134,
             0.0218,  0.0333, -0.0140,  0.0053, -0.0346,  0.0312,  0.0322,  0.0046,
             0.0344, -0.0294, -0.0068, -0.0018, -0.0324,  0.0271,  0.0263,  0.0123,
            -0.0278,  0.0357, -0.0165,  0.0255,  0.0143,  0.0134, -0.0034, -0.0048,
             0.0292, -0.0210,  0.0116,  0.0053,  0.0032,  0.0317,  0.0139, -0.0245,
            -0.0308,  0.0148,  0.0132, -0.0287, -0.0030,  0.0274, -0.0146, -0.0112,
             0.0143, -0.0158,  0.0348, -0.0047, -0.0076,  0.0105,  0.0290,  0.0119,
             0.0177, -0.0177,  0.0037,  0.0258,  0.0187, -0.0307, -0.0141, -0.0047,
            -0.0284,  0.0166,  0.0218,  0.0156,  0.0018,  0.0190, -0.0153,  0.0005,
            -0.0234,  0.0141,  0.0308, -0.0212, -0.0332,  0.0209, -0.0095, -0.0133,
            -0.0253, -0.0102,  0.0297,  0.0129,  0.0206, -0.0304,  0.0304,  0.0333,
            -0.0035,  0.0146, -0.0260,  0.0345,  0.0121, -0.0099, -0.0237,  0.0070,
             0.0276, -0.0247, -0.0053,  0.0221, -0.0091, -0.0124, -0.0283, -0.0100,
             0.0251, -0.0169, -0.0142, -0.0002,  0.0275, -0.0118,  0.0345,  0.0324,
             0.0232,  0.0227, -0.0022, -0.0108, -0.0273, -0.0060,  0.0256,  0.0102],
           requires_grad=True)

For custom initialization, we want to modify these tensors in place. Once we have the tensors, we can fill them with zeros (for biases) or random normal values.

```python
# Set biases to all zeros
net.fc1.bias.data.fill_(0)
```

    tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0.])

```python
# sample from random normal with standard dev = 0.01
net.fc1.weight.data.normal_(std=0.01)
```

    tensor([[ 6.4400e-03,  1.0824e-02,  1.0809e-02,  ..., -1.0652e-03,
              4.1703e-03,  2.0442e-02],
            [-2.8747e-02, -7.4504e-03,  1.6087e-03,  ...,  1.6905e-04,
             -1.6302e-03, -5.7481e-03],
            [-1.6169e-03,  8.1673e-03,  1.6747e-02,  ..., -1.0482e-02,
              1.9876e-03,  8.7329e-03],
            ...,
            [-7.5507e-03, -9.2924e-03,  6.6852e-03,  ..., -1.0514e-02,
              1.7884e-02,  1.3018e-02],
            [ 9.7859e-03, -1.6901e-03,  5.5006e-05,  ...,  4.0575e-03,
             -1.3077e-02,  4.5288e-06],
            [-2.5877e-02, -5.0783e-03,  1.5483e-03,  ...,  7.1621e-03,
              5.5963e-04,  4.8401e-03]])

### Forward pass

Now that we have a network, let's see what happens when we pass in an image.

For a better visualization, we define the following function

```python
def view_classify(img, ps):
    ''' Function for viewing an image and it's predicted classes.
    '''
    ps = ps.data.numpy().squeeze()

    fig, (ax1, ax2) = plt.subplots(figsize=(6,7), ncols=2)
    ax1.imshow(img.numpy().squeeze())
    ax1.axis('off')
    ax2.barh(np.arange(10), ps)
    ax2.set_aspect(0.1)
    ax2.set_yticks(np.arange(10))
    ax2.set_yticklabels(np.arange(10).astype(int), size='large');
    ax2.set_title('Digit Probability')
    ax2.set_xlim(0, 1.1)
```

```python
# Grab some data
dataiter = iter(trainloader)
inputs, labels = dataiter.next()
print(inputs.size(), labels.size())
inputs.resize_(64, 1, 784)

# Forward pass through the network
img_idx = 0
logits = net(inputs[img_idx,:])

# Predict the class from the network output
ps = F.softmax(logits)

img = images[img_idx]
view_classify(img.resize_(1, 28, 28), ps)
```

```python
inputs.size()
```

    torch.Size([64, 1, 784])

As you can see above, our network has basically no idea what this digit is. It's because we haven't trained it yet, all the weights are random! Next up, we'll look at training this network so it learns how to properly classify these digits.


This post is a continuation of the [Neural Network](http://lakshgupta.github.io/2015/06/12/NeuralNetwork/) post where we learned about the basics of a neural network and applied it on the handwritten digit recognition problem. Here we'll cover the following topics which can help our neural network to perform better in terms of the accuracy of the model.
- [Rectified linear unit (ReLU) function](https://en.wikipedia.org/wiki/Rectifier_(neural_networks))
- [Softmax function](https://en.wikipedia.org/wiki/Softmax_function)
- [Mini-batch gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)

So let's get started!

```julia
using MNIST
using PyPlot
```

ReLU Activation Function : $$f(x) = max(0, x)$$

We have already seen the sigmoid function instead of which we'll use ReLU activation function for the input and hidden layers in the current neural network architecture because it is faster and does not suffer from the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem).

>- Biological plausibility: One-sided, compared to the antisymmetry of [tanh](https://en.wikipedia.org/wiki/Hyperbolic_function#Tanh).
>- Sparse activation: For example, in a randomly initialized network, only about 50% of hidden units are activated (having a non-zero output).
>- Efficient gradient propagation: No [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) or exploding effect.
>- Efficient computation: Only comparison, addition and multiplication.
>
>For the first time in 2011, the use of the rectifier as a non-linearity has been shown to enable training deep supervised neural networks without requiring unsupervised pre-training. Rectified linear units, compared to [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) function or similar activation functions, allow for faster and effective training of deep neural architectures on large and complex datasets.
>
>Potential problems:
Non-differentiable at zero: however it is differentiable at any point arbitrarily close to 0.

- Wikipedia

>"What neuron type should I use?" Use the ReLU non-linearity, be careful with your learning rates and possibly monitor the fraction of "dead" units in a network. If this concerns you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but expect it to work worse than ReLU/Maxout.

- Andrej Karpathy

```julia
# ReLU function
function relu(z::Matrix{Float64})
    return max(0.0, z);
end
```

    relu (generic function with 1 method)

```julia
# sigmoid function
function sigmoid(z::Matrix{Float64})
    g = 1.0 ./ (1.0 + exp(-z));
    return g;
end
```

    sigmoid (generic function with 1 method)

Softmax Function : $$f_i(x) = \frac{e^{x_i}}{\sum_k e^{x_k}}$$

Softmax function gives us normalized class probabilities. It takes the input, exponentiates it to generate unnormalized probabilities and then it uses a normalization factor to result in normalized probabilities. The output for each class lies between $0$ and $1$, and the sum of all the class probabilities is equal to $1$.

Forward Process

I have re-structured the program to have replaceable components to make the experiment easy. There is a small change in the architecture of the neural network:
- output layer will always be using the Softmax activation function
- rest of all the layers we'll either use ReLU activation function

With the introduction of the ReLU and Softmax function we can look at our forward step.
At the first input layer:
$$a^{(1)} = x$$

At the hidden layer:
$$
\begin{array}{ll}
z^{(2)} = \theta^{(1)}*a^{(1)} + bias^{(1)} \\[2ex]
a^{(2)} = ReLU(z^{(2)})
\end{array}
$$

At the output layer:
$$
\begin{array}{ll}
z^{(3)} = \theta^{(2)}*a^{(2)} + bias^{(2)} \\[2ex]
\hat y = softmax(z^{(3)})
\end{array}
$$

Each entry in $p$ vector defines our output normalized probability for that specific class.

```julia
function forwardNN(activationFn::Function, x::Matrix{Float64})
    global network;
    # collect input for each layer
    # the last element will be the output from the neural network
    activation = Matrix{Float64}[];
    # initialize activation vector with the actual data
    push!(activation, x);
    for layer in 1:length(network)-1
        push!(activation, activationFn((activation[layer]*network[layer][1]) .+ network[layer][2]))
    end
    # softmax on last layer
    score = activation[length(network)]*network[length(network)][1] .+ network[length(network)][2]
    exp_scores = exp(score);
    yCap = exp_scores ./ sum(exp_scores, 2);

    return activation, yCap;
end
```

    forwardNN (generic function with 1 method)

Cost Function: $J$

The [last time](http://lakshgupta.github.io/2015/06/12/NeuralNetwork/) we converted each output to an array of size $10$ with $1$ on the index representing the actual output and $0$ on the rest of the indices. Therefore we used a special case of the cross-entropy cost function where number of classes is equal to 2, assuming all of the output classes are independent of each other:

$$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m} \sum_{i=1}^{k}[ y^{(i)}_k\log{(h_{\theta}(x^{(i)})_k)} + (1-y^{(i)}_k)\log({1-(h_{\theta}(x^{(i)}))_k)}]$$

>If we have multiple independent binary attributes by which to classify the data, we can use a network with multiple logistic outputs and cross-entropy error. For multinomial classification problems (1-of-n, where n > 2) we use a network with n outputs, one corresponding to each class, and target values of 1 for the correct class, and 0 otherwise. Since these targets are not independent of each other, however, it is no longer appropriate to use logistic output units. The corect generalization of the logistic sigmoid to the multinomial case is the softmax activation function.

- Genevieve (Jenny) B. Orr

Since we are using softmax in the output layer, the probability for one class is divided by the sum of probabilities for all the classes. As a result, we will be using the generalized cross entropy cost function:

\begin{align}
J(\theta) = - \left[ \sum_{i=1}^{m} \sum_{k=1}^{K}  1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right]
\end{align}

>In the probabilistic interpretation, we are therefore minimizing the negative log likelihood of the correct class, which can be interpreted as performing Maximum Likelihood Estimation (MLE).

- Andrej Karpathy

More detailed information can be found [here](http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/).

```julia
function costFunction(truth::Vector{Float64}, probability::Matrix{Float64})
    global network;
    # average cross-entropy loss
    m = size(truth,1)

    corect_logprobs = [-log(probability[j,convert(Int32, truth[j])]) for j in 1:m];
    data_loss = sum(corect_logprobs)/m;

    #L2 regularization
    reg_loss = 0;
    for j in 1:length(network)
        reg_loss = reg_loss + 0.5*lambda*sum(network[j][1].^2)/m;
    end

    loss = data_loss + reg_loss;
    return loss;
end
```

    costFunction (generic function with 1 method)

```julia
# gradient of the sigmoid function evaluated at a
function sigmoidGradient(a::Matrix{Float64})
  return a.*(1-a);
end
```

    sigmoidGradient (generic function with 1 method)

```julia
# gradient of the ReLU function evaluated at a
function reluGradient(a::Matrix{Float64})
    grad = ones(a);
    grad[a.<=0] = 0;
    return grad;
end
```

    reluGradient (generic function with 1 method)

Backward process

The algorithm to learn the parameters is still the same, backpropagation. We need to calculate the parameter gradients to update them using the chain rule. For simplicity consider the cost function for a single input,

\begin{align}
J(\theta) &= - \left[ \sum_{k=1}^{K} y_k* \log (\hat y_k) \right]
\end{align}
where,

$$\hat y_k = softmax(z^3_k) = \frac{\exp(z^3_k)}{\sum_{j=1}^K \exp(z^3_j) }$$
and $y_k$ is either $0$ or $1$ as per the probability of the correct class.

therefore,
$$ \frac{\partial J}{\partial \hat y_k} = − \frac{y_k}{\hat y_k} $$
$$ \frac{\partial \hat y_k}{\partial z^{(3)}_i} = \begin{cases}
\hat y_k(1-\hat y_k),  & \text{i = k} \\[2ex]
-\hat y_i \hat y_k, & \text{i $\neq$ k} \\
\end{cases} \\
$$

$$\begin{eqnarray}
\frac{\partial J}{\partial z^{(3)}_i}
&=&\sum_{k = 1}^{K}\frac{\partial J}{\partial \hat y_k}\frac{\partial \hat y_k}{\partial z^{(3)}_i} \\ \nonumber
&=& \underbrace{\frac{\partial J}{\partial \hat y_i}\frac{\partial \hat y_i}{\partial x_i}}_{i = k}
+ \underbrace{\sum_{k \neq i}\frac{\partial J(\theta)}{\partial \hat y_k}\frac{\partial \hat y_k}{\partial x_i}}_{i \neq k} \\ \nonumber
&=&-y_i(1 - \hat y_i) + \sum_{k \neq i} y_k \hat y_k \\ \nonumber
&=&-y_i + \sum_{k} y_k \hat y_k \\ \nonumber
&=& \hat y_i - y_i \\
\end{eqnarray}
$$

The correct output element in the vector $y$ is always $1$ else $0$ since we are now dealing with the normalized class probabilities.

For the softmax output layer:

$$
\begin{eqnarray}
\dfrac{\partial J}{\partial \theta^{(2)}}
&=& \dfrac{\partial J}{\partial z^{(3)}} \dfrac{\partial z^{(3)}}{\partial \theta^{(2)}} \\
&=& (\hat y - y)* a^{(2)}
\end{eqnarray}
$$

$$
\begin{eqnarray}
\dfrac{\partial J}{\partial bias^{(2)}}
&=& \dfrac{\partial J}{\partial z^{(3)}} \dfrac{\partial z^{(3)}}{\partial bias^{(2)}} \\
&=& (\hat y - y)* 1
\end{eqnarray}
$$

For the hidden layer with the ReLU activation function:

$$\begin{eqnarray}
\frac{\partial J}{\partial \theta^{(1)}}
&=&\frac{\partial J}{\partial z^{(3)}}\frac{\partial z^{(3)}}{\partial g(z^{(2)})}\frac{\partial g(z^{(2)})}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial \theta^{(1)}}\\ \nonumber
&=& (\hat y - y)* \theta^{(2)} * g'(z^{(2)})*a^{(1)} \\ \nonumber
\end{eqnarray}$$

$$\begin{eqnarray}
\frac{\partial J}{\partial bias^{(1)}}
&=&\frac{\partial J}{\partial z^{(3)}}\frac{\partial z^{(3)}}{\partial g(z^{(2)})}\frac{\partial g(z^{(2)})}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial bias^{(1)}}\\ \nonumber
&=& (\hat y - y)* \theta^{(2)} * g'(z^{(2)})*1 \\ \nonumber
\end{eqnarray}$$

Now we can update the weights as:

$$\theta^{(l)} \leftarrow \theta^{(l)} - \alpha \dfrac{\partial J}{\partial \theta^{l}}$$

All the above calculations is being performed by the backwordNN and the updateThetas method below.

```julia
function backwardNN(activationFnGrad::Function, a::Vector{Matrix{Float64}}, y::Vector{Float64}, dscores::Matrix{Float64})
    global network;
    m = size(y,1);
    delta = Array(Matrix{Float64}, 1,length(network));
    # start from the last layer to backpropagate the error
    # compute the gradient on scores
    for j in 1:size(dscores,1)
        dscores[j,convert(Int32, y[j])] -= 1;
    end
    delta[length(network)] = dscores/m; # normalization factor
    # backpropagate
    for j in length(network)-1:-1:1
        delta[j] = delta[j+1]*network[j+1][1]'.*activationFnGrad(a[j+1]);
    end
    return delta;
end
```

    backwardNN (generic function with 1 method)

```julia
function updateThetas(a::Vector{Matrix{Float64}}, delta::Matrix{Matrix{Float64}})
    global network;
    for j in 1:length(network)
        # update theta
        network[j][1] = network[j][1] - alpha*(a[j]'*delta[j] + lambda*network[j][1])
        # update bias
        network[j][2] =  network[j][2] - alpha*sum(delta[j],1);
    end
end
```

    updateThetas (generic function with 1 method)

```julia
function predict(activationFn, data::Matrix{Float64})
    activation, probs = forwardNN(activationFn, data);
    predicted_class = [indmax(probs[i,:]) for i in 1:size(probs,1)]
    return predicted_class;
end
```

    predict (generic function with 1 method)

```julia
function accuracy(truth, prediction)
    correct = 0;
    for i in 1:length(truth)
        if truth[i] == prediction[i]
            correct = correct + 1;
        end
    end
    println("training accuracy: ", correct/length(truth)*100);
end
```

    accuracy (generic function with 1 method)

Training a model

Based on the new structure of the program, we'll build up our neural network and train our model.

Load Data

```julia
# ===================
# load training data
# ===================
X,Y = traindata(); #X:(784x60000), y:(60000x1)
X /= 255.0; # scale the input between 0 and 1
X = X'; #X:(60000X784)
Y = Y+1; # adding 1 to handle index, hence value 1 represent digit 0 now
# number of instances
println(size(Y,1));
```

    60000

Define Network

```julia
inputLayerSize = size(X,2);
hiddenLayerSize = 100;
outputLayerSize = 10;
```

```julia
# define a network
network = [];

# add first layer to the network
layer1 = Array(Matrix{Float64}, 1,2)
#theta1
layer1[1] = 0.01*randn(inputLayerSize, hiddenLayerSize);
#bias1
layer1[2] = zeros(1, hiddenLayerSize);
push!(network,layer1);

# add second layer to the network
layer2 = Array(Matrix{Float64}, 1,2)
#theta2
layer2[1] = 0.01*randn(hiddenLayerSize, outputLayerSize);
#bias2
layer2[2] = zeros(1, outputLayerSize);
push!(network,layer2);
```

Training: Mini-batch gradient descent

The last time we used gradient descent to train our model. In this implementation we'll switch to Mini-Batch gradient descent algorithm which is not much different from regular gradient descent. It's just that instead of working on the whole dataset we'll work on a smaller dataset in each iteration. Because of this change our training algorithm will become computationally faster since large datasets are difficult to handle in memory which makes vectorization much less efficient.

>In particular, suppose that our error function is particularly pernicious and has a bunch of little valleys. If we used the entire training set to compute each gradient, our model would get stuck in the first valley it fell into (since it would register a gradient of 0 at this point). If we use smaller mini-batches, on the other hand, we'll get more noise in our estimate of the gradient. This noise might be enough to push us out of some of the shallow valleys in the error function.
>
- Quora

One thing to take care in the while training is that mini-batches need to be balanced for classes otherwise the estimation of the gradient using mini-batch gradient descent would be way off then the gradient calculated using the whole dataset.

```julia
alpha = 1e-0; # step size
lambda = 1e-3; # regularization factor
numItr = 1500;
sampleCostAtEveryItr = 10;
batchSize = convert(Int32,floor(size(X,1)/10));
J = [];
for itr in 1:numItr
    # take next batch of random instances
    minibatch = collect(rand(1:size(X,1), batchSize));
    # feedforward
    activations, probs = forwardNN(relu, X[minibatch,:]);
    # cost
    if itr%sampleCostAtEveryItr == 1
        activationsX, probsX = forwardNN(relu, X);
        push!(J, costFunction(Y, probsX));
    end
    # backpropagation
    newThetas = backwardNN(reluGradient, activations, Y[minibatch], probs);
    # update parameters
    updateThetas(activations, newThetas);
end
```

The code above is using ReLU activation function for the hidden layers but the program is modular enough to let us experiment with different activation functions, for example sigmoid.

Prediction and Accuracy

```julia
accuracy(Y, predict(relu, X));
```

    training accuracy: 97.975

This is approximately 10% improvement over our previous implementation.

```julia
# plot the cost per iteration
sampleIdxJ = [1+sampleCostAtEveryItr*i for i in 0:length(J)-1]
plot(sampleIdxJ, J)
xlabel("Sampled Iterations")
ylabel("Cost")
grid("on")
```

The spikes in the graph are due to the use of mini-batch gradient descent, which estimates the cost over the whole dataset hence sometimes moves away from the minima but in the end converges satisfactorily.

```julia
# test data
XTest,YTest = testdata();
XTest /= 255.0;
XTest = XTest';
YTest = YTest+1;
accuracy(YTest, predict(relu, XTest));
```

    training accuracy: 97.24000000000001

References:

- [Rectified linear unit (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks))
- [Learning representations by back-propagating errors](http://www.cs.toronto.edu/~hinton/absps/naturebp.pdf)
- [Practical recommendations for gradient-based training of deep architectures](http://arxiv.org/abs/1206.5533)
- [Efficient BackProp](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)
- [Deep Learning](http://www.iro.umontreal.ca/~bengioy/dlbook/mlp.html)
- [word2vec Parameter Learning Explained](http://arxiv.org/abs/1411.2738)
- [binary cross entropy cost function with softmax?](https://www.reddit.com/r/cs231n/comments/45u13l/binary_cross_entropy_cost_function_with_softmax/)
- [Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy](http://arxiv.org/abs/cond-mat/0512017)
- [word2vec gradients](https://courses.cs.ut.ee/MTAT.03.277/2015_fall/uploads/Main/word2vec.pdf)
- [Softmax Regression](http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/)


Make sure you remove `raise NotImplementedError()` and fill in any place that says `# YOUR CODE HERE`, as well as your `NAME`, `ID`, and `LAB_SECTION` below:

```python
NAME = "FAHMID BIN KIBRIA"
ID = "19201063"
LAB_SECTION = "02"
```

---

# Floating Point Representation

## 8 bit Floating Point Representation

We will implement an 8 bit Floating point number system in this part, where First bit is a sign bit, next three bits are exponents,
and the rest 4 bits are mantissas (significands).

Numbers are divided into two categories: normals and subnormals.

For normals the encoding is:
\begin{equation}
(-1)^{sign} \times 2^{(e-3)} \times 1.fraction
\end{equation}
For subnormals the encoding is:
\begin{equation}
(-1)^{sign} \times 2^{(1-3)} \times 0.fraction
\end{equation}

Example:
\begin{aligned}
    0 001 0000 &= 2^{(1-3)} \times 1.0 = 0.25\\
    0 010 0000 &= 2^{(2-3)} \times 1.0 = 0.5\\
    0 011 0000 &= 2^{(3-3)} \times 1.0 = 1.0\\
    0 011 0100 &= 2^{(3-3)} \times 1.25 = 1.25
\end{aligned}

If the exponent is 0 then the number is considered subnormal.

Example:
\begin{aligned}
    0 000 0000 &= 2^{-2} \times 0   = 0.0\\
    0 000 1000 &= 2^{-2} \times 0.5 = 0.125\\
    0 000 1100 &= 2^{-2} \times 0.75 = 0.1875
\end{aligned}

If the exponent is 7 then and mantissa is zero then the number is
considered infinity (inf), otherwise it is not-a-number (nan).

Example:
\begin{aligned}
    0 111 0000 &= +\infty\\
    1 111 0000 &= -\infty\\
    X 111 XXXX &= NaN
\end{aligned}

### Instructions
In this task, you need to implement the `Float8()` class. Some parts of the class are already written for you. You only need to modify `__init__(self, bitstring)` and `calculate_value(self)` fuctions, where it says `# YOUR CODE HERE`. You may ignore `raise NotImplementedError()`.

```python
import math
import itertools
import matplotlib.pyplot as plt
```

```python
class Float8():

    def __init__(self, bitstring):
        '''Constructor
        takes a 8-bit string of 0's and 1's as input and stores the sub-strings
        accordingly.
        Usage: Float8('00011110')
        '''

        # Make sure the input consists of exactly 8-bits.
        assert(len(bitstring)==8)

        # Assign the sign bit
        # self.sign = bitstring[?]

        # Assign the exponent part
        # self.exponent = bitstring[?]

        # Assign the mantissa
        # self.mantissa = bitstring[?]

        # YOUR CODE HERE
        self.sign = bitstring[0]
        self.exponent = bitstring[1:4]
        self.mantissa = bitstring[4:8]

        self.val = self.calculate_value()

    def __str__(self):
        return f'Sign bit value: {self.sign}\n' + \
            f'Exponent value: {self.exponent}\n' + \
            f'Mantissa value: {self.mantissa}\n' + \
            f'Floating value: {self.val}\n'

    def tobitstring(self):
        return self.sign + self.exponent + self.mantissa

    def toformattedstring(self):
        return ' '.join([self.sign, self.exponent, self.mantissa])

    def calculate_value(self):
        '''Calculate the value of the number from bits'''

        # Initialize with zero
        val = 0.0

        # YOUR CODE HERE
        if int(self.exponent,2)==7 and int(self.mantissa,2)==0:
            if self.sign=="0":
                val="inf"
            elif self.sign=="1":
                val="-inf"

        elif int(self.exponent,2)==7 and int(self.mantissa,2)!=0:
            val="nan"

        elif int(self.exponent,2)!=0:
            mantissa="."+self.mantissa
            val=((-1)**(int(self.sign)))*(2**(int(self.exponent,2)-3))*float((1+(int(mantissa[1:], 2) / 2.**(len(mantissa) - 1))))

        else:
            mantissa="."+self.mantissa
            val=((-1)**(int(self.sign)))*(2**-2)*float((0+(int(mantissa[1:], 2) / 2.**(len(mantissa) - 1))))

        return val
```

```python
'''28 test cases for 8-bit floating points'''
import numpy as np

data = [ '00000000', '00000001', '00001001', '00010000',
         '00010001', '00011000', '00011011', '00100000',
         '00101101', '00110000', '00110101', '01000011',
         '01000000', '01010000', '01011100', '01100000',
         '01110111', '01110000', '10000000', '10000001',
         '11110001', '11110000', '10110001', '10111101',
         '11100000', '11101011', '11010000', '11000000']
results = ['(0, 1)', '(1, 64)', '(9, 64)', '(1, 4)', '(17, 64)', '(3, 8)', '(27, 64)',
          '(1, 2)', '(29, 32)', '(1, 1)', '(21, 16)', '(19, 8)', '(2, 1)', '(4, 1)',
          '(7, 1)', '(8, 1)', 'nan', 'inf', '(0, 1)', '(-1, 64)', 'nan', '-inf',
          '(-17, 16)', '(-29, 16)', '(-8, 1)', '(-27, 2)', '(-4, 1)', '(-2, 1)']

test = [Float8(x).val for x in data]

for index in range(len(test)):
    d = test[index]
    try:
        test[index] = str(d.as_integer_ratio())
    except Exception:
        test[index] = str(d)

np.testing.assert_equal(np.array(test), np.array(results))
```

## 16 Bit Floating Point Representation

Over here, we will implement a 16 bit Floating point number system,
where the first bit is a sign bit, next four bits are exponents,
and the rest 11 bits are mantissas (significands).

Numbers are divided into two categories: normals and subnormals.

For normals the encoding is:
\begin{equation}
(-1)^{sign} \times 2^{(e-7)} \times 1.fraction
\end{equation}
For subnormals the encoding is:
\begin{equation}
(-1)^{sign} \times 2^{(1-7)} \times 0.fraction
\end{equation}

Example:
\begin{aligned}
    0 0101 00000000000 &= 2^{(5-7)} \times 1.0  = 0.25\\
    0 0110 00000000000 &= 2^{(6-7)} \times 1.0  = 0.5\\
    0 0111 00000000000 &= 2^{(3-7)} \times 1.0  = 1.0\\
    0 0111 01000000000 &= 2^{(3-7)} \times 1.25 = 1.25
\end{aligned}

If the exponent is 0 then the number is considered subnormal.

Example:
\begin{aligned}
    0 0000 00000000000 &= 2^{-6} \times 0    = +0.0\\
    0 0000 10000000000 &= 2^{-6} \times 0.5  = +0.0078125\\
    0 0000 11000000000 &= 2^{-6} \times 0.75 = +0.01171875
\end{aligned}

If the exponent is 15 (all 1's) then and mantissa is zero then the number is considered infinity (inf), otherwise it is not-a-number (nan).

Example:
\begin{aligned}
    0 1111 00000000000 &= +\infty\\
    1 1111 00000000000 &= -\infty\\
    X 1111 XXXXXXXXXXX &= NaN
\end{aligned}

### Instructions
In this task, you need to implement the `Float16()` class. Some parts of the class are already written for you. You only need to modify `__init__(self, bitstring)` and `calculate_value(self)` fuctions, where it says `# YOUR CODE HERE`. You may ignore `raise NotImplementedError()`.

```python
class Float16():

    def __init__(self, bitstring):
        '''Constructor
        takes a 16-bit string of 0's and 1's as input and stores the sub-strings
        accordingly.
        Usage: Float16('0001111000011110')
        '''

        # Make sure the input consists of exactly 16-bits.
        assert(len(bitstring)==16)

        # Assign the sign bit
        # self.sign = bitstring[?]

        # Assign the exponent part
        # self.exponent = bitstring[?]

        # Assign the mantissa
        # self.mantissa = bitstring[?]

        # YOUR CODE HERE
        self.sign = bitstring[0]
        self.exponent = bitstring[1:5]
        self.mantissa = bitstring[5:16]

        self.val = self.calculate_value()

    def __str__(self):
        return f'Sign bit value: {self.sign}\n' + \
            f'Exponent value: {self.exponent}\n' + \
            f'Mantissa value: {self.mantissa}\n' + \
            f'Floating value: {self.val}\n'

    def tobitstring(self):
        return self.sign + self.exponent + self.mantissa

    def toformattedstring(self):
        return ' '.join([self.sign, self.exponent, self.mantissa])

    def calculate_value(self):
        '''Calculate the value of the number from bits'''

        # Initialize with zero
        val = 0.0

        # YOUR CODE HERE
        if int(self.exponent,2)==15 and int(self.mantissa,2)==0:
            if self.sign=="0":
                val="inf"
            elif self.sign=="1":
                val="-inf"
        elif int(self.exponent,2)==15 and int(self.mantissa,2)!=0:
            val="nan"
        elif int(self.exponent,2)!=0:
            mantissa="."+self.mantissa
            val=((-1)**(int(self.sign)))*(2**(int(self.exponent,2)-7))*float((1+(int(mantissa[1:], 2) / 2.**(len(mantissa) - 1))))
        else:
            mantissa="."+self.mantissa
            val=((-1)**(int(self.sign)))*(2**-6)*float((0+(int(mantissa[1:], 2) / 2.**(len(mantissa) - 1))))

        return val
```

```python
'''28 test cases for 16-bit floating points'''
data = [ '0011100000000010', '0100000000000000', '1100000000000000', '0100010000000000',
         '1100010000000000', '0100100000000000', '1100100000000000', '0100101000000000',
         '1100101000000000', '0100110000000000', '1100110000000000', '0101101110000000',
         '0010010000000000', '0000000000000001', '0000011111111111', '0000100000000000',
         '0111011111111111', '0000000000000000', '1000000000000000', '0111100000000000',
         '1111100000000000', '0111100000000001', '0111110000000001', '0111111111111111',
         '0010101010101011', '0100010010010001', '0011100000000000', '0011100000000001']
results = ['(1025, 1024)', '(2, 1)', '(-2, 1)', '(3, 1)', '(-3, 1)', '(4, 1)', '(-4, 1)',
           '(5, 1)', '(-5, 1)', '(6, 1)', '(-6, 1)', '(23, 1)', '(3, 16)', '(1, 131072)',
           '(2047, 131072)', '(1, 64)', '(4095, 16)', '(0, 1)', '(0, 1)', 'inf', '-inf',
           'nan', 'nan', 'nan', '(2731, 8192)', '(3217, 1024)', '(1, 1)', '(2049, 2048)']

test = [Float16(x).val for x in data]

for index in range(len(test)):
    d = test[index]
    try:
        test[index] = str(d.as_integer_ratio())
    except Exception:
        test[index] = str(d)

np.testing.assert_equal(np.array(test), np.array(results))
```


```python
import numpy as np

solar_mass_in_seconds = 4.92686088e-6

def phase(f, Mc):

        phase = -np.pi/4. + ( 3./( 128. * pow(Mc*np.pi*f, 5./3.) ) )
        return phase

def htilde(f, Mc):
        Mc *= solar_mass_in_seconds
        htilde = pow(f, -7./6.) * pow(Mc,5./6.) * np.exp(1j*phase(f,Mc))
        return htilde
```

In this note I will show how we use the empirical interpolant of a function to make a "reduced order quadrature" (ROQ). The exercise is to compute the inner product between a data set d (which plays the role of $A(x,y)$ in the coupling constant work) and the function h. The standard inner product is:
\begin{equation}
\langle\, d | h(M_c)\, \rangle = \Re \Delta f \sum_{i=1}^{N}\,d^{*}(f_{min} + i \Delta f)\,h(M_c;f_{min} +i \Delta f)\,,
\end{equation}
In this quadrature rule there are N terms. In general N is the number of sample points of the discretely sampled data and function h and could be very large for some problems. If we substitute the empirical interpolant of h into the above equation for the inner product then we get the ROQ. To recap, the empirical interpolant of h is
\begin{equation}
\mathcal{I}[h](M_c;f) = \sum_{j=1}^m B_j (f) h(M_c;F_j)
\end{equation}
and in the previous note we computed $\{B_j\}_{i=1}^{n}$ and $\{F_j\}_{i=1}^{n}$. Substituting the empirical interpolant for h in the inner product we get
\begin{equation}
\langle\,d | \mathcal{I}[h]\,\rangle = \Re \Delta f\,\sum_{j=1}^{m}\sum_{i=1}^{N} d^{*}(f_{min} + i \Delta f) B_j(f_{min} + i \Delta f) h(M_c;F_j)\,.
\end{equation}
Notice that the sum over frequency now decouples from the dependence on $M_c$! Thus we can do all the sums over frequency once-and-for-all, and evaluating the inner product should be cheaper. Thus we write the reduced order quadrature for the inner product as
\begin{equation}
\langle\, d | h(M_c)\, \rangle_{\mathrm{ROQ}} = \Re \sum_{j=1}^{m}\,w_j\,h(M_c;F_j)\,,
\end{equation}

where

\begin{equation}
w_j = \Delta f\sum_{i=1}^{N} d^{*}(f_{min} + i \Delta f) B_j(f_{min} + i \Delta f)\,.
\end{equation}

Notice that the ROQ has only m terms in the sum, whereas the original had N. Thus we get a compression of N/m. For this specific problem, the original waveforms which we built the bases for had N=5000 samples. The number of bases is around m=300, so in this case we should expect a compression of N/m = 5000/300 = 17. Let's try it out.

First I'll create a data set. It doesn't matter what it is, so I'll just let it be a random vector of length 5000. I'll start by loading in the frequency series and frequency steps $\Delta f$, and then generate the data:

```python
fseries = np.loadtxt("fseries.dat")
df = np.loadtxt("df.dat")

data = np.random.rand(len(fseries)) + 1j*np.random.rand(len(fseries))
```

Next I'll load in the $B_j$s and the empirical interpolation nodes $F_j$.

```python
B_re = np.loadtxt("B_re.dat")
B_im = np.loadtxt("B_im.dat")

B = B_re + 1j*B_im

nodes = np.loadtxt("nodes.dat")
```

Now I have everything I need to make the weights $\omega_j$ which is done below

```python
# multipy data by \Delta f
data *= df

weights = np.inner(B, data.conjugate().T)
```

Now I'll compute the regular inner product and the ROQ

```python
d_dot_h = np.vdot(data, htilde(fseries, 1.5)).real # regular inner product

ROQ = np.dot(weights, htilde(nodes, 1.5)).real # ROQ inner product

print "regular inner product = %.15e"%d_dot_h
print "ROQ = %.15e"%ROQ
```

    regular inner product = 5.805838449180775e-07
    ROQ = 5.805838449179256e-07

so the ROQ is essentially error free at round off. Next I'll time the regular inner product and the ROQ.

```python
import time

Mc = 2

t1 = time.time()
for i in range(50000):
    np.vdot(data, htilde(fseries, Mc)).real # regular inner product
e1 = time.time()

t2 = time.time()
for i in range(50000):
    np.dot(weights, htilde(nodes, Mc)).real # ROQ inner product
e2 = time.time()

print "regular inner product took %f s"%((e1-t1)/50000.)
print "ROQ took %f s"%((e2-t2)/50000.)
print "speedup = %f"%((e1-t1) / (e2-t2))
```

    regular inner product took 0.000582 s
    ROQ took 0.000068 s
    speedup = 8.501153

This seems to give a speed up between 8 and 9, which isn't quite the predicted speed up of 16: presumably there is some overhead in passing the vectors to np.vdot and np.dot which is on the order of the time it takes to compute the inner products. Anyway, hopefully this illustrates the main point which is that if you have an empirical interpolant then computing inner products of the kind we considered can be significantly sped up. This is what we should do for the coupling coefficient work!

```python

```

ROQ/Building a reduced order quadrature (ROQ).ipynb