# 16 PDEs: Solution with Time Stepping

## Heat Equation
The **heat equation** can be derived from Fourier's law and energy conservation (see the [lecture notes on the heat equation (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs_LectureNotes_HeatEquation.pdf))

$$
\frac{\partial T(\mathbf{x}, t)}{\partial t} = \frac{K}{C\rho} \nabla^2 T(\mathbf{x}, t),
$$

## Problem: insulated metal bar (1D heat equation)
A metal bar of length $L$ is insulated along it lengths and held at 0ºC at its ends. Initially, the whole bar is at 100ºC. Calculate $T(x, t)$ for $t>0$.

### Analytic solution
Solve by separation of variables and power series: The general solution that obeys the boundary conditions $T(0, t) = T(L, t) = 0$ is

$$
T(x, t) = \sum_{n=1}^{+\infty} A_n \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right), \quad k_n = \frac{n\pi}{L}
$$

The specific solution that satisfies $T(x, 0) = T_0 = 100^\circ\text{C}$ leads to $A_n = 4 T_0/n\pi$ for $n$ odd:

$$
T(x, t) = \sum_{n=1,3,5,\dots}^{+\infty} \frac{4 T_0}{n \pi} \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right)
$$

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

Analytical solution:

```python
def T_bar(x, t, T0, L, K=237, C=900, rho=2700, nmax=1000):
    T = np.zeros_like(x)
    eta = K / (C*rho)
    for n in range(1, nmax, 2):
        kn = n*np.pi/L
        T += 4*T0/(np.pi * n) * np.sin(kn*x) * np.exp(-kn*kn * eta * t)
    return T
```

```python
T0 = 100.
L = 1.0
X = np.linspace(0, L, 100)
for t in np.linspace(0, 3000, 50):
    plt.plot(X, T_bar(X, t, T0, L))
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ ($^\circ$C)");
```

### Numerical solution: Leap frog
Discretize (finite difference):

For the time domain we only have the initial values so we use a simple forward difference for the time derivative:

$$
\frac{\partial T(x,t)}{\partial t} \approx \frac{T(x, t+\Delta t) - T(x, t)}{\Delta t}
$$

For the spatial derivative we have initially all values so we can use the more accurate central difference approximation:

$$
\frac{\partial^2 T(x, t)}{\partial x^2} \approx \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

Thus, the heat equation can be written as the finite difference equation

$$
\frac{T(x, t+\Delta t) - T(x, t)}{\Delta t} = \frac{K}{C\rho} \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

which can be reordered so that the RHS contains only known terms and the LHS future terms. Index $i$ is the spatial index, and $j$ the time index: $x = x_0 + i \Delta x$, $t = t_0 + j \Delta t$.

$$
T_{i, j+1} = (1 - 2\eta) T_{i,j} + \eta(T_{i+1,j} + T_{i-1, j}), \quad \eta := \frac{K \Delta t}{C \rho \Delta x^2}
$$

Thus we can step forward in time ("leap frog"), using only known values.

### Solve the 1D heat equation numerically for an iron bar
* $K = 237$ W/mK
* $C = 900$ J/K
* $\rho = 2700$ kg/m3
* $L = 1$ m
* $T_0 = 373$ K and $T_b = 273$ K
* $T(x, 0) = T_0$ and $T(0, t) = T(L, t) = T_b$

#### Key considerations

The key line is the computation of the new temperature field at time step $j+1$ from the temperature distribution at time step $j$. It can be written purely with numpy array operations (see last lecture!):

```python
T[1:-1] = (1 - 2*eta) * T[1:-1] + eta * (T[2:] + T[:-2])
```

Note that the range operator `T[start:end]` *excludes* `end`, so in order to include `T[1], T[2], ..., T[-2]` (but not the rightmost `T[-1]`) we have to use `T[1:-1]`.

The *boundary conditions* are fixed for all times:
```python
T[0] = T[-1] = Tb
```

The *initial conditions* (at time step `j=0`)
```python
T[1:-1] = T0
```
are only used to compute the distribution of temperatures at the next step `j=1`.

#### Solution

```python
import numpy as np

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

For 3D rotatable output:

```python
%matplotlib widget
```

For HTML/nbviewer output, use inline:

```python
%matplotlib inline
```

Numerical solution:

```python
L_rod = 1.    # m
t_max = 3000. # s

Dx = 0.02   # m
Dt = 2    # s

Nx = int(L_rod // Dx)
Nt = int(t_max // Dt)

Kappa = 237 # W/(m K)
CHeat = 900 # J/K
rho = 2700  # kg/m^3

T0 = 373 # K
Tb = 273 # K

eta = Kappa * Dt / (CHeat * rho * Dx**2)
eta2 = 1 - 2*eta

step = 20  # plot solution every n steps

print("Nx = {0}, Nt = {1}".format(Nx, Nt))
print("eta = {0}".format(eta))

T = np.zeros(Nx)
T_plot = np.zeros((Nt//step + 1, Nx))

# initial conditions
T[1:-1] = T0
# boundary conditions
T[0] = T[-1] = Tb

t_index = 0
T_plot[t_index, :] = T
for jt in range(1, Nt):
    T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        T_plot[t_index, :] = T
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
```

    Nx = 49, Nt = 1500
    eta = 0.4876543209876543
    Completed  1499 iterations: t=2998 s

#### Visualization
Visualize (you can use the code as is).

Note how we are making the plot use proper units by mutiplying with `Dt * step` and `Dx`.

```python
X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
Z = T_plot[X, Y]
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
ax.set_xlabel(r"time $t$ (s)")
ax.set_ylabel(r"position $x$ (m)")
ax.set_zlabel(r"temperature $T$ (K)")
fig.tight_layout()
```

2D as above for the analytical solution…

```python
X = Dx * np.arange(T_plot.shape[1])
plt.plot(X, T_plot.T)
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ (K)");
```

## Stability of the solution

### Empirical investigation of the stability
Investigate the solution for different values of `Dt` and `Dx`. Can you discern patters for stable/unstable solutions?

Report `Dt`, `Dx`, and `eta`
* for 3 stable solutions
* for 3 unstable solutions

```python
def calculate_T(L_rod=1, t_max=3000, Dx=0.02, Dt=2, T0=373, Tb=273,
               step=20):
    Nx = int(L_rod // Dx)
    Nt = int(t_max // Dt)

    Kappa = 237 # W/(m K)
    CHeat = 900 # J/K
    rho = 2700  # kg/m^3

    eta = Kappa * Dt / (CHeat * rho * Dx**2)
    eta2 = 1 - 2*eta

    print("Nx = {0}, Nt = {1}".format(Nx, Nt))
    print("eta = {0}".format(eta))

    T = np.zeros(Nx)
    T_plot = np.zeros((int(np.ceil(Nt/step)) + 1, Nx))

    # initial conditions
    T[1:-1] = T0
    # boundary conditions
    T[0] = T[-1] = Tb

    t_index = 0
    T_plot[t_index, :] = T
    for jt in range(1, Nt):
        T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            T_plot[t_index, :] = T
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
    return T_plot

def plot_T(T_plot, Dx, Dt, step):
    X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
    Z = T_plot[X, Y]
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
    ax.set_xlabel(r"time $t$ (s)")
    ax.set_ylabel(r"position $x$ (m)")
    ax.set_zlabel(r"temperature $T$ (K)")
    fig.tight_layout()
    return ax
```

```python
Dx, Dt, step = 0.01, 2, 20
T_plot = calculate_T(Dx=Dx, Dt=Dt, step=step)
plot_T(T_plot, Dx, Dt, step)
```

Note that *decreasing* the value of $\Delta x$ made the solution *unstable*. This is strange, we have gotten used to the idea that working on a finer mesh will increase the detail (until we hit round-off error) and just become computationally more expensive. But here the algorithm suddenly becomes unstable (and it is not just round-off).

For certain combination of values of $\Delta t$ and $\Delta x$ the solution become unstable. Empirically, bigger $\eta$ leads to instability. (In fact, $\eta \geq \frac{1}{2}$ is unstable for the leapfrog algorithm as we will see.)

### Von Neumann stability analysis

If the difference equation solution diverges then we *know* that we have a bad approximation to the original PDE.

Von Neumann stability analysis starts from the assumption that *eigenmodes* of the difference equation can be written as

$$
T_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

with the unknown wave vectors $k=2\pi/\lambda$ and unknown complex functions – the *amplification factors* – $\xi(k)$.

Solutions of the difference equation can be written as linear superpositions of these basis functions. But they are only stable if the eigenmodes are stable, i.e., will not grow in time (with $j$). This is the case when

$$
|\xi(k)| < 1
$$

for all $k$.

Insert the eigenmodes into the finite difference equation

$$
T_{m, j+1} = (1 - 2\eta) T_{m,j} + \eta(T_{m+1,j} + T_{m-1, j})
$$

to obtain

\begin{align}
\xi(k)^{j+1} e^{ikm\Delta x} &= (1 - 2\eta) \xi(k)^{j} e^{ikm\Delta x}
    + \eta(\xi(k)^{j} e^{ik(m+1)\Delta x} + \xi(k)^{j} e^{ik(m-1)\Delta x})\\
\xi(k) &= (1 - 2\eta) + \eta(e^{ik\Delta x} + e^{-ik\Delta x})\\
\xi(k) &= 1 - 2\eta + 2\eta \cos k\Delta x\\
\xi(k) &= 1 + 2\eta\big(\cos k\Delta x - 1\big)
\end{align}

For $|\xi(k)| < 1$ (and all possible $k$):

\begin{align}
|\xi(k)| < 1 \quad &\Leftrightarrow \quad \xi^2(k) < 1\\
(1 + 2y)^2 = 1 + 4y + 4y^2 &< 1 \quad \text{with}\ \  y = \eta(\cos k\Delta x - 1)\\
y(1 + y) &< 0 \quad \Leftrightarrow \quad -1 < y < 0\\
\eta(\cos k\Delta x - 1) &\leq 0 \quad \forall k \quad (\eta > 0, -1 \leq \cos x \leq 1)\\
\eta(\cos k\Delta x - 1) &> -1\\
\eta &< \frac{1}{1 - \cos k\Delta x}\\
\eta = \frac{K \Delta t}{C \rho \Delta x^2} &< \frac{1}{2} \le \frac{1}{1 - \cos k\Delta x}
\end{align}

Thus, solutions are only stable for $\eta < 1/2$. In particular, decreasing $\Delta t$ will always improve stability, But decreasing $\Delta x$ requires an quadratic *increase* in $\Delta t$!

Note
* Perform von Neumann stability analysis when possible (depends on PDE and the specific discretization).
* Test different combinations of $\Delta t$ and $\Delta x$.
* Not guarantee that decreasing both will lead to more stable solutions!

Check my inputs:

This was stable and it conforms to the stability criterion:

```python
Dt = 2
Dx = 0.02
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```

    0.4876543209876543

... and this was unstable, despite a seemingly small change:

```python
Dt = 2
Dx = 0.01
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```

    1.9506172839506173

```python

```


1) Given the array split the array at the middle (if it is odd length consider next higher integer), add reverse the array.

Ex:
if input is [12,10,5,6,52,36]
output should be [6,52,36,12,10,5]

if input is  [12,10,5,6,52,36,34]
output should be [6,52,36,34,12,10,5]

```python
import math
# Complete this function to get the desired result
def reverseatCenter(arr):
    n = len(arr)  #length of the array
    m = n//2      #Middle element
    arr2 = arr[m:]+arr[:m]     #Reversing the sub-lists
    return arr2
```

```python
# Print wroking examples
print("Example 1 - [14,9,5,6,52,36,34]")
print(reverseatCenter([14,9,5,6,52,36,34,23,111]))
print("Example 2 - [14,9,5,6,52,36]")
print(reverseatCenter([14,9,5,6,52,36]))
```

    Example 1 - [14,9,5,6,52,36,34]
    [52, 36, 34, 23, 111, 14, 9, 5, 6]
    Example 2 - [14,9,5,6,52,36]
    [6, 52, 36, 14, 9, 5]

2) Given a list of numbers, return a list where all adjacent duplicate elements have been removed.

Ex:

2, 2, 2, 3, 2 returns 2, 3, 2.

```python
# Complete the function to get the desired result
def remove_adjacent(b):
    b2 = [b[0]]
    for i in range(1,len(b)):
        #Checking consecutive repitions
        if b2[-1]!=b[i]:
            b2.append(b[i])
    return b2
```

```python
# Print working examples
print("Example 1 - [2, 2, 2, 3, 2]")
print(remove_adjacent([2, 2, 2, 3, 2]))
print("\n")
print("Example 2 - [1,2,3,3,2,3,1,1,1,2,1,2,2]")
print(remove_adjacent([1,2,3,3,2,3,1,1,1,2,1,2,2]))
```

    Example 1 - [2, 2, 2, 3, 2]
    [2, 3, 2]

    Example 2 - [1,2,3,3,2,3,1,1,1,2,1,2,2]
    [1, 2, 3, 2, 3, 1, 2, 1, 2]

3) given matrix of 7x7 full of ones, create a square with given side length (center same as original square(7x7)) replace ones with zeors at the edges for example,

$$
\begin{align}
A = \begin{bmatrix}
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
\end{bmatrix}
\end{align}
$$
After modification
$$
\begin{align}
A = \begin{bmatrix}
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 0 & 0 & 0 & 1 & 1\\
1 & 1 & 0 & 1 & 0 & 1 & 1\\
1 & 1 & 0 & 0 & 0 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
1 & 1 & 1 & 1 & 1 & 1 & 1\\
\end{bmatrix}
\end{align}
$$

```python
import numpy as np
# write programm to get the desired result
def matrix_square(A,l):
    n = A.shape[0]  #Size of the matrix
    #the length of the smaller square should be lesser than the size of the matrix, both of which
    #are odd
    assert l<=n and n%2==1 and l%2==1

    #start and end of inner square
    start = (n-l)//2
    end = (n+l)//2

    for i in range(start,end):
        for j in range(start,end):
            #Making the elements in the edges of the smaller square 0
            if (i==start or i==end-1) or (j==start or j==end-1):
                A[i][j] = 0

    return A.astype(int)

```

```python
#Print working examples

#Example 1 - side length 3
print("Example 1 - side length 3")
A = np.ones((7,7))  #7x7 matrix of ones
l = 3  #Side length = 3
print(matrix_square(A,l))
print("\n")

#Example 2 - side length 5
print("Example 2 - side length 5")
A = np.ones((7,7))  #7x7 matrix of ones
l = 5  #Side length = 5
print(matrix_square(A,l))
print("\n")

#Example 3 - side length 7
print("Example 3 - side length 7")
A = np.ones((7,7))  #7x7 matrix of ones
l = 7  #Side length = 7
print(matrix_square(A,l))
```

    Example 1 - side length 3
    [[1 1 1 1 1 1 1]
     [1 1 1 1 1 1 1]
     [1 1 0 0 0 1 1]
     [1 1 0 1 0 1 1]
     [1 1 0 0 0 1 1]
     [1 1 1 1 1 1 1]
     [1 1 1 1 1 1 1]]

    Example 2 - side length 5
    [[1 1 1 1 1 1 1]
     [1 0 0 0 0 0 1]
     [1 0 1 1 1 0 1]
     [1 0 1 1 1 0 1]
     [1 0 1 1 1 0 1]
     [1 0 0 0 0 0 1]
     [1 1 1 1 1 1 1]]

    Example 3 - side length 7
    [[0 0 0 0 0 0 0]
     [0 1 1 1 1 1 0]
     [0 1 1 1 1 1 0]
     [0 1 1 1 1 1 0]
     [0 1 1 1 1 1 0]
     [0 1 1 1 1 1 0]
     [0 0 0 0 0 0 0]]

4) Paragraph present in data.txt is encoded such that each alphabet in word is incremented to next ascii value. Decode the paragraph present in the data_encoded.txt (Hint: decrease the ascii value of each character in the word)

```python
def sentence_decode(str):

    dec_words = ""   #initializing empty string

    for character in str:
        #Copying spaces as it is
        if character==" ":
            dec_words+=" "
        #Decoding - identifying the previous ascii value
        else:
            dec_words += chr(ord(character)-1)
    return dec_words
```

```python
with open ("data_encoded.txt", "r") as myfile:
    data=myfile.read()
    print(data)
    print(" ")
    print(sentence_decode(data))
```

    ebub bobmztjt jt b qspdftt pg jotqfdujoh- dmfbotjoh- usbotgpsnjoh boe npefmjoh ebub xjui uif hpbm pg ejtdpwfsjoh vtfgvm jogpsnbujpo- jogpsnjoh dpodmvtjpot boe tvqqpsujoh efdjtjpo.nbljoh/ ebub bobmztjt ibt nvmujqmf gbdfut boe bqqspbdift- fodpnqbttjoh ejwfstf ufdiojrvft voefs b wbsjfuz pg obnft- boe jt vtfe jo ejggfsfou cvtjoftt- tdjfodf- boe tpdjbm tdjfodf epnbjot/ jo upebz(t cvtjoftt xpsme- ebub bobmztjt qmbzt b spmf jo nbljoh efdjtjpot npsf tdjfoujgjd boe ifmqjoh cvtjofttft pqfsbuf npsf fggfdujwfmz/

    data analysis is a process of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, informing conclusions and supporting decision-making. data analysis has multiple facets and approaches, encompassing diverse techniques under a variety of names, and is used in different business, science, and social science domains. in today's business world, data analysis plays a role in making decisions more scientific and helping businesses operate more effectively.

```python
# Has to print the decoded paragraph.
```


# IHA1 - Assignment

Welcome to the first individual home assignment!

This assignment consists of two parts:
 * Python and NumPy exercises
 * Build a deep neural network for forward propagation

The focus of this assignment is for you to gain practical knowledge with implementing forward propagation of deep neural networks without using any deep learning framework. You will also gain practical knowledge in two of Python's scientific libraries [NumPy](https://docs.scipy.org/doc/numpy-1.13.0/index.html) and [Matplotlib](https://matplotlib.org/devdocs/index.html).

Skeleton code is provided for most tasks and every part you are expected to implement is marked with **TODO**

We expect you to search and learn by yourself any commands you think are useful for these tasks. Don't limit yourself to only what was taught in CL1. Use the help function, [stackoverflow](https://stackoverflow.com/), google, the [python documentation](https://docs.python.org/3.5/library/index.html) and the [NumPy](https://docs.scipy.org/doc/numpy-1.13.0/index.html) documentation to your advantage.

**IMPORTANT NOTE**: The tests available are not exhaustive, meaning that if you pass a test you have avoided the most common mistakes, but it is still not guaranteed that you solution is 100% correct.

Lets start by importing the necessary libraries below

```python
import numpy as np
import matplotlib.pyplot as plt
from utils.tests.iha1Tests import *
```

## 1. Lists and arrays introduction
First, we will warm up with a Python exercise and few NumPy exercises

### 1.1 List comprehensions
Examine the code snippet provided below

```python
myList = []
for i in range(25):
    if i % 2 == 0:
        myList.append(i**2)

print(myList)
```

    [0, 4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]

This is not a very "[pythonic](http://docs.python-guide.org/en/latest/writing/style/)" way of writing. Lets re-write the code above using a [list comprehension](https://docs.python.org/2/tutorial/datastructures.html#list-comprehensions). The result will be less code, more readable and elegant. Your solution should be able to fit into one line of code.

```python
myList = [i**2 for i in range(0,25,2) ]
print(myList)
```

    [0, 4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]

```python
# sample output from cell above for reference
```

### 1.2 Numpy array vs numpy vectors
Run the cell below to create a numpy array.

```python
myArr = np.array([1, 9, 25, 49, 81, 121, 169, 225, 289, 361, 441, 529])
print(myArr)
print(myArr.shape)
```

    [  1   9  25  49  81 121 169 225 289 361 441 529]
    (12,)

One of the core features of numpy is to efficiently perform linear algebra operations.
There are two types of one-dimensional representations in numpy: arrays of shape (x,) and vectors of shape (x,1)

The the above result indicates that **myArr** is an array of 12 elements with shape (12,).

Numpy's arrays and vectors both have the type of `numpy.ndarray` but have in some cases different characteristics and it is important to separate the two types because it will save a lot of debugging time later on. Read more about numpy shapes [here](https://stackoverflow.com/a/22074424)

Run the code below to see how the transpose operation behaves differently between an array and vector

```python
# print the shape of an array and the shape of a transposed array
print('myArr is an array of shape:')
print(myArr.shape)
print('The transpose of myArr has the shape:')
print(myArr.T.shape)

# print the shape of a vector and the transpose of a vector
myVec = myArr.reshape(12,1)
print('myVec is a vector of shape:')
print(myVec.shape)
print('The transpose of myVec has the shape:')
print(myVec.T.shape)
```

    myArr is an array of shape:
    (12,)
    The transpose of myArr has the shape:
    (12,)
    myVec is a vector of shape:
    (12, 1)
    The transpose of myVec has the shape:
    (1, 12)

### 1.3 Numpy exercises
Now run the cell below to create the numpy array `numbers` and then complete the exercises sequentially

```python
numbers = np.arange(24)
print(numbers)
```

    [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]

```python
# Reshape numbers into a 6x4 matrix
numbers = numbers.reshape(6,4)
print(numbers)
```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22 23]]

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_reshape(numbers)
```

    Test passed

```python
# Set the element of the last row of the last column to zero
numbers[-1,-1]=0;
print(numbers)
```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22  0]]

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_neg_ix(numbers)
```

    Test passed

```python
# Set every element of the 0th row to 0
numbers[0,:]=0;
print(numbers)
```

    [[ 0  0  0  0]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22  0]]

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_row_ix(numbers)
```

    Test passed

```python
# TODO: append a 1x4 row vector of zeros to `numbers`,
# resulting in a 7x4 matrix where the new row of zeros is the last row
# Hint: A new matrix must be created in the procedure. Numpy arrays are not dynamic.
numbers = np.vstack((numbers, np.array([0,0,0,0])))
print(numbers)
print(numbers.shape)
```

    [[ 0  0  0  0]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]
     [16 17 18 19]
     [20 21 22  0]
     [ 0  0  0  0]]
    (7, 4)

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_append_row(numbers)
```

    Test passed

```python
# TODO: set all elements above 10 to the value 1
numbers[numbers>10] = 1
print(numbers)
```

    [[ 0  0  0  0]
     [ 4  5  6  7]
     [ 8  9 10  1]
     [ 1  1  1  1]
     [ 1  1  1  1]
     [ 1  1  1  0]
     [ 0  0  0  0]]

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_bool_matrix(numbers)
```

    Test passed

```python
# Compute the sum of every row and replace `numbers` with the answer
# `numbers` will be a (7,) array as a result
numbers = np.sum(numbers,axis=1)
print(numbers.shape)
print(numbers)
```

    (7,)
    [ 0 22 28  4  4  3  0]

```python
# sample output from cell above for reference
```

```python
# test case
test_numpy_sum(numbers)
```

    Test passed

## 2 Building your deep neural network
It is time to start implementing your first feed-forward neural network. In this lab you will only focus on implementing the forward propagation procedure.

When using a neural network, you can not forward propagate the entire dataset at once. Therefore, you divide the dataset into a number of sets/parts called batches. A batch will make up for the first dimension of every input to a layer and the notation `(BATCH_SIZE, NUM_FEATURES)` simply means the dimension of a batch of samples where every sample has `NUM_FEATURES` features

### 2.1 activation functions
You will start by defining a few activation functions that are later needed by the neural network.

#### 2.1.1 ReLU
The neural network will use the ReLU activation function in every layer except for the last. ReLU does element-wise comparison of the input matrix. For example, if the input is `X`, and `X[i,j] == 2` and `X[k,l] == -1`, then after applying ReLU, `X[i,j] == 2` and `X[k,l] == 0` should be true.

The formula for implementing ReLU for a single neuron $i$ is:
\begin{equation}
relu(z_i) =
    \begin{cases}
      0, & \text{if}\ z_i \leq 0 \\
      z_i, & \text{otherwise}
    \end{cases}
\end{equation}

Now implement `relu` in vectorized form

```python
def relu(z):
    """ Implement the ReLU activation function

    Arguments:
    z - the input of the activation function. Has a type of `numpy.ndarray`

    Returns:
    a - the output of the activation function. Has a type of numpy.ndarray and the same shape as `z`
    """

    return np.maximum(0,z)
```

```python
# test case
test_relu(relu)
```

    Test passed

#### 2.1.2 Sigmoid
The sigmoid activation function is common for binary classification. This is because it squashes its input to the range [0,1].
Implement the activation function `sigmoid` using the formula:
\begin{equation}
    \sigma(z) = \frac{1}{1 + e^{-z}}
\end{equation}

```python
def sigmoid(z):
    """ Implement the sigmoid activation function

    Arguments:
    z - the input of the activation function. Has a type of `numpy.ndarray`

    Returns:
    a - the output of the activation function. Has a type of `numpy.ndarray` and the same shape as `z`
    """

    a = 1/(1+np.exp(-z))

    return a
```

```python
# test case
test_sigmoid(sigmoid)
```

    Test passed

#### 2.1.3 Visualization
Make a plot using matplotlib to visualize the activation functions between the input interval [-3,3]. The plot should have the following properties
 * one plot should contain a visualization of both `ReLU` and `sigmoid`
 * x-axis: range of values between [-3,3], **hint**: np.linspace
 * y-axis: the value of the activation functions at a given input `x`
 * a legend explaining which line represents which activation function

```python
# A plot of ReLU and sigmoid values in the interval [-3,3]
from matplotlib import pyplot as plt
%matplotlib inline
import numpy as np
fig,axes = plt.subplots()
xData = np.linspace(-3,3)
yRelu = relu(xData)
ySigmoid = sigmoid(xData)
hRelu, = plt.plot(xData,yRelu,label='relu')
hSigmoid, = plt.plot(xData,ySigmoid,label='sigmoid')
plt.legend(handles=(hRelu,hSigmoid),loc=0);
```

```python
# sample output from cell above for reference
```

#### 2.1.4 Softmax
You will use the softmax activation function / classifier as the final layer of your neural network later in the assignment. Implement `softmax` according the the formula below. The subtraction of the maximum value is there solely to avoid overflows in a practical implementation.
\begin{equation}
softmax(z_i) = \frac{e^{z_i - max(\mathbf{z})}}{ \sum^j e^{z_j - max(\mathbf{z})}}
\end{equation}

```python
def softmax(z):
    """ Implement the softmax activation function

    Arguments:
    z - the input of the activation function, shape (BATCH_SIZE, FEATURES) and type `numpy.ndarray`

    Returns:
    a - the output of the activation function, shape (BATCH_SIZE, FEATURES) and type numpy.ndarray
    """
    # Store maximum value of z
    zMax = np.max(z);

    # Preallocate a
    a = np.empty(z.shape);
    print(z)
    for c,zColumn in enumerate(z):
        print(zColumn)
        print(np.exp(zColumn-zMax))
        a[c,:] = np.exp(zColumn-zMax);
        a[c,:] =a[c,:]/np.sum(a[c,:]);
    return a
```

```python
# test case
test_softmax(softmax)
```

    [[0 0]]
    [0 0]
    [1. 1.]
    [[1 1]]
    [1 1]
    [1. 1.]
    [[1.    0.001]]
    [1.    0.001]
    [1.        0.3682475]
    [[3.  1.  0.2]]
    [3.  1.  0.2]
    [1.         0.13533528 0.06081006]
    [[1 2 3 6]
     [2 4 5 6]
     [3 8 7 6]]
    [1 2 3 6]
    [0.00091188 0.00247875 0.00673795 0.13533528]
    [2 4 5 6]
    [0.00247875 0.01831564 0.04978707 0.13533528]
    [3 8 7 6]
    [0.00673795 1.         0.36787944 0.13533528]
    [[0.31323624 0.7810351  0.26183059 0.09174578 0.09806706]
     [0.28981829 0.03154328 0.99442807 0.4591928  0.42556593]
     [0.06799825 0.89438807 0.68276332 0.89185543 0.37638809]
     [0.49131144 0.03873597 0.91306311 0.2533448  0.24115072]
     [0.38297911 0.23184308 0.88202174 0.42546236 0.78325552]]
    [0.31323624 0.7810351  0.26183059 0.09174578 0.09806706]
    [0.50601355 0.80783862 0.48065887 0.40548058 0.40805186]
    [0.28981829 0.03154328 0.99442807 0.4591928  0.42556593]
    [0.49430142 0.38178991 1.         0.58553152 0.56616929]
    [0.06799825 0.89438807 0.68276332 0.89185543 0.37638809]
    [0.39596486 0.90480123 0.73222697 0.90251259 0.53899985]
    [0.49131144 0.03873597 0.91306311 0.2533448  0.24115072]
    [0.60464327 0.38454591 0.92185719 0.47659735 0.47082098]
    [0.38297911 0.23184308 0.88202174 0.42546236 0.78325552]
    [0.54256415 0.46645908 0.89368105 0.56611066 0.80963435]
    Test passed

### 2.2 Initialize weights
You will implement a helper function that takes the shape of a layer as input, and returns an initialized weight matrix $\mathbf{W}$ and bias vector $\mathbf{b}$ as output. $\mathbf{W}$ should be sampled from a normal distribution of mean 0 and standard deviation 2, and $\mathbf{b}$ should be initialized to all zeros.

```python
def initialize_weights(layer_shape):
    """ Implement initialization of the weight matrix and biases

    Arguments:
    layer_shape - a tuple of length 2, type (int, int), that determines the dimensions of the weight matrix: (input_dim, output_dim)

    Returns:
    w - a weight matrix with dimensions of `layer_shape`, (input_dim, output_dim), that is normally distributed with
        properties mu = 0, stddev = 2. Has a type of `numpy.ndarray`
    b - a vector of initialized biases with shape (1,output_dim), all of value zero. Has a type of `numpy.ndarray`
    """
    w = np.random.normal(loc=0,scale=2,size=layer_shape)
    b = np.zeros((1,layer_shape[1]))

    return w, b
```

```python
# test case
test_initialize_weights(initialize_weights)
```

    Test passed

### 2.3 Feed-forward neural network layer module
To build a feed-forward neural network of arbitrary depth you are going to define a neural network layer as a module that can be used to stack upon eachother.

Your task is to complete the `Layer` class by following the descriptions in the comments.

Recall the formula for forward propagation of an arbitrary layer $l$:

\begin{equation}
\mathbf{a}^{[l]} = g(\mathbf{z}^{[l]}) = g(\mathbf{a}^{[l-1]}\mathbf{w}^{[l]} +\mathbf{b}^{[l]})
\end{equation}

$g$ is the activation function given by `activation_fn`, which can be relu, sigmoid or softmax.

```python
import numpy as np
class Layer:
    """
    TODO: Build a class called Layer that satisfies the descriptions of the methods
    Make sure to utilize the helper functions you implemented before
    """

    def __init__(self, input_dim, output_dim, activation_fn=relu):
        """
        Arguments:
        input_dim - the number of inputs of the layer. type int
        output_dim - the number of outputs of the layer. type int
        activation_fn - a reference to the activation function to use. Should be `relu` as a default
                        possible values are the `relu`, `sigmoid` and `softmax` functions you implemented earlier.
                        Has the type `function`

        Attributes:
        w - the weight matrix of the layer, should be initialized with `initialize_weights`
            and has the shape (INPUT_FEATURES, OUTPUT_FEATURES) and type `numpy.ndarray`
        b - the bias vector of the layer, should be initialized with `initialize_weights`
            and has the shape (1, OUTPUT_FEATURES) and type `numpy.ndarray`
        activation_fn - a reference to the activation function to use.
                        Has the type `function`
        """
        self.w, self.b = initialize_weights((input_dim, output_dim))

        self.activation_fn = activation_fn

    def forward_prop(self, a_prev):
        """ Implement the forward propagation module of the neural network layer
        Should use whatever activation function that `activation_fn` references to

        Arguments:
        a_prev - the input to the layer, which may be the data `X`, or the output from the previous layer.
            a_prev has the shape of (BATCH_SIZE, INPUT_FEATURES) and the type `numpy.ndarray`

        Returns:
        a - the output of the layer when performing forward propagation. Has the type `numpy.ndarray`
        """
        z = np.matmul(a_prev,self.w)+self.b
        a = self.activation_fn(z);

        return a
```

```python
# test case, be sure that you pass the previous activation function tests before running this test
test_layer(Layer, relu, sigmoid, softmax)
```

    [[ 1.68047376 -2.14613104  1.00556785 -1.255238  ]
     [ 2.93861903 -2.0227981  -0.01599629 -2.02901342]
     [ 2.3772233  -4.38036742  2.80134077 -1.90467409]]
    [ 1.68047376 -2.14613104  1.00556785 -1.255238  ]
    [0.28418062 0.00619043 0.144706   0.01508798]
    [ 2.93861903 -2.0227981  -0.01599629 -2.02901342]
    [1.         0.007003   0.0520987  0.00695961]
    [ 2.3772233  -4.38036742  2.80134077 -1.90467409]
    [5.70412365e-01 6.62833689e-04 8.71727631e-01 7.88105806e-03]
    Test passed

### 2.4 Logistic regression
Binary logistic regression is a classifier where classification is performed by applying the sigmoid activation function to a linear combination of input values. You will now try out your neural network layer by utilizing it as a linear combination of input values and apply the sigmoid activation function to classify a simple problem.

The cell below defines a dataset of 5 points of either class `0` or class `1`. Your assignment is to:
1. Create an instance of a `Layer` with sigmoid activation function
2. Manually tune the weights `w` and `b` of your layer

You can use `test_logistic` to visually inspect how your classifier is performing.

```python
# Run this cell to create the dataset
X_s = np.array([[1, 2],
               [5, 3],
               [8, 8],
               [7, 5],
               [3, 6]])
Y_s = np.array([0,0,1,0,1])

test_logistic(X_s, Y_s)
```

```python
# create an instance of layer
l = Layer(2,1,sigmoid)
# Manually tune weights and bias  i.e. find parameters such that the different classes are separated on different sides of the line descrived by those param
l.w = np.array([[-1],[2]])
l.b = -6;

# testing your choice of weights with this function
test_logistic(X_s,Y_s,l,sigmoid)
```

```python
# sample output from cell above for reference
```

### 2.5 Feed-forward neural network
Now define the actual neural network class. It is an L-layer neural network, meaning that the number of layers and neurons in each layer is specified as input by the user. Once again, you will only focus on implementing the forward propagation part.

Read the descriptions in the comments and complete the todos

```python
class NeuralNetwork:
    """
    Implement an L-layer neural network class by utilizing the Layer module defined above
    Each layer should use `relu` activation function, except for the output layer, which should use `softmax`
    """

    def __init__(self, input_n, layer_dims):
        """
        Arguments:
        input_n    - the number of inputs to the network. Should be the same as the length of a data sample
                     Has type int
        layer_dims - a python list or tuple of the number of neurons in each layer. Layer `l` should have a weight matrix
                     with the shape (`layer_dims[l-1]`, `layer_dims[l]`).
                     `layer_dims[-1]` is the dimension of the output layer.
                     Layer 1 should have the dimensions (`input_n`, `layer_dims[0]`).
                     len(layer_dims) is the depth of the neural network
        Attributes:
        input_n - the number of inputs to the network. Has type int
        layers  - a python list of each layer in the network. Each layer should use the `relu` activation function,
                  except for the last layer, which should use `softmax`.
                  Has type `list` containing layers of type `Layer`
        """
        Layers = [None]*(len(layer_dims))
        Layers[0] = Layer( input_n, layer_dims[0], activation_fn=relu)

        for l in range(1,len(layer_dims)-1):
            Layers[l] = Layer( layer_dims[l-1], layer_dims[l], activation_fn=relu)
        Layers[-1] = Layer( layer_dims[-2], layer_dims[-1], activation_fn=softmax)

        self.input_n = input_n
        self.layers = Layers

    def forward_prop(self, x):
        """
        Implement the forward propagation procedure through the entire network, from input to output.
        You will now connect each layer's forward propagation function into a chain of layer-wise forward propagations.

        Arguments:
        x - the input data, which has the shape (BATCH_SIZE, NUM_FEATURES) and type `numpy.ndarray`

        Returns:
        a - the output of the last layer after forward propagating through the every layer in `layers`.
            Should have the dimension (BATCH_SIZE, layers[-1].w.shape[1]) and type `numpy.ndarray`
        """
        for layer in self.layers:
            x = layer.forward_prop(x)
        a=x
        return a
```

```python
# test case
test_neuralnetwork(NeuralNetwork)
```

    [[ 4.70955644 -2.05463245  0.86114311]
     [ 1.48952066 -0.64983136  0.27235908]]
    [ 4.70955644 -2.05463245  0.86114311]
    [1.         0.00115438 0.02131353]
    [ 1.48952066 -0.64983136  0.27235908]
    [0.03995363 0.00470378 0.01182904]
    Test passed

## 3 Making predictions with a neural network
In practice, its common to load weights to your neural network that has already been trained.
In this section, you will create an instance of your neural network, load trained weights from disk, and perform predictions.

### 3.1 Load weights from disk
Create an instance of `NeuralNetwork` with input size $28 \times 28 = 784$, two hidden layers of size 100 and an output layer of size 10. Thereafter, load the weights contained in `./utils/ann_weights.npz` to your network.

```python
ann = NeuralNetwork(input_n=784, layer_dims=(100,100,10)) # Create instance of ann

# load weights
weights = np.load('./utils/ann_weights.npz')
for l in range(len(ann.layers)):
    ann.layers[l].w = weights['w' + str(l)]
    ann.layers[l].b = weights['b' + str(l)]
```

### 3.2 Prediction
Now, implement the function `predict_and_correct` which does the following:
1. Load `./utils/test_data.npz` from disk
2. Extract test data `X` and `Y` from file
2. Perform for every pair of data:
    a. plot the image `x`
    b. make a prediction using your neural network by forward propagating and picking the most probable class
    c. check wether the prediction is correct (compare with the ground truth number `y`)
    d. print the predicted label and wether it was correct or not

```python
def predict_and_correct(ann):
    """ Load test data from file and predict using your neural network.
    Make a prediction for every data sample and print it along with wether it was a correct prediction or not

    Arguments:
    ann - the neural network to use for prediction. Has type `NeuralNetwork`

    Returns: # for test case purposes
    A `numpy.ndarray` of predicted classes (integers [0-9]) with shape (11,)
    """
    data = np.load('./utils/test_data.npz');
    X, cls = data['X'], data['Y'];

    cls_preds = np.argmax(ann.forward_prop(X),axis=1); # Make a predicted number for every image in X
    print(cls_preds)
    for i in range(len(X)):
        plt.imshow(X[i].reshape(28,28), cmap='gray')
        plt.show()
        correct = cls_preds[i] == cls[i]
        print('The prediction was {0}, it was {1}!'.format(cls_preds[i], 'correct' if correct else 'incorrect'))

    return cls_preds

cls_pred = predict_and_correct(ann)
```

```python
# final test case
test_predict_and_correct_answer(cls_pred)
```

    Test passed, you have implemented your neural network correctly!

```python
# sample output from cell above for reference
```

## Congratulations!
You have successfully implemented a neural network from scratch using only NumPy!


# Derivative Formulas
http://tutorial.math.lamar.edu/pdf/Calculus_Cheat_Sheet_All.pdf

$\newcommand{\dv}{\displaystyle \frac{d}{dx}}$
$\newcommand{\ds}{\displaystyle}$

----

### Limits - Definition
#### Precise Definition
$\displaystyle\lim_{x\to\alpha} f(x) = L$
if for every $\epsilon > 0$, there is  $\delta > 0$, such that whenever $0 < |x-\alpha| < \delta$ then $|f(x) - L| < \epsilon$
#### Working Definition
We say $\displaystyle\lim_{x\to\alpha} f(x) = L$
if we can make f(x) as close to L as we want by taking x sufficiently close to $\alpha$ (on either side of $\alpha$) without letting x = $\alpha$.
#### Right Hand Limit Definition
$\displaystyle\lim_{x\to\alpha^+} f(x) = L$,
same definition as above, abd x>$\alpha$
#### Left Hand Limit Definition
$\displaystyle\lim_{x\to\alpha^+} f(x) = L$,
same definition as above, and x<$\alpha$

----
### Basic Derivatives
$\ds \begin{equation} \label{eq1}
\begin{split}
(c f)^\prime & = c f^{\prime}(x), c\in\mathbb{R} \ \\
\\
(f \pm g)^\prime & = f^{\prime}(x) \pm g^{\prime}(x) \\
\\
(f g)^\prime & = f^{\prime}g + fg^{\prime} \\
\\
\left(\frac{f}{g}\right)^\prime & = \frac{f^{\prime}g - fg^{\prime}}{g^2} \\
\\
\dv\Big( f\big(g(x)\big)\Big) & = f^{\prime}\big(g(x)\big)\;g^{\prime}(x) \\
\end{split}
\end{equation}$

----

| Formula Example Formula Example | Derivative Formula Derivative Formula |
| -- | -- |
| $\dv f( c ) = 0 $ | $\dv f( sin\;x ) = \ds cos\;x $ |
| $\dv f( x ) = 1 $ | $\dv f( cos\;x ) = \ds -sin\;x $ |
| $\dv f( x^a ) = ax^{a-1} $ | $\dv f( tan\;x ) = \ds sec^2\;x $ |
| $\dv f( a^x ) = a^x\ln(a) $ | $\dv f( sec\;x ) = \ds sec\;x\;tan\;x $ |
| $\dv f( e^x ) = e^x $ | $\dv f( csc\;x ) = \ds -csc\;x\;cot\;x $ |
| $\dv f( ln(x) ) = \frac{1}{x}, x > 0 $ | $\dv f( cot\;x ) = \ds -csc^2\;x $ |
| $\dv f( ln|x| ) = \frac{1}{x}, x \ne 0 $ |
| $\dv f( log_a(x) ) = \frac{1}{x ln(a)}, x > 0 $ |

### Notes
| Formula Rule | Formula Example Formula Example | Related Derivative Formula | Where Condition |
| -- | -- | -- | -- |
| | $\ds f(x) = ax^b + c$ | $\ds \frac{d}{dx}f(x) = abx^{b-1} dx $ |
| | $\ds f(x) = a_1x^b_1 + a_2x^b_2 + c$ | $\displaystyle \frac{d}{dx}f(x) = a_1b_1x^{b_1-1} + a_2b_2x^{b_2-1} dx $ |
| | $\ds f(x) = e^x + c $ | $\ds \frac{d}{dx}f(x) = e ^x dx $ |
| | $\ds f(x) = ln(x) + c $ | $\ds \frac{d}{dx}f(x) = x^{-1} dx $ |
| | $\ds \big(\alpha\;f(x)\big)^{\prime} = \alpha\;f^{\prime}(x) $ | $\dv \big(\alpha\;f(x)\big) = \alpha\;\dv f(x)$ | $\alpha\in\mathbb{R}$ |
| Sum Rule | $\ds (f \pm g)^{\prime} $ | $\ds f^{\prime}(x) \pm g^{\prime}(x)$ | |
| | $\dv \Big( \sum_{i=1}^{n} f_i(x) \Big)$ | $\ds \sum_{i=1}^{n} \dv f_i(x) $ | |

----
### Basic Integrals
$\newcommand{\dv}{\displaystyle \frac{d}{dx}}$
$\newcommand{\ds}{\displaystyle}$

$\ds \begin{equation} \label{eq1}
\begin{split}
\text{Integratation by parts} \\
\int u.\frac{d(v)}{dx} &= uv - \int v.\frac{d(u)}{dx} dx \\
\end{split}
\end{equation}$

----

### Integration by Parts
#### Derivation
$\newcommand{\dv}{\displaystyle \frac{d}{dx}}$
$\newcommand{\ds}{\displaystyle}$
$\ds \begin{equation} \label{eq1}
\begin{split}
y & = u v \\
\frac{dy}{dx} & = \frac{d(uv)}{dx} \\
\frac{d(uv)}{dx} & = u.\frac{d(v)}{dx} + v.\frac{d(u)}{dx} \\
\text{rearrange...} \\
u.\frac{d(v)}{dx} & = \frac{d(uv)}{dx} - v.\frac{d(u)}{dx} \\
\text{integrate both sides} \\
\int u.\frac{dv}{dx} dx & = \int \Big( \frac{d(uv)}{dx} - v.\frac{du}{dx} \Big) dx \\
\int u.\frac{dv}{dx} dx & = \int \frac{d(uv)}{dx} dx - \int v.\frac{du}{dx} dx \\
\text{formula:} \\
\int u.\frac{dv}{dx} dx & = u.v - \int v.\frac{du}{dx} dx \\
\end{split}
\end{equation}$

---

### Quadratic Formula
#### Derivation
$\newcommand{\dv}{\displaystyle \frac{d}{dx}}$
$\newcommand{\ds}{\displaystyle}$
(from https://www.mathsisfun.com/algebra/completing-square.html)

$\ds \begin{equation} \label{quadratic}
\begin{split}
ax^2 + bx + c & = 0 \\
x^2 + \frac{b}{a}x + \frac{c}{a} & = 0  \\
x^2 + \frac{b}{a}x & = - \frac{c}{a}  \\
\text{complete the square} \\
\Big(x + \frac{b}{2a}\Big)^2 - \Big(\frac{b}{2a}\Big)^2 & = - \frac{c}{a}  \\
\text{complete the square} \\
\Big(x + \frac{b}{2a}\Big)^2 & = - \frac{c}{a} + \Big(\frac{b}{2a}\Big)^2 \\
\Big(x + \frac{b}{2a}\Big)^2 & = - \frac{c}{a} + \frac{b^2}{(2a)^2} \\
\Big(x + \frac{b}{2a}\Big)^2 & = - \frac{4ac}{4a^2} + \frac{b^2}{(2a)^2} \\
\Big(x + \frac{b}{2a}\Big)^2 & = \frac{b^2 - 4ac}{(2a)^2} \\
x + \frac{b}{2a} & = \pm \sqrt{ \frac{b^2 - 4ac}{(2a)^2} } \\
x & = \frac{\pm \sqrt{ b^2 - 4ac}}{2a} - \frac{b}{2a}\\
x & = \frac{-b \pm \sqrt{ b^2 - 4ac}}{2a}\\
\end{split}
\end{equation}$

```python

```


# Linear Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

### Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1,

for example $y = 2(x) + 3$.

* 2 is Slope
* 3 is inter

```python
x = np.arange(-5.0, 5.0, 0.1)
print("Size of X Data is:",x.size)
x
```

    Size of X Data is: 100

    array([-5.00000000e+00, -4.90000000e+00, -4.80000000e+00, -4.70000000e+00,
           -4.60000000e+00, -4.50000000e+00, -4.40000000e+00, -4.30000000e+00,
           -4.20000000e+00, -4.10000000e+00, -4.00000000e+00, -3.90000000e+00,
           -3.80000000e+00, -3.70000000e+00, -3.60000000e+00, -3.50000000e+00,
           -3.40000000e+00, -3.30000000e+00, -3.20000000e+00, -3.10000000e+00,
           -3.00000000e+00, -2.90000000e+00, -2.80000000e+00, -2.70000000e+00,
           -2.60000000e+00, -2.50000000e+00, -2.40000000e+00, -2.30000000e+00,
           -2.20000000e+00, -2.10000000e+00, -2.00000000e+00, -1.90000000e+00,
           -1.80000000e+00, -1.70000000e+00, -1.60000000e+00, -1.50000000e+00,
           -1.40000000e+00, -1.30000000e+00, -1.20000000e+00, -1.10000000e+00,
           -1.00000000e+00, -9.00000000e-01, -8.00000000e-01, -7.00000000e-01,
           -6.00000000e-01, -5.00000000e-01, -4.00000000e-01, -3.00000000e-01,
           -2.00000000e-01, -1.00000000e-01, -1.77635684e-14,  1.00000000e-01,
            2.00000000e-01,  3.00000000e-01,  4.00000000e-01,  5.00000000e-01,
            6.00000000e-01,  7.00000000e-01,  8.00000000e-01,  9.00000000e-01,
            1.00000000e+00,  1.10000000e+00,  1.20000000e+00,  1.30000000e+00,
            1.40000000e+00,  1.50000000e+00,  1.60000000e+00,  1.70000000e+00,
            1.80000000e+00,  1.90000000e+00,  2.00000000e+00,  2.10000000e+00,
            2.20000000e+00,  2.30000000e+00,  2.40000000e+00,  2.50000000e+00,
            2.60000000e+00,  2.70000000e+00,  2.80000000e+00,  2.90000000e+00,
            3.00000000e+00,  3.10000000e+00,  3.20000000e+00,  3.30000000e+00,
            3.40000000e+00,  3.50000000e+00,  3.60000000e+00,  3.70000000e+00,
            3.80000000e+00,  3.90000000e+00,  4.00000000e+00,  4.10000000e+00,
            4.20000000e+00,  4.30000000e+00,  4.40000000e+00,  4.50000000e+00,
            4.60000000e+00,  4.70000000e+00,  4.80000000e+00,  4.90000000e+00])

```python
##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 0.2 * np.random.normal(size=x.size)
ydata = y + y_noise
```

```python
plt.figure(figsize=(16,8))
plt.scatter(x, ydata)
plt.plot(x,y,'r')
plt.title("Strongly Postive Linear Regression Line")
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

```python
x = np.linspace(-1,-4,100)
plt.figure(figsize=(16,8))
plt.scatter(x, ydata)
plt.plot(x,y,'r')
plt.title("Strongly Negative Linear Regression Line")
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 5 * np.random.normal(size=x.size)
ydata = y + y_noise

#Ploting My X and Y
plt.figure(figsize=(16,9))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'k')
plt.title('cubic of Poly')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Log

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.log(1*(x**3) + 1*(x**2) + 1*x + 3)

y_noise = 0.9* np.random.normal(size=x.size)

ydata = y + y_noise

#Ploting My X and Y
plt.figure(figsize=(16,9))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'k')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise =1  * np.random.normal(size=x.size)
ydata = y + y_noise

plt.figure(figsize=(16,9))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.title("Quadratic or plob")
plt.show()
```

```python
x = np.linspace(-9,9,400)
y = np.linspace(-5,5,400)
x,y= np.meshgrid(x,y)
```

```python
def axes():
    plt.figure(figsize=(16,9))
    plt.axhline(0,alpha=.2)
    plt.axvline(0,alpha=.9)
```

```python
a = .4
b = .2
axes()
plt.contour(x, y, (x**2/a**2 - y**2/b**2),[1],colors='y')
plt.show()
```

```python
axes()
plt.contour(x, y, (y**2-4*a*x),[0],colors='y')
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)
#print(.shape)
##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)
y_noise = 5 * np.random.normal(size=X.size)
ydata = Y + y_noise
#print(ydata.shape)
```

```python
plt.figure(figsize=(16,9))
plt.plot(X, ydata,  'yo')
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

y_noise = 0.2*np.random.normal(size=X.size)
ydata = Y + y_noise

plt.figure(figsize=(16,9))
plt.plot(X, ydata,  'bo')
plt.plot(X,Y)

plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ e^{(X-d)}}$$

or

$$ Y = \frac{1}{1+ e^{(-Z)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))
y_noise = 0.10*np.random.normal(size=X.size)
ydata = Y + y_noise

plt.figure(figsize=(16,9))
plt.plot(X,ydata,'bo')
plt.plot(X,Y)

plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

```python
a = np.logspace(1,1)  # Weekly Corelated
plt.plot(a)
```

```python
plt.plot(np.logspace(1,4)) # exp
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd
import pandas_profiling

df = pd.read_csv("DataSets/china_gdp.csv")
df.head()
```

```python
pandas_profiling.ProfileReport(df)
```

```python
import seaborn as sns
sns.pairplot(df)
```

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it deaccelerates slightly in the 2010s.

```python
x_data = df["Year"].values
y_data = df["Value"].values

plt.figure(figsize=(16,9))
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
    y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
    return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.figure(figsize=(16,9))
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
print(xdata)
```

```python
ydata =y_data/max(y_data)
ydata
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
```

```python
print(popt)
print(pcov)
```

```python
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

Now we plot our resulting regresssion model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(16,8))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'k*', label='data')
plt.plot(x,y, linewidth=5.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

train_x.shape
```

```python
test_x.shape
```

```python
train_y.shape
```

```python
test_y.shape
```

```python
# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
```

```python
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```


## Workshop 8: Bayesian Inference with Regression

*Note:* Chrome displays a vertical bar after all maths expressions. This is not intentional, it's a display bug. If this annoys you, please use a different browser.

In this workshop we will be looking at Bayesian linear regression. Briefly, this involves learning a linear regression model from a training set of *(x, y)* pairs, where *x* is a vector representing a data point and *y* is a real-valued response variable. Earlier we looked at ridge regression, which involved:
1. assuming a linear relationship between inputs and outputs, i.e., $y \approx \mathbf{x}' \mathbf{w}$ for all pairs
1. minimising the *residual sum of squares error*, that is finding the parameters $\hat{\mathbf{w}}$ that give the best fit to the training responses (subject to added *regularisation* term)
1. using $\hat{\mathbf{w}}$ to make test inferences

Today we will look at *Bayesian* inference for the above model. In this case we don't follow steps 2 and 3 above, but rather formulate the *posterior* over the parameters, and make test inferences using *all settings of the parameters* weighted by their posterior probability. These operations can be solved exactly, using linear algebra.

### (Desperately) seeking posterior

Before diving into the implementation, let's recap the maths behind Bayesian linear regression. We assume the data was generated from a Normal distribution, with its mean a linear function of the input vector and constant variance $\sigma^2$ (assumed known, herein).

$$ \newcommand\ys{\mathbf{y}}
\newcommand\xs{\mathbf{x}}
\newcommand\Xs{\mathbf{X}}
\newcommand\ws{\mathbf{w}}
\newcommand\Vs{\mathbf{V}}
\newcommand\Is{\mathbf{I}}
\begin{align*}
y &\sim \textrm{Normal}(\xs' \ws, \sigma^2) & \mbox{Likelihood}\\
\ws &\sim \textrm{Normal}(\mathbf{0}, \gamma^2 \mathbf{I}_D) & \mbox{Prior}
\end{align*}$$

The prior over the weights encourages low-magnitude weights, with parameter $\gamma^2$ controlling the variance.

Given this formulation, we next step is to find an expression for the posterior

$$
\begin{align*}
p(\ws | \Xs, \ys, \sigma^2) = \frac{ p(\ys | \Xs, \ws, \sigma^2) p(\ws) }{ p(\ys | \Xs) } \propto p(\ys | \Xs, \ws, \sigma^2) p(\ws)
\end{align*}
$$

where $\Xs$ are the training points (a matrix) and $\ys$ are a vector of the response values for each training point.

Recall that this can be solved, resulting in a Normal distribution for the posterior. We'll come back to this later on.

## Data set

We'll generate a simple synthetic dataset with a tiny handful of points drawn from a simple quadratic function,
$$ y = 5(x-\frac{1}{2})^2 $$
with a small amount of added noise.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
# generate a few points
N = 5
# each x is a scalar, uniformly drawn between [0, 1]
X = np.sort(np.random.random(size=N))
# reshape X into a matrix, such that each instance is a row
X = X[:,np.newaxis]

#Fixed Run#3 for more clear uncertainty plot
X2 = [ [0.18615756], [0.59675711] ,[0.67832224] ,[0.70492722] ,[0.89709031]]
X = np.array(X2)
print(X)

# generate the target response values using the quadratic function
# and additive noise
sigma2 = 0.1**2
y = 5*(X-0.5)**2 + sigma2**0.5 * np.random.normal(size=(N,1))

# plot the training data
plt.plot(X, y, 'bo')

# and plot the true function (without noise)
Xp = np.arange(-0.2, 1.2, 0.01)
Xp = Xp[:,np.newaxis]
yp_gold = 5*(Xp-0.5)**2
plt.plot(Xp, yp_gold, 'g')
```

### Polynomial basis functions

We'll fit this data with a polynomial model, considering a few different orders. Recall the basis function trick whereby you can augment the features of the input with new columns to implement a richer model class. Here we'll create a *basis matrix* for a $d^{th}$ order polynomial as follows

$$ \Phi = \begin{bmatrix}
    1 & x_1 & x_1^2 & x_1^3 & \ldots & x_1^d \\
    1 & x_2 & x_2^2 & x_2^3 & \ldots & x_2^d \\
    \vdots &    \vdots &    \vdots &\vdots &    \ddots & \vdots \\
    1 & x_n & x_n^2 & x_n^3 & \ldots & x_n^d \\
\end{bmatrix} $$

Observe how each row is a training point raised to a different power, from *0* to *d*. This means that when we take a row from $\Phi$ and take the dot product with $\mathbf{w}$, a parameter vector of size *d+1*, we obtain

$$ \begin{align}
\Phi_{i} \mathbf{w} &= \sum_{j=0}^d \Phi_{ij} w_j \\
&= 1 \times w_0 + x_i \times w_1 + x_i^2 \times w_2 + \ldots + x_i^d \times w_d
\end{align}$$

I.e., a $d^{th}$ order polynomial expression with $\mathbf{w}$ giving the coefficients.

```python
def polynomial_basis(X, Xtest, d):
    Phi = np.hstack([X ** i for i in range(0,d+1)])
    Phi_test = np.hstack([Xtest ** i for i in range(0,d+1)])
    return Phi, Phi_test

Phi, Phi_p = polynomial_basis(X, Xp, 9)
print (Phi.shape)
```

    (5, 10)

**Discussion**: How does this basis trick relate to kernel methods?

### Computing the posterior

Returning to the maths, the next step is to formulate the posterior. We plug in the Normal defintions of the likelihood and prior to solve for the unnormalised posterior

$$
\begin{align*}
p(\ws | \Xs, \ys, \sigma^2) & \propto \textrm{Normal}(\ws | \mathbf{0}, \gamma^2 \mathbf{I}_D)
\textrm{Normal}(\ys | \Xs \ws, \sigma^2 \mathbf{I}_N) \\
& \propto  \textrm{Normal}(\ws |  \ws_N, \mathbf{V}_N) \\
\mbox{where} ~ \ws_N &= \frac{1}{\sigma^2} \Vs_N \Phi' \ys \\
\Vs_N &= \sigma^2 ( \Phi' \Phi + \frac{\sigma^2}{\gamma^2} \Is_D )^{-1}
\end{align*}
$$

which is (proportional to) a Normal distribution. As we know the normalising term for a Gaussian [the denominator of the PDF](https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Non-degenerate_case), this means we have an exact normalised solution for the posterior, i.e.,  $p(\ws | \Xs, \ys, \sigma^2) = \textrm{Normal}(\ws |  \ws_N, \mathbf{V}_N)$.

Let's compute the mean vector and covariance matrix for the posterior based on the above expression.

```python
gamma2 = 100 # large number = more permissive, prior is broader
VN = sigma2 * np.linalg.inv(np.dot(Phi.T, Phi) + sigma2/gamma2 * np.eye(Phi.shape[1]))
wN = 1/sigma2 * np.dot(VN, np.dot(Phi.T, y))
print(wN.shape, VN.shape)
```

    (10, 1) (10, 10)

Next, we can plot the prior and posterior to see how they differ; given we have several weights, we'll just look at a couple of these, $w_0$ and $w_1$.

```python
# set up a 2d plot mesh
plt.figure(1)
delta = 0.05
xpl = np.arange(-3, 3, delta)
ypl = np.arange(-3, 3, delta)
Xpl, Ypl = np.meshgrid(xpl, ypl)

# plot a bivariate normal for the prior
Zpl = mlab.bivariate_normal(Xpl, Ypl, gamma2 ** 0.5,gamma2 ** 0.5, 0, 0, 0)
CS = plt.contour(Xpl, Ypl, Zpl)
plt.clabel(CS, inline=1, fontsize=10)
plt.title('Prior p(w) indices 0,1')

# plot a bivariate normal for the posterior
plt.figure(2)
Zpl = mlab.bivariate_normal(Xpl, Ypl, VN[0,0] ** 0.5, VN[1,1] ** 0.5, wN[0], wN[1], VN[0,1])
CS = plt.contour(Xpl, Ypl, Zpl) #, levels=[0.001, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5])
plt.clabel(CS, inline=1, fontsize=10)
plt.title('Posterior p(w|X,y) indices 0,1')
plt.plot(wN[0], wN[1], 'rx')
```

**Discussion question**: Can you explain why the prior and the posterior are so different? How is this related to the dataset? Why are the elipses in the posterior not aligned to the axes? *You might want to change the parameter indices from 0,1 to other pairs to get a better idea of the full posterior.*

### Bayesian inference

The conceptually simplest means of doing inference is to draw a few parameter vectors from the posterior (sampling from a Gaussian). Let's do this 10 times and plot the results.

```python
# plot the data points
plt.plot(X, y, 'o')
# plot the mean prediction
yp = np.dot(Phi_p, wN)
plt.plot(Xp, yp, 'g-')

# generate some samples from the posterior
samples = []
ps = []
for s in range(10):
    # draw a weight vector
    w_sampled = np.random.multivariate_normal(wN.flatten(), VN, 1).flatten()
    # plot the predictions for this weight vector
    yp_sampled = np.dot(Phi_p, w_sampled)
    p = plt.plot(Xp.flatten(), yp_sampled.flatten(), ':', lw=3)
    samples.append(w_sampled)
    ps.append(p[0])

plt.ylim(-1,2)
```

It's interesting to see what happens near the data points, and away from them, in particular the edges of the plot. We'll come back to this.

But there's a more elegant solution, as the predictive distribution can be found in closed form. Namely

$$
\begin{align*}
p(y_{*} | \xs_{*}, \Xs, \ys, \sigma^2) &= \textrm{Normal}(y_{*} | \xs_{*}'\ws_N, \sigma^2_N(\xs_{*})) \\
\sigma^2_N(\xs_{*}) & = \sigma^2 + \xs_{*}' \Vs_N \xs_{*}
\end{align*}
$$

Note that the predictive mean is a simple application of the posterior mean to the data point, but the predictive variance is  a bit more complicated. Let's compute the mean and variance on the test points, and visualise the results.

```python
# compute the mean from above
yp = np.dot(Phi_p, wN).flatten()

# compute the variance using formula above
s2s = np.zeros(Xp.shape[0])
for i in range(Xp.shape[0]):
    xs = Phi_p[i,:]
    s2s[i] = sigma2 + np.dot(xs.T, np.dot(VN, xs))

# plot the data
plt.plot(X, y, 'o')
# plot the 95% confidence interval
plt.fill_between(Xp.flatten(), yp+2*s2s ** 0.5, yp-2*s2s ** 0.5, facecolor='blue', alpha=0.1)
# plot the mean as a green dotted line
plt.plot(Xp, yp, 'g:')
# plot the ground truth in red
plt.plot(Xp, yp_gold, 'r')

plt.xlabel('x')
plt.ylabel('y')
plt.ylim(-2,5)
```

**Discussion**: How does the uncertainty plot compare to the samples above? How does the uncertainty change relative to the distance from training points? Can you explain why?

**Practical**: How does the setting of *gamma2* affect the fit? How about the number of data points, *N*. Try some other values and see what happens.

**Discussion**: Is a 9th order polynomial a good choice for this problem? Based on the results above, would you recommend this model, or make a different choice?

### Bayesian model selection

Now let's revisit the assumption of having a 9th order polynomial. The evidence gives us a good way of evaluating the quality of the fit.

We can compute the evidence, $p(\ys|\Xs, \sigma^2, \gamma^2)$ based on slide 31 of deck 14 (note the notation key, for mapping between Bishop to Murphy notation)

```python
def log_evidence(m, Phi, sigma2, gamma2):
    N, M = Phi.shape
    alpha, beta = 1/float(gamma2), 1/float(sigma2)
    rss = np.sum((y - np.dot(Phi, m)) ** 2)
    wpen = np.dot(m.T, m)
    E = beta/2.0 * rss + alpha/2.0 * wpen
    A = alpha * np.eye(M) + beta * np.dot(Phi.T, Phi)
    lE = M/2.0 * np.log(alpha) + N/2.0 * np.log(beta) - E \
        - 0.5 * np.log(np.linalg.det(A)) - N/2.0 * np.log(2.0 * np.pi)

    # return both the evidence, and the RSS term (the raw quality of fit)
    return lE.flatten()[0], rss

# what's the evidence for our 9th order model?
print(log_evidence(wN, Phi, sigma2, gamma2))
```

    (-8.2235382714206917, 3.5178227360987758e-05)

So what happens if we use a lower order model, e.g., a 3rd order model?

```python
Phi3, Phi_p3 = polynomial_basis(X, Xp, 3)
VN3 = sigma2 * np.linalg.inv(np.dot(Phi3.T, Phi3) + sigma2/gamma2 * np.eye(Phi3.shape[1]))
wN3 = 1/sigma2 * np.dot(VN3, np.dot(Phi3.T, y))

yp3 = np.dot(Phi_p3, wN3).flatten()
s2s = np.zeros(Phi_p3.shape[0])
for i in range(Phi_p3.shape[0]):
    xs = Phi_p3[i,:]
    s2s[i] = sigma2 + np.dot(xs.T, np.dot(VN3, xs))

plt.plot(X, y, 'o')
plt.fill_between(Xp.flatten(), yp3+2*s2s ** 0.5, yp3-2*s2s ** 0.5, facecolor='blue', alpha=0.1)
plt.plot(Xp, yp3, 'g:')
plt.plot(Xp, yp_gold, 'r')
plt.xlabel('x')
plt.ylabel('y')
```

**Discussion**: does that look like a better fit to you? Consider both the area [0,1] near the training points, and those outside this range.

Let's see what the *evidence* says, and compare this to the above result:

```python
log_evidence(wN3, Phi3, sigma2, gamma2)
```

    (-6.4222742732425395, 0.0096638048424639595)

The RSS has barely changed, but the evidence is much higher. We can look at various model orders to see which has the best *evidence* to perform Bayesian model selection:

```python
evs = []
gamma2 = 1000
for order in range(0,10):
    Phio, Phipo = polynomial_basis(X, Xp, order)
    Vo = sigma2 * np.linalg.inv(np.dot(Phio.T, Phio) + sigma2/gamma2 * np.eye(Phio.shape[1]))
    wo = 1.0/sigma2 * np.dot(Vo, np.dot(Phio.T, y))
    lEo = log_evidence(wo, Phio, sigma2, gamma2)
    print('order', order, 'log evidence', lEo[0], 'rss', lEo[1])
    evs.append((order,) + lEo)
```

    order 0 log evidence -20.5140120924 rss 0.417419791083
    order 1 log evidence -23.8041797097 rss 0.380971069447
    order 2 log evidence -9.37459173066 rss 0.0193299468997
    order 3 log evidence -9.90348064632 rss 0.00186445700331
    order 4 log evidence -10.7310556967 rss 0.000123843689711
    order 5 log evidence -11.4136164863 rss 2.07166464993e-05
    order 6 log evidence -11.9172319959 rss 1.46544834059e-05
    order 7 log evidence -12.2916612224 rss 1.82179774615e-05
    order 8 log evidence -12.5763208064 rss 2.2612626367e-05
    order 9 log evidence -12.7976083372 rss 2.62690480193e-05

```python
#plot the above log evidence values against the model order

evs = np.vstack(evs)
plt.plot(evs[:,0], evs[:,1])
plt.xlabel('Model Order')
plt.ylabel('Log Evidence')
plt.show()

plt.plot(evs[:,0], evs[:,2])
plt.xlabel('Model Order')
plt.ylabel('RSS')

```

**Discussion**: So which model class will be chosen? Is this a reasonable situation?

**Practical**: Rerun the code with a new random training set, or different values of *N*, such as 2 or 3 points or 20; the results may be different. Can you explain why the outcome might be different?


## Logistic regression

Logistic regression is classification method which is a bit similar to linear regression, but used for categorical dependent variable. Instead of predicting a value of the target variable, which is assumed to be linear, logistic regression tries to predict the probabilities of a binary response. Some common examples are:

* marking an email as a spam
* marking given medical image as possible illness

For the classification we use a logistic function:

\begin{align}
p = \frac{1}{1 + e^{-(b_{0} + b_{1}x)}}
\end{align}

Let's consider the same dataset like in the example of linear regression. This time, we are not going to predict an exact price, but instead would like to know if given zone is expensive when it comes to a median of value. First of all, we need to consider which examples should be thought to be expensive.

```python
# Show some statistics about the dataset
from sklearn.datasets import load_boston
import pandas as pd
import numpy as np

# Loading the dataset with pandas
boston_data = load_boston()
boston_housing_df = pd.DataFrame(boston_data.data,columns=boston_data.feature_names)
boston_housing_df["MEDV"] = boston_data.target
boston_housing_df.head()

boston_housing_df.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      CRIM
      506.0
      3.613524
      8.601545
      0.00632
      0.082045
      0.25651
      3.677083
      88.9762

      ZN
      506.0
      11.363636
      23.322453
      0.00000
      0.000000
      0.00000
      12.500000
      100.0000

      INDUS
      506.0
      11.136779
      6.860353
      0.46000
      5.190000
      9.69000
      18.100000
      27.7400

      CHAS
      506.0
      0.069170
      0.253994
      0.00000
      0.000000
      0.00000
      0.000000
      1.0000

      NOX
      506.0
      0.554695
      0.115878
      0.38500
      0.449000
      0.53800
      0.624000
      0.8710

      RM
      506.0
      6.284634
      0.702617
      3.56100
      5.885500
      6.20850
      6.623500
      8.7800

      AGE
      506.0
      68.574901
      28.148861
      2.90000
      45.025000
      77.50000
      94.075000
      100.0000

      DIS
      506.0
      3.795043
      2.105710
      1.12960
      2.100175
      3.20745
      5.188425
      12.1265

      RAD
      506.0
      9.549407
      8.707259
      1.00000
      4.000000
      5.00000
      24.000000
      24.0000

      TAX
      506.0
      408.237154
      168.537116
      187.00000
      279.000000
      330.00000
      666.000000
      711.0000

      PTRATIO
      506.0
      18.455534
      2.164946
      12.60000
      17.400000
      19.05000
      20.200000
      22.0000

      B
      506.0
      356.674032
      91.294864
      0.32000
      375.377500
      391.44000
      396.225000
      396.9000

      LSTAT
      506.0
      12.653063
      7.141062
      1.73000
      6.950000
      11.36000
      16.955000
      37.9700

      MEDV
      506.0
      22.532806
      9.197104
      5.00000
      17.025000
      21.20000
      25.000000
      50.0000

A mean of *medv* is 22.768769. By a simple thresholding we'll add another feature to our dataset, called **is_expensive** which will be set to 1 whenever the median value is higher than the mean.

```python
# Get the mean value of the medv as a threshold
IS_EXPENSIVE_THRESHOLD = boston_housing_df["MEDV"].mean()

# Append a new column to the dataset
boston_housing_df["is_expensive"] = boston_housing_df["MEDV"]\
    .map(lambda x: int(x > IS_EXPENSIVE_THRESHOLD))
boston_housing_df.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      MEDV
      is_expensive

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0
      1

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6
      0

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7
      1

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4
      1

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2
      1

The perfect predictor for the *is_expensive* variable is naturally *medv* feature, but we are not going to consider it. In one of the previous examples we found out that column *lstat* has the highest absolute correlation with *medv*, so for the 2D example we will consider it as an independent variable.

### 2D logistic regression

For the intuition, we are going to plot the *lstat* feature with *is_expensive* first, and then apply logistic regression to find, what is a threshold found by the algorithm itself.

```python
# Display a scatter plot: lstat vs is_expensive
boston_housing_df.plot.scatter(x="LSTAT", y="is_expensive", color="blue")
```

```python
from sklearn.linear_model import LogisticRegression

# Create an instance of LogisticRegression and fit it
logistic_regression = LogisticRegression()
logistic_regression.fit(X=boston_housing_df[["LSTAT"]],
                        y=boston_housing_df["is_expensive"])

# Check the coefficients of the created model
logistic_regression.coef_, logistic_regression.intercept_
```

    (array([[-0.43645866]]), array([4.33565677]))

```python
import matplotlib.pyplot as plt

# Calculating the value of the logistic function
x_values = np.linspace(0, 40)
y_values = 1.0 / (1.0 + np.exp(-(logistic_regression.coef_[0] * x_values + logistic_regression.intercept_[0])))

# Display a scatter plot: lstat vs is_expensive and logistic function
boston_housing_df.plot(x="LSTAT", y="is_expensive", color="blue", kind="scatter")
plt.plot(x_values, y_values, color="red", linestyle="dashed")
```

With a simple thresholing we may perform a classification - if the *is_expensive* value returned by the classifier is higher than 50%, we can consider it to be expensive.

### Multidimensional logistic regression

Once again, we will consider an example with two independent variables, in order to visualize the predictions in the 3-dimensional space. As the most correlated values are typically the best predictors for the linear models, we are going to consider *lstat* together with *rm*.

```python
from sklearn.preprocessing import StandardScaler

# Scale the original dataset first, and then run regression
scaler = StandardScaler()
boston_housing_lstatrm_scaled = scaler.fit_transform(
    boston_housing_df[["LSTAT", "RM"]])

# Create an instance of LogisticRegression and fit it
logistic_regression = LogisticRegression()
logistic_regression.fit(X=boston_housing_lstatrm_scaled,
                        y=boston_housing_df["is_expensive"])

# Check the coefficients of the created model
logistic_regression.coef_, logistic_regression.intercept_
```

    (array([[-2.42328339,  0.87415551]]), array([-0.94370746]))

```python
from mpl_toolkits.mplot3d import Axes3D

# Save the coefficients in a single array
coefficients = np.append(logistic_regression.coef_[0],
                         logistic_regression.intercept_[0])

# Calculate the values for a selected range
x = np.linspace(-2, 2)
y = np.linspace(-2, 2)
x_values, y_values = np.meshgrid(x, y)
z_values = 1.0 / (1.0 + np.exp(-(coefficients[0] * x_values + coefficients[1] * y_values + coefficients[2])))

# Display 3D scatter: rm, lstat vs is_expensive and logistic function
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.scatter(boston_housing_lstatrm_scaled[:, 0],
           boston_housing_lstatrm_scaled[:, 1],
           boston_housing_df["is_expensive"], c="blue")
ax.plot_surface(x_values, y_values, z_values, linewidth=0.2,
                color="red", alpha=0.5)
plt.show()
```