# AutoGenU for Jupyter

An Automatic C++ Code Generator for Nonlinear Model Predictive Control (NMPC)

---

## Formulation of the optimal control problem: Mobile robot

### Model description

The above figure shows the model of a mobile robot. The kinematics of the robot is given by
$$ \dot{X} = V \cos{\theta}, $$
$$ \dot{Y} = V \sin{\theta}. $$
Physical limitations of the robot are given by
$$ V_{\rm min} \leq V \leq V_{\rm max}, $$
$$ w_{\rm min} \leq \dot{\theta} \leq w_{\rm max}. $$

### Control Objective
We consider tracjectory tracking problem. We set the reference trajectory as
$$ x_{\rm ref} (t) = \begin{bmatrix} t v_{\rm ref} \\ 0 \end{bmatrix} .$$
We also assume that there are two circular obstacles to be avoided whose center and radius are $(X_1, Y_1)$, $R_1$ and $(X_2, Y_2)$, $R_2$, respectively.

### Formulation of the optimal control problem
We define the state vector as
$$ x = \begin{bmatrix} X \\ Y \\ \theta \end{bmatrix} .$$
We assume that $V=u_1$ and $\theta = u_2$. The state equation is then given as
$$ \dot{x} = f(t, x, u) = \begin{bmatrix} u_1 \cos{\theta} \\ u_1 \sin{\theta} \\ u_2 \end{bmatrix} $$

Next, we set the cost function
$$ J = \varphi(t, x) + \int_{t}^{t+T} L(t, x, u) .$$
To achieve the aforementioned goal, we set the terminal cost as
$$\varphi(t,x) = \frac{1}{2} q_1 (x_1 - t v_{\rm ref})^2 + \frac{1}{2} q_2 x_2 ^2 + \frac{1}{2} q_3 x_3 ^2 $$
and the stage cost as
$$ L(t, x, u) = \frac{1}{2} q_1 (x_1 - t v_{\rm ref})^2 + \frac{1}{2} q_2 x_2 ^2 + \frac{1}{2} q_3 x_3 ^2 + \frac{1}{2} r_1 (u_1 \cos{x_3} - v_{\rm ref})^2 + \frac{1}{2} r_2 u_2 ^2 $$
where $q_1, q_2, q_3$, $r_1, r_2$ are positive real constant.
From the physical limitations and the obstacles, there are 6 inequality constraints
$$
h(x, u) = \begin{bmatrix} R_1 ^2 - (x_1 - X_1)^2 - (x_2 - Y_1)^2 \\
                          R_2 ^2 - (x_1 - X_2)^2 - (x_2 - Y_2)^2 \\
                          v_{\rm min} - u_1 \\
                          u_1 - v_{\rm max} \\
                          w_{\rm min} - u_2 \\
                          u_2 - w_{\rm max}
                          \end{bmatrix} \leq 0
$$
In this example, these inequality constraints are considered by the semi-smooth Fischer-Burmeister method.
In the optimization problem, the complementary conditions are imposed for inequality constraints, e.g., for $g(x, u) \in \mathbb{R}$,
$$
    g(x, u) \leq 0 ,
$$
$$
    \nu \geq 0 ,
$$
$$
    \nu g(x, u) = 0 .
$$
The semi-smooth Fischer-Burmeister method transforms these conditions into
$$
    \Phi(\nu, - g(x, u)) = 0, \;\; \Phi(a, b) = \sqrt{a^2 + b^2 + \epsilon ^2} - (a + b) .
$$
where $\epsilon > 0$ is a regularization term.
After all, in the optimal control problem, we find the solution satisfying the following conditions
$$
    \left( \frac{\partial H}{\partial u} \right)^{\rm T} = \left( \frac{\partial L}{\partial u} \right)^{\rm T} + \left( \frac{\partial f}{\partial u} \right)^{\rm T} \lambda + \left( \frac{\partial h}{\partial u} \right)^{\rm T} \nu,
$$
$$
    \Phi(\nu_i, - h_i(x, u)) = 0 ,
$$
where $\lambda$ and $\nu$ are the Lagrange multipliers and the latter condition is imposed on elementwise.

### References
For more details about the semi-smooth Fischer-Burmeister method in the optimal control problem, especially with the C/GMRES method, see the following references.
- [M. Huang, et. al, Nonlinear Model Predictive Control of a Diesel Engine Air Path: A Comparison of Constraint Handling and Computational Strategies, IFAC-PapersOnLine, Vol. 48, No. 23, pp. 372-379 (2015)](https://doi.org/10.1016/j.ifacol.2015.11.308)
- [M. Huang, Low Complexity Model Predictive Control of a Diesel Engine Airpath, Ph. D. diss., University of Michigan (2016)](https://deepblue.lib.umich.edu/handle/2027.42/120832)
- [S. Alamdari, S. Amin, Stochastic Model Predictive Control for Eco-Driving Assistance Systems in Electric Vehicles, Ph. D. diss., University of Luxembourg (2018)](http://orbilu.uni.lu/handle/10993/36164)
- [D. Liao-McPherson, M. Huang and I. Kolmanovsky, A Regularized and Smoothed Fischer–Burmeister Method for Quadratic Programming With Applications to Model Predictive Control, IEEE Transactions on Automatic Control, Vol. 64, No. 7, pp. 2937-2944 (2019)](https://doi.org/10.1109/TAC.2018.2872201)

### Numerical solver of NMPC
In the following example, we compute the solution of NMPC using the multiple-shooting-based C/GMRES method.

---

# Generate C++ codes of NMPC model

## Import modules

```python
from autogenu import autogenu
```

## Set dimensions and define `AutoGenU`
- `dimx`: Dimension of the state vector $x$
- `dimu`: Dimension of the control input vector $u$
- `model_name`: Name of the model. Used as the name of the directory containing the generated C++ source files.

```python
dimx = 3
dimu = 2
model_name = 'mobilerobot'
ag = autogenu.AutoGenU(model_name, dimx, dimu)
```

## Generate t, x, and u, necessary variables to formulate of the optimal control problem
- `t`: Time parameter $t$
- `x`: The state vector $x$
- `u`: The control input vector $u$

```python
t = ag.define_t()
x = ag.define_x()
u = ag.define_u()
```

## Define variables used in the state equation, constraints, and cost function
- Define a scalar variable whose name is "var" as
`var = ag.define_scalar_var('var') `

- Define scalar variables whose names are "var\_1", "var\_2", ..., "var\_n" as
`var_1, var_2, ..., var_n = ag.define_scalar_vars('var_1', 'var_2', ..., 'var_n')`

- Define array variables whose name is "vec" and dimension is dim_vec as
`vec = ag.define_array_var('vec', dim_vec)`

```python
# Define variables used in the state function
vx_ref = ag.define_scalar_var('vx_ref')
v_min, v_max = ag.define_scalar_vars('v_min', 'v_max')
w_min, w_max = ag.define_scalar_vars('w_min', 'w_max')
X_1, Y_1, R_1 = ag.define_scalar_vars('X_1', 'Y_1', 'R_1')
X_2, Y_2, R_2 = ag.define_scalar_vars('X_2', 'Y_2', 'R_2')
xx_ref = vx_ref * t

# Variables used in the cost function
q = ag.define_array_var('q', dimx)
r = ag.define_array_var('r', dimu)
x_ref = ag.define_array_var('x_ref', dimx)
```

## Define the state equation, constraints, the stage cost, and the terminal cost
- `f`: The state equation $ f(t, x, u)$
- `C`: The equality constraisnts $C(t, x, u) = 0 $
- `h`: The inequality constraisnts $h(t, x, u) \leq 0$ considered by semi-smooth Fischer-Burumeister method
- `L`: The stage cost $L(t, x, u)$
- `phi`: The terminal cost $\phi (t, x)$

Note: array indices start with 0

```python
# Import symbolic math from sympy. If you need more functions, import same as below.
from sympy import sin, cos, tan, exp, log, sinh, cosh, tanh, diff, sqrt
```

```python
# Define the state equation
f = [u[0] * cos(x[2]),
     u[0] * sin(x[2]),
     u[1]]

# Define the equality constraints C(x, u) = 0
C = []

# Define the inequality constraints considered using the semi-smooth Fischer-Burmeister method
h = [R_1**2 - (x[0]-X_1)**2 - (x[1]-Y_1)**2,
     R_2**2 - (x[0]-X_2)**2 - (x[1]-Y_2)**2,
     v_min - u[0],
     u[0] - v_max,
     w_min - u[1],
     u[1] - w_max]

# Define the stage cost
L = (q[0]*(x[0]-xx_ref)**2 + q[1]*x[1]**2 + q[2]*x[2]**2) / 2 + (r[0]*(u[0]*cos(x[2])-vx_ref)**2 + r[1]*u[1]**2) / 2

# Define the terminal cost
phi = (q[0]*(x[0]-xx_ref)**2 + q[1]*x[1]**2 + q[2]*x[2]**2)/2
```

```python
# Set functions
ag.set_functions(f, C, h, L, phi)
```

## Set Parameters
Set values for all parameters you defined.
- Set `value` for a scalar variable whose name is "var" as
`ag.set_scalar_var('var', value) `

- Set `value_1`, `value_2`, ..., `value_n`, for scalar variables whose names are "var\_1", "var\_2", ..., "var\_n" as
`ag.set_scalar_vars(['var_1', value_1], ['var_2', value_2], ..., ['var_n', value_n])`

- Set array variables `var_1`, `var_2`, ..., `var_n` whose name is "vec" and dimension is n as
`ag.define_array_var('vec', [var_1, var_2, ..., var_n])`

- If you define the inequality constraints using Fischer-Brumeister method `h`, then you have to define related regularization term additionaly. The dimension of the regularization term must be same as the dimension of `h`. If the dimension is `dimh`, you need to set the regularization as `ag.set_FB_epsilon([eps_1, ..., eps_dimh])`

```python
ag.set_scalar_vars(['vx_ref', 0.4], ['X_1', 1], ['Y_1', 0.25], ['R_1', 0.5],
                   ['X_2', 2], ['Y_2', -0.25], ['R_2', 0.5],
                   ['v_min', -0.5], ['v_max', 0.5],
                   ['w_min', -0.75], ['w_max', 0.75])
ag.set_array_var('q', [10, 1, 0.01])
ag.set_array_var('r', [0.1, 0.1])
ag.set_array_var('x_ref', [0, 0, 0])

ag.set_FB_epsilon([0.01, 0.01, 0.0001, 0.0001, 0.0001, 0.0001])
```

## Generate C++ codes of NMPC model
Generate `nmpc_model.hpp` and `nmpc_model.cpp`, C++ source files of NMPC problem settings.
- `use_simplification`: The flag for simplification. If `True`, symbolic functions are simplified. However, if functions are too complicated, it takes too much time. Default is `False`.
- `use_case`: The flag for common subexpression elimination. If `True`, common subexpressions in fxu, phix, hx, and hu are eliminated when `nmpc_model.cpp` is generated. Default is `False`.

```python
use_simplification = False
use_cse = True

ag.generate_source_files(use_simplification, use_cse)
```

---
# Generate C++ codes for numerical simulation

## Set solvers
Set which solvers you use.
- `ContinuationGMRES` : The continuation/GMRES method (the original C/GMRES method, single shooting).
- `MultipleShootingCGMRES` : The multiple shooting based C/GMRES method with condensing of the state and the Lagragne multipliers with respect to the state equation.
- `MSCGMRESWithInputSaturation` : The multiple shooting based C/GMRES method with condensing of the state, the Lagragne multipliers with respect to the state equation, and variables with respect to the constraints on the saturation function on the control input.

Note that if you have to define the solver type as `autogenu.SolverType.ContinuationGMRES`, `autogenu.SolverType.MultipleShootingCGMRES`, or `autogenu.SolverType.MSCGMRESWithInputSaturation`

```python
solver_type = autogenu.SolverType.MultipleShootingCGMRES
ag.set_solver_type(solver_type)
```

## Set saturaions on the control input if you choose `MSCGMRESWithInputSaturation`
- Saturation on the control input: $u_{i, {\rm min}} \leq u_i \leq u_{i, {\rm max}}$
$u_i \in \mathbb{R}$ : a constrained component of the control input $u$
- Transformed equality constraint: $(u_i - \frac{u_{i, {\rm max}} + u_{i, {\rm min}}}{2})^2 - ( \frac{u_{i, {\rm max}} - u_{i, {\rm min}}}{2})^2 + {u_d}_i ^2 = 0$
${u_d}_i \in \mathbb{R}$ : a dummy input for the transformation of the saturation into the equality constraint
- Additional term in the stage cost $L(x, u)$ with respect to the saturation of $u_i$: $- {r_d}_i {u_d}_i + \frac{1}{2} {r_q}_i {u_d}_i ^2$
   ${r_d}_i > 0$: a weight parameter to avoid failure of numerical computation, ${r_q}_i \geq 0$: a weight parameter to increase mergin of the saturation

- Parameters:
    - `index`: $i$
    - `u_min`: $u_{i, {\rm min}}$
    - `u_max`: $u_{i, {\rm max}}$
    - `dummy_weight`: ${r_d}_i > 0$
    - `quadratic_weight` :  ${r_q}_i \geq 0$

```python
# Add saturation_list as follows.
# ag.add_control_input_saturation(index , u_min, u_max, dummy_weight, quadratic_weight)
```

## Set parameters for the C/GMRES method

- `T_f`, `alpha`: Parameters for the length of the horizon $T(t)$: $T(t) = T_f (1 - e^{-\alpha t})$. The length of the horizon converges to `T_f` and the speed of the convergence is determied by `alpha`.
- `N`: The division number of the horzion for the numerical computation.
- `finite_difference_increment`: Step length of a finite difference approximations of hessian-vector products in C/GMRES.
- `zeta`: A stabilization parameter of the C/GMRES method. Basically, must be set by the reciprocal of the sampling time.
- `kmax`: The maximam number of the iteration of the GMRES.

```python
T_f = 1.5
alpha = 1.0
N = 50
finite_difference_increment = 1.0e-08
zeta = 1000
kmax = 15

ag.set_solver_parameters(T_f, alpha, N, finite_difference_increment, zeta, kmax)
```

## Set parameters for the initialization of the solution
- `solution_initial_guess`: The initial guess of the solution of the optimal control problem (OCP) for initialization of the solution of NMPC.
- `newton_residual_torelance`: The residual torelance of the solution of the OCP for the initialization of the solution of NMPC. The Newton iteration terminates when the optimality error is less than this velue.
- `max_newton_iteration`: The maxmum number of Newton iteration for the initialization of the solution of NMPC.
- `initial_Lagrange_multiplier`: An optional parameter for `MSCGMRESWithInputSaturation`. This is a part of the initial guess of the solution, the initial guess of the Lagrange multiplier with respect the constraints on the saturation function of the control input. NOTE that this parameter has to be defined as list. If you do not specify this parameter, the all of initial Lagrange multiplier are set by 1e-03.

```python
solution_initial_guess = [0.1, 0.1, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01]
newton_residual_torelance = 1.0e-06
max_newton_iteration = 50

ag.set_initialization_parameters(solution_initial_guess, newton_residual_torelance, max_newton_iteration)
```

## Set parameters for numerical simulation
- `initial_time`: Initial time of the numerical simulation.
- `initial_state`: Initial state vector of the system.
- `simulation_time`: Simulation time of the numerical simulation.
- `sampling_time`: The sampling time of the numerical simulation.

```python
initial_time = 0
initial_state = [0, 0, 0]
simulation_time = 10
sampling_time = 0.001

ag.set_simulation_parameters(initial_time, initial_state, simulation_time, sampling_time)
```

## Generate main.cpp and CMakeLists.txt

```python
ag.generate_main()
ag.generate_cmake()
```

## Build and run simulation

- `generator`: An optional variable for Windows user to choose the generator of CMake. If 'MSYS', then 'MSYS Makefiles' is used. If 'MinGW', then 'MinGW Makefiles' is used. The default value is 'Auto' and the generator is selected automatically. If sh.exe exists in your PATH, MSYS is choosed, and otherwise MinGW is used.
- `remove_build_dir`: If `True`, the existing build directory is removed. If `False`, the build directory is not removed. Need to be set `True` if you change the generator. Default is `False`.

```python
generator = 'Auto'
remove_build_dir = False
ag.build(generator, remove_build_dir)
ag.run_simulation()
```

## Plot the simulation results

```python
import autogenu.simulation_plottor as simplot
```

```python
plot = simplot.SimulationPlottor(model_name)
plot.set_scales(2,5,2)
# plot.show_plots()
plot.save_plots()
```

## Draw animations of the simulation results

```python
import autogenu.animation_generator as animgen
```

```python
vx_ref = 0.4
X1, Y1, R1 = 1, 0.25, 0.5
X2, Y2, R2 = 2, -0.25, 0.5

anim = animgen.MobileRobot(model_name, vx_ref, X1, Y1, R1, X2, Y2, R2)
anim.set_skip_frames(10)
anim.generate_animation()
```


# Project 4, deadline  May 2

**Stock Market model**

Date: **Spring semester 2018**

## Theoretical background and description of the system

The aim of this project is to simulate financial transactions among financial agents
using Monte Carlo methods. The final goal is to extract a distribution of income  as function
of the income $m$.   From Pareto's work ([V. Pareto, 1897](http://www.institutcoppet.org/2012/05/08/cours-deconomie-politique-1896-de-vilfredo-pareto)) it is known from empirical studies
that the higher end of the distribution of money follows a distribution

$$
w_m\propto m^{-1-\alpha},
$$

with $\alpha\in [1,2]$. We will here follow the analysis made by [Patriarca and collaborators](http://www.sciencedirect.com/science/article/pii/S0378437104004327).

Here we will study numerically the relation between the micro-dynamic relations among financial
agents and the  resulting macroscopic money distribution.

We assume we have $N$ agents that exchange money in pairs $(i,j)$. We assume also that all agents
start with the same amount of money $m_0 > 0$. At a given 'time step', we choose randomly a pair
of agents $(i,j)$ and let a transaction take place. This means that agent $i$'s money $m_i$ changes
to $m_i'$ and similarly we have $m_j\rightarrow m_j'$.
Money is conserved during a transaction, meaning that

$$
\begin{equation}
  m_i+m_j=m_i'+m_j'.
\label{eq:conserve} \tag{1}
\end{equation}
$$

The change is done via a random reassignement (a random number) $\epsilon$, meaning that

$$
m_i' = \epsilon(m_i+m_j),
$$

leading to

$$
m_j'= (1-\epsilon)(m_i+m_j).
$$

The number $\epsilon$ is extracted from a uniform distribution.
In this simple model, no agents are left with a debt, that is $m\ge 0$.
Due to the conservation law above, one can show that the system relaxes toward an equilibrium
state given by a Gibbs distribution

$$
w_m=\beta \exp{(-\beta m)},
$$

with

$$
\beta = \frac{1}{\langle m\rangle},
$$

and $\langle m\rangle=\sum_i m_i/N=m_0$, the average money.
It means that after equilibrium has been reached that the majority of agents is left with a small
number of money, while the number of richest agents, those with $m$ larger than a specific value $m'$,
exponentially decreases with $m'$.

We assume that we have $N=500$ agents.   In each simulation, we need a sufficiently large number of transactions, say $10^7$. Our aim is find the final equilibrium distribution $w_m$. In order to do that we would need
several runs of the above simulations, at least $10^3-10^4$ runs (experiments).

### Project 4a): Simulation of Transactions

Your task is to first set up an algorithm which simulates the above transactions with an initial
  amount $m_0$.
  The challenge here is to figure out a Monte Carlo  simulation  based on the
  above equations.
  You will in particular need to make an algorithm which sets up a histogram as function of $m$.
  This histogram contains the number of times a value $m$ is registered and represents
  $w_m\Delta m$. You will need to set up a value for the interval $\Delta m$  (typically $0.01-0.05$).
  That means you need to account for the number of times you register an income in the interval
  $m,m+\Delta m$. The number of times you register this income, represents the value that enters the histogram.
  You will also need to find a criterion for when the equilibrium situation has been reached.

### Project 4b): Recognizing the distribution

Make thereafter a plot of  $\log{(w_m)}$ as function of $m$
  and see if you get a straight line.
  Comment the result.

### Project 4c): Transactions and savings

We can then change our model to allow for a saving criterion, meaning that the agents save
  a fraction $\lambda$ of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.

  The conservation law of Eq. ([eq:conserve](#eq:conserve)) holds, but the money to be shared in a transaction between
  agent $i$ and agent $j$ is now $(1-\lambda)(m_i+m_j)$. This means that we have

$$
m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
$$

and

$$
m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
$$

which can be written as

$$
m_i'=m_i+\delta m
$$

and

$$
m_j'=m_j-\delta m,
$$

with

$$
\delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
$$

showing how money is conserved during a transaction.
  Select values of $\lambda =0.25,0.5$ and $\lambda=0.9$ and try to extract the corresponding
  equilibrium distributions and compare these with the Gibbs distribution. Comment your results.
Extract a parametrization of the above curves, see for example [Patriarca and collaborators](http://www.sciencedirect.com/science/article/pii/S0378437104004327) and see if you can parametrize the high-end tails of the distributions in terms of power laws. Comment your results.

### Project 4d): Nearest neighbor interactions

In the rest of this project we will follow the work of [Goswami and Sen](http://www.sciencedirect.com/science/article/pii/S0378437114006967).
In the studies above the agents were selected randomly, irrespective of whether we allowed for
saving or not during a transaction. What is often observed is that various agents tend to make preferences for for whom to interact with. We will now study the evolution of the distribution of wealth $w_m$ by assuming that there is a likelihood

$$
p_{ij} \propto \vert m_i-m_j\vert^{-\alpha},
$$

for an interaction between agents $i$ and $j$ with respective wealths $m_i$ and $m_j$. The parameter  $\alpha >  0$. For $\alpha=0$ we recover our model from part 5a).
Perform the same analysis as previously with $N=500$ as well as with $N=1000$ agents and study the distribution of wealth for $\alpha =0.5$, $\alpha =1.0$, $\alpha =1.5$ and $\alpha =2.0$.
You should try to reproduce Figure 1 of [Goswami and Sen](http://www.sciencedirect.com/science/article/pii/S0378437114006967).
Extract the tail of the distribution and see if it follows a Pareto distribution

$$
w_m\propto m^{-1-\alpha}.
$$

What happens if $\alpha \gg 1$?

Perform the analysis with and without a saving $\lambda$ on each transaction and comment your results.
### Project 4e): Nearest neighbors and former  transactions

We add to the previous probability the possibility that two agents who interact have performed similar transactions earlier. That is, in addition to being financially close, we assume that the likelihood for interacting increases if two agents have interacted earlier.
We add this feature by modifying the previous likelihood to

$$
p_{ij} \propto \vert m_i-m_j\vert^{-\alpha}\left(c_{ij}+1\right)^{\gamma},
$$

where $c_{ij}$ represents the number of  previous interactions that have taken place between $i$ and $j$. The factor $1$ is added in order to ensure that if they have not interacted earlier they can still interact.  Perform similar studies as above with $N=1000$, $\alpha=1.0$ and $\alpha=2.0$ using $\gamma = 0.0, 1.0, 2.0, 3.0$ and $4.0$. Plot the wealth distributions for these cases and try to extract eventual power law tails with and without a saving $\lambda$ in each transaction. Comment your results and compare them with figures 5 and 6 of [Goswami and Sen](http://www.sciencedirect.com/science/article/pii/S0378437114006967).

Finally, (this part is optional) if you have time, which features would you add to these models in order to make them even more realistic?

## Background literature

  * [V. Pareto, Cours d'economie politique, Lausanne, 1897](http://www.institutcoppet.org/2012/05/08/cours-deconomie-politique-1896-de-vilfredo-pareto).

  * [M. Patriarca, A. Chakraborti, K. Kaski, Physica A **340**, 334 (2004)](http://www.sciencedirect.com/science/article/pii/S0378437104004327).

  * [S. Goswami and P. Sen, Physica A **415**, 514 (2014)](http://www.sciencedirect.com/science/article/pii/S0378437114006967).

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use your github repository to upload your report. Indicate where the report is by creating for example a **Report** folder. Please send us as soon as possible your github username.

  * Place your programs in a folder called for example **Programs** or **src**, in order to indicate where your programs are. You can use a README file to tell us how your github folders are organized.

  * In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, with score and detailed feedback will be emailed to you.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.


# Inferential Statistics Ib - Frequentism

## Learning objectives

Welcome to the second Frequentist inference mini-project! Over the course of working on this mini-project and the previous frequentist mini-project, you'll learn the fundamental concepts associated with frequentist inference. The following list includes the topics you will become familiar with as you work through these two mini-projects:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate confidence intervals and p-values
* how those confidence intervals and p-values allow you to perform hypothesis (or A/B) tests

## Prerequisites

* what a random variable is
* what a probability density function (pdf) is
* what the cumulative density function is
* a high-level sense of what the Normal distribution

If these concepts are new to you, please take a few moments to Google these topics in order to get a sense of what they are and how you might use them.

These two notebooks were designed to bridge the gap between having a basic understanding of probability and random variables and being able to apply these concepts in Python. This second frequentist inference mini-project focuses on a real-world application of this type of inference to give you further practice using these concepts.

In the previous notebook, we used only data from a known normal distribution. You'll now tackle real data, rather than simulated data, and answer some relevant real-world business problems using the data.

## Hospital medical charges

Imagine that a hospital has hired you as their data analyst. An administrator is working on the hospital's business operations plan and needs you to help them answer some business questions. This mini-project, as well as the bootstrap and Bayesian inference mini-projects also found in this unit are designed to illustrate how each of the inferential statistics methods have their uses for different use cases. In this assignment notebook, you're going to use frequentist statistical inference on a data sample to answer the questions:
* has the hospital's revenue stream fallen below a key threshold?
* are patients with insurance really charged different amounts than those without?
Answering that last question with a frequentist approach makes some assumptions, or requires some knowledge, about the two groups. In the next mini-project, you'll use bootstrapping to test that assumption. And in the final mini-project of the unit, you're going to create a model for simulating _individual_ charges (not a sampling distribution) that the hospital can use to model a range of scenarios.

We are going to use some data on medical charges obtained from [Kaggle](https://www.kaggle.com/easonlai/sample-insurance-claim-prediction-dataset). For the purposes of this exercise, assume the observations are the result of random sampling from our one hospital. Recall in the previous assignment, we introduced the Central Limit Theorem (CLT), and how it tells us that the distributions of sample statistics approach a normal distribution as $n$ increases. The amazing thing about this is that it applies to the sampling distributions of statistics that have been calculated from even highly non-normal distributions of data. Remember, also, that hypothesis testing is very much based on making inferences about such sample statistics. You're going to rely heavily on the CLT to apply frequentist (parametric) tests to answer the questions in this notebook.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import t
from numpy.random import seed
import seaborn as sns
medical = pd.read_csv('data/insurance2.csv')
```

```python
medical.shape
```

    (1338, 8)

```python
medical.head()
```

      age
      sex
      bmi
      children
      smoker
      region
      charges
      insuranceclaim

      0
      19
      0
      27.900
      0
      1
      3
      16884.92400
      1

      1
      18
      1
      33.770
      1
      0
      2
      1725.55230
      1

      2
      28
      1
      33.000
      3
      0
      2
      4449.46200
      0

      3
      33
      1
      22.705
      0
      0
      1
      21984.47061
      0

      4
      32
      1
      28.880
      0
      0
      1
      3866.85520
      1

__Q:__ Plot the histogram of charges and calculate the mean and standard deviation. Comment on the appropriateness of these statistics for the data.

__A:__

```python
# Let's look at the max value so we can use a number around it for our
# xticks
max(medical.charges)
```

    63770.42801

```python
sns.set()
plt.figure(figsize=[12,8])
plt.hist(medical.charges, bins = 50, color='C4')
plt.xlabel('Charges')
plt.ylabel('Count')
plt.title('Medical Charges and Counts')
plt.xticks(range(0,65000,2500), rotation = 'vertical')
plt.show()
```

```python
print('The mean of the medical charges is {} '.format(round(np.mean(medical.charges),2)))
print('The standard deviation of the medical charges is {} '.format(round(np.std(medical.charges),2)))
```

    The mean of the medical charges is 13270.42
    The standard deviation of the medical charges is 12105.48

 The combination of mean and standard deviation is appropriate. The median wouldn't be suitable as it looks like more people pay under 15000 for medical charges. Also, the distribution is not normal and skewed right.

__Q:__ The administrator is concerned that the actual average charge has fallen below 12000, threatening the hospital's operational model. On the assumption that these data represent a random sample of charges, how would you justify that these data allow you to answer that question? And what would be the most appropriate frequentist test, of the ones discussed so far, to apply?

__A:__ The data is not normally distributed and it looks like it is skewed to the right. We have a large sample size as well which means that the CLT should be ok to use. The mean is 13270.42 which isn't much higher than their concern average amount. Since we don't have the population mean in this example, I think the most appropriate frequentist test would be to use the CLT to resample the means and work toward a better distribution

__Q:__ Given the nature of the administrator's concern, what is the appropriate confidence interval in this case? A one-sided or two-sided interval? Calculate the critical value and the relevant 95% confidence interval for the mean and comment on whether the administrator should be concerned?

__A:__  I think this should be a one sided interval since we are concerned about the actual average charge fallen below 12000 and not concerned if it is over 12000. The mean is 13270.42 for the sample collected and should not be concerned at first glance.

```python
# Critical Value with sample size of 1338
n = len(medical.charges)
dof = n - 1
p = 0.95
critical_t = t.ppf(p, dof)
print('The critical t value for one tailed 95% confidence interval is: {} '.format(critical_t))
```

    The critical t value for one tailed 95% confidence interval is: 1.6459941145571317

```python
# Standard Error of the Mean (a.k.a. the standard deviation of the sampling distribution of the sample mean!
se = (np.std(medical.charges)) / (np.sqrt(n))
moe = critical_t * se # Margin of Error
print('The Margin of error is: {} '.format(moe))
```

    The Margin of error is: 544.7314053390934

```python
lower = (np.mean(medical.charges)) - moe
lower
#print('The lower is {}'.format(lower) + "\n""The administrator shouldn't be concerened as we are 95 percent confident it lies above {}".format(lower))
```

    12725.690859802164

The administrator shouldn't be concerened as we are 95% confident that the actual average charge is above 12725.690859802164

The administrator then wants to know whether people with insurance really are charged a different amount to those without.

__Q:__ State the null and alternative hypothesis here. Use the _t_-test for the difference between means where the pooled standard deviation of the two groups is given by
\begin{equation}
s_p = \sqrt{\frac{(n_0 - 1)s^2_0 + (n_1 - 1)s^2_1}{n_0 + n_1 - 2}}
\end{equation}

and the *t* test statistic is then given by

\begin{equation}
t = \frac{\bar{x}_0 - \bar{x}_1}{s_p \sqrt{1/n_0 + 1/n_1}}.
\end{equation}

What assumption about the variances of the two groups are we making here?

__A:__
1. The H0 would be that people with insurance are charged the same.
2. The Ha would be that they are NOT charged the same.

The null hypothesis (H0) is that there is no difference between people with insurance and people without insurance, that they are charged the same.

The alternative hypothesis is that the people with insurance are charged differently than those without insurance.

My assumptions are:

When we plot the data, we would see a normal distribution (bell-shaped distribution curve).
Equal variance exists when the standard deviations of samples are equal or approximately equal.

__Q:__ Perform this hypothesis test both manually, using the above formulae, and then using the appropriate function from [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests) (hint, you're looking for a function to perform a _t_-test on two independent samples). For the manual approach, calculate the value of the test statistic and then its probability (the p-value). Verify you get the same results from both.

__A:__

```python
#print(medical.columns)
# This is a binary column so 1 means it has insurance claim 0 means it has no claims
#print(medical.insuranceclaim.head())
```

```python
insured_charges = medical.loc[medical['insuranceclaim'] == 1, 'charges']
uninsured_charges = medical.loc[medical['insuranceclaim'] == 0, 'charges']
n1 = sum(medical['insuranceclaim'] == 1)
x1 = (np.mean(insured_charges))
s1 = (np.std(insured_charges,ddof = 1))

n0 = sum(medical['insuranceclaim'] == 0)
x0 = (np.mean(uninsured_charges))
s0 = (np.std(uninsured_charges, ddof =1))

print('The Uninsured: \n','count', n0, 'mean', x0, 'standard deviation', s0)
print('The Insured: \n', 'count', n1, 'mean', x1, 'standard deviation', s1)
```

    The Uninsured:
     count 555 mean 8821.421892306294 standard deviation 6446.510126811736
    The Insured:
     count 783 mean 16423.928276537663 standard deviation 14045.928418802127

```python
# breaking down the formula

a = (n0 - 1) * s0 **2 # first part of the formula
b = (n1 - 1) * s1 **2 # second part of the formula

dof = (n0 + n1 - 2)

sp = np.sqrt((a + b)/dof) # pooled standard deviation

print('sp:', sp)

t_test = ((x0 - x1)/(sp*(np.sqrt(1/n0 + 1/n1))))

print('t test:', t_test)
```

    sp: 11520.034268775256
    t test: -11.89329903087671

```python
# For a two-sided test, if the value of the test statistic from your sample is negative, then the p-value is equal to
# two times the p-value for the lower-tailed p-value (i.e. 2 * cdf(ts))
p_value = 2 * t.cdf(t_test, dof)
print(f'The p value is: ', p_value)
```

    The p value is:  4.461230231620972e-31

The p value is less than 0.05 so we reject the null hypothesis (H0). This means that people with insurance are charged differently!

```python
# Using the appropriate function from scipy.stats to Calculate the T-test for the means of two independent samples.
from scipy.stats import ttest_ind

# (stats.ttest_ind) This is a two-sided test for the null hypothesis that 2 independent samples have identical average
# This test assumes that the populations have identical variances by default.
print(ttest_ind(uninsured,insured))

```

    Ttest_indResult(statistic=-11.893299030876712, pvalue=4.461230231620717e-31)

Congratulations! Hopefully you got the exact same numerical results. This shows that you correctly calculated the numbers by hand. Secondly, you used the correct function and saw that it's much easier to use. All you need to do pass your data to it.

__Q:__ In the above calculations, we assumed the sample variances were equal. We may well suspect they are not (we'll explore this in another assignment). The calculation becomes a little more complicated to do by hand in this case, but we now know of a helpful function. Check the documentation for the function to tell it not to assume equal variances and perform the test again.

__A:__

```python
# Let's not assume equal variances and perform this test again
print(ttest_ind(uninsured,insured, equal_var = False))
```

    Ttest_indResult(statistic=-13.298031957975649, pvalue=1.1105103216309125e-37)

__Q:__ Conceptual question: look through the documentation for statistical test functions in scipy.stats. You'll see the above _t_-test for a sample, but can you see an equivalent one for performing a *z*-test from a sample? Comment on your answer.

```python
#import scipy
#help(scipy.stats.zscore)

```

__A:__ The scipy.stats has a zscore method that we can use.

zscore(a, axis=0, ddof=0)
    Calculate the z score of each value in the sample, relative to the
    sample mean and standard deviation.

## Learning outcomes

Having completed this project notebook, you now have good hands-on experience:
* using the central limit theorem to help you apply frequentist techniques to answer questions that pertain to very non-normally distributed data from the real world
* performing inference using such data to answer business questions
* forming a hypothesis and framing the null and alternative hypotheses
* testing this using a _t_-test


```python
from sympy import *
```

```python
x, u, y, v, Ax, Bu, Ay, Bv, Q, R, xf, dt, c1, c2 = symbols('x u y v Ax Bu Ay Bv Q R xf dt c1 c2')
```

```python
xm = x/2 + y/2 + dt/8*(Ax*x + Bu*u + c1) - dt/8*(Ay*y + Bv*v + c2)
um = (u+v)/2
l = (xm - xf)*Q*(xm - xf)/2 + um*R*um/2
```

Scalar results for stage cost l(xm,um) expansion

```python
diff(l,x)
```

    Q*(Ax*dt/4 + 1)*(dt*(Ax*x + Bu*u + c1)/8 - dt*(Ay*y + Bv*v + c2)/8 + x/2 - xf + y/2)/2

```python
diff(l,u)
```

    Bu*Q*dt*(dt*(Ax*x + Bu*u + c1)/8 - dt*(Ay*y + Bv*v + c2)/8 + x/2 - xf + y/2)/8 + R*(u/2 + v/2)/2

```python
diff(l,y)
```

    Q*(-Ay*dt/4 + 1)*(dt*(Ax*x + Bu*u + c1)/8 - dt*(Ay*y + Bv*v + c2)/8 + x/2 - xf + y/2)/2

```python
diff(l,v)
```

    -Bv*Q*dt*(dt*(Ax*x + Bu*u + c1)/8 - dt*(Ay*y + Bv*v + c2)/8 + x/2 - xf + y/2)/8 + R*(u/2 + v/2)/2

```python
diff(l,x,x)
```

    Q*(Ax*dt/4 + 1)*(Ax*dt + 4)/16

```python
diff(l,u,u)
```

    (Bu**2*Q*dt**2 + 16*R)/64

```python
diff(l,y,y)
```

    Q*(Ay*dt/4 - 1)*(Ay*dt - 4)/16

```python
diff(l,v,v)
```

    (Bv**2*Q*dt**2 + 16*R)/64

```python
diff(l,x,u)
```

    Bu*Q*dt*(Ax*dt/4 + 1)/16

```python
diff(l,x,y)
```

    -Q*(Ax*dt/4 + 1)*(Ay*dt - 4)/16

```python
diff(l,x,v)
```

    -Bv*Q*dt*(Ax*dt/4 + 1)/16

```python
diff(l,u,y)
```

    -Bu*Q*dt*(Ay*dt - 4)/64

```python
diff(l,u,v)
```

    (-Bu*Bv*Q*dt**2 + 16*R)/64

```python
diff(l,y,v)
```

    Bv*Q*dt*(Ay*dt/4 - 1)/16

```python

```

dev_notebooks/First Order Hold (new derivation).ipynb


# Computing the expectation value of $\langle x^4 \rangle$ for the 1-dimensional particle-in-a-box

### Here is an example of calculating $\langle x \rangle$  and $\langle x^2 \rangle$ for the 1-dimensional particle-in-a-box.

The average position is expected to be

$$
\langle x \rangle =\tfrac{a}{2}
$$

We can confirm this by explicit integration,

$$
\begin{align}
\langle x \rangle &= \int_0^a \psi_n^*(x)\, x \,\psi_n(x) dx \\
&= \int_0^a \left(\sqrt{\tfrac{2}{a}} \sin\left(\tfrac{n \pi x}{a} \right)\right) x \left(\sqrt{\tfrac{2}{a}}\sin\left(\tfrac{n \pi x}{a} \right)\right) dx \\
&= \tfrac{2}{a} \int_0^a  x \sin^2\left(\tfrac{n \pi x}{a} \right) dx \\
&= \tfrac{2}{a} \left[ \tfrac{x^2}{4} - x \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{4 n \pi}{a}}
- \tfrac{\cos \tfrac{2n \pi x}{a}}{\tfrac{8 n^2 \pi^2}{a^2}}
 \right]_0^a \\
 &= \tfrac{2}{a} \left[ \tfrac{a^2}{4} - 0 - 0 \right] \\
 &= \tfrac{a}{2}
\end{align}
$$

Similarly, we expect that the expectation value of $\langle x^2 \rangle$ will be proportional to $a^2$. We can confirm this by explicit integration,

$$
\begin{align}
\langle x^2 \rangle &= \int_0^a \psi_n^*(x)\, x^2 \,\psi_n(x) dx \\
&= \int_0^a \left(\sqrt{\tfrac{2}{a}} \sin\left(\tfrac{n \pi x}{a} \right)\right) x^2 \left(\sqrt{\tfrac{2}{a}}\sin\left(\tfrac{n \pi x}{a} \right)\right) dx \\
&= \tfrac{2}{a} \int_0^a  x^2 \sin^2\left(\tfrac{n \pi x}{a} \right) dx \\
&= \tfrac{2}{a} \left[ \tfrac{x^3}{6}
 - x^2 \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{4 n \pi}{a}}
 - x \tfrac{\cos \tfrac{2n \pi x}{a}}{\tfrac{4 n^2 \pi^2}{a^2}}
 - \tfrac{\sin \tfrac{2n \pi x}{a}}{\tfrac{8 n^3 \pi^3}{a^3}}
 \right]_0^a \\
 &= \tfrac{2}{a} \left[ \tfrac{a^3}{6} - 0 - \tfrac{a}{{\tfrac{4 n^2 \pi^2}{a^2}}} - 0 \right] \\
  &= \tfrac{2}{a} \left[ \tfrac{a^3}{6} - \tfrac{a^3}{4 n^2 \pi^2} \right] \\
  &= a^2\left[ \tfrac{1}{3} - \tfrac{1}{2 n^2 \pi^2} \right]
\end{align}
$$

### Task:
Use the above example to calculate $\langle x^4 \rangle$. You can follow next steps:
1. Complete the `compute_wavefunction` function by coding the expression for calculating the wavefunction. Don't forget the normalization constant. Also, take into account that $x$ values could be negative and greater than $a$, and that $x$ values could be a simple real number (a `float`) or a list of numbers (a `numpy.ndarray`).
2. Using the `compute_wavefunction` function, complete the `compute_probability` function.
3. Fill the gaps in `compute_moments` function; this is especially easy if you use the `compute_probability` function.
4. Complete the function `check_moment`.
5. Now you are ready to calculate the average. Use the [quad](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.quad.html) function for numerical integration and complete the `calc_average` function.
6. Use your functions to calculate  the $\langle x^2 \rangle$ and compare it with the analytic value. We encourage you to play with different values of $n$ and $a$.
7. Submit your notebook as described at the [first tutorial](Tutorial_0.ipynb).

```python
import numpy as np
from scipy.integrate import quad

# Define a function for the wavefunction
def compute_wavefunction(x, n, a):
    """Compute 1-dimensional particle-in-a-box wave-function value(s).

    Parameters
    ----------
    x: float or np.ndarray
        Position of the particle.
    n: int
        Quantum number value.
    a: float
        Length of the box.
    """
    # check argument n
    if not (isinstance(n, int) and n > 0):
        raise ValueError("Argument n should be a positive integer.")
    # check argument a
    if a <= 0.0:
        raise ValueError("Argument a should be positive.")
    # check argument x
    if not (isinstance(x, float) or hasattr(x, "__iter__")):
        raise ValueError("Argument x should be a float or an array!")

    # compute wave-function

    ### START YOU CODE HERE
    value = np.sqrt(2/a)*np.sin(np.pi*x*n/a)
    ### END YOUR CODE HERE

    # set wave-function values out of the box equal to zero

    ### START YOU CODE HERE
    if hasattr(x, "__iter__"):
      value[np.logigal_or(x>a, x<0)] = 0
    elif x<0 or x>a:
      value = 0

    ### END YOUR CODE HERE
    return value

# Define a function for the wavefunction squared
def compute_probability(x, n, a):
    """Compute 1-dimensional particle-in-a-box probablity value(s).

    See `compute_wavefunction` parameters.
    """
    ### START YOUR CODE HERE
    probability = compute_wavefunction(x, n, a)**2
    ### END YOUR CODE HERE
    return probability

def compute_moment(x, n, a, power):
    """Compute the x^power moment of the 1-dimensional particle-in-a-box.

    See `compute_wavefunction` parameters.
    """
    return compute_probability(x, n, a)*x**power

#Compute <x^power>, the expectation value of x^power
def calc_average(n, a, power):
    """
    Compute the average value by explicit integrating
    """

    ### START YOUR CODE HERE
    avg, error = quad(compute_moment, 0, a, args=(n, a, power))
    ### END YOUR CODE HERE

    return avg

#This next bit of code just prints out the values.
def check_moments(n, a):
    #check the computed values of the moments against the analytic formula

    ### START YOUR CODE HERE
    power = 2
    avg_r2 = calc_average(n, a, power)
    ### END YOUR CODE HERE

    print(f"<r^2> computed = {avg_r2:.5f}")
    print(f"<r^2> analytic = {a**2*(1/3 - 1./(2*n**2*np.pi**2))}")

#Principle quantum number:
n = 1

#Box length:
a = 1

check_moments(a, n)
```

    <r^2> computed = 0.28267
    <r^2> analytic = 0.2826727415121644


```python
%load_ext autoreload
%autoreload 2  # automatically reload modules when changed
import simba
from sympy import symbols, simplify, init_printing, Matrix, MatrixSymbol, Rational, adjoint, Symbol
init_printing()
```

The unstable filter transfer function is given by,

$$
G(s) = \frac{s - 2}{s + 2}
$$

```python
s = symbols('s')
tf = (s - 2) / (s + 2)
tf
```

```python
ss = simba.transfer_function_to_state_space(tf).extended_to_quantum()
a, b, c, d = ss
ss
```

$$\displaystyle A=\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right],\,B=\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right],\,C=\left[\begin{matrix}4 & 0\\0 & 4\end{matrix}\right],\,D=\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

Convert to a physically realisable state space

```python
ss = ss.to_physically_realisable()
assert ss.is_physically_realisable
ss
```

$$\displaystyle A=\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right],\,B=\left[\begin{matrix}0 & 2\\2 & 0\end{matrix}\right],\,C=\left[\begin{matrix}0 & 2\\2 & 0\end{matrix}\right],\,D=\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

```python
slh = ss.to_slh('a')
slh
```

$$\displaystyle \left(I_{2\times2}, \left[\begin{matrix}0 & 2\\2 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\end{matrix}\right], \frac{1}{2} \left[\begin{matrix}\overline{a_{1}} & a_{1}\end{matrix}\right] \left[\begin{matrix}0 & 0\\0 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\end{matrix}\right]\right)$$


## Define no - belly regime for N' and N'' (increasing with depth)

$N(z)=N_0 + N'(z-z_0) + N''(z-z_0)^2$, where $N_0$ is the concentration at $z_0$. I will start with $z_0=H_s$, so $N_0$ is a reference concentration at shelf break depth.

 Non-dimensional profile:

 $\hat{N(z)}=\frac{N(z)}{N_0} = 1 + \hat{N'}(z-Hs) + \hat{N''}(z-H_s)^2$,

 where $\hat{}$ variables are divided by $N_0$.

 I can't use a non-dim profile for my model so I need o define dimensional regimes where N' and N'' give reasonable concentrations. For the non-dimesional case I found $0.0<\hat{N'}<0.006$ and $0.0001<\hat{N''}<0.00001$.

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sympy as sym
import seaborn as sns
sym.init_printing() # enable fancy printing
```

    /Users/Karina/anaconda3/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      warnings.warn(self.msg_depr % (key, alt_key))

```python
# Set appearance options seaborn
sns.set_style('white')
sns.set_context('notebook')
```

```python
# Constants and scales from canyon bathy

#L = 6400.0 # canyon length
#R = 5000.0 # Upstream radius of curvature
#g = 9.81 # accel. gravity
#Wsb = 13000 # Width at shelf break
#Hs = 147.5 # Shelf break depth
#Hh = 97.5 #
#Hr = 130.0 # rim depth at dn station
# NOTE: The default values of all functions correspond to the base case

```

```python
C1,C2,z,Co,Hs,tau,Z = sym.symbols('C1,C2,z,Co,Hs,tau, Z')
func = Co + C1*(z-Hs) + C2*((z-Hs)**2)

```

```python
func
```

A reasonable number for $\hat{C'}$ is 0.006. This comes from the slope of the linear nitrate profile I use $d\hat{C}/dz \approx ((32\mu M-3 \mu M)/(150 m))/32 \mu M$. Using a higher value gives negative concetrations at surface.

```python
func = 7.4 + C1*(z-Hs) + C2*((z-Hs)**2)
func = func.subs({Hs:147.5,C2:0})
hand =sym.plot(func.subs(C1,0.0*7.4),
               func.subs(C1,0.003*7.4),
               func.subs(C1,0.006*7.4),
               func.subs(C1,0.0067*7.4),(z, 0, 400),
               xlabel='Depth (m)',
               ylabel='Concentration',
               title='%s' %func,
               show=False)
hand[1].line_color='r'
hand[2].line_color='g'
hand[3].line_color='purple'

hand.show()
```

### What should be the value of $\hat{C''}$ to get stable profiles given an $\hat{C'}$?

i.e. Profiles without 'bellies' (max, mins or inflection points). This might not be necessary but I want to classify them.

There are bellies wherever

$d\hat{C}/dz=\hat{C'}+2\hat{C''}(z-Hs)=0$. (1)

I don't want these points (I'll call them $z_{belly}$) to be within my profile ($0\le z \le 400$), so I have the conditions:

$z_{belly} \le 0 $ ,  $z_{belly} \ge 400$ (2),

but solving (1) for $z_{belly}$ gives,

$z_{belly}=\frac{-\hat{C'}}{2\hat{C''}}+Hs$ (3).

Taking the equalities in conditions (2) and equating to (3) gives:

(4) $\frac{\hat{C'}}{2Hs}\ge {\hat{C''}}$ and (5) $\frac{\hat{C'}}{2(Hs-400)}\le {\hat{C''}}$,

for a given $ 0\le\hat{C'}$ (since I want increasing profiles)

The region that satisfies conditions (4) and (5) is the triangle plotted below. Also, these hold for the dimensional parameters $C'$ and $C''$, where $  0 \le C'$

```python
p2 = sym.plot_implicit(sym.And(C1/(2*(-400+147.5)) < C2,C1/(2*147.5) > C2),
                       (C1,0,0.05),(C2,-0.0003,0.0003),
                      title='C2 values as a function of C1 values')

```

Let's see if it works...

```python
func = 0 + C1*(z-Hs) + C2*((z-Hs)**2)
hand = sym.plot(func.subs({C1:0.00005,Hs:147.5,C2:0.00005/(2*147.5)}),
               func.subs({C1:0.003,Hs:147.5,C2:0.003/(2*147.5)}),
               func.subs({C1:0.005,Hs:147.5,C2:0.005/(2*147.5)}),
               func.subs({C1:0.01,Hs:147.5,C2:0.0067/(2*147.5)}),(z, 0, 400),
               xlabel='Depth (m)',
               ylabel='Concentration (nd)',
               title='Using the upper bound of C2',
               show=False)
hand[1].line_color='r'
hand[2].line_color='g'
hand[3].line_color='purple'

hand.show()
```

```python
func = 0 + C1*(z-Hs) + C2*((z-Hs)**2)
hand = sym.plot(func.subs({C1:0.00005,Hs:147.5,C2:0.00005/(2*(147.5-400))}),
               func.subs({C1:0.003,Hs:147.5,C2:0.003/(2*(147.5-400))}),
               func.subs({C1:0.005,Hs:147.5,C2:0.005/(2*(147.5-400))}),
               func.subs({C1:0.01,Hs:147.5,C2:0.0067/(2*(147.5-400))}),(z, 0, 400),
               xlabel='Depth (m)',
               ylabel='Concentration (nd)',
               title='Using the lower bound of C2',
               show=False)
hand[1].line_color='r'
hand[2].line_color='g'
hand[3].line_color='purple'

hand.show()
```

```python
func = 0 + C1*(z-Hs) + C2*((z-Hs)**2)
hand = sym.plot(func.subs({C1:0.00005,Hs:147.5,C2:0.5e-7}),
               func.subs({C1:0.003,Hs:147.5,C2:0.5e-5}),
               func.subs({C1:0.005,Hs:147.5,C2:-0.7E-5}),
               func.subs({C1:0.01,Hs:147.5,C2:0.00002}),(z, 0, 400),
               xlabel='Depth (m)',
               ylabel='Concentration (nd)',
               title='Using some C2 values within the range of C2(C1)',
               show=False)
hand[1].line_color='r'
hand[2].line_color='g'
hand[3].line_color='purple'

hand.show()

```

## $\tau_v$ as a function of N' and N''

$$\frac{\tau}{Z}=\frac{\delta^2_vN}{\delta_vN}=Z\frac{\delta^2_vN}{\delta_vN}\rvert_{z=Hs}=\frac{2N''}{N'}$$

We have some bounds for N' and N'': we imposed $0\le N'$ and $N'/2Hs\ge N''$ and $N''\ge N'/(2(Hs-400))$ to have a profile with no max, mins or inflection points above 400 m.

Using these bounds, we can now bound the values of \tau_v/Z:

$$\frac{\tau}{Z}=\frac{N''}{N'} \le \frac{2N'}{2HsN'} = \frac{1}{Hs}$$

and

$$\frac{\tau}{Z}=\frac{N''}{N'} \ge \frac{2N'}{2N'(Hs-400)} = \frac{1}{Hs-400}$$

These bounds do not depend on N' or N'', only on Hs.

So,

$$ \frac{1}{Hs-400} \le \frac{\tau}{Z} \le \frac{1}{Hs}$$

Hs=0 and 400 m are asymptotes, but that is fine. If Hs = 0 it means you don't even have a shelf and if Hs=400 m you probbaly care about not having bellies deeper than that, so 400 m wouldn't even be an asymptote in the first place.

```python
p2 = sym.plot_implicit(sym.And(1/Hs > tau, 1/(Hs-400)< tau), (Hs, 0, 400),(tau,-0.1,0.1),
                      title='Tau/Z as a function of Hs',
                      ylabel='Tau/Z')

```

In the idealized bathymetry Hs=147.5 m, so the bounds are: $-0.004  \le \frac{\tau}{Z} \le 0.007$, units are m$^{-1}$. If I say that Z can be between 0 and 100 m then the values of $\tau_v$ can be:

```python
p2 = sym.plot_implicit(sym.And(Z/147.5 > tau, Z/(147.5-400)< tau), (Z, 0, 100),(tau,-0.5,0.7),
                      title='Tau as a function of Z',
                      ylabel='Tau')

```

### Regime definition

Now we have all the elements to define a range of values for $N'$, $N''$ and $\tau_v$:

(1) Linear profiles that increase with depth:

$$0 \le N'$$.

(2) No maximums, minimums or inflection points above 400 m (No bellies) given an $N'$:

$$\frac{\hat{N'}}{2Hs}\ge {\hat{N''}} \,and \,  \frac{\hat{N'}}{2(Hs-400)}\le {\hat{N''}}$$.

Using these bounds, the regime of values of $\tau/Z$ is:

$$ \frac{1}{Hs-400} \le \frac{\tau}{Z} \le \frac{1}{Hs}$$.

For our value of Hs,

$$ -0.004 \le \frac{\tau}{Z} \le 0.007$$.

```python
Hs = 147.5
No = 0

Nprime = np.array([0.1, 0.05,0.02,0.01,0.005,0.001])
z = np.linspace(0,400,50)

labels = ['C1=%1.1e'% Nprime[0],'C1=%1.1e'% Nprime[1],'C1=%1.1e'% Nprime[2],
          'C1=%1.1e'% Nprime[3],'C1=%1.1e'% Nprime[4],'C1=%1.1e'% Nprime[5]]
colors = ['blue','green','purple','red','orange','pink']
fig,ax = plt.subplots(1,1,figsize=(6,8))

for N1,lab,cc in zip(Nprime,labels,colors):
    N2min = N1/(2*(Hs-400))
    N2max = N1/(2*Hs)

    Nmin = No + N1*(z-Hs) + N2min*(z-Hs)**2
    Nmax = No + N1*(z-Hs) + N2max*(z-Hs)**2

    ax.plot(Nmin,z, '--', color=cc)
    ax.plot(Nmax,z,'-', color=cc, label=lab)

    Tau_max = 2*N2max/(N1)
    Tau_min = 2*N2min/(N1)

    print('N1=%1.4f, N2min=%1.3e, Tau/Z=%1.3e \n N1=%1.4f, N2max=%1.3e, Tau/Z=%1.3e \n '
          %(N1,N2min,Tau_min,N1,N2max,Tau_max))

ax.set_xlabel('Concentration ($\mu$M)')
ax.set_ylabel('Depth (m)')
ax.invert_yaxis()
ax.axvline(0, color='k')
ax.legend(loc=0)
```

The linear profile I have used for all runs has:

$C'=(2.2\mu M-45.3\mu M)/(0m-1200m)=0.359 \mu M m^{-1}$

so, ${C'}=0.3 \mu M m^{-1}$

```python
Hs = 147.5
No = 7.49
Nprime = np.array([0.3/No])
z = np.linspace(0,400,50)

labels = ['C1=0.047']
colors = ['blue','green','purple','red','orange','pink']
fig,ax = plt.subplots(1,1,figsize=(6,8))

for N1,lab,cc in zip(Nprime,labels,colors):
    N2min = N1/(2*(Hs-400))
    N2max = N1/(2*Hs)
    N2mean = (N2max-N2min)/2

    Nmean = No + N1*(z-Hs)
    Nmax = No + N1*(z-Hs) + N2max*(z-Hs)**2
    Nmin = No + N1*(z-Hs) + N2min*(z-Hs)**2

    ax.plot(Nmean,z,'-', color=cc, label='lin profile')
    ax.plot(Nmax,z,':', color=cc, label='max curv')
    ax.plot(Nmin,z,'--', color=cc, label='min curv')

    Tau_mean = 2*N2mean/(N1)

    print('N1=%1.4f, N2mean=%1.3e, Tau/Z=%1.3e \n'
          %(N1,N2min,Tau_mean))

ax.set_xlabel('Concentration ($\mu$M)')
ax.set_ylabel('Depth (m)')
ax.invert_yaxis()
ax.axvline(0, color='k')
ax.legend(loc=0)
```

```python

```


# Problem 1: Basics of Neural Networks
* Learning Objective: In the entrance exam, we asked you to implement a K-NN classifier to classify some tiny images extracted from CIFAR-10 dataset. Probably many of you noticed that the performances were quite bad. In this problem, you are going to implement a basic multi-layer fully connected neural network to perform the same classification task.
* Provided Code: We provide the skeletons of classes you need to complete. Forward checking and gradient checkings are provided for verifying your implementation as well.
* TODOs: You are asked to implement the forward passes and backward passes for standard layers and loss functions, various widely-used optimizers, and part of the training procedure. And finally we want you to train a network from scratch on your own.

```python
from lib.fully_conn import *
from lib.layer_utils import *
from lib.grad_check import *
from lib.datasets import *
from lib.optim import *
from lib.train import *
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2
```

## Loading the data (CIFAR-10)
Run the following code block to load in the properly splitted CIFAR-10 data.

```python
data = CIFAR10_data()
for k, v in data.iteritems():
    print "Name: {} Shape: {}".format(k, v.shape)
```

    Name: data_train Shape: (49000, 3, 32, 32)
    Name: data_val Shape: (1000, 3, 32, 32)
    Name: data_test Shape: (1000, 3, 32, 32)
    Name: labels_train Shape: (49000,)
    Name: labels_val Shape: (1000,)
    Name: labels_test Shape: (1000,)

## Implement Standard Layers
You will now implement all the following standard layers commonly seen in a fully connected neural network. Please refer to the file layer_utils.py under the directory lib. Take a look at each class skeleton, and we will walk you through the network layer by layer. We provide results of some examples we pre-computed for you for checking the forward pass, and also the gradient checking for the backward pass.

## FC Forward
In the class skeleton "fc", please complete the forward pass in function "forward", the input to the fc layer may not be of dimension (batch size, features size), it could be an image or any higher dimensional data. Make sure that you handle this dimensionality issue.

```python
# Test the fc forward function
input_bz = 3
input_dim = (6, 5, 4)
output_dim = 4

input_size = input_bz * np.prod(input_dim)
weight_size = output_dim * np.prod(input_dim)

single_fc = fc(np.prod(input_dim), output_dim, init_scale=0.02, name="fc_test")

x = np.linspace(-0.1, 0.5, num=input_size).reshape(input_bz, *input_dim)
w = np.linspace(-0.2, 0.3, num=weight_size).reshape(np.prod(input_dim), output_dim)
b = np.linspace(-0.3, 0.1, num=output_dim)

single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

out = single_fc.forward(x)

correct_out = np.array([[0.70157129, 0.83483484, 0.96809839, 1.10136194],
                        [1.86723094, 2.02561647, 2.18400199, 2.34238752],
                        [3.0328906,  3.2163981,  3.3999056,  3.5834131]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)
```

    Difference:  2.48539291792e-09

## FC Backward
Please complete the function "backward" as the backward pass of the fc layer. Follow the instructions in the comments to store gradients into the predefined dictionaries in the attributes of the class. Parameters of the layer are also stored in the predefined dictionary.

```python
# Test the fc backward function
x = np.random.randn(10, 2, 2, 3)
w = np.random.randn(12, 10)
b = np.random.randn(10)
dout = np.random.randn(10, 10)

single_fc = fc(np.prod(x.shape[1:]), 10, init_scale=5e-2, name="fc_test")
single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

dx_num = eval_numerical_gradient_array(lambda x: single_fc.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: single_fc.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: single_fc.forward(x), b, dout)

out = single_fc.forward(x)
dx = single_fc.backward(dout)
dw = single_fc.grads[single_fc.w_name]
db = single_fc.grads[single_fc.b_name]

# The error should be around 1e-10
print "dx Error: ", rel_error(dx_num, dx)
print "dw Error: ", rel_error(dw_num, dw)
print "db Error: ", rel_error(db_num, db)
```

    dx Error:  8.52174876315e-10
    dw Error:  1.22843425191e-09
    db Error:  8.82086649315e-11

## ReLU Forward
In the class skeleton "relu", please complete the forward pass.

```python
# Test the relu forward function
x = np.linspace(-1.0, 1.0, num=12).reshape(3, 4)
relu_f = relu(name="relu_f")

out = relu_f.forward(x)
correct_out = np.array([[0.,          0.,        0.,         0.        ],
                        [0.,          0.,        0.09090909, 0.27272727],
                        [0.45454545, 0.63636364, 0.81818182, 1.        ]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)
```

    Difference:  5.00000005012e-09

## ReLU Backward
Please complete the backward pass of the class relu.

```python
# Test the relu backward function
x = np.random.randn(10, 10)
dout = np.random.randn(*x.shape)
relu_b = relu(name="relu_b")

dx_num = eval_numerical_gradient_array(lambda x: relu_b.forward(x), x, dout)

out = relu_b.forward(x)
dx = relu_b.backward(dout)

# The error should not be larger than 1e-10
print "dx Error: ", rel_error(dx_num, dx)
```

    dx Error:  3.27561972634e-12

## Dropout Forward
In the class "dropout", please complete the forward pass. Remember that the dropout is only applied during training phase, you should pay attention to this while implementing the function.

```python
x = np.random.randn(100, 100) + 5.0

print "----------------------------------------------------------------"
for p in [0.25, 0.50, 0.75]:
    dropout_f = dropout(p)
    out = dropout_f.forward(x, True)
    out_test = dropout_f.forward(x, False)

    print "Dropout p = ", p
    print "Mean of input: ", x.mean()
    print "Mean of output during training time: ", out.mean()
    print "Mean of output during testing time: ", out_test.mean()
    print "Fraction of output set to zero during training time: ", (out == 0).mean()
    print "Fraction of output set to zero during testing time: ", (out_test == 0).mean()
    print "----------------------------------------------------------------"
```

    ----------------------------------------------------------------
    Dropout p =  0.25
    Mean of input:  5.01578563647
    Mean of output during training time:  4.99372767041
    Mean of output during testing time:  5.01578563647
    Fraction of output set to zero during training time:  0.7504
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------
    Dropout p =  0.5
    Mean of input:  5.01578563647
    Mean of output during training time:  5.03282467518
    Mean of output during testing time:  5.01578563647
    Fraction of output set to zero during training time:  0.5004
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------
    Dropout p =  0.75
    Mean of input:  5.01578563647
    Mean of output during training time:  4.96014951445
    Mean of output during testing time:  5.01578563647
    Fraction of output set to zero during training time:  0.2584
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------

## Dropout Backward
Please complete the backward pass. Again remember that the dropout is only applied during training phase, handle this in the backward pass as well.

```python
x = np.random.randn(5, 5) + 5
dout = np.random.randn(*x.shape)

p = 0.75
dropout_b = dropout(p, seed=100)
out = dropout_b.forward(x, True)
dx = dropout_b.backward(dout)
dx_num = eval_numerical_gradient_array(lambda xx: dropout_b.forward(xx, True), x, dout)

# The error should not be larger than 1e-9
print 'dx relative error: ', rel_error(dx, dx_num)
```

    dx relative error:  3.00311469865e-11

## Testing cascaded layers: FC + ReLU
Please find the TestFCReLU function in fully_conn.py under lib directory.

You only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU two-layer-mini-network where the parameters of them match the given x, w, and b

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup

```python
x = np.random.randn(2, 3, 4)  # the input features
w = np.random.randn(12, 10)   # the weight of fc layer
b = np.random.randn(10)       # the bias of fc layer
dout = np.random.randn(2, 10) # the gradients to the output, notice the shape

tiny_net = TestFCReLU()

tiny_net.net.assign("fc_w", w)
tiny_net.net.assign("fc_b", b)

out = tiny_net.forward(x)
dx = tiny_net.backward(dout)

dw = tiny_net.net.get_grads("fc_w")
db = tiny_net.net.get_grads("fc_b")

dx_num = eval_numerical_gradient_array(lambda x: tiny_net.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: tiny_net.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: tiny_net.forward(x), b, dout)

# The errors should not be larger than 1e-7
print "dx error: ", rel_error(dx_num, dx)
print "dw error: ", rel_error(dw_num, dw)
print "db error: ", rel_error(db_num, db)
```

    dx error:  1.13598437673e-10
    dw error:  1.14872404377e-10
    db error:  3.27562531714e-12

## SoftMax Function and Loss Layer
In the layer_utils.py, please first complete the function softmax, which will be use in the function cross_entropy. Please refer to the lecture slides of the mathematical expressions of the cross entropy loss function, and complete its forward pass and backward pass.

```python
num_classes, num_inputs = 5, 50
x = 0.001 * np.random.randn(num_inputs, num_classes)
y = np.random.randint(num_classes, size=num_inputs)

test_loss = cross_entropy()

dx_num = eval_numerical_gradient(lambda x: test_loss.forward(x, y), x, verbose=False)

loss = test_loss.forward(x, y)
dx = test_loss.backward()

# Test softmax_loss function. Loss should be around 1.609
# and dx error should be at the scale of 1e-8 (or smaller)
print "Cross Entropy Loss: ", loss
print "dx error: ", rel_error(dx_num, dx)
```

    Cross Entropy Loss:  1.60945846468
    dx error:  2.77559987163e-09

## Test a Small Fully Connected Network
Please find the SmallFullyConnectedNetwork function in fully_conn.py under lib directory.

Again you only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU --> FC --> ReLU network where the shapes of parameters match the given shapes

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup

```python
model = SmallFullyConnectedNetwork()
loss_func = cross_entropy()

N, D, = 4, 4  # N: batch size, D: input dimension
H, C  = 30, 7 # H: hidden dimension, C: output dimension
std = 0.02
x = np.random.randn(N, D)
y = np.random.randint(C, size=N)

print "Testing initialization ... "
w1_std = abs(model.net.get_params("fc1_w").std() - std)
b1 = model.net.get_params("fc1_b").std()
w2_std = abs(model.net.get_params("fc2_w").std() - std)
b2 = model.net.get_params("fc2_b").std()

assert w1_std < std / 10, "First layer weights do not seem right"
assert np.all(b1 == 0), "First layer biases do not seem right"
assert w2_std < std / 10, "Second layer weights do not seem right"
assert np.all(b2 == 0), "Second layer biases do not seem right"
print "Passed!"

print "Testing test-time forward pass ... "
w1 = np.linspace(-0.7, 0.3, num=D*H).reshape(D, H)
w2 = np.linspace(-0.3, 0.4, num=H*C).reshape(H, C)
b1 = np.linspace(-0.1, 0.9, num=H)
b2 = np.linspace(-0.9, 0.1, num=C)

model.net.assign("fc1_w", w1)
model.net.assign("fc1_b", b1)
model.net.assign("fc2_w", w2)
model.net.assign("fc2_b", b2)

feats = np.linspace(-5.5, 4.5, num=N*D).reshape(D, N).T
scores = model.forward(feats)
correct_scores = np.asarray([[4.20670862, 4.87188359, 5.53705856, 6.20223352, 6.86740849, 7.53258346, 8.19775843],
                             [4.74826036, 5.35984681, 5.97143326, 6.58301972, 7.19460617, 7.80619262, 8.41777907],
                             [5.2898121,  5.84781003, 6.40580797, 6.96380591, 7.52180384, 8.07980178, 8.63779971],
                             [5.83136384, 6.33577326, 6.84018268, 7.3445921,  7.84900151, 8.35341093, 8.85782035]])
scores_diff = np.sum(np.abs(scores - correct_scores))
assert scores_diff < 1e-6, "Your implementation might went wrong!"
print "Passed!"

print "Testing the loss ...",
y = np.asarray([0, 5, 1, 4])
loss = loss_func.forward(scores, y)
dLoss = loss_func.backward()
correct_loss = 2.90181552716
assert abs(loss - correct_loss) < 1e-10, "Your implementation might went wrong!"
print "Passed!"

print "Testing the gradients (error should be no larger than 1e-7) ..."
din = model.backward(dLoss)
for layer in model.net.layers:
    if not layer.params:
        continue
    for name in sorted(layer.grads):
        f = lambda _: loss_func.forward(model.forward(feats), y)
        grad_num = eval_numerical_gradient(f, layer.params[name], verbose=False)
        print '%s relative error: %.2e' % (name, rel_error(grad_num, layer.grads[name]))
```

    Testing initialization ...
    Passed!
    Testing test-time forward pass ...
    Passed!
    Testing the loss ... Passed!
    Testing the gradients (error should be no larger than 1e-7) ...
    fc1_b relative error: 2.85e-09
    fc1_w relative error: 7.76e-09
    fc2_b relative error: 6.71e-08
    fc2_w relative error: 3.03e-09

## Test a Fully Connected Network regularized with Dropout
Please find the DropoutNet function in fully_conn.py under lib directory.

For this part you don't need to design a new network, just simply run the following test code

If something goes wrong, you might want to double check your dropout implementation

```python
N, D, C = 3, 15, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))
seed = 123

for dropout_p in [0., 0.25, 0.5]:
    print "Dropout p =", dropout_p
    model = DropoutNet(dropout_p=dropout_p, seed=seed)
    loss_func = cross_entropy()
    output = model.forward(X, True)
    loss = loss_func.forward(output, y)
    dLoss = loss_func.backward()
    dX = model.backward(dLoss)
    grads = model.net.grads
    print "Loss (should be ~2.30) : ", loss

    print "Error of gradients should be no larger than 1e-5"
    for name in sorted(model.net.params):
        f = lambda _: loss_func.forward(model.forward(X, True), y)
        grad_num = eval_numerical_gradient(f, model.net.params[name], verbose=False, h=1e-5)
        print "{} relative error: {}".format(name, rel_error(grad_num, grads[name]))
    print
```

    Dropout p = 0.0
    Loss (should be ~2.30) :  2.30285163514
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 3.86706745047e-08
    fc1_w relative error: 1.68428083818e-06
    fc2_b relative error: 3.81209866093e-09
    fc2_w relative error: 2.26785962239e-06
    fc3_b relative error: 1.52481732925e-10
    fc3_w relative error: 1.16962221954e-07

    Dropout p = 0.25
    Loss (should be ~2.30) :  2.30423200279
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 8.88557372487e-08
    fc1_w relative error: 9.65553469517e-06
    fc2_b relative error: 1.99592245205e-07
    fc2_w relative error: 1.46154121819e-06
    fc3_b relative error: 1.11483204553e-10
    fc3_w relative error: 1.87136172477e-08

    Dropout p = 0.5
    Loss (should be ~2.30) :  2.29994690876
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 1.00992963045e-07
    fc1_w relative error: 5.24658134615e-06
    fc2_b relative error: 1.67052460093e-08
    fc2_w relative error: 1.41351178592e-05
    fc3_b relative error: 6.01233850074e-11
    fc3_w relative error: 2.63122521848e-07

## Training a Network
In this section, we defined a TinyNet class for you to fill in the TODO block in fully_conn.py.
* Here please design a two layer fully connected network for this part.
* Please read the train.py under lib directory carefully and complete the TODO blocks in the train_net function first.
* In addition, read how the SGD function is implemented in optim.py, you will be asked to complete three other optimization methods in the later sections.

```python
# Arrange the data
data_dict = {
    "data_train": (data["data_train"], data["labels_train"]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}
```

```python
model = TinyNet()
loss_f = cross_entropy()
optimizer = SGD(model.net, 1e-4)
```

### Now train the network to achieve at least 50% validation accuracy

```python
results = None
#############################################################################
# TODO: Use the train_net function you completed to train a network         #
#############################################################################
#train_net(data,model,loss_func,optimizer,batch_size,max_epochs,lr_decay,lr_decay_every,show_every,verbose)
results = train_net(data_dict, model, loss_f, optimizer, 100, 200, 2.0, 1000, 10, False)
#############################################################################
#                             END OF YOUR CODE                              #
#############################################################################
opt_params, loss_hist, train_acc_hist, val_acc_hist = results
```

```python
# Take a look at what names of params were stored
print opt_params.keys()
```

    ['fc1_w', 'fc2_b', 'fc1_b', 'fc2_w']

```python
# Demo: How to load the parameters to a newly defined network
model = TinyNet()
model.net.load(opt_params)
val_acc = compute_acc(model, data["data_val"], data["labels_val"])
print "Validation Accuracy: {}%".format(val_acc*100)
test_acc = compute_acc(model, data["data_test"], data["labels_test"])
print "Testing Accuracy: {}%".format(test_acc*100)
```

    Loading Params: fc1_w Shape: (3072, 100)
    Loading Params: fc1_b Shape: (100,)
    Loading Params: fc2_b Shape: (10,)
    Loading Params: fc2_w Shape: (100, 10)
    Validation Accuracy: 50.3%
    Testing Accuracy: 47.6%

```python
# Plot the learning curves
plt.subplot(2, 1, 1)
plt.title('Training loss')
loss_hist_ = loss_hist[1::100] # sparse the curve a bit
plt.plot(loss_hist_, '-o')
plt.xlabel('Iteration')

plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(train_acc_hist, '-o', label='Training')
plt.plot(val_acc_hist, '-o', label='Validation')
plt.plot([0.5] * len(val_acc_hist), 'k--')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)
plt.show()
```

## Different Optimizers
There are several more advanced optimizers than vanilla SGD, you will implement three more sophisticated and widely-used methods in this section. Please complete the TODOs in the optim.py under lib directory.

## SGD + Momentum
The update rule of SGD plus momentum is as shown below:

\begin{equation}
v_t: velocity \\
\gamma: momentum \\
\eta: learning\ rate \\
v_t = \gamma v_{t-1} + \eta \nabla_{\theta}J(\theta) \\
\theta = \theta - v_t
\end{equation}
Complete the SGDM() function in optim.py

```python
# SGD with momentum
model = TinyNet()
loss_f = cross_entropy()
optimizer = SGD(model.net, 1e-4)
```

```python
# Test the implementation of SGD with Momentum
N, D = 4, 5
test_sgd = sequential(fc(N, D, name="sgd_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
v = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

test_sgd.layers[0].params = {"sgd_fc_w": w}
test_sgd.layers[0].grads = {"sgd_fc_w": dw}

test_sgd_momentum = SGDM(test_sgd, 1e-3, 0.9)
test_sgd_momentum.velocity = {"sgd_fc_w": v}
test_sgd_momentum.step()

updated_w = test_sgd.layers[0].params["sgd_fc_w"]
velocity = test_sgd_momentum.velocity["sgd_fc_w"]

expected_updated_w = np.asarray([
  [ 0.1406,      0.20738947,  0.27417895,  0.34096842,  0.40775789],
  [ 0.47454737,  0.54133684,  0.60812632,  0.67491579,  0.74170526],
  [ 0.80849474,  0.87528421,  0.94207368,  1.00886316,  1.07565263],
  [ 1.14244211,  1.20923158,  1.27602105,  1.34281053,  1.4096    ]])
expected_velocity = np.asarray([
  [ 0.5406,      0.55475789,  0.56891579, 0.58307368,  0.59723158],
  [ 0.61138947,  0.62554737,  0.63970526,  0.65386316,  0.66802105],
  [ 0.68217895,  0.69633684,  0.71049474,  0.72465263,  0.73881053],
  [ 0.75296842,  0.76712632,  0.78128421,  0.79544211,  0.8096    ]])

print 'updated_w error: ', rel_error(updated_w, expected_updated_w)
print 'velocity error: ', rel_error(expected_velocity, velocity)
```

    updated_w error:  8.88234703351e-09
    velocity error:  4.26928774328e-09

Run the following code block to train a multi-layer fully connected network with both SGD and SGD plus Momentum. The network trained with SGDM optimizer should converge faster.

```python
# Arrange a small data
num_train = 4000
small_data_dict = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

model_sgd      = FullyConnectedNetwork()
model_sgdm     = FullyConnectedNetwork()
loss_f_sgd     = cross_entropy()
loss_f_sgdm    = cross_entropy()
optimizer_sgd  = SGD(model_sgd.net, 1e-2)
optimizer_sgdm = SGDM(model_sgdm.net, 1e-2, 0.9)

print "Training with Vanilla SGD..."
results_sgd = train_net(small_data_dict, model_sgd, loss_f_sgd, optimizer_sgd, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with SGD plus Momentum..."
results_sgdm = train_net(small_data_dict, model_sgdm, loss_f_sgdm, optimizer_sgdm, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_sgd,  loss_hist_sgd,  train_acc_hist_sgd,  val_acc_hist_sgd  = results_sgd
opt_params_sgdm, loss_hist_sgdm, train_acc_hist_sgdm, val_acc_hist_sgdm = results_sgdm

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
```

## RMSProp
The update rule of RMSProp is as shown below:

\begin{equation}
\gamma: decay\ rate \\
\epsilon: small\ number \\
g_t^2: squared\ gradients \\
\eta: learning\ rate \\
E[g^2]_t: decaying\ average\ of\ past\ squared\ gradients\ at\ update\ step\ t \\
E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g_t^2 \\
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t+\epsilon}}
\end{equation}
Complete the RMSProp() function in optim.py

```python
# Test RMSProp implementation; you should see errors less than 1e-7
N, D = 4, 5
test_rms = sequential(fc(N, D, name="rms_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
cache = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

test_rms.layers[0].params = {"rms_fc_w": w}
test_rms.layers[0].grads = {"rms_fc_w": dw}

opt_rms = RMSProp(test_rms, 1e-2, 0.99)
opt_rms.cache = {"rms_fc_w": cache}
opt_rms.step()

updated_w = test_rms.layers[0].params["rms_fc_w"]
cache = opt_rms.cache["rms_fc_w"]

expected_updated_w = np.asarray([
  [-0.39223849, -0.34037513, -0.28849239, -0.23659121, -0.18467247],
  [-0.132737,   -0.08078555, -0.02881884,  0.02316247,  0.07515774],
  [ 0.12716641,  0.17918792,  0.23122175,  0.28326742,  0.33532447],
  [ 0.38739248,  0.43947102,  0.49155973,  0.54365823,  0.59576619]])
expected_cache = np.asarray([
  [ 0.5976,      0.6126277,   0.6277108,   0.64284931,  0.65804321],
  [ 0.67329252,  0.68859723,  0.70395734,  0.71937285,  0.73484377],
  [ 0.75037008,  0.7659518,   0.78158892,  0.79728144,  0.81302936],
  [ 0.82883269,  0.84469141,  0.86060554,  0.87657507,  0.8926    ]])

print 'updated_w error: ', rel_error(expected_updated_w, updated_w)
print 'cache error: ', rel_error(expected_cache, opt_rms.cache["rms_fc_w"])
```

    updated_w error:  9.50264522989e-08
    cache error:  1.06132471212e-08

## Adam
The update rule of Adam is as shown below:

\begin{equation}
g_t: gradients\ at\ update\ step\ t \\
m_t = \beta_1m_{t-1} + (1-\beta_1)g_t \\
v_t = \beta_2v_{t-1} + (1-\beta_1)g_t^2 \\
\hat{m_t}: bias\ corrected\ m_t \\
\hat{v_t}: bias\ corrected\ v_t \\
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v_t}}+\epsilon}
\end{equation}
Complete the Adam() function in optim.py

```python
# Test Adam implementation; you should see errors around 1e-7 or less
N, D = 4, 5
test_adam = sequential(fc(N, D, name="adam_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
m = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
v = np.linspace(0.7, 0.5, num=N*D).reshape(N, D)

test_adam.layers[0].params = {"adam_fc_w": w}
test_adam.layers[0].grads = {"adam_fc_w": dw}

opt_adam = Adam(test_adam, 1e-2, 0.9, 0.999, t=5)
opt_adam.mt = {"adam_fc_w": m}
opt_adam.vt = {"adam_fc_w": v}
opt_adam.step()

updated_w = test_adam.layers[0].params["adam_fc_w"]
mt = opt_adam.mt["adam_fc_w"]
vt = opt_adam.vt["adam_fc_w"]

expected_updated_w = np.asarray([
  [-0.40094747, -0.34836187, -0.29577703, -0.24319299, -0.19060977],
  [-0.1380274,  -0.08544591, -0.03286534,  0.01971428,  0.0722929],
  [ 0.1248705,   0.17744702,  0.23002243,  0.28259667,  0.33516969],
  [ 0.38774145,  0.44031188,  0.49288093,  0.54544852,  0.59801459]])
expected_v = np.asarray([
  [ 0.69966,     0.68908382,  0.67851319,  0.66794809,  0.65738853,],
  [ 0.64683452,  0.63628604,  0.6257431,   0.61520571,  0.60467385,],
  [ 0.59414753,  0.58362676,  0.57311152,  0.56260183,  0.55209767,],
  [ 0.54159906,  0.53110598,  0.52061845,  0.51013645,  0.49966,   ]])
expected_m = np.asarray([
  [ 0.48,        0.49947368,  0.51894737,  0.53842105,  0.55789474],
  [ 0.57736842,  0.59684211,  0.61631579,  0.63578947,  0.65526316],
  [ 0.67473684,  0.69421053,  0.71368421,  0.73315789,  0.75263158],
  [ 0.77210526,  0.79157895,  0.81105263,  0.83052632,  0.85      ]])

print 'updated_w error: ', rel_error(expected_updated_w, updated_w)
print 'mt error: ', rel_error(expected_m, mt)
print 'vt error: ', rel_error(expected_v, vt)
```

    updated_w error:  1.13956917985e-07
    mt error:  4.21496319311e-09
    vt error:  4.20831403811e-09

## Comparing the optimizers
Run the following code block to compare the plotted results among all the above optimizers

```python
model_rms      = FullyConnectedNetwork()
model_adam     = FullyConnectedNetwork()
loss_f_rms     = cross_entropy()
loss_f_adam    = cross_entropy()
optimizer_rms  = RMSProp(model_rms.net, 5e-4)
optimizer_adam = Adam(model_adam.net, 5e-4)

print "Training with RMSProp..."
results_rms = train_net(small_data_dict, model_rms, loss_f_rms, optimizer_rms, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with Adam..."
results_adam = train_net(small_data_dict, model_adam, loss_f_adam, optimizer_adam, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_rms,  loss_hist_rms,  train_acc_hist_rms,  val_acc_hist_rms  = results_rms
opt_params_adam, loss_hist_adam, train_acc_hist_adam, val_acc_hist_adam = results_adam

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_rms, 'o', label="RMSProp")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_rms, '-o', label="RMSProp")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_rms, '-o', label="RMSProp")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_adam, 'o', label="Adam")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_adam, '-o', label="Adam")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_adam, '-o', label="Adam")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Training a Network with Dropout
Run the following code blocks to compare the results with and without dropout

```python
# Train two identical nets, one with dropout and one without
num_train = 500
data_dict_500 = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

solvers = {}
dropout_ps = [0, 0.25]  # you can try some dropout prob yourself

results_dict = {}
for dropout_p in dropout_ps:
    results_dict[dropout_p] = {}

for dropout_p in dropout_ps:
    print "Dropout =", dropout_p
    model = DropoutNetTest(dropout_p=dropout_p)
    loss_f = cross_entropy()
    optimizer = SGDM(model.net, 1e-4)
    results = train_net(data_dict_500, model, loss_f, optimizer, batch_size=100,
                        max_epochs=20, show_every=100, verbose=True)
    opt_params, loss_hist, train_acc_hist, val_acc_hist = results
    results_dict[dropout_p] = {
        "opt_params": opt_params,
        "loss_hist": loss_hist,
        "train_acc_hist": train_acc_hist,
        "val_acc_hist": val_acc_hist
    }
```

    Dropout = 0
    (Iteration 1 / 100) loss: 2.54822405595
    (Epoch 1 / 20) Training Accuracy: 0.092, Validation Accuracy: 0.096
    (Epoch 2 / 20) Training Accuracy: 0.116, Validation Accuracy: 0.101
    (Epoch 3 / 20) Training Accuracy: 0.144, Validation Accuracy: 0.112
    (Epoch 4 / 20) Training Accuracy: 0.174, Validation Accuracy: 0.128
    (Epoch 5 / 20) Training Accuracy: 0.196, Validation Accuracy: 0.135
    (Epoch 6 / 20) Training Accuracy: 0.226, Validation Accuracy: 0.145
    (Epoch 7 / 20) Training Accuracy: 0.23, Validation Accuracy: 0.161
    (Epoch 8 / 20) Training Accuracy: 0.236, Validation Accuracy: 0.163
    (Epoch 9 / 20) Training Accuracy: 0.24, Validation Accuracy: 0.168
    (Epoch 10 / 20) Training Accuracy: 0.25, Validation Accuracy: 0.171
    (Epoch 11 / 20) Training Accuracy: 0.26, Validation Accuracy: 0.18
    (Epoch 12 / 20) Training Accuracy: 0.278, Validation Accuracy: 0.184
    (Epoch 13 / 20) Training Accuracy: 0.286, Validation Accuracy: 0.192
    (Epoch 14 / 20) Training Accuracy: 0.292, Validation Accuracy: 0.196
    (Epoch 15 / 20) Training Accuracy: 0.296, Validation Accuracy: 0.201
    (Epoch 16 / 20) Training Accuracy: 0.308, Validation Accuracy: 0.201
    (Epoch 17 / 20) Training Accuracy: 0.31, Validation Accuracy: 0.203
    (Epoch 18 / 20) Training Accuracy: 0.318, Validation Accuracy: 0.207
    (Epoch 19 / 20) Training Accuracy: 0.328, Validation Accuracy: 0.215
    (Epoch 20 / 20) Training Accuracy: 0.332, Validation Accuracy: 0.219
    Dropout = 0.25
    (Iteration 1 / 100) loss: 3.0082675448
    (Epoch 1 / 20) Training Accuracy: 0.13, Validation Accuracy: 0.096
    (Epoch 2 / 20) Training Accuracy: 0.13, Validation Accuracy: 0.107
    (Epoch 3 / 20) Training Accuracy: 0.158, Validation Accuracy: 0.12
    (Epoch 4 / 20) Training Accuracy: 0.168, Validation Accuracy: 0.126
    (Epoch 5 / 20) Training Accuracy: 0.196, Validation Accuracy: 0.14
    (Epoch 6 / 20) Training Accuracy: 0.208, Validation Accuracy: 0.15
    (Epoch 7 / 20) Training Accuracy: 0.218, Validation Accuracy: 0.153
    (Epoch 8 / 20) Training Accuracy: 0.234, Validation Accuracy: 0.159
    (Epoch 9 / 20) Training Accuracy: 0.24, Validation Accuracy: 0.16
    (Epoch 10 / 20) Training Accuracy: 0.25, Validation Accuracy: 0.167
    (Epoch 11 / 20) Training Accuracy: 0.276, Validation Accuracy: 0.16
    (Epoch 12 / 20) Training Accuracy: 0.274, Validation Accuracy: 0.169
    (Epoch 13 / 20) Training Accuracy: 0.292, Validation Accuracy: 0.176
    (Epoch 14 / 20) Training Accuracy: 0.308, Validation Accuracy: 0.176
    (Epoch 15 / 20) Training Accuracy: 0.3, Validation Accuracy: 0.18
    (Epoch 16 / 20) Training Accuracy: 0.332, Validation Accuracy: 0.181
    (Epoch 17 / 20) Training Accuracy: 0.328, Validation Accuracy: 0.179
    (Epoch 18 / 20) Training Accuracy: 0.326, Validation Accuracy: 0.193
    (Epoch 19 / 20) Training Accuracy: 0.338, Validation Accuracy: 0.183
    (Epoch 20 / 20) Training Accuracy: 0.376, Validation Accuracy: 0.176

```python
# Plot train and validation accuracies of the two models
train_accs = []
val_accs = []
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    train_accs.append(curr_dict["train_acc_hist"][-1])
    val_accs.append(curr_dict["val_acc_hist"][-1])

plt.subplot(3, 1, 1)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["train_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Train accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["val_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Val accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

### Inline Question: Describe what you observe from the above results and graphs
#### Ans: The overall loss is lesser in the case of training without dropout. Also, we achieve higher training accuracy when training the network with dropout of 0.25 as compared to the one with no dropout. But for the reverse is true in the case of validation accuracy.

## Plot the Activation Functions
In each of the activation function, use the given lambda function template to plot their corresponding curves.

```python
left, right = -10, 10
X  = np.linspace(left, right, 100)
XS = np.linspace(-5, 5, 10)
lw = 4
alpha = 0.1
elu_alpha = 0.5
selu_alpha = 1.6732
selu_scale = 1.0507

#########################
####### YOUR CODE #######
#########################
sigmoid = lambda x: 1/(1 + np.exp(-x))
leaky_relu = lambda x: ((0.1*x) * (x<0))+(x*(x>=0))
relu = lambda x: x * (x > 0)
elu = lambda x: ((elu_alpha*(np.exp(x)-1)) * (x<0))+(x*(x>=0))
selu = lambda x: selu_scale*(((selu_alpha*(np.exp(x)-1)) * (x<0))+(x *(x>=0)))
tanh = lambda x: np.tanh(x)
#########################
### END OF YOUR CODE ####
#########################

activations = {
    "Sigmoid": sigmoid,
    "LeakyReLU": leaky_relu,
    "ReLU": relu,
    "ELU": elu,
    "SeLU": selu,
    "Tanh": tanh
}

# Ground Truth activations
GT_Act = {
    "Sigmoid": [0.00669285092428, 0.0200575365379, 0.0585369028744, 0.158869104881, 0.364576440742,
                0.635423559258, 0.841130895119, 0.941463097126, 0.979942463462, 0.993307149076],
    "LeakyReLU": [-0.5, -0.388888888889, -0.277777777778, -0.166666666667, -0.0555555555556,
                  0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "ReLU": [-0.0, -0.0, -0.0, -0.0, -0.0, 0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "ELU": [-0.4966310265, -0.489765962143, -0.468911737989, -0.405562198581, -0.213123289631,
            0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "SeLU": [-1.74618571868, -1.72204772347, -1.64872296837, -1.42598202974, -0.749354802287,
             0.583722222222, 1.75116666667, 2.91861111111, 4.08605555556, 5.2535],
    "Tanh": [-0.999909204263, -0.999162466631, -0.992297935288, -0.931109608668, -0.504672397722,
             0.504672397722, 0.931109608668, 0.992297935288, 0.999162466631, 0.999909204263]
}

for label in activations:
    fig = plt.figure(figsize=(4,4))
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(X, activations[label](X), color='darkorchid', lw=lw, label=label)
    assert rel_error(activations[label](XS), GT_Act[label]) < 1e-9, \
           "Your implementation of {} might be wrong".format(label)
    ax.legend(loc="lower right")
    ax.axhline(0, color='black')
    ax.axvline(0, color='black')
    ax.set_title('{}'.format(label), fontsize=14)
    plt.xlabel(r"X")
    plt.ylabel(r"Y")
    plt.show()
```

# Phew! You're done for problem 1 now, but 3 more to go... LOL