###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth. Partly based on David Ketcheson's pendulum lesson, also under CC-BY.

# Phugoid Oscillation

Welcome back! This is the second Jupyter Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/info)

In the [first notebook](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_01_Phugoid_Theory.ipynb), _"Phugoid Motion"_, we described the physics of an aircraft's oscillatory trajectory, seen as an exchange of kinetic and potential energy. This analysis goes back to Frederick Lanchester, who published his book _"Aerodonetics"_ on aircraft stability in 1909. We concluded that first exposure to our problem of interest by plotting the flight paths predicted by Lanchester's analysis, known as _phugoids_.

Here, we will look at the situation when an aircraft is initially moving on the straight-line phugoid (obtained with the parameters $C=2/3$, $\cos\theta=1$, and $z=z_t$ in the previous analysis), and experiences a small upset, a wind gust that slightly perturbs its path. It will then enter into a gentle oscillation around the previous straight-line path: a _phugoid oscillation_.

If the aircraft experiences an upward acceleration of $-d^2z/dt^2$, and we assume that the perturbation is small, then $\cos\theta=1$ is a good approximation and Newton's second law in the vertical direction is:

$$
\begin{equation}
L - W = - \frac{W}{g}\frac{d^2 z}{dt^2}
\end{equation}
$$

In the previous notebook, we saw that the following relation holds for the ratio of lift to weight, in terms of the trim velocity $v_t$:

$$
\begin{equation}
\frac{L}{W}=\frac{v^2}{v_t^2}
\end{equation}
$$

This will be useful: we can divide Equation (1) by the weight and use Equation (2) to replace $L/W$. Another useful relation from the previous notebook expressed the conservation of energy (per unit mass) as $v^2 = 2 gz$. With this, Equation (1) is rearranged as:

$$
\begin{equation}
\frac{d^2z}{dt^2} + \frac{gz}{z_t} = g
\end{equation}
$$

Look at Equation (3) for a moment. Does it ring a bell? Do you recognize it?

If you remember from your physics courses the equation for _simple harmonic motion_, you should see the similarity!

Take the case of a simple spring. Hooke's law is $F=-kx$, where $F$ is a restoring force, $x$ the displacement from a position of equilibrium and $k$ the spring constant. This results in the following ordinary differential equation for the displacement:

$$
\begin{equation}
 \frac{d^2 x}{dt^2}= -\frac{k}{m}x
\end{equation}
$$

which has the solution $x(t) = A \cos(\omega t- \phi)$, representing simple harmonic motion with an angular frequency $\omega=\sqrt{k/m}=2\pi f$ and phase angle $\phi$.

Now look back at Equation (3): it has nearly the same form and it represents simple harmonic motion with angular frequency $\omega=\sqrt{g/z_t}$ around mean height $z_t$.

Think about this for a moment ... we can immediately say what the period of the oscillation is: exactly $2 \pi \sqrt{z_t/g}$ — or, in terms of the trim velocity, $\pi \sqrt{2} v_t/g$.

_This is a remarkable result!_ Think about it: we know nothing about the aircraft, or the flight altitude, yet we can obtain the period of the phugoid oscillation simply as a function of the trim velocity. For example, if trim velocity is 200 knots, we get a phugoid period of about 47 seconds—over that time, you really would not notice anything if you were flying in that aircraft.

Next, we want to be able to compute the trajectory of the aircraft for a given initial perturbation. We will do this by numerically integrating the equation of motion.

## Prepare to integrate

We want to integrate the differential equation and plot the trajectory of the aircraft. Are you ready?

The equation for the phugoid oscillation is a second-order, ordinary differential equation (ODE). Let's represent the time derivative with a prime, and write it like this:

$$
\begin{equation}
z(t)'' + \frac{g \,z(t)}{z_t}=g
\end{equation}
$$

There's a convenient trick when we work with ODEs: we can turn this 2nd-order equation into a system of two 1st-order equations. Like this:

$$
\begin{eqnarray}
z'(t) &=& b(t)\\
b'(t) &=& g\left(1-\frac{z(t)}{z_t}\right)
\end{eqnarray}
$$

Are you following? Make sure you are following the derivations, even if it means writing the equations down in your own notes! (Yes, the old-fashioned paper way.)

Another way to look at a system of two 1st-order ODEs is by using vectors. You can make a vector with your two independent variables,

$$
\begin{equation}
\vec{u}  = \begin{pmatrix} z \\ b \end{pmatrix}
\end{equation}
$$

and write the differential system as a single vector equation:

$$
\begin{equation}
\vec{u}'(t)  = \begin{pmatrix} b\\ g-g\frac{z(t)}{z_t} \end{pmatrix}
\end{equation}
$$

If you call the right-hand-side $\vec{f}(\vec{u})$, then the equation is very short: $\vec{u}'(t) = \vec{f}(\vec{u})$—but let's drop those arrows to denote vectors from now on, as they are a bit cumbersome: just remember that $u$ and $f$ are vectors in the phugoid equation of motion.

Next, we'll prepare to solve this problem numerically.

## Initial value problems

Let's step back for a moment. Suppose we have a first-order ODE $u'=f(u)$. You know that if we were to integrate this, there would be an arbitrary constant of integration. To find its value, we do need to know one point on the curve $(t, u)$. When the derivative in the ODE is with respect to time, we call that point the _initial value_ and write something like this:

$$
u(t=0)=u_0
$$

In the case of a second-order ODE, we already saw how to write it as a system of first-order ODEs, and we would need an initial value for each equation: two conditions are needed to determine our constants of integration. The same applies for higher-order ODEs: if it is of order $n$, we can write it as $n$ first-order equations, and we need $n$ known values. If we have that data, we call the problem an _initial value problem_.

Remember the definition of a derivative? The derivative represents the slope of the tangent at a point of the curve $u=u(t)$, and the definition of the derivative $u'$ for a function is:

$$
u'(t) = \lim_{\Delta t\rightarrow 0} \frac{u(t+\Delta t)-u(t)}{\Delta t}
$$

If the step $\Delta t$ is already very small, we can _approximate_ the derivative by dropping the limit. We can write:

$$
\begin{equation}
u(t+\Delta t) \approx u(t) + u'(t) \Delta t
\end{equation}
$$

With this equation, and because we know $u'(t)=f(u)$, if we have an initial value, we can step by $\Delta t$ and find the value of $u(t+\Delta t)$, then we can take this value, and find $u(t+2\Delta t)$, and so on: we say that we _step in time_, numerically finding the solution $u(t)$ for a range of values: $t_1, t_2, t_3 \cdots$, each separated by $\Delta t$. The numerical solution of the ODE is simply the table of values $t_i, u_i$ that results from this process.

## Discretization

In order to execute the process described above and find the numerical solution of the ODE, we start by choosing the values $t_1,t_2,t_3 \cdots t_n$—we call these values our *grid* in time. The first point of the grid is given by our _initial value_, and the small difference between two consecutive times is called the _time step_, denoted by $\Delta t$.  The solution value at time $t_n$ is denoted by $u_n$.

Let's build a time grid for our problem. We first choose a final time $T$ and the time step $\Delta t$. In code, we'll use readily identifiable variable names: `T` and `dt`, respectively. With those values set, we can calculate the number of time steps that will be needed to reach the final time; we call that variable `N`.

Let's write some code. The first thing we do in Python is load our favorite libraries: NumPy for array operations, and the Pyplot module in Matplotlib, to later on be able to plot the numerical solution. The line `%matplotlib inline` tells Jupyter Notebook to show the plots inline.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

Now, initialize `T` and `dt`, calculate `N` and build a NumPy array with all the values of time that make up the grid.

```python
# Create the time grid.
T = 100.0  # length of the time-interval
dt = 0.02  # time-step size
N = int(T / dt) + 1  # number of time steps
t = numpy.linspace(0.0, T, num=N)  # time grid using linspace()
#t = numpy.arange(start=0.0, stop=T+1, step=dt) #time grid using arange()
```

We have our grid! Now it's time to apply the numerical time stepping represented by Equation (10).

##### Challenge!

* Write the code above using the NumPy function `arange()` instead of `linspace()`. If you need to, read the documentation for these functions.

##### Pro tip:

Enter a question mark followed by any function, e.g., `?numpy.linspace`, into a code cell and execute it, to get a help pane on the notebook.

## Euler's method

The approximate solution at time $t_n$ is $u_n$, and the numerical solution of the differential equation consists of computing a sequence of approximate solutions by the following formula, based on Equation (10):

$$
\begin{equation}
u_{n+1} = u_n + \Delta t \,f(u_n)
\end{equation}
$$

This formula is called **Euler's method**.

For the equations of the phugoid oscillation, Euler's method gives the following algorithm that we need to implement in code:

$$
\begin{align}
z_{n+1} & = z_n + \Delta t \, b_n \\
b_{n+1} & = b_n + \Delta t \left(g - \frac{g}{z_t} \, z_n \right)
\end{align}
$$

### And solve!

To apply the numerical solution method, we need to set things up in code: define the parameter values needed in the model, initialize a NumPy array to hold the discrete solution values, and initialize another array for the elevation values.

```python
# Set the initial conditions.
z0 = 100.0  # altitude
b0 = 16.0  # upward velocity resulting from gust
zt = 100.0  # trim altitude
g = 9.81  # acceleration due to gravity

# Set the initial value of the numerical solution.
u = numpy.array([z0, b0])

# Create an array to store the elevation value at each time step.
z = numpy.zeros(N)
z[0] = z0
```

You should pay attention to a couple of things: (1) See how there is a dot after the numbers used to define our parameters? We just want to be explicit (as a good habit) that these variables are real numbers, called "floats." (2) We both _created_ and _initialized_ with zeros everywhere the solution vector `z`. Look up the documentation for the handy NumPy function `zeros()`, if you need to. (3) In the last line above, we assign the _initial value_ to the first element of the solution vector: `z[0]`.

Now we can step in time using Euler's method. Notice how we are time stepping the two independent variables at once in the time iterations.

```python
# Temporal integration using Euler's method.
for n in range(1, N):
    rhs = numpy.array([u[1], g * (1 - u[0] / zt)])
    u = u + dt * rhs
    z[n] = u[0]
```

Make sure you understand what this code is doing. This is a basic pattern in numerical methods: iterations in a time variable that apply a numerical scheme at each step.

## Plot the solution

If the code is correct, we have stored in the array `z` the position of the glider at each time. Let's use Matplotlib to examine the flight path of the aircraft.

You should explore the [Matplotlib tutorial](https://matplotlib.org/users/pyplot_tutorial.html) (if you need to) and familiarize yourself with the command-style functions that control the size, labels, line style, and so on. Creating good plots is a useful skill: it is about communicating your results effectively.

Here, we set the figure size, the limits of the vertical axis, the format of tick-marks, and axis labels. The final line actually produces the plot, with our chosen line style (continuous black line).

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16

# Plot the solution of the elevation.
pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title('Elevation of the phugoid over the time')  # set the title
pyplot.xlabel('Time [s]')  # set the x-axis label
pyplot.ylabel('Elevation [m]')  # set the y-axis label
pyplot.xlim(t[0], t[-1])  # set the x-axis limits
pyplot.ylim(40.0, 160.0)  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.plot(t, z, color='C0', linestyle='-', linewidth=2);
```

##### Explore and think

Try changing the value of `v` in the initial conditions.

* What happens when you have a larger gust?
* What about a smaller gust?
* What happens if there isn't a gust (`v = 0`)?

## Exact solution

The equation for phugoid oscillations is a 2nd-order, linear ODE and it has an exact solution of the following form:

$$
\begin{equation}
z(t) = A \sin \left(\sqrt{\frac{g}{z_t}} t \right) + B \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

where $A$ and $B$ are constants that we solve for using initial conditions.

Our numerical solution used the initial conditions:

$$
\begin{eqnarray}
z(0) = z_0 \\
b(0) = b_0
\end{eqnarray}
$$

Applying these to the exact solution and solving for $A$ and $B$, we get:

$$
\begin{equation}
z(t) = b_0 \sqrt{\frac{z_t}{g}} \sin \left(\sqrt{\frac{g}{z_t}} t \right) + (z_0-z_t) \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

We already defined all of these variables for our numerical solution, so we can immediately compute the exact solution.

##### Pro tip:

The expression is a bit long —if you don't feel like scrolling left and right, you can cast the value of the variable between parenthesis and insert line breaks and Python will treat the next line as a continuation of the first.

```python
z_exact = (b0 * (zt / g)**0.5 * numpy.sin((g / zt)**0.5 * t) +
           (z0 - zt) * numpy.cos((g / zt)**0.5 * t) + zt)
```

## Plot the exact solution

Now we can plot our exact solution!  Even better, we can plot _both_ the numerical solution *and* the exact solution to see how well Euler's method approximated the phugoid oscillations.

To add another curve to a plot, simply type a second `pyplot.plot()` statement.  We also add a legend using `pyplot.legend()`.

```python
# Plot the numerical solution and the exact solution.
pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title('Elevation of the phugoid over the time')  # set the title
pyplot.xlabel('Time [s]')  # set the x-axis label
pyplot.ylabel('Elevation [m]')  # set the y-axis label
pyplot.xlim(t[0], t[-1])  # set the x-axis limits
pyplot.ylim(40.0, 160.0)  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.plot(t, z, label='Numerical',
            color='C0', linestyle='-', linewidth=2)
pyplot.plot(t, z_exact, label='Analytical',
            color='C1', linestyle='-', linewidth=2)
pyplot.legend();  # set the legend
```

That looks like pretty good agreement, but what's happening towards the end? We'll come back to this. For now, re-run the previous steps with a different time step, say $dt=0.01$ and pay attention to the difference.

Euler's method, like all numerical methods, introduces some errors.  If the method is *convergent*, the approximation will get closer and closer to the exact solution as we reduce the size of the step, $\Delta t$. The error in the numerical method should tend to zero, in fact, when $\Delta t\rightarrow 0$—when this happens, we call the method _consistent_. We'll define these terms more carefully in the theory components of this course.

## Convergence

To compare the two solutions, we need to use a **norm** of the difference, like the $L_1$ norm, for example.

$$
E = \Delta t \sum_{n=0}^N \left|z(t_n) - z_n\right|
$$

The $L_1$ norm is the sum of the individual differences between the exact and the numerical solutions, at each mesh point. In other words, $E$ is the discrete representation of the integral over the interval $T$ of the (absolute) difference between the computed $z$ and $z_{\rm exact}$:

$$
E = \int \vert z-z_\rm{exact}\vert dt
$$

We check for convergence by calculating the numerical solution using progressively smaller values of `dt`. We already have most of the code that we need.  We just need to add an extra loop and an array of different $\Delta t$ values to iterate through.

##### Warning

The cell below can take a little while to finish (the last $\Delta t$ value alone requires 1 million iterations!).  If the cell is still running, the input label will say `In [*]`.  When it finishes, the `*` will be replaced by a number.

```python
# Set the list of time-step sizes.
dt_values = [0.1, 0.05, 0.01, 0.005, 0.001, 0.0001]

# Create an empty list that will contain the solution of each grid.
z_values = []

for dt in dt_values:
    N = int(T / dt) + 1  # number of time-steps
    t = numpy.linspace(0.0, T, num=N)  # time grid
    # Set the initial conditions.
    u = numpy.array([z0, b0])
    z = numpy.empty_like(t)
    z[0] = z0
    # Temporal integration using Euler's method.
    for n in range(1, N):
        rhs = numpy.array([u[1], g * (1 - u[0] / zt)])
        u = u + dt * rhs
        z[n] = u[0]  # store the elevation at time-step n+1
    z_values.append(z)  # store the elevation over the time
```

```python
#We will write Euler's method as a function to avoid using a nested for loop

# Set the list of time-step sizes.
dt_values = [0.1, 0.05, 0.01, 0.005, 0.001, 0.0001]

# Create an empty list that will contain the solution of each grid.
z_values = []

def Euler(T,dt):
    N = int(T / dt) + 1  # number of time-steps
    t = numpy.linspace(0.0, T, num=N)  # time grid
    # Set the initial conditions.
    u = numpy.array([z0, b0])
    z = numpy.empty_like(t)
    z[0] = z0
    for n in range(1,N):
        rhs = numpy.array([u[1], g * (1 - u[0] / zt)])
        u = u + dt * rhs
        z[n] = u[0]
    return z
for dt in dt_values:
    # Temporal integration using Euler's method.
    result = Euler(T,dt)
    z_values.append(result)

```

### Calculate the error

We now have numerical solutions for each $\Delta t$ in the array `z_values`.  To calculate the error corresponding to each $\Delta t$, we can write a function!

```python
def l1_error(z, z_exact, dt):
    """
    Computes and returns the error
    (between the numerical and exact solutions)
    in the L1 norm.

    Parameters
    ----------
    z : numpy.ndarray
        The numerical solution as an array of floats.
    z_exact : numpy.ndarray
        The analytical solution as an array of floats.
    dt : float
        The time-step size.

    Returns
    -------
    error: float
        L1-norm of the error with respect to the exact solution.
    """
    error = dt * numpy.sum(numpy.abs(z - z_exact))
    return error
```

**Note**: in the first line of the function, we perform an 'array operation':

`z - z_exact`

We are *not* subtracting one value from another.  Instead, we are taking the difference between elements at each corresponding index in both arrays.  Here is a quick example:

```python
a = numpy.array([1, 2, 3])
b = numpy.array([4, 4, 4])

b - a
```

    array([3, 2, 1])

Now, we iterate through each $\Delta t$ value and calculate the corresponding error.
In the following code cell, we use the built-in function [`zip`](https://docs.python.org/3/library/functions.html#zip) to iterate over the two lists `z_values` and `dt_values`.

```python
# Create an empty list to store the errors on each time grid.
error_values = []

for z, dt in zip(z_values, dt_values):
    N = int(T / dt) + 1  # number of time-steps
    t = numpy.linspace(0.0, T, num=N)  # time grid
    # Compute the exact solution.
    z_exact = (b0 * (zt / g)**0.5 * numpy.sin((g / zt)**0.5 * t) +
               (z0 - zt) * numpy.cos((g / zt)**0.5 * t) + zt)
    # Calculate the L1-norm of the error for the present time grid.
    error_values.append(l1_error(z, z_exact, dt))
```

Remember, *if* the method is convergent then the error should get smaller as  $\Delta t$ gets smaller.  To visualize this, let's plot $\Delta t$ vs. error.  If you use `pyplot.plot` you won't get a very useful result.  Instead, use `pyplot.loglog` to create the same plot with a log-log scale.  This is what we do almost always to assess the errors of a numerical scheme graphically.

```python
# Plot the error versus the time-step size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.title('L1-norm error vs. time-step size')  # set the title
pyplot.xlabel('$\Delta t$')  # set the x-axis label
pyplot.ylabel('Error')  # set the y-axis label
pyplot.grid()
pyplot.loglog(dt_values, error_values,
              color='C0', linestyle='--', marker='o')  # log-log plot
pyplot.axis('equal');  # make axes scale equally
```

This is the kind of result we like to see!  As $\Delta t$ shrinks (towards the left), the error gets smaller and smaller, like it should.

##### Challenge!

We calculated the solution for several different time-step sizes using two nested `for` loops.
That worked, but whenever possible, we like to re-use code (and not just copy and paste it!).

Create a function that implements Euler's method and re-write the code cell that computes the solution for different time-step sizes using your function.

---

###### The cell below loads the style of this notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Impulse Response

The response $y(t)$ of a linear time-invariant (LTI) system $\mathcal{H}$ to an arbitrary input signal $x(t)$ is derived in the following. The input signal can be represented as an integral when applying the [sifting-property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse)

\begin{equation}
x(t) = \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau
\end{equation}

The output signal of the system is then given as

\begin{equation}
y(t) = \mathcal{H} \left\{ \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau \right\}
\end{equation}

The integration and system response operator can be exchanged under the assumption that the system is linear

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot \mathcal{H} \left\{ \delta(t-\tau) \right\} \; d \tau
\end{equation}

where $\mathcal{H} \{\cdot\}$ was only applied to the Dirac impulse, since $x(\tau)$ can be regarded as constant factor with respect to the time $t$. It is obvious that the response of a system to a Dirac impulse plays an important role in the calculation of the output signal for arbitrary input signals.

The response of a system to a Dirac impulse as input signal is denoted as [*impulse response*](https://en.wikipedia.org/wiki/Impulse_response). It is defined as

\begin{equation}
h(t) = \mathcal{H} \left\{ \delta(t) \right\}
\end{equation}

If the system is time-invariant, the response to a shifted Dirac impulse is $\mathcal{H} \left\{ \delta(t-\tau) \right\} = h(t-\tau)$. Hence, for an LTI system we finally get

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot h(t-\tau) \; d \tau
\end{equation}

Due to its relevance in the theory of LTI systems, this operation is explicitly termed as [*convolution*](https://en.wikipedia.org/wiki/Convolution). It is commonly abbreviated by $*$, hence for above integral we get $y(t) = x(t) * h(t)$.

The properties of an LTI system are entirely characterized by its impulse response. The response $y(t)$ of a system to an arbitrary input signal $x(t)$ is given by the convolution of the input signal $x(t)$ with its impulse response $h(t)$.

**Example**

The following example considers an LTI system whose relation between input $x(t)$ and output $y(t)$ is given by an ordinary differential equation (ODE) with constant coefficients

\begin{equation}
y(t) + \frac{d}{dt} y(t) = x(t)
\end{equation}

The system response is computed for $x(t) = e^{- 2 t} \cdot \epsilon(t)$ by

1. explicitly solving the ODE and by
2. computing the impulse response $h(t)$ and convolution with the input signal.

The solution should fulfill the initial conditions $y(t)\big\vert_{t = 0-} = 0$ and $\frac{d}{dt}y(t)\big\vert_{t = 0-} = 0$ due to causality.

First the ODE is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(y + y.diff(t) , x)
ode
```

The ODE is solved for the given input signal in order to calculate the output signal. Note that the integration constant is set to zero to fulfill the initial conditions

```python
solution = sym.dsolve(ode.subs(x, sym.exp(-2*t)*sym.Heaviside(t)))
y1 = solution.rhs.subs('C1', 0)
y1
```

Lets plot the output signal derived by explicit solution of the ODE

```python
sym.plot(y1, (t,-1,10), ylabel=r'$y(t)$');
```

The impulse response $h(t)$ is computed by solving the ODE for a Dirac impulse as input signal, $x(t) = \delta(t)$

```python
solution2 = sym.dsolve(ode.subs(x, sym.DiracDelta(t)))
h = solution2.rhs.subs('C1', 0)
h
```

Lets plot the impulse response $h(t)$ of the LTI system

```python
sym.plot(h, (t,-1,10), ylabel=r'$h(t)$');
```

As alternative to the explicit solution of the ODE, the system response is computed by evaluating the convolution integral. Since `SymPy` cannot handle the Heaviside function properly in integrands, the convolution integral is first simplified. Both the input signal $x(t)$ and the impulse response $h(t)$ are causal signals. Hence, the convolution integral degenerates to

\begin{equation}
y(t) = \int_{0}^{t} x(\tau) \cdot h(t - \tau) \; d\tau
\end{equation}

for $t \geq 0$. Note that $y(t) = 0$ for $t<0$.

```python
tau = sym.symbols('tau', real=True)

y2 = sym.integrate(sym.exp(-2*tau) * h.subs(sym.Heaviside(t), 1).subs(t, t-tau), (tau, 0, t))
y2
```

Lets plot the output signal derived by evaluation of the convolution

```python
sym.plot(y2, (t,0,10), ylabel=r'$y(t)$');
```

**Exercise**

* Compare the output signal derived by explicit solution of the ODE with the signal derived by convolution. Are both equal?
* Check if the impulse response $h(t)$ is a solution of the ODE by manual calculation. Hint $\frac{d}{dt} \epsilon(t) = \delta(t)$.
* Check the solution of the convolution integral by manual calculation including the Heaviside functions.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


## RKR Computation
All Code and Markdown written by Gary Zeri, Chapman University Student and member of the LaRue Cat Lab

All equations and information within this notebook originated from The Computation of RKR Potential Energy Curves of Diatomic Molecules using Matematica, written by Peter Senn.

The RKR method is a procedure used to determine the potential energy curves of diatomic molecules by calculating the classical turning points, $r_-$ and $r_+$ from the molecule's diatomic constants.

The turning points, $r_{\pm}$, can be computed using the following equation:
$$r_{\pm} = \frac{f(v)}{2} \cdot \left[\sqrt{1 + \frac{1}{f(v)g(v)}} \pm 1\right]$$

It should be noted that in the above equation the fraction in the square root has a one in the numerator while in the paper by Peter Senn the numerator has a four. The four is not present in the Matematica code in Senn's paper and in Lecture #21: Construction of Potential Curves by the Rydberg-Klein-Rees Method (RKR) from MIT OpenCourseWare, the four is not included as well, leading to the omission of the four in the equation presented here to provide an exact equation to that used within the below code.

$f(v)$ and $g(v)$ are defined as follows:
$$f(v) = \int^v_{\frac{-1}{2}}{ \left[ \frac{dv^{'}}{\sqrt{E(v)-E(v^{'})}{}} \right] }$$

$$g(v) = \int_{\frac{-1}{2}}^v{\left[ \frac{\frac{\partial E}{\partial J}}{\sqrt{E(v) - E(v^{'})}}\right]dv^{'}}$$

In the original paper by Senn, there are constants in front of the integrals that are omitted here as they are used to scale the values to a desired unit system. Since this RKR code will use wavenumbers for energy and angstroms for bond distance, the constant of $\frac{8.211609}{\sqrt{\mu}}$ will be multiplied by the final answers to perform this conversion.

For vibronic states with no rotation and $J=0$, $E(v)$ and $\frac{\partial E}{\partial J}$ can be represented as:
$$E(v) = \omega_e(v+ \frac{1}{2}) - \omega_ex_e(v+\frac{1}{2})^2 + \omega_ey_e(v+\frac{1}{2})^3 + \omega_ez_e(v+\frac{1}{2})^4 + \dots$$

$$\frac{\partial E}{\partial J} = B(v) = B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2 + \dots$$

An important computational issue to note with the RKR method is that the integrand term $\frac{dv^{'}}{\sqrt{E(v) - E(v^{'})}}$ will become one over zero since the integration is from $\frac{-1}{2}$ to $v$, thus the integrand will become $\frac{1}{\sqrt{E(v) - E(v)}}$ when $v^{'} = v$ at the end of the integration.

In order to deal with this issue, the integral is changed from $\int^{v}_{\frac{-1}{2}}$ to $\int^{v-\delta}_{\frac{-1}{2}}$, where $\delta$ is some extremely small value. A correction term is then added to both $f(v)$ and $g(v)$ to account for the missing endpoint of the integration resulting in new and more easily computable versions of $f(v)$ and $g(v)$:

$$f(v) = \int^{v-\delta}_{\frac{-1}{2}}{\frac{dv^{'}}{\sqrt{E(v) - E(v^{'})}} + \sqrt{\frac{\delta}{Q_v}}}$$

$$g(v) = \int^{v-\delta}_{\frac{-1}{2}}{\frac{B(v^{'})}{\sqrt{E(v)-E(v^{'})}}dv^{'} } + 2B(v)\sqrt{\frac{\delta}{Q_v}} $$

Where $Q_v$ is the following series:
$$Q_v = w_e-2\omega_ex_e(v+\frac{1}{2}) + 3\omega_ey_e(v+\frac{1}{2})^2 + 4\omega_ez_e(v+\frac{1}{2})^3 + \dots$$

The following code only uses the terms listed above in the computation for all series listed.

## RKR With Excited States

In order to allow for RKR computations with excited states, $E(v)$ and $\frac{\partial E}{\partial J}$ must include the $J$ term. The full $E(v)$ equation is included below:
\begin{align}
    E(v) &= \omega_e(v+\frac{1}{2}) - \omega_ex_e(v+\frac{1}{2})^2 + \omega_ey_e(v+\frac{1}{2})^3 + \omega_ez_e(v+\frac{1}{2}) + \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot J(J+1)-D_eJ^2(J+1)^2 \\
    E(v) &= c(v) + \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot J(J+1)-D_eJ^2(J+1)^2 \\
    E(v) &= c(v) + f(v, J) - g(J)
\end{align}

\begin{align}
    \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}\left[ c(v) + f(v, J) - g(J)\right] \\
    \frac{\partial E}{\partial J} &= 0 + \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
    \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
\end{align}

\begin{align}
    \frac{\partial }{\partial J}f(v, J) &= \frac{\partial }{\partial J}\left[d(v)\cdot h(J)\right] \\
    &= d(v) \frac{\partial}{\partial J}h(v) \\
    &= d(v) \frac{\partial}{\partial J}\left[J(J+1)\right] \\
    &= d(v) \cdot \left(2J + 1\right) \\
    &= \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot \left(2J + 1\right) \\
    &= B_e \cdot \left(2J + 1\right) - \alpha_e(v+\frac{1}{2})\left(2J + 1\right) + y_e(v+\frac{1}{2})^2 \left(2J + 1\right) \\
    &= 2JB_e + B_e -2J\alpha_e(v+\frac{1}{2}) -2\alpha_e(v+\frac{1}{2} ) + 2Jy_e(v+\frac{1}{2})^2 + y_e(v+\frac{1}{2})^2 \\
    &= 2J\left[B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] + B_e - \alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2 \\
    &= 2Jd(v) + d(v) \\
    \frac{\partial}{\partial J}f(v, J) &= d(v)\left(2J + 1\right)
\end{align}

\begin{align}
    \frac{\partial }{\partial J}g(J) &= \frac{\partial}{\partial J} \left[D_e J^2(J+1)^2 \right] \\
        &= D_e \frac{\partial}{\partial J}\left[ J^2(J+1)^2\right] \\
        &= D_e \left[ \frac{\partial}{\partial J}[J^2] \cdot (J+1)^2 + J^2 \cdot \frac{\partial}{\partial J}[(J+1)^2]\right] \\
        &= D_e \left[ 2J \cdot (J+1)^2 + J^2 \cdot 2(J+1)\right] \\
        &= 2J(J+1)D_e \left[ (J+1) + J\right] \\
        &= 2J(J+1)D_e\left[ 2J + 1 \right] \\
    \frac{\partial}{\partial J}g(J) &= 2D_eJ(J+1)(2J+1)
\end{align}

\begin{align}
        \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
       \frac{\partial E}{\partial J} &= d(v)(2J + 1) - 2D_eJ(J+1)(2J+1) \\
       \frac{\partial E}{\partial J} &= \left[B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right](2J+1) - 2D_eJ(J+1)(2J+1)
\end{align}

```python
#Allow Notebook to Import from Comp_Chem_Package
import sys
sys.path.append("..\\Comp_Chem_Package")

from compChemGlobal import *
from nistScraper import getDiatomicConstants

#Global variables to be modified by the user

#Distance from v that the integration should stop at
#Should be a very very small number strictly greater than 0
#But, the smaller the value, the slower the integrals will take to compute
delta = pow(10, -3)

resolution = 10

#provide the name of the diatomic constants that are desired
dc = getDiatomicConstants("CO")
```

```python
#Define All Functions Here

def E(v, J=0):
    term = v + 0.5
    sumValue = (dc["w"] * term) - (dc["wx"]*pow(term, 2)) + (dc["wy"]*pow(term, 3)) + (dc["wz"]*pow(term,4))
    return sumValue * (2*J + 1) - 2*dc["D"]*J*(J+1)*(2*J+1)

def EPrime(v, J=0):
    term = v + 0.5
    return dc["w"] - 2*dc["wx"]*term + 3*dc["wy"]*pow(term, 2) + 4*dc["wz"]*pow(term, 3)

def B(v):
    term = v + 0.5
    return dc["B"] - (dc["a"] * term)  + (dc["y"]*pow(term, 2))

#Used in the correctionFactor calculation
def Q(v):
    term = v + 0.5
    return dc["w"] - (2*dc["wx"]*term) + (3*dc["wy"]*pow(term, 2)) + (4*dc["wz"]*pow(term,3))

#Used to correct integrals that stop delta away from v
def correctionFactor(v):
    return 2 * sqrt(delta / Q(v))

def integralRadical(v, vPrime):
    return sqrt(E(v) - E(vPrime))

def f(v):
    integrand = lambda vPrime: 1 / integralRadical(v, vPrime)
    return integrate(integrand, -0.5, v-delta) + correctionFactor(v)

def g(v):
    integrand = lambda vPrime : B(vPrime) / integralRadical(v, vPrime)
    return integrate(integrand, -0.5, v-delta) + (B(v)*correctionFactor(v))

#v refers to a float energy level for the potential well
#returns the tuple (r+, r-)
def RKR(v):

    fValue = f(v)

    c0 = (8.211609 * fValue ) / (2 * sqrt(dc["u"]))

    radicand = 1 / (fValue * g(v))
    c1 = sqrt(1 + radicand)

    return c0 * (c1 + 1), c0 * (c1 - 1)
```

```python
#Graphing and Computation Code
import numpy as np

#Lists to hold data
r = []
EList = []

ddr = []
ddE = []
ddr2 = []
ddE2 = []
leftAsympCutOff = False

v = -0.499
dv = 1 / resolution

loadingBar =  widgets.FloatProgress(
                  value = v,
                  min = -EPrime(v),
                  max = 0,
             )
display(widgets.HBox([
              widgets.Label(value="Building RKR Potential"),
              loadingBar
             ]))

#Loop to generate all data to graph
#for v in tqdm(np.arange(vStart, 17, 1/resolution)):
while(EPrime(v) > 0):

    energy = E(v)
    rPoints = RKR(v)

    if(not leftAsympCutOff and len(r) != 0):
        #Compute First Derivative
        ddr.append( (rPoints[1] + r[-2:][-1]) / 2 )
        ddE.append( (energy-EList[-1]) / ( rPoints[1] - r[-2:][-1] ))

        if(len(ddr) > 1):
            #Compute 2nd Derivative
            ddr2.append( (ddr[-2] + ddr[-1]) / 2 )
            ddE2.append( (ddE[-1] - ddE[-2]) / (ddr[-1] - ddr[-2]) )

            #Determine if Cutoff should be used
            if(ddE2[-1] <= 0):
                leftAsympCutOff = True

    if(not leftAsympCutOff):
        r.extend( rPoints )
        EList.extend( [energy] * 2 )
    else:
        r.append(max(rPoints))
        EList.append(energy)

    v += dv
    loadingBar.value = -EPrime(v)

#Manually add in re point
r.append(dc["re"])
EList.append(0)

#Prepare data for graphing
figure = {
    "data":[
        {
            "type":"scatter",
            "x":r,
            "y":EList,
            "connectgaps":True,
            "mode":"markers",
            "name":"RKR Potential",
        }
    ],

    "layout":{
        "xaxis":{"title":"r in Angstroms"},
        "yaxis":{"title":"Energy in Wavenumbers"},
        "title":{"text":"RKR Generated Potential Energy Surface"}
    }
}

display(plot.go.FigureWidget(data=figure))
```

    HBox(children=(Label(value='Building RKR Potential'), FloatProgress(value=-0.499, max=0.0, min=-2169.78700338)…

    FigureWidget({
        'data': [{'connectgaps': True,
                  'mode': 'markers',
                  'name': 'RKR …

```python
#Test for the RKR Class
#Allow Notebook to Import from Comp_Chem_Package
import sys
sys.path.append("..\\Comp_Chem_Package")
from rkr import rkr as rkrClass

from compChemGlobal import *
from nistScraper import getDiatomicConstants

dc = getDiatomicConstants("CO")

rkr = rkrClass(dc)
rkr.compute(resolution=50, delta = pow(10, -3))
print("Graphing Potential")
rkr.graph()
```

    HBox(children=(Label(value='Computing RKR Surface'), FloatProgress(value=-2169.78700338, max=0.0, min=-2169.78…

    Graphing Potential

    VBox(children=(FigureWidget({
        'data': [{'hoverlabel': {'font': {'size': 16}},
                  'hovertemplate…

```python

```


# 15 Partial Differential Equations — 1

## Solving Laplace's or Poisson's equation

**Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

Solutions depend on the **boundary conditions**:

* the *value of the potential* on the *boundary* or
* the *electric field* (i.e. the derivative of the potential, $\mathbf{E} = -\nabla\Phi$ *normal to the surface* ($\mathbf{n}\cdot\mathbf{E}$), which directly follows from the charge distribution).

### Example: 2D Laplace equation
$$
\frac{\partial^2 \Phi(x,y)}{\partial x^2} + \frac{\partial^2 \Phi(x,y)}{\partial y^2} = 0
$$
("elliptic PDE")

Boundary conditions:
* square area surrounded by wires
* three wires at ground (0 V), one wire at 100 V

## Finite difference algorithm for Poisson's equation
Discretize space on a lattice (2D) and solve for $\Phi$ on each lattice site.

Taylor-expansion of the four neighbors of $\Phi(x, y)$:

\begin{align}
\Phi(x \pm \Delta x, y) &= \Phi(x, y) \pm \Phi_x \Delta x + \frac{1}{2} \Phi_{xx} \Delta x^2 + \dots\\
\Phi(x, y \pm \Delta y) &= \Phi(x, y) \pm \Phi_y \Delta x + \frac{1}{2} \Phi_{yy} \Delta x^2 + \dots\\
\end{align}

Add equations in pairs: odd terms cancel, and **central difference approximation** for 2nd order partial derivatives (to $\mathcal{O}(\Delta^4)$):

\begin{align}
\Phi_{xx}(x,y) = \frac{\partial^2 \Phi}{\partial x^2} & \approx
  \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) - 2\Phi(x,y)}{\Delta x^2} \\
\Phi_{yy}(x,y) = \frac{\partial^2 \Phi}{\partial y^2} &\approx
  \frac{\Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) - 2\Phi(x,y)}{\Delta y^2}
\end{align}

Take $x$ and $y$ grids of equal spacing $\Delta$: Discretized Poisson equation

$$
\begin{split}
\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) &+ \\
   +\, \Phi(x,y-\Delta y) - 4\Phi(x,y) &= -4\pi\rho(x,y)\,\Delta^2
   \end{split}
$$

Defines a system of $N_x \times N_y$ simultaneous algebraic equations for $\Phi_{ij}$ to be solved.

Can be solved directly via matrix approaches (and then is the best solution) but can be unwieldy for large grids.

Alternatively: **iterative solution**:

$$
\begin{split}
4\Phi(x,y) &= \Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\\
 &+ \Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) + 4\pi\rho(x,y)\,\Delta^2
\end{split}
$$

Or written for lattice sites $(i, j)$ where

$$
x = x_0 + i\Delta\quad\text{and}\quad y = y_0 + j\Delta, \quad 0 \leq i,j < N_\text{max}
$$

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

* Converged solution at $(i, j)$ will be the average potential from the four neighbor sites + charge density contribution.
* *Not a direct solution*: iterate and hope for convergence.

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

### Solution via relaxation (Gauss-Seidel)

Solve the box-wire problem on a lattice.

Note: $\rho=0$ inside the box.

Note for Jupyter notebook use:
* For interactive 3D plots, select
  ```
  %matplotlib notebook
  ```
* For standard inline figures (e.g. for exporting the notebook to LaTeX/PDF or html) use
  ```
  %matplotlib inline
  ```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
#%matplotlib inline
%matplotlib notebook
```

```python
%matplotlib inline
```

#### Wire on a box: Solution of Laplace's equation with the Gauss-Seidel algorithm

```python
Nmax = 100
Max_iter = 70
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V

for n_iter in range(Max_iter):
    for xi in range(1, Nmax-1):
        for yj in range(1, Nmax-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
```

#### Visualization of the potential

```python
# plot Phi(x,y)
x = np.arange(Nmax)
y = np.arange(Nmax)
X, Y = np.meshgrid(x, y)

Z = Phi[X, Y]
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')

ax.view_init(elev=40, azim=20)
```

Nicer plot (use this code for other projects):

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

surf = ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm, rstride=2, cstride=2, alpha=0.3)
cset = ax.contourf(X, Y, Z, zdir='z', offset=-50, cmap=plt.cm.coolwarm)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')
ax.set_zlim(-50, 100)
ax.view_init(elev=40, azim=20)

cb = fig.colorbar(surf, shrink=0.5, aspect=5)
cb.set_label(r"potential $\Phi$ (V)")
```

```python

```


```python
from itertools import product
import numpy as np
```

# Introducing DenseNet

Ming Li

Data Scientist
Open Source Contributor (pandas, scikit-learn, etc).

A single Dense Block of 5 layers. Huang et al 2016.

* Novel connectivity pattern to increase network connections to quadratic in relation to layers: $\frac{𝐿(𝐿+1)}{2}$
* Feature reuse.
* Parameters and computation efficient.
* Outperform current state-of-the-art results across various benchmarks.
* Easy and efficient (Pleiss et al 2017) implementation available.

## Convolution
In continuous domain of $\tau$, convolution is defined as:

$$(f * g)(\tau) = \int_{0}^{t} f(\tau) g(\tau - t) d\tau$$
In discrete coordinate space $[h, w]$, this is equivalently defined as:

$$(f * g)[h, w] = \sum_{i}\sum_{j} f(h, w)g(h - i, w - j)$$

```python
f = np.array([[1, 1, 1, 0, 0], [0, 1, 1, 1, 0], [0, 0, 1, 1, 1], [0, 0, 1, 1, 0], [0, 1, 1, 0, 0]])
g = np.array([[1, 0, 1], [0, 1, 0], [1, 0, 1]])
```

Convolution during Forward Propagation, source of image

```python
def element_conv():
    h, w = 3, 3
    element_conv = np.zeros_like(g, dtype=np.float32)
    for i in range(h):
        for j in range(w):
            kernel = f[i:h+i, j:w+j]
            element_conv[i, j] = np.sum(kernel * g)
    return element_conv

%timeit -n 1000 element_conv()
```

    92.2 µs ± 10.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
element_conv()
```

    array([[4., 3., 4.],
           [2., 4., 3.],
           [2., 3., 4.]], dtype=float32)

Implementation as Matrix Multiplication:

```python
def matmul_conv():
    h, w = 3, 3
    col = np.zeros([9, 9])
    for i, j in product(range(h), range(w)):
        col[i*w+j] = f[i:h+i, j:w+j].flatten()
    return (g.flatten() @ col).reshape(g.shape)

%timeit -n 1000 matmul_conv()
```

    23.9 µs ± 6.66 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
# check identity
assert np.allclose(element_conv(), matmul_conv())
```

## Dense Block

A Dense Block in DenseNet is a block of hidden layers where subsequent layer reuses feature from preceding layers, through concatenation of feature maps along the depth. More concretely,
$$\begin{align*}
x_{l} &= f_{composite}(x_{0})\\
x_{2} &= f_{composite}([x_{0}, x_{1}])\\
\dots\\
x_{l} &= f_{composite}([x_{0}, x_{1}, x_{2}, \dots, x_{l-1}])
\end{align*}$$

- inspired by "identity skip connection" in ResNet
- avoids informaiton loss by using concatenation (not summation).
- bottleneck and compression techniques to control growth.

An illustration of Dense Block of 2 layers

## Composite Function

$f_{composite}$ consists of 3 functions inspired from 'pre-activation' in ResNet: Batch Normalization, ReLU, Convolution. Concretely:

$$
f_{composite}(x) = Conv(ReLU(BN(x)))
$$

full pre-activation as in ResNet, note that "weight" indicates conv. He et al 2016

**Batch Normalizing Transform** is defined as:
$$\begin{align}
\hat{x_{i}} &= \frac{x_{i} - \mu_{B}}{\sqrt{\sigma_{B}^2 + \epsilon}}\\
BN(x_{i}; \gamma, \beta) &= \gamma \hat{x_{i}} + \beta
\end{align}$$

It has the benefit of reducing internal covariance shift (moments of activations), accelerating training, regularizing. In addition, with DenseNet, it produces data augmentation every time a feature map is reused.

Test Acurracy and Distribution of Logits over time. Ioffe and Szegedy 2015

**Rectified Linear Unit (ReLU)** is defined as: $$f(x) = \max({0, x})$$

- faster evaluation than sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$
- $\frac{\partial{f(x)}}{\partial{x}} \in \{0, 1\}$ is more favourable to $\frac{\partial{\sigma(x)}}{\partial{x}} \in [0., 0.25]$ in regards to reducing gradient vanishing.
- may cause 'dead neurons', unable to recover due to $\forall x \in (-\infty, 0), \ \frac{\partial{f(x)}}{\partial{x}} = 0$.

Saturated layer slowing down learning. Glorot and Bengio et al 2010

## Dense Connectivity

- direct connections allow Jacobians to travel further with reduced gradient vanishing, so deeper network is sensibly possible.
- fewer parameters required, less prone to overfitting.
- pseudo-augmentation due to feature resuse and Batch Normalization

A DenseNet with multiple dense blocks.

a _very_ illustrative example:

$$\begin{align*}
\delta{w_1} &= \frac{\partial{E_{1}}}{\partial{w_1}} + \frac{\partial{E_{2}}}{\partial{w_1}} + \dots \\
\delta{w_1} &= \frac{\partial{E_{1}}}{\partial{f(x_1)}}\frac{\partial{f(x_1)}}{\partial{x_1}}\frac{\partial{x_1}}{\partial{w_1}} + \frac{\partial{E_{2}}}{\partial{f([x_1, x_2])}}\frac{\partial{f([x_1, x_2])}}{\partial{x_1}}\frac{\partial{x_1}}{\partial{w_1}} + \dots\\
\text{then}\\
\hat{w_1} &= w_1 - \lambda\delta{w_1}
\end{align*}$$

## Implementation

- Pleiss et al 2017 reduces memory consumption exploiting cheap concatenation and BN operations.
- Two pre-allocated shared memory storages for intermediate outputs from concatenation and BN.

efficient implementation. Pleiss et al 2017

## Not Mentioned

- bottleneck
- transition layers
- compression

## Reference

- Huang et al 2016. Densely Connected Convolutional Networks. accessible at: https://arxiv.org/abs/1608.06993
- Glorot and Bengio et al 2010. Understanding the difficulty of training deep feedforward neural networks. accessible at: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
- He et al 2016. Identity Mappings in Deep Residual Networks. accessible at: https://arxiv.org/abs/1603.05027
- Ioffe and Szegedy 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. accessible at: https://arxiv.org/abs/1502.03167
- Pleiss et al 2017. Memory-Efficient Implementation of DenseNets. accessible at: https://arxiv.org/abs/1707.06990

## Reads
- Wan et al 2018. Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout. accessible at: https://arxiv.org/abs/1810.00091
- Zhang et al 2015. Character-level Convolutional Networks for Text Classification. accessible at: https://arxiv.org/abs/1509.01626
- He et al 2015. Deep Residual Learning for Image Recognition. accessible at: https://arxiv.org/abs/1512.03385
- Krizhevsky et al 2012. Deep Convolutional Neural Networks. accessible at : https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
- Simonyan and Zisserman 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. accessible at: https://arxiv.org/pdf/1409.1556.pdf
- Szegedy et al 2014. Going Deeper with Convolutions. accessible at https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf

## Slides
https://github.com/minggli/densenet


# Detrending, Stylized Facts and the Business Cycle

In an influential article, Harvey and Jaeger (1993) described the use of unobserved components models (also known as "structural time series models") to derive stylized facts of the business cycle.

Their paper begins:

    "Establishing the 'stylized facts' associated with a set of time series is widely considered a crucial step
    in macroeconomic research ... For such facts to be useful they should (1) be consistent with the stochastic
    properties of the data and (2) present meaningful information."

In particular, they make the argument that these goals are often better met using the unobserved components approach rather than the popular Hodrick-Prescott filter or Box-Jenkins ARIMA modeling techniques.

statsmodels has the ability to perform all three types of analysis, and below we follow the steps of their paper, using a slightly updated dataset.

```python
%matplotlib inline

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt

from IPython.display import display, Latex
```

## Unobserved Components

The unobserved components model available in statsmodels can be written as:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{\gamma_{t}}_{\text{seasonal}} + \underbrace{c_{t}}_{\text{cycle}} + \sum_{j=1}^k \underbrace{\beta_j x_{jt}}_{\text{explanatory}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

see Durbin and Koopman 2012, Chapter 3 for notation and additional details. Notice that different specifications for the different individual components can support a wide range of models. The specific models considered in the paper and below are specializations of this general equation.

### Trend

The trend component is a dynamic extension of a regression model that includes an intercept and linear time-trend.

$$
\begin{align}
\underbrace{\mu_{t+1}}_{\text{level}} & = \mu_t + \nu_t + \eta_{t+1} \qquad & \eta_{t+1} \sim N(0, \sigma_\eta^2) \\\\
\underbrace{\nu_{t+1}}_{\text{trend}} & = \nu_t + \zeta_{t+1} & \zeta_{t+1} \sim N(0, \sigma_\zeta^2) \\
\end{align}
$$

where the level is a generalization of the intercept term that can dynamically vary across time, and the trend is a generalization of the time-trend such that the slope can dynamically vary across time.

For both elements (level and trend), we can consider models in which:

- The element is included vs excluded (if the trend is included, there must also be a level included).
- The element is deterministic vs stochastic (i.e. whether or not the variance on the error term is confined to be zero or not)

The only additional parameters to be estimated via MLE are the variances of any included stochastic components.

This leads to the following specifications:

|                                                                      | Level | Trend | Stochastic Level | Stochastic Trend |
|----------------------------------------------------------------------|-------|-------|------------------|------------------|
| Constant                                                             | ✓     |       |                  |                  |
| Local Level
 (random walk)                                     | ✓     |       | ✓                |                  |
| Deterministic trend                                                  | ✓     | ✓     |                  |                  |
| Local level with deterministic trend
 (random walk with drift) | ✓     | ✓     | ✓                |                  |
| Local linear trend                                                   | ✓     | ✓     | ✓                | ✓                |
| Smooth trend
 (integrated random walk)                         | ✓     | ✓     |                  | ✓                |

### Seasonal

The seasonal component is written as:

$$
\gamma_t = - \sum_{j=1}^{s-1} \gamma_{t+1-j} + \omega_t \qquad \omega_t \sim N(0, \sigma_\omega^2)
$$

The periodicity (number of seasons) is `s`, and the defining character is that (without the error term), the seasonal components sum to zero across one complete cycle. The inclusion of an error term allows the seasonal effects to vary over time.

The variants of this model are:

- The periodicity `s`
- Whether or not to make the seasonal effects stochastic.

If the seasonal effect is stochastic, then there is one additional parameter to estimate via MLE (the variance of the error term).

### Cycle

The cyclical component is intended to capture cyclical effects at time frames much longer than captured by the seasonal component. For example, in economics the cyclical term is often intended to capture the business cycle, and is then expected to have a period between "1.5 and 12 years" (see Durbin and Koopman).

The cycle is written as:

$$
\begin{align}
c_{t+1} & = c_t \cos \lambda_c + c_t^* \sin \lambda_c + \tilde \omega_t \qquad & \tilde \omega_t \sim N(0, \sigma_{\tilde \omega}^2) \\\\
c_{t+1}^* & = -c_t \sin \lambda_c + c_t^* \cos \lambda_c + \tilde \omega_t^* & \tilde \omega_t^* \sim N(0, \sigma_{\tilde \omega}^2)
\end{align}
$$

The parameter $\lambda_c$ (the frequency of the cycle) is an additional parameter to be estimated by MLE. If the seasonal effect is stochastic, then there is one another parameter to estimate (the variance of the error term - note that both of the error terms here share the same variance, but are assumed to have independent draws).

### Irregular

The irregular component is assumed to be a white noise error term. Its variance is a parameter to be estimated by MLE; i.e.

$$
\varepsilon_t \sim N(0, \sigma_\varepsilon^2)
$$

In some cases, we may want to generalize the irregular component to allow for autoregressive effects:

$$
\varepsilon_t = \rho(L) \varepsilon_{t-1} + \epsilon_t, \qquad \epsilon_t \sim N(0, \sigma_\epsilon^2)
$$

In this case, the autoregressive parameters would also be estimated via MLE.

### Regression effects

We may want to allow for explanatory variables by including additional terms

$$
\sum_{j=1}^k \beta_j x_{jt}
$$

or for intervention effects by including

$$
\begin{align}
\delta w_t \qquad \text{where} \qquad w_t & = 0, \qquad t < \tau, \\\\
& = 1, \qquad t \ge \tau
\end{align}
$$

These additional parameters could be estimated via MLE or by including them as components of the state space formulation.

## Data

Following Harvey and Jaeger, we will consider the following time series:

- US real GNP, "output", ([GNPC96](https://research.stlouisfed.org/fred2/series/GNPC96))
- US GNP implicit price deflator, "prices", ([GNPDEF](https://research.stlouisfed.org/fred2/series/GNPDEF))
- US monetary base, "money", ([AMBSL](https://research.stlouisfed.org/fred2/series/AMBSL))

The time frame in the original paper varied across series, but was broadly 1954-1989. Below we use data from the period 1948-2008 for all series. Although the unobserved components approach allows isolating a seasonal component within the model, the series considered in the paper, and here, are already seasonally adjusted.

All data series considered here are taken from [Federal Reserve Economic Data (FRED)](https://research.stlouisfed.org/fred2/). Conveniently, the Python library [Pandas](https://pandas.pydata.org/) has the ability to download data from FRED directly.

```python
# Datasets
from pandas_datareader.data import DataReader

# Get the raw data
start = '1948-01'
end = '2008-01'
us_gnp = DataReader('GNPC96', 'fred', start=start, end=end)
us_gnp_deflator = DataReader('GNPDEF', 'fred', start=start, end=end)
us_monetary_base = DataReader('AMBSL', 'fred', start=start, end=end).resample('QS').mean()
recessions = DataReader('USRECQ', 'fred', start=start, end=end).resample('QS').last().values[:,0]

# Construct the dataframe
dta = pd.concat(map(np.log, (us_gnp, us_gnp_deflator, us_monetary_base)), axis=1)
dta.columns = ['US GNP','US Prices','US monetary base']
dta.index.freq = dta.index.inferred_freq
dates = dta.index._mpl_repr()
```

To get a sense of these three variables over the timeframe, we can plot them:

```python
# Plot the data
ax = dta.plot(figsize=(13,3))
ylim = ax.get_ylim()
ax.xaxis.grid()
ax.fill_between(dates, ylim[0]+1e-5, ylim[1]-1e-5, recessions, facecolor='k', alpha=0.1);
```

## Model

Since the data is already seasonally adjusted and there are no obvious explanatory variables, the generic model considered is:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{c_{t}}_{\text{cycle}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

The irregular will be assumed to be white noise, and the cycle will be stochastic and damped. The final modeling choice is the specification to use for the trend component. Harvey and Jaeger consider two models:

1. Local linear trend (the "unrestricted" model)
2. Smooth trend (the "restricted" model, since we are forcing $\sigma_\eta = 0$)

Below, we construct `kwargs` dictionaries for each of these model types. Notice that rather that there are two ways to specify the models. One way is to specify components directly, as in the table above. The other way is to use string names which map to various specifications.

```python
# Model specifications

# Unrestricted model, using string specification
unrestricted_model = {
    'level': 'local linear trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Unrestricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# local linear trend model with a stochastic damped cycle:
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': True, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }

# The restricted model forces a smooth trend
restricted_model = {
    'level': 'smooth trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Restricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# smooth trend model with a stochastic damped cycle. Notice
# that the difference from the local linear trend model is that
# `stochastic_level=False` here.
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': False, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }
```

We now fit the following models:

1. Output, unrestricted model
2. Prices, unrestricted model
3. Prices, restricted model
4. Money, unrestricted model
5. Money, restricted model

```python
# Output
output_mod = sm.tsa.UnobservedComponents(dta['US GNP'], **unrestricted_model)
output_res = output_mod.fit(method='powell', disp=False)

# Prices
prices_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **unrestricted_model)
prices_res = prices_mod.fit(method='powell', disp=False)

prices_restricted_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **restricted_model)
prices_restricted_res = prices_restricted_mod.fit(method='powell', disp=False)

# Money
money_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **unrestricted_model)
money_res = money_mod.fit(method='powell', disp=False)

money_restricted_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **restricted_model)
money_restricted_res = money_restricted_mod.fit(method='powell', disp=False)
```

Once we have fit these models, there are a variety of ways to display the information. Looking at the model of US GNP, we can summarize the fit of the model using the `summary` method on the fit object.

```python
print(output_res.summary())
```

                                Unobserved Components Results
    =====================================================================================
    Dep. Variable:                        US GNP   No. Observations:                  241
    Model:                    local linear trend   Log Likelihood                 769.632
                       + damped stochastic cycle   AIC                          -1527.263
    Date:                       Thu, 06 Aug 2020   BIC                          -1506.455
    Time:                               13:04:43   HQIC                         -1518.876
    Sample:                           01-01-1948
                                    - 01-01-2008
    Covariance Type:                         opg
    ====================================================================================
                           coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------------
    sigma2.irregular  1.091e-06   7.29e-06      0.150      0.881   -1.32e-05    1.54e-05
    sigma2.level      4.077e-06   4.93e-05      0.083      0.934   -9.26e-05       0.000
    sigma2.trend      2.962e-06    1.4e-06      2.112      0.035    2.14e-07    5.71e-06
    sigma2.cycle       3.91e-05   2.56e-05      1.525      0.127   -1.12e-05    8.94e-05
    frequency.cycle      0.4454      0.047      9.479      0.000       0.353       0.537
    damping.cycle        0.8685      0.042     20.732      0.000       0.786       0.951
    ===================================================================================
    Ljung-Box (L1) (Q):                   0.00   Jarque-Bera (JB):                 9.29
    Prob(Q):                              0.96   Prob(JB):                         0.01
    Heteroskedasticity (H):               0.27   Skew:                            -0.05
    Prob(H) (two-sided):                  0.00   Kurtosis:                         3.96
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

For unobserved components models, and in particular when exploring stylized facts in line with point (2) from the introduction, it is often more instructive to plot the estimated unobserved components (e.g. the level, trend, and cycle) themselves to see if they provide a meaningful description of the data.

The `plot_components` method of the fit object can be used to show plots and confidence intervals of each of the estimated states, as well as a plot of the observed data versus the one-step-ahead predictions of the model to assess fit.

```python
fig = output_res.plot_components(legend_loc='lower right', figsize=(15, 9));
```

Finally, Harvey and Jaeger summarize the models in another way to highlight the relative importances of the trend and cyclical components; below we replicate their Table I. The values we find are broadly consistent with, but different in the particulars from, the values from their table.

```python
# Create Table I
table_i = np.zeros((5,6))

start = dta.index[0]
end = dta.index[-1]
time_range = '%d:%d-%d:%d' % (start.year, start.quarter, end.year, end.quarter)
models = [
    ('US GNP', time_range, 'None'),
    ('US Prices', time_range, 'None'),
    ('US Prices', time_range, r'$\sigma_\eta^2 = 0$'),
    ('US monetary base', time_range, 'None'),
    ('US monetary base', time_range, r'$\sigma_\eta^2 = 0$'),
]
index = pd.MultiIndex.from_tuples(models, names=['Series', 'Time range', 'Restrictions'])
parameter_symbols = [
    r'$\sigma_\zeta^2$', r'$\sigma_\eta^2$', r'$\sigma_\kappa^2$', r'$\rho$',
    r'$2 \pi / \lambda_c$', r'$\sigma_\varepsilon^2$',
]

i = 0
for res in (output_res, prices_res, prices_restricted_res, money_res, money_restricted_res):
    if res.model.stochastic_level:
        (sigma_irregular, sigma_level, sigma_trend,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
    else:
        (sigma_irregular, sigma_level,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
        sigma_trend = np.nan
    period_cycle = 2 * np.pi / frequency_cycle

    table_i[i, :] = [
        sigma_level*1e7, sigma_trend*1e7,
        sigma_cycle*1e7, damping_cycle, period_cycle,
        sigma_irregular*1e7
    ]
    i += 1

pd.set_option('float_format', lambda x: '%.4g' % np.round(x, 2) if not np.isnan(x) else '-')
table_i = pd.DataFrame(table_i, index=index, columns=parameter_symbols)
table_i
```

      $\sigma_\zeta^2$
      $\sigma_\eta^2$
      $\sigma_\kappa^2$
      $\rho$
      $2 \pi / \lambda_c$
      $\sigma_\varepsilon^2$

      Series
      Time range
      Restrictions

      US GNP
      1948:1-2008:1
      None
      40.77
      29.62
      391
      0.87
      14.11
      10.91

      US Prices
      1948:1-2008:1
      None
      41.22
      32.84
      0.02
      0.92
      9.98
      8.3

      $\sigma_\eta^2 = 0$
      22.11
      -
      49.12
      0.89
      15.35
      5.03

      US monetary base
      1948:1-2008:1
      None
      70.08
      18.8
      192.1
      0.89
      23.21
      0

      $\sigma_\eta^2 = 0$
      18.94
      -
      247
      0.89
      23.83
      0


## Nonlinear Dimensionality Reduction
G. Richards (2016, 2018, 2020) based on materials from Connolly, Miller, Leighly, VanderPlas, and Ivezic Chapter 7.

Today we will talk about the concepts of
* manifold learning
* nonlinear dimensionality reduction

Specifically using the following algorithms
* local linear embedding (LLE)
* isometric mapping (IsoMap)
* t-distributed Stochastic Neighbor Embedding (t-SNE)

Let's start by my echoing the brief note of caution given in Adam Miller's notebook: "astronomers will often try to derive physical insight from PCA eigenspectra or eigentimeseries, but this is not advisable as there is no physical reason for the data to be linearly and orthogonally separable".  Moreover, physical components are (generally) positive definite.  So, PCA is great for dimensional reduction, but for doing physics there are generally better choices.

While NMF "solves" the issue of negative components, it is still a linear process.  For data with non-linear correlations, an entire field, known as [Manifold Learning](http://scikit-learn.org/stable/modules/manifold.html) and [nonlinear dimensionality reduction]( https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction), has been developed, with several algorithms available via the [`sklearn.manifold`](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.manifold) module.

For example, if your data set looks like this:

Then PCA is going to give you something like this.

Clearly not very helpful!

What you were really hoping for is something more like the results below.  For more examples see
[Vanderplas & Connolly 2009](http://iopscience.iop.org/article/10.1088/0004-6256/138/5/1365/meta;jsessionid=48A569862A424ECCAEECE2A900D9837B.c3.iopscience.cld.iop.org)

## Local Linear Embedding

[Local Linear Embedding](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.LocallyLinearEmbedding.html#sklearn.manifold.LocallyLinearEmbedding) attempts to embed high-$D$ data in a lower-$D$ space.  Crucially it also seeks to preserve the geometry of the local "neighborhoods" around each point.  In the case of the "S" curve, it seeks to unroll the data.  The steps are

Step 1: define local geometry
- local neighborhoods determined from $k$ nearest neighbors.
- for each point calculate weights that reconstruct a point from its $k$ nearest
neighbors via
$$
\begin{equation}
  \mathcal{E}_1(W) = \left|X - WX\right|^2,
\end{equation}
$$
where $X$ is an $N\times K$ matrix and $W$ is an $N\times N$ matrix that minimizes the reconstruction error.

Essentially this is finding the hyperplane that describes the local surface at each point within the data set. So, imagine that you have a bunch of square tiles and you are trying to tile the surface with them.

Step 2: embed within a lower dimensional space
- set all $W_{ij}=0$ except when point $j$ is one of the $k$ nearest neighbors of point $i$.
- $W$ becomes very sparse for $k \ll N$ (only $Nk$ entries in $W$ are non-zero).
- minimize
>$\begin{equation}
  \mathcal{E}_2(Y) = \left|Y - W Y\right|^2,
\end{equation}
$

with $W$ fixed to find an $N$ by $d$ matrix ($d$ is the new dimensionality).

Step 1 requires a nearest-neighbor search.

Step 2 requires an
eigenvalue decomposition of the matrix $C_W \equiv (I-W)^T(I-W)$.

LLE has been applied to data as diverse as galaxy spectra, stellar spectra, and photometric light curves.   It was introduced by [Roweis & Saul (2000)](https://www.ncbi.nlm.nih.gov/pubmed/11125150).

Skikit-Learn's call to LLE is as follows, with a more detailed example already being given above.

```python
import numpy as np
from sklearn.manifold import LocallyLinearEmbedding
X = np.random.normal(size=(1000,2)) # 1000 points in 2D
R = np.random.random((2,10)) # projection matrix
X = np.dot(X,R) # now a 2D linear manifold in 10D space
k = 5 # Number of neighbors to use in fit
n = 2 # Number of dimensions to fit
lle = LocallyLinearEmbedding(n_neighbors=k,n_components=n)
lle.fit(X)
proj = lle.transform(X) # 100x2 projection of the data
```

See what LLE does for the digits data, using the 7 nearest neighbors and 2 components.

```python
# Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print(grid_data)
X = digits.data
y = digits.target
```

```python
#LLE
from sklearn.manifold import LocallyLinearEmbedding
k = ____ # Number of neighbors to use in fit
n = ____ # Number of dimensions to fit
lle = LocallyLinearEmbedding(n_neighbors=____,n_components=____)
lle.fit(X)
X_reduced = lle.transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

## Isometric Mapping

is based on multi-dimensional scaling (MDS) framework.  It was introduced in the same volume of *Science* as the article above.  See [Tenenbaum, de Silva, & Langford (2000)](https://www.ncbi.nlm.nih.gov/pubmed/?term=A+Global+Geometric+Framework+for+Nonlinear+Dimensionality+Reduction).
Geodestic curves are used to recover non-linear structure.

In Scikit-Learn [IsoMap](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.Isomap.html) is implemented as follows:

```python
# Execute this cell
import numpy as np
from sklearn.manifold import Isomap
XX = np.random.normal(size=(1000,2)) # 1000 points in 2D
R = np.random.random((2,10)) # projection matrix
XX = np.dot(XX,R) # X is a 2D manifold in 10D space
k = 5 # number of neighbors
n = 2 # number of dimensions
iso = Isomap(n_neighbors=k,n_components=n)
iso.fit(XX)
proj = iso.transform(XX) # 1000x2 projection of the data
```

Try 7 neighbors and 2 dimensions on the digits data.

```python
# IsoMap
from sklearn.manifold import Isomap
k = 7 # Number of neighbors to use in fit
n = 2 # Number of dimensions to fit
iso = Isomap(n_neighbors=k,n_components=n)
iso.fit(X)
X_reduced = iso.____(____)

plt.scatter(____[:,0], ____[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

## t-SNE

[t-distributed Stochastic Neighbor Embedding (t-SNE)](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding) is not discussed in the book, Scikit-Learn does have a [t-SNE implementation](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) and it is well worth mentioning this manifold learning algorithm too.  SNE itself was developed by [Hinton & Roweis](http://www.cs.toronto.edu/~fritz/absps/sne.pdf) with the "$t$" part being added by [van der Maaten & Hinton](http://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf).  It works like the other manifold learning algorithms.

 Try it on the digits data.  You'll need to import `TSNE` from `sklearn.manifold`, instantiate it with 2 components, then do a `fit_transform` on the original data.

```python
# t-SNE
from sklearn.manifold import TSNE
tsne = ___(___=___,learning_rate=200)
X_reduced = ___.___(___)

plt.scatter(___, ___, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

You'll know if you have done it right if you understand Adam Miller's comment "Holy freakin' smokes.  That is magic.  (It's possible we just solved science)."

Personally, I think that some exclamation points may be needed in there!

What's even more illuminating is to make the plot using the actual digits to plot the points.  Then you can see why certain digits are alike or split into multiple regions.  Can you explain the patterns you see here?

```python
# Execute this cell
from matplotlib import offsetbox

#----------------------------------------------------------------------
# Scale and visualize the embedding vectors
def plot_embedding(X):
    x_min, x_max = np.min(X, 0), np.max(X, 0)
    X = (X - x_min) / (x_max - x_min)

    plt.figure(figsize=(10, 10))
    ax = plt.subplot(111)
    for i in range(X.shape[0]):
      #plt.text(X[i, 0], X[i, 1], str(digits.target[i]), color=plt.cm.Set1(y[i] / 10.), fontdict={'weight': 'bold', 'size': 9})
      plt.text(X[i, 0], X[i, 1], str(digits.target[i]), color=plt.cm.nipy_spectral(y[i]/9.))

    shown_images = np.array([[1., 1.]])  # just something big
    for i in range(digits.data.shape[0]):
        dist = np.sum((X[i] - shown_images) ** 2, 1)
        if np.min(dist) < 4e-3:
            # don't show points that are too close
            continue
        shown_images = np.r_[shown_images, [X[i]]]
        imagebox = offsetbox.AnnotationBbox(offsetbox.OffsetImage(digits.images[i], cmap=plt.cm.gray_r), X[i])
        ax.add_artist(imagebox)
    plt.xticks([]), plt.yticks([])

plot_embedding(X_reduced)
plt.show()
```

We often use dimensionality reduction to help with data visualization, so it seems appropriate to talk about some different ways to improve our data visualization using plotting tools. The first example is [seaborn.pairplot](https://seaborn.pydata.org/generated/seaborn.pairplot.html).

```python
import seaborn as sns
penguins = sns.load_dataset("penguins")
sns.pairplot(penguins)
```

The second example is an illustration of the power of [Bokeh](https://bokeh.org/).

```python
import numpy as np
from bokeh.plotting import *
from bokeh.models import ColumnDataSource

# prepare some data
N = 300
x0 = penguins['bill_depth_mm']
x1 = penguins['flipper_length_mm']
y = penguins['body_mass_g']
species = penguins['species']

# output to static HTML file
output_file("linked_brushing.html")

# NEW: create a column data source for the plots to share
source = ColumnDataSource(data=dict(x0=x0, x1=x1, y=y, species=species))

TOOLS = "pan,wheel_zoom,box_zoom,reset,save,box_select,lasso_select"

TOOLTIPS = [
    ("index", "$index"),
    ("(x,y)", "($x, $y)"),
    ("species", "@species"),
]

# create a new plot and add a renderer
left = figure(tools=TOOLS, tooltips=TOOLTIPS, width=350, height=350, title=None,\
             x_axis_label ="bill_depth_mm",y_axis_label ="body_mass_g")
left.circle('x0', 'y', source=source)

# create another new plot and add a renderer
right = figure(tools=TOOLS, tooltips=TOOLTIPS, width=350, height=350, title=None, \
              x_axis_label ="flipper_length_mm",y_axis_label ="body_mass_g")
right.circle('x1', 'y', source=source)

# put the subplots in a gridplot
p = gridplot([[left, right]])

# show the results
show(p,browser="chrome")
```

Finally for today -- if irises, penguins, or hand-written digits aren't your thing, then have a look through some of these public data repositories:

- [https://github.com/caesar0301/awesome-public-datasets?utm_content=buffer4245d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer](https://github.com/caesar0301/awesome-public-datasets?utm_content=buffer4245d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer)
- [http://www.datasciencecentral.com/m/blogpost?id=6448529%3ABlogPost%3A318739](http://www.datasciencecentral.com/m/blogpost?id=6448529%3ABlogPost%3A318739)
- [http://www.kdnuggets.com/2015/04/awesome-public-datasets-github.html](http://www.kdnuggets.com/2015/04/awesome-public-datasets-github.html)


```python
import os, sys
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from varform1D import solver
import sympy as sym
x, b = sym.symbols('x b')
f = 1

# Compute basis functions and their derivatives
N = 4
psi = {0: [x**(i+1) for i in range(N+1)]}
psi[1] = [sym.diff(psi_i, x) for psi_i in psi[0]]

# Galerkin

def integrand_lhs(psi, i, j):
    return psi[1][i]*psi[1][j]

def integrand_rhs(psi, i):
    return -f*psi[0][i]

Omega = [0, 1]

u, c = solver(integrand_lhs, integrand_rhs, psi, Omega,
              verbose=True, symbolic=True)
print(('Galerkin solution u:', sym.simplify(sym.expand(u))))

# Least squares
psi = {0: [x**(i+2) for i in range(N+1)]}
psi[1] = [sym.diff(psi_i, x) for psi_i in psi[0]]
psi[2] = [sym.diff(psi_i, x) for psi_i in psi[1]]

def integrand_lhs(psi, i, j):
    return psi[2][i]*psi[2][j]

def integrand_rhs(psi, i):
    return -f*psi[2][i]

Omega = [0, 1]

```

    ...evaluating matrix... (0,0): 1
    (0,1): 2*x
    (0,2): 3*x**2
    (0,3): 4*x**3
    (0,4): 5*x**4
    rhs: -x
    (1,1): 4*x**2
    (1,2): 6*x**3
    (1,3): 8*x**4
    (1,4): 10*x**5
    rhs: -x**2
    (2,2): 9*x**4
    (2,3): 12*x**5
    (2,4): 15*x**6
    rhs: -x**3
    (3,3): 16*x**6
    (3,4): 20*x**7
    rhs: -x**4
    (4,4): 25*x**8
    rhs: -x**5

    A:
     Matrix([[1, 1, 1, 1, 1], [1, 4/3, 3/2, 8/5, 5/3], [1, 3/2, 9/5, 2, 15/7], [1, 8/5, 2, 16/7, 5/2], [1, 5/3, 15/7, 5/2, 25/9]])
    b:
     Matrix([[-1/2], [-1/3], [-1/4], [-1/5], [-1/6]])
    coeff: [-1, 1/2, 0, 0, 0]
    approximation: x**2/2 - x
    ('Galerkin solution u:', x*(x - 2)/2)

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/06_CABLE_XN.ipynb


```python
%matplotlib inline
```

# The Johnson-Lindenstrauss bound for embedding with random projections

The `Johnson-Lindenstrauss lemma`_ states that any high dimensional
dataset can be randomly projected into a lower dimensional Euclidean
space while controlling the distortion in the pairwise distances.

Theoretical bounds
==================

The distortion introduced by a random projection `p` is asserted by
the fact that `p` is defining an eps-embedding with good probability
as defined by:

\begin{align}(1 - eps) \|u - v\|^2 < \|p(u) - p(v)\|^2 < (1 + eps) \|u - v\|^2\end{align}

Where u and v are any rows taken from a dataset of shape [n_samples,
n_features] and p is a projection by a random Gaussian N(0, 1) matrix
with shape [n_components, n_features] (or a sparse Achlioptas matrix).

The minimum number of components to guarantees the eps-embedding is
given by:

\begin{align}n\_components >= 4 log(n\_samples) / (eps^2 / 2 - eps^3 / 3)\end{align}

The first plot shows that with an increasing number of samples ``n_samples``,
the minimal number of dimensions ``n_components`` increased logarithmically
in order to guarantee an ``eps``-embedding.

The second plot shows that an increase of the admissible
distortion ``eps`` allows to reduce drastically the minimal number of
dimensions ``n_components`` for a given number of samples ``n_samples``

Empirical validation
====================

We validate the above bounds on the digits dataset or on the 20 newsgroups
text document (TF-IDF word frequencies) dataset:

- for the digits dataset, some 8x8 gray level pixels data for 500
  handwritten digits pictures are randomly projected to spaces for various
  larger number of dimensions ``n_components``.

- for the 20 newsgroups dataset some 500 documents with 100k
  features in total are projected using a sparse random matrix to smaller
  euclidean spaces with various values for the target number of dimensions
  ``n_components``.

The default dataset is the digits dataset. To run the example on the twenty
newsgroups dataset, pass the --twenty-newsgroups command line argument to this
script.

For each value of ``n_components``, we plot:

- 2D distribution of sample pairs with pairwise distances in original
  and projected spaces as x and y axis respectively.

- 1D histogram of the ratio of those distances (projected / original).

We can see that for low values of ``n_components`` the distribution is wide
with many distorted pairs and a skewed distribution (due to the hard
limit of zero ratio on the left as distances are always positives)
while for larger values of n_components the distortion is controlled
and the distances are well preserved by the random projection.

Remarks
=======

According to the JL lemma, projecting 500 samples without too much distortion
will require at least several thousands dimensions, irrespective of the
number of features of the original dataset.

Hence using random projections on the digits dataset which only has 64 features
in the input space does not make sense: it does not allow for dimensionality
reduction in this case.

On the twenty newsgroups on the other hand the dimensionality can be decreased
from 56436 down to 10000 while reasonably preserving pairwise distances.

```python
print(__doc__)

import sys
from time import time
import numpy as np
import matplotlib.pyplot as plt
from sklearn.random_projection import johnson_lindenstrauss_min_dim
from sklearn.random_projection import SparseRandomProjection
from sklearn.datasets import fetch_20newsgroups_vectorized
from sklearn.datasets import load_digits
from sklearn.metrics.pairwise import euclidean_distances

# Part 1: plot the theoretical dependency between n_components_min and
# n_samples

# range of admissible distortions
eps_range = np.linspace(0.1, 0.99, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(eps_range)))

# range of number of samples (observation) to embed
n_samples_range = np.logspace(1, 9, 9)

plt.figure()
for eps, color in zip(eps_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples_range, eps=eps)
    plt.loglog(n_samples_range, min_n_components, color=color)

plt.legend(["eps = %0.1f" % eps for eps in eps_range], loc="lower right")
plt.xlabel("Number of observations to eps-embed")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_samples vs n_components")

# range of admissible distortions
eps_range = np.linspace(0.01, 0.99, 100)

# range of number of samples (observation) to embed
n_samples_range = np.logspace(2, 6, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(n_samples_range)))

plt.figure()
for n_samples, color in zip(n_samples_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples, eps=eps_range)
    plt.semilogy(eps_range, min_n_components, color=color)

plt.legend(["n_samples = %d" % n for n in n_samples_range], loc="upper right")
plt.xlabel("Distortion eps")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_components vs eps")

# Part 2: perform sparse random projection of some digits images which are
# quite low dimensional and dense or documents of the 20 newsgroups dataset
# which is both high dimensional and sparse

if '--twenty-newsgroups' in sys.argv:
    # Need an internet connection hence not enabled by default
    data = fetch_20newsgroups_vectorized().data[:500]
else:
    data = load_digits().data[:500]

n_samples, n_features = data.shape
print("Embedding %d samples with dim %d using various random projections"
      % (n_samples, n_features))

n_components_range = np.array([300, 1000, 10000])
dists = euclidean_distances(data, squared=True).ravel()

# select only non-identical samples pairs
nonzero = dists != 0
dists = dists[nonzero]

for n_components in n_components_range:
    t0 = time()
    rp = SparseRandomProjection(n_components=n_components)
    projected_data = rp.fit_transform(data)
    print("Projected %d samples from %d to %d in %0.3fs"
          % (n_samples, n_features, n_components, time() - t0))
    if hasattr(rp, 'components_'):
        n_bytes = rp.components_.data.nbytes
        n_bytes += rp.components_.indices.nbytes
        print("Random matrix with size: %0.3fMB" % (n_bytes / 1e6))

    projected_dists = euclidean_distances(
        projected_data, squared=True).ravel()[nonzero]

    plt.figure()
    plt.hexbin(dists, projected_dists, gridsize=100, cmap=plt.cm.PuBu)
    plt.xlabel("Pairwise squared distances in original space")
    plt.ylabel("Pairwise squared distances in projected space")
    plt.title("Pairwise distances distribution for n_components=%d" %
              n_components)
    cb = plt.colorbar()
    cb.set_label('Sample pairs counts')

    rates = projected_dists / dists
    print("Mean distances rate: %0.2f (%0.2f)"
          % (np.mean(rates), np.std(rates)))

    plt.figure()
    plt.hist(rates, bins=50, normed=True, range=(0., 2.), edgecolor='k')
    plt.xlabel("Squared distances rate: projected / original")
    plt.ylabel("Distribution of samples pairs")
    plt.title("Histogram of pairwise distance rates for n_components=%d" %
              n_components)

    # TODO: compute the expected value of eps and add them to the previous plot
    # as vertical lines / region

plt.show()
```


```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

Linear regression and logistic regression can be viewed as simplest neural network without hidden layer.

The basic steps are similar: **feedforward**, **compute cost**, **Backpropagate**, **update weights**

Neural network units can perform as logic gates, and with billion logic gates we can build a computer.

| x1| x2|and|or |nand|nor|xor|
|:--:|:-:|:-:|:-:|:-:|:-:|:-:|
| 0 | 0 | 0 | 0 | 1 | 1 | 0 |
| 0 | 1 | 0 | 1 | 1 | 0 | 1 |
| 1 | 0 | 0 | 1 | 1 | 0 | 1 |
| 1 | 1 | 1 | 1 | 0 | 0 | 0 |

## Neural network

Build logic gates with neural network (sigmoid as active function).

### Step 1: single unit

### 1. Feedforward
\begin{equation}
\mathbf{z} = XW
\end{equation}

\begin{equation}
\mathbf{\hat{y}} = \sigma(\mathbf{z})
\end{equation}

\begin{equation}
\sigma(z) = \frac{1}{1+e^{-z}}
\end{equation}

### 2. Compute cost function
\begin{equation}
L(\mathbf{y}, \mathbf{\hat{y}}) = -\frac{1}{m}\big(\mathbf{y}^T\log(\mathbf{\hat{y}})+(1-\mathbf{y})^T\log(1-\mathbf{\hat{y}})\big)
\end{equation}

### 3. Backpropagation
\begin{equation}
\frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial W}
= \frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial \mathbf{\hat{y}}} \cdot \frac{\partial \mathbf{\hat{y}}}{\partial W}
= \frac{1}{m}X^T(\mathbf{\hat{y}} - \mathbf{y})
\end{equation}

### 4. Gradient descent
\begin{equation}
W = W - \alpha  \frac{\delta Loss(y, \hat{y})}{\delta W}
\end{equation}

```python
class NeuralNetwork:
    """ Simple neural network """
    def __init__(self):
        pass

    def _init_params(self, x, y, iterations, learning_rate):
        """ Initilize parameters.

        ----------
        W : ndarray, shape (n_features+1,)
            Coefficient vector
        """
        self._X = np.hstack([np.ones((x.shape[0], 1)), x])
        self._y = y
        self._learning_rate = learning_rate

        self.weights_ = np.random.rand(self._X.shape[1], self._y.shape[1])
        self.costs_ = np.zeros(iterations)

    def _sigmoid(self, x):
        """ Computes sigmoid function. """
        return 1.0/(1 + np.exp(-x))

    def _feedforward(self):
        """ Computes np.dot(X, W). """
        self.output = self._sigmoid(self._X.dot(self.weights_))

    def _backprop(self):
        """ Update weights. """
        self.weights_ -= self._learning_rate * self._X.T.dot(self.output-self._y)/len(self._y)

    def _get_cost(self):
        """ Compute loss. """
        return -1/len(self._y) *(self._y.T.dot(np.log(self.output))+(1-self._y.T).dot(np.log(1-self.output)))

    def fit(self, x, y, iterations=1000, learning_rate=0.5):
        """ Fit model.

        ----------
        x : ndarray, shape (n_samples, n_features)
            Training data
        y : ndarray, shape (n_samples,)
            Target data
        """
        self._init_params(x, y, iterations, learning_rate)

        # train model
        for i in range(iterations):
            self._feedforward()
            self._backprop()
            self.costs_[i] = self._get_cost()

        return self
```

```python
# input vector
x = np.array([[0,0], [0,1], [1,0], [1,1]])
# logic gates
gates = {
    'AND':  np.array([[0],[0],[0],[1]]),
    'OR':   np.array([[0],[1],[1],[1]]),
    'NAND': np.array([[1],[1],[1],[0]]),
    'NOR':  np.array([[1],[0],[0],[0]]),
    'XOR':  np.array([[0],[1],[1],[0]])
}

for key,value in gates.items():
    nn = NeuralNetwork()
    model = nn.fit(x, value)
    print(f"{key}:\n", model.output)
    print(f'loss: {model.costs_[-1]:.2f}')
```

Neural network without hidden layer can be viewed as logistic regression unit. It's a simple linear classifier.

## Neural network

Build logic gates with neural network.

### Step 2: with 1 hidden layer

### 1. Feedforward
\begin{equation}
z_1 = W_1^TX
\end{equation}

\begin{equation}
a_1 = \sigma(z_1)
\end{equation}

\begin{equation}
z_2 = W_2^Ta_1
\end{equation}

\begin{eqnarray}
\hat{y} &=& \sigma(z_2) \\
&=& \sigma(W_2^T\sigma(W_1^TX))
\end{eqnarray}

\begin{equation}
\sigma(z) = \frac{1}{1+e^{-z}}
\end{equation}

### 2. Compute cost function
\begin{equation}
Loss(y, \hat{y}) = \frac{1}{2n}(\hat{y} - y)^T \cdot (\hat{y} - y)
\end{equation}

### 3. Backpropagation
\begin{eqnarray}
\frac{\delta Loss(y, \hat{y})}{\delta W_2} &=& \frac{\delta Loss(y, \hat{y})}{\delta \hat{y}}\cdot\frac{\delta \hat{y}}{\delta z_2}\cdot\frac{\delta z_2}{\delta W_2} \\
&=& \frac{1}{n}(\hat{y}-y)\cdot \hat{y}(1-\hat{y})\cdot a_1
\end{eqnarray}

\begin{eqnarray}
\frac{\delta Loss(y, \hat{y})}{\delta W_1} &=& \frac{\delta Loss(y, \hat{y})}{\delta \hat{y}}\cdot\frac{\delta \hat{y}}{\delta z_2}\cdot\frac{\delta z_2}{\delta y_1}\cdot\frac{\delta y_1}{\delta z_1}\cdot\frac{\delta z_1}{\delta W_1} \\
&=& \frac{1}{n}(\hat{y}-y)\cdot \hat{y}(1-\hat{y})\cdot W_2 \cdot a_1(1-a_1)\cdot X
\end{eqnarray}

### 4. Gradient descent
\begin{equation}
W_1 = W_1 - \alpha  \frac{\delta Loss(y, \hat{y})}{\delta W_1}
\end{equation}

\begin{equation}
W_2 = W_2 - \alpha  \frac{\delta Loss(y, \hat{y})}{\delta W_2}
\end{equation}

```python
class NeuralNetwork:
    """ Simple neural network with 1 hidden layer """
    def __init__(self, neurons):
        self._layers = len(neurons)
        self._weights = [np.random.randn(nex, pre+1) for pre, nex in zip(neurons[:-1], neurons[1:])]

    def _init_params(self, x, y, iterations, learning_rate):
        """ Initilize parameters. """
        self._x = x.T
        self._y = y.T
        self._learning_rate = learning_rate
        self.costs_ = np.zeros(iterations)

    def _sigmoid(self, x):
        """ Computes sigmoid function. """
        return 1.0/(1 + np.exp(-x))

    def _feedforward(self, a):
        a_s = []
        for w in self._weights:
            a = np.vstack([np.ones((1, a.shape[1])), a])
            a_s.append(a)
            z = w.dot(a)
            a = self._sigmoid(z)
        a_s.append(a)
        return a_s

    def _backprop(self, a_s):
        """ Update weights. """
        delta_weights = [np.zeros(w.shape) for w in self._weights]
        # Update last layer delta
        delta = a_s[-1] - self._y
        delta_weights[-1] = delta.dot(a_s[-2].T)
        # Update all but the last layer delta
        for L in range(2, self._layers):
            # [1:] Ignore bias term
            delta = (self._weights[-L+1].T.dot(delta)*a_s[-L]*(1-a_s[-L]))[1:]
            delta_weights[-L] = delta.dot(a_s[-L-1].T)
        # Update all weights
        self._weights = [w - self._learning_rate * dw for w, dw in zip(self._weights, delta_weights)]

    def _get_cost(self, a):
        """ Compute loss. """
        m = self._y.shape[1]
        return 0.5/m * np.sum((a - self._y)**2)

    def fit(self, x, y, iterations=1000, learning_rate=0.5):
        """ Fit model.

        ----------
        x : ndarray, shape (n_samples, n_features)
            Training data
        y : ndarray, shape (n_samples,)
            Target data
        """
        self._init_params(x, y, iterations, learning_rate)

        # train model
        for i in range(iterations):
            a_s = self._feedforward(self._x)
            self._backprop(a_s)
            # Check output
            self.output = a_s[-1].T
            self.costs_[i] = self._get_cost(a_s[-1])

        return self
```

```python
hidden_neurons = 2

# input vector
x = np.array([[0,0], [0,1], [1,0], [1,1]])
# logic gates
gates = {
    'AND':  np.array([[0],[0],[0],[1]]),
    'OR':   np.array([[0],[1],[1],[1]]),
    'NAND': np.array([[1],[1],[1],[0]]),
    'NOR':  np.array([[1],[0],[0],[0]]),
    'XOR':  np.array([[0],[1],[1],[0]])
}

for key,value in gates.items():
    x_features = x.shape[1]
    y_features = value.shape[1]
    nn = NeuralNetwork([x_features, hidden_neurons, y_features])
    model = nn.fit(x, value)
    print(f"{key}:\n", model.output)
    print(f'loss: {model.costs_[-1]:.2f}')

plt.figure(figsize=(10,5))
ax = plt.gca()
ax.grid(color='#b7b7b7', linestyle='-', linewidth=0.5, alpha=0.5)
plt.plot(model.costs_, color='#212121')
```