###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2018 L.A. Barba, G.F. Forsyth [CFD Python](https://github.com/barbagroup/CFDPython#cfd-python), (c)2014 L.A. Barba, I. Hawke, B. Knaepen [Practical Numerical Methods with Python](https://github.com/numerical-mooc/numerical-mooc#practical-numerical-methods-with-python), also under CC-BY.

```python
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, 'r').read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 2D Poisson Equation

In the [previous notebook](http://nbviewer.ipython.org/urls/github.com/daniel-koehn/Differential-equations-earth-system/tree/master/07_BVP_2D/01_2D_Laplace_Equation.ipynb), you learned to use Jacobi iterations to solve Laplace's equation. The iterations *relax* the solution from an initial guess to the final, steady-state solution. You also saw again that the way we treat boundary conditions can influence our solution. Using a first-order approximation of the Neumann boundary messed up our spatial convergence in the whole domain! (We expected second-order spatial convergence from the central difference scheme, but we got closer to first order.) This was easily fixed by using a second-order scheme for the Neumann boundary. *It's always good to check that you get the expected order of convergence.*

A word of warning: in this course module, we will introduce a different use of the word *"convergence"*. Before, we used it to refer to the decay of the truncation errors (in space and time) with a decrease in the grid spacing ($\Delta x$ and $\Delta t$). Now, we also have a relaxation scheme, and we use the word convergence to mean that the iterative solution approaches the exact solution of the linear system. Sometimes, this is called *algebraic convergence*. We'll concern ourselves with this in the next lesson. But first, let's play with Poisson.

## Poisson equation

The **Poisson equation** has a forcing function that drives the solution to its steady state. Unlike the Laplace equation, Poisson's equation involves imposed values inside the field (a.k.a., sources):

$$
\frac{\partial ^2 p}{\partial x^2} + \frac{\partial ^2 p}{\partial y^2} = b \tag{1}
$$

In discretized form, this looks almost the same as [the Laplace Equation](http://nbviewer.ipython.org/urls/github.com/daniel-koehn/Differential-equations-earth-system/tree/master/07_BVP_2D/01_2D_Laplace_Equation.ipynb), except for the source term on the right-hand side:

$$
\frac{p_{i+1,j}^{k}-2p_{i,j}^{k}+p_{i-1,j}^{k}}{\Delta x^2}+\frac{p_{i,j+1}^{k}-2 p_{i,j}^{k}+p_{i,j-1}^{k}}{\Delta y^2}=b_{i,j}^{k} \notag
$$

As before, we rearrange this to obtain an equation for $p$ at point $i,j$, based on its neighbors:

$$
p_{i,j}^{k+1}=\frac{(p_{i+1,j}^{k}+p_{i-1,j}^{k})\Delta y^2+(p_{i,j+1}^{k}+p_{i,j-1}^{k})\Delta x^2-b_{i,j}^{k}\Delta x^2\Delta y^2}{2(\Delta x^2+\Delta y^2)} \tag{2}
$$

It's slightly more complicated than the Laplace equation, but nothing we can't handle.

### An example problem

Let's consider the following Poisson equation:

$$
\begin{equation}
\nabla^2 p = -2\left(\frac{\pi}{2}\right)^2\sin\left( \frac{\pi x}{L_x} \right) \cos\left(\frac{\pi y}{L_y}\right) \notag
\end{equation}
$$

in the domain

$$
\left\lbrace \begin{align*}
0 &\leq x\leq 1  \\
-0.5 &\leq y \leq 0.5
\end{align*} \right.
$$

where $L_x = L_y = 1$ and with Dirichlet boundary conditions

$$p=0 \text{ at } \left\lbrace
\begin{align*}
x&=0\\
y&=0\\
y&=-0.5\\
y&=0.5
\end{align*} \right.$$

To solve this equation, we assume an initial state of $p=0$ everywhere, apply the boundary conditions and then iteratively relax the system until we converge on a solution.

To start, let's import the libraries and set up our spatial mesh.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline

# import JIT from Numba
from numba import jit
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
# Set parameters.
nx = 41  # number of points in the x direction
ny = 41  # number of points in the y direction
xmin, xmax = 0.0, 1.0  # domain limits in the x direction
ymin, ymax = -0.5, 0.5  # domain limits in the y direction
Lx = (xmax - xmin)  # domain length in the x direction
Ly = (ymax - ymin)  # domain length in the y direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dy = Ly / (ny - 1)  # grid spacing in the y direction

# Create the gridline locations.
x = numpy.linspace(xmin, xmax, num=nx)
y = numpy.linspace(ymin, ymax, num=ny)
```

```python
def poisson_source(x, y, Lx, Ly):
    """
    Computes and returns the source term (right-hand side)
    of the Poisson equation.

    Parameters
    ----------
    x : numpy.ndarray
        The gridline locations in the x direction
        as a 1D array of floats.
    y : numpy.ndarray
        The gridline locations in the y direction
        as a 1D array of floats.
    Lx : float
        Domain length in the x direction.
    Ly : float
        Domain length in the y direction.

    Returns
    -------
    b : numpy.ndarray of floats
        The forcing function as a 2D array.
    """
    X, Y = numpy.meshgrid(x, y)
    b = (-2.0 * numpy.pi / Lx * numpy.pi / Ly *
         numpy.sin(numpy.pi * X / Lx) *
         numpy.cos(numpy.pi * Y / Ly))
    return b
```

The Jacobi iterations need an exit condition, based on some norm of the difference between two consecutive iterations. We can use the same relative L2-norm that we wrote for the Laplace exit condition, so we saved the function into a helper Python file (`helper.py`) for easy importing.

```python
from helper import l2_norm
```

Now, what value to choose for the exit condition? We saw in the previous notebook that with an exit tolerance of $10^{-8}$, we could converge well for the different grids we tried, and observe second-order spatial convergence (with the second-order Neumann BC). We speculated in the end that we might be able to use a less stringent exit tolerance, since the spatial error was a lot larger (around $0.0002$ for the finer grid). Here, we'll try with $2\times 10^{-7}$. Go ahead and try with different values and see what you get!

It's time to write the function to solve the Poisson equation.

First, we put the spatial finite difference approximation of the Poisson equation (2) into a function `poisson` and apply Just-In-Time (JIT) compilation from the Numba package.

```python
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def poisson(p,pn,b,nx,ny,dx,dy):

    for i in range(1,nx-1):
        for j in range(1,ny-1):

            # FD solution of Poisson eq. (2)
            p[j, i] = (((pn[j, i+1] + pn[j, i-1]) * dy**2 +
                        (pn[j+1, i] + pn[j-1, i]) * dx**2 -
                              b[j, i] * dx**2 * dy**2) /
                             (2.0 * (dx**2 + dy**2)))

    return p
```

Finally, we code the solution of the Laplace equation using the Jacobi method in a function `poisson_2d_jacobi`.

Notice that all of the boundaries in this problem are Dirichlet boundaries, so no BC updates required!

There's also one extra piece we're adding in here.  To later examine the convergence of the iterative process, we will save the L2-norm of the difference between successive solutions. A plot of this quantity with respect to the iteration number will be an indication of how fast the relaxation scheme is converging.

```python
def poisson_2d_jacobi(p0, b, dx, dy, maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Poisson equation for a given forcing term
    using Jacobi relaxation method.

    The function assumes Dirichlet boundary conditions with value zero.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    b : numpy.ndarray
        The forcing term as a 2D array of floats.
    dx : float
        Grid spacing in the x direction.
    dy : float
        Grid spacing in the y direction.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    conv : list
        The convergence history as a list of floats.
    """

    # estimate nx, ny
    ny, nx = numpy.shape(p0)

    # copy initial guess p0 -> p
    p = p0.copy()

    conv = []  # convergence history
    diff = rtol + 1.0  # initial difference
    ite = 0  # iteration index

    # Jacobi iteration
    while diff > rtol and ite < maxiter:

        # store old p-values in pn
        pn = p.copy()

        # Update the solution at interior points of the Poisson equation
        poisson(p,pn,b,nx,ny,dx,dy)

        # Dirichlet boundary conditions at automatically enforced.
        # Compute and record the relative L2-norm of the difference.
        diff = l2_norm(p, pn)
        conv.append(diff)
        ite += 1

    return p, ite, conv
```

We can use the `plot_3d` function we wrote in the previous notebook to explore the field $p$, before and after the relaxation. We saved this plotting function into the helper Python file, so we can re-use it here.

```python
from helper import plot_3d
```

Now we initialize all of the problem variables and plot!

```python
# Set the initial conditions.
p0 = numpy.zeros((ny, nx))

# Compute the source term.
b = poisson_source(x, y, Lx, Ly)

# Plot the initial scalar field.
plot_3d(x, y, p0)
```

That looks suitably boring. Zeros everywhere and boundaries held at zero. If this were a Laplace problem we would already be done!

But the Poisson problem has a source term that will evolve this zero initial guess to something different. Let's run our relaxation scheme and see what effect the forcing function has on `p`.

```python
# Compute the solution using Jacobi relaxation method.
p, ites, conv = poisson_2d_jacobi(p0, b, dx, dy, rtol=2e-7)
print('Jacobi relaxation: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(conv[-1]))
```

    Jacobi relaxation: 3125 iterations to reach a relative difference of 1.9958631078740742e-07

It took 3,125 iterations to converge to the exit criterion (that's quite a lot, don't you think? Let's now take a look at a plot of the final field:

```python
# Plot the solution.
plot_3d(x, y, p)
```

Something has definitely happened. That looks good, but what about the error? This problem has the following analytical solution:

$$
\begin{equation}
p(x,y) = \sin{\left(\frac{x\pi}{L_x} \right)}\cos{\left(\frac{y\pi}{L_y} \right)} \notag
\end{equation}
$$

Time to compare the calculated solution to the analytical one. Let's do that.

```python
def poisson_solution(x, y, Lx, Ly):
    """
    Computes and returns the analytical solution of the Poisson equation
    on a given two-dimensional Cartesian grid.

    Parameters
    ----------
    x : numpy.ndarray
        The gridline locations in the x direction
        as a 1D array of floats.
    y : numpy.ndarray
        The gridline locations in the y direction
        as a 1D array of floats.
    Lx : float
        Length of the domain in the x direction.
    Ly : float
        Length of the domain in the y direction.

    Returns
    -------
    p : numpy.ndarray
        The analytical solution as a 2D array of floats.
    """
    X, Y = numpy.meshgrid(x, y)
    p = numpy.sin(numpy.pi * X / Lx) * numpy.cos(numpy.pi * Y / Ly)
    return p
```

```python
# Compute the analytical solution.
p_exact = poisson_solution(x, y, Lx, Ly)
```

```python
# Compute the relative L2-norm of the error.
l2_norm(p, p_exact)
```

    0.00044962635351969427

That seems small enough. Of course, each application problem can have different accuracy requirements.

### Algebraic convergence

Remember that we saved the L2-norm of the difference between two consecutive iterations. The purpose of that was to look at how the relaxation scheme *converges*, in algebraic sense: with consecutive solutions getting closer and closer to each other. Let's use a line plot for this.

```python
# Plot the convergence history.
pyplot.figure(figsize=(9.0, 4.0))
pyplot.xlabel('Iterations')
pyplot.ylabel('Relative $L_2$-norm\nof the difference')
pyplot.grid()
pyplot.semilogy(conv, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0, len(conv));
```

It looks like in the beginning, iterations started converging pretty fast, but they quickly adopted a slower rate. As we saw before, it took more than 3,000 iterations to get to our target difference between two consecutive solutions (in L2-norm). That is a *lot* of iterations, and we would really like to relax faster! No worries, we'll learn to do that in the next notebook.

### Spatial convergence

For a sanity check, let's make sure the solution is achieving the expected second-order convergence in space.

```python
# List of the grid sizes to investigate.
nx_values = [11, 21, 41, 81]

# Create an empty list to record the error on each grid.
errors = []

# Compute the solution and error for each grid size.
for nx in nx_values:
    ny = nx  # same number of points in all directions
    dx = Lx / (nx - 1)  # grid spacing in the x direction
    dy = Ly / (ny - 1)  # grid spacing in the y direction
    # Create the gridline locations.
    x = numpy.linspace(xmin, xmax, num=nx)
    y = numpy.linspace(ymin, ymax, num=ny)
    # Set the initial conditions.
    p0 = numpy.zeros((ny, nx))
    # Compute the source term.
    b = poisson_source(x, y, Lx, Ly)
    # Relax the solution.
    # We do not return number of iterations
    # or the convergence history.
    p, ites, _ = poisson_2d_jacobi(p0, b, dx, dy, rtol=2e-7)
    print('[nx = {}] Number of Jacobi iterations: {}'.format(nx, ites))
    # Compute the analytical solution.
    p_exact = poisson_solution(x, y, Lx, Ly)
    # Compute and record the relative L2-norm of the error.
    errors.append(l2_norm(p, p_exact))
```

    [nx = 11] Number of Jacobi iterations: 249
    [nx = 21] Number of Jacobi iterations: 892
    [nx = 41] Number of Jacobi iterations: 3125
    [nx = 81] Number of Jacobi iterations: 10708

```python
# Plot the error versus the grid-spacing size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.xlabel(r'$\Delta x$')
pyplot.ylabel('Relative $L_2$-norm\nof the error')
pyplot.grid()
dx_values = Lx / (numpy.array(nx_values) - 1)
pyplot.loglog(dx_values, errors,
              color='black', linestyle='--', linewidth=2, marker='o')
pyplot.axis('equal');
```

That looks pretty much second order! Remember that the boundary conditions can adversely affect convergence, but Dirichlet boundaries are "exact" and will never impact your convergence.

## Final word

We have used the difference between two consecutive solutions in the iterative process as a way to indicate convergence. However, this is *not* in general the best idea. For some problems and some iterative methods, you could experience iterates *stagnating* but the solution *not converging*.

Convergence of an iterative solution of a system $A \mathbf{x} = \mathbf{b}$ means that:

$$
\begin{equation}
\lim_{k \rightarrow \infty} \mathbf{x}^k = \mathbf{x} \notag
\end{equation}
$$

The error in the solution is actually $\mathbf{x}-\mathbf{x}^k$, but we're looking at $\mathbf{x}^{k+1}-\mathbf{x}^k$ for our exit criterion. They are not the same thing and the second could tend to zero (or machine precision) without the first being comparably small.

A discussion of better ways to apply stopping criteria for iterative methods is a more advanced topic than we want cover in this course module. Just keep this in mind as you continue your exploration of numerical methods in the future!


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import control
from ipywidgets import interact

```

# Error de Estado Estacionario

En el bucle de control, el error se calcula como la diferencia entre la referencia y la variable controlada.

Los cambios en la señal de referencia provocan errores durante los transitorios y pueden generar un error estacionario.

Considerando un sistema de control con realimentación unitaria y función de transferencia de lazo abierto $G(s) = G_c(s)G_a(s)G_p(s)$:

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^N(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)}$$

Este sistema tiene:
- $c$ ceros definidos por $T_{c1}$, $T_{c2}$, $T_{c3}$, ..., $T_{cc}$
- $N$ polos en $0$
- $p$ polos definidos por $T_{p1}$, $T_{p2}$, $T_{p3}$, ..., $T_{pp}$

La función de transferencia de lazo cerrado desde la referencia hacia la variable controlada es:

$$\frac{C(s)}{R(s)} = \frac{G(s)}{1 + G_(s)}$$

La función de transferencia de lazo cerrado desde la referencia hacia el error es:

$$\frac{E(s)}{R(s)} = \frac{R(s)-C(s)}{R(s)} = 1- \frac{C(s)}{R(s)} = 1 - \frac{G(s)}{1 + G(s)} = \frac{1}{1+G(s)}$$

El **error de estado estacionario** puede calcularse como:
$$E(s) = \int_0^\infty e(t)e^{-st}dt$$

$$E_{ee} = \lim_{t \rightarrow \infty} e(t) = \lim_{s \rightarrow 0} sE(s) = \lim_{s \rightarrow 0}\frac{sR(s)}{1+G(s)}$$

## Error de estado estacionario ante una entrada escalón (posición)

Dada esta señal como referencia:

$$R(s) = \frac{1}{s}$$

y

$$E_{ee} = \lim_{s \rightarrow 0}\frac{s \frac{1}{s} }{1+G(s)} = \frac{1}{1+ \lim_{s \rightarrow 0} G(s) }$$

**Ejemplo**

Suponga:

$$G(s) = \frac{1}{5s + 1}$$

Entonces:

$$E_{ee} =\frac{1}{1+ \underbrace{\lim_{s \rightarrow 0} \frac{1}{5s + 1}}_{1} }$$

Si se introduce una ganancia $K_p$ (controlador proporcional):

$$E_{ee} =\frac{1}{1+ \underbrace{\lim_{s \rightarrow 0} \frac{K_p}{5s + 1}}_{K_p} }$$

```python
# Se define la función de transferencia
G = control.tf(1, [5,1])
G
```

$$\frac{1}{5 s + 1}$$

Análisis $E_{ee}$

$$E_{ee} =\frac{1}{1+ \underbrace{\lim_{s \rightarrow 0} \frac{K_p}{5s + 1}}_{K_p} } = \frac{1}{1+K_p}$$

```python
ts = np.linspace(0, 10, 2000)

Kp = [0.5,0.75,1,2,3,5,10]
etiq = []#lista para etiquetas
e = []#lista para errores
for k in Kp:

    GLC = control.feedback(k*G,1)
    _, yLC = control.step_response(GLC, ts)
    e.append(1/(1+k))
    plt.plot(ts, yLC)
    etiq.append('k = '+str(k))
etiq.append('referencia')
plt.plot([ts[0],ts[-1]], [1,1])
plt.legend(etiq)
plt.grid(True)
print(e)
```

¿Qué pasa si el sistema tiene una dinámica integradora?

$$G(s) = \frac{1}{s(5s + 1)}$$

```python
# Se define la función de transferencia
G = control.tf(1, [5,1,0])
G
```

$$\frac{1}{5 s^2 + s}$$

Análisis $E_{ee}$

$$E_{ee} =\frac{1}{1+ \underbrace{\lim_{s \rightarrow 0} \frac{1}{s(5s + 1)}}_{\infty} } = 0$$

```python
ts = np.linspace(0, 50, 2000)

Kp = [0.5,3,5,10]
etiq = []
e = []
for k in Kp:

    GLC = control.feedback(k*G,1)
    _, yLC = control.step_response(GLC, ts)
    e.append(0)
    plt.plot(ts, yLC)
    etiq.append('k = '+str(k))
etiq.append('referencia')
plt.plot([ts[0],ts[-1]], [1,1])
plt.legend(etiq)
plt.grid(True)
print(e)
```

Observe que la presencia de una dinámica integradora hace que el sistema en lazo cerrado tenga error de estado estable nulo. La cantidad de polos integradores determina el **tipo** del sistema:

### Sistemas tipo cero

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{c3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \frac{1}{1+ \lim_{s \rightarrow 0} G(s) } \\
& = \frac{1}{1+ \lim_{s \rightarrow 0} K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} } \\
& = \frac{1}{1+ K }
\end{align}

### Sistemas tipo uno

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \frac{1}{1+ \lim_{s \rightarrow 0} G(s) } \\
& = \frac{1}{1+ \lim_{s \rightarrow 0} K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} } \\
& = 0
\end{align}

### Sistemas tipo dos

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \frac{1}{1+ \lim_{s \rightarrow 0} G(s) } \\
& = \frac{1}{1+ \lim_{s \rightarrow 0} K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} } \\
& = 0
\end{align}

## Error de estado estacionario ante una entrada rampa (velocidad)
$$R(s) = \frac{1}{s^2} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0}\frac{sR(s)}{1+G(s)} \\
&=  \lim_{s \rightarrow 0}\frac{s\frac{1}{s^2}}{1+G(s)} \\
&=  \lim_{s \rightarrow 0}\frac{1}{s+sG(s)} \\
&= \lim_{s \rightarrow 0} \frac{1}{sG(s)}
\end{align}

### Sistemas tipo cero

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{sG(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{sK\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& = \infty
\end{align}

**Ejemplo**

Suponga:

$$G(s) = \frac{1}{5s + 1}$$

```python
# Se define la función de transferencia
G = control.tf(1, [5,1])
G
```

$$\frac{1}{5 s + 1}$$

```python
ts = np.linspace(0, 100, 2000)

Kp = [0.5,5,10]
etiq = []
e = []
for k in Kp:

    GLC = control.feedback(k*G,1)
    _, yLC = control.step_response(control.tf(1,[1,0])*GLC, ts)
    plt.plot(ts, yLC)
    etiq.append('k = '+str(k))
etiq.append('referencia')
plt.plot([ts[0],ts[-1]], [0,ts[-1]])
plt.legend(etiq)
plt.grid(True)
print(e)
```

### Sistemas tipo uno

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{p3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{c3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{sG(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{sK\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& = \frac{1}{K}
\end{align}

```python
# Se define la función de transferencia
G = control.tf(1, [5,1,0])
G
```

$$\frac{1}{5 s^2 + s}$$

```python
ts = np.linspace(0, 50, 2000)

Kp = [0.1,1,10]
etiq = []
e = []
for k in Kp:

    GLC = control.feedback(k*G,1)
    _, yLC = control.step_response(control.tf(1,[1,0])*GLC, ts)
    e.append(1/(k))
    plt.plot(ts, ts-yLC)
    etiq.append('k = '+str(k))
# etiq.append('referencia')
# plt.plot([ts[0],ts[-1]], [0,ts[-1]])
plt.legend(etiq)
plt.grid(True)
print(e)
```

### Sistemas tipo dos

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{c3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{sG(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{sK\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& = \frac{s}{K} = 0
\end{align}

```python
## Desarrolle su propio ejemplo
G = control.tf([10,1], [5,1,0,0])
G

```

$$\frac{10 s + 1}{5 s^3 + s^2}$$

```python
rlist,klist = control.root_locus(G,plot = True)
```

```python
ts = np.linspace(0, 50, 2000)

Kp = [0.1,1,10]
etiq = []
e = []
for k in Kp:

    GLC = control.feedback(k*G,1)
    _, yLC = control.step_response(control.tf(1,[1,0])*GLC, ts)
    plt.plot(ts, yLC)
    etiq.append('k = '+str(k))
# etiq.append('referencia')
# plt.plot([ts[0],ts[-1]], [0,ts[-1]])
plt.legend(etiq)
plt.grid(True)
print(e)
```

## Error de estado estacionario ante una entrada parábola (aceleración)

$$R(s) = \frac{1}{s^3} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0}\frac{sR(s)}{1+G(s)} \\
&=  \lim_{s \rightarrow 0}\frac{s\frac{1}{s^3}}{1+G(s)} \\
&=  \lim_{s \rightarrow 0}\frac{1}{s^2+s^2G(s)} \\
&= \lim_{s \rightarrow 0} \frac{1}{s^2G(s)}
\end{align}

### Sistemas tipo cero

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{s^2G(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{s^2K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& = \infty
\end{align}

```python
## Desarrolle su propio ejemplo

```

### Sistemas tipo uno

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{p3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{c3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{s^2G(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{s^2K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& =  \infty
\end{align}

```python
## Desarrolle su propio ejemplo

```

### Sistemas tipo dos

$$G(s) = K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{c3}s+1)...(T_{pp}s+1)} $$

\begin{align}
E_{ee} &= \lim_{s \rightarrow 0} \frac{1}{sG(s)} \\
& = \lim_{s \rightarrow 0} \frac{1}{s^2K\frac{(T_{c1}s+1)(T_{c2}s+1)(T_{c3}s+1)...(T_{cc}s+1)}{s^2(T_{p1}s+1)(T_{p2}s+1)(T_{p3}s+1)...(T_{pp}s+1)} }\\
& = \frac{1}{K}
\end{align}

```python
## Desarrolle su propio ejemplo

```

## Tabla

|error   | $$\frac{1}{s}$$|$$\frac{1}{s^2}$$|$$\frac{1}{s^3}$$|
|--------|----------------|-----------------|-----------------|
| tipo 0 |$$\frac{1}{1+K}$$ |$$\infty$$     | $$\infty$$      |
| tipo 1 |$$0$$           |$$ \frac{1}{K}$$ | $$\infty$$      |
| tipo 2 |$$0$$           |$$ 0 $$          | $$ \frac{1}{K}$$|

## Ejemplo

Suponga un proceso con el siguiente modelo:

$$G_p(s) = \frac{2.5}{s^2+6s+5}$$

Analice el comportamiento del sistema

```python
Gp = control.tf(2.5,[1,6,5])
Gp
```

$$\frac{2.5}{s^2 + 6 s + 5}$$

```python
Gp.pole()
```

    array([-5., -1.])

```python
Gp.zero()
```

    array([], dtype=float64)

```python
rlist,klist = control.root_locus(Gp,plot = True)
```

```python
ts = np.linspace(0, 10, 2000)
e = 0.05

_, y = control.step_response(Gp, ts)
plt.plot(ts, y)
plt.plot([ts[0],ts[-1]], [y[-1],y[-1]],':k',
         [ts[0],ts[-1]], [y[-1]*(1-e),y[-1]*(1-e)],':k',
         [ts[0],ts[-1]], [y[-1]*(1+e),y[-1]*(1+e)],':k')
# etiq.append('referencia')
# plt.plot([ts[0],ts[-1]], [0,ts[-1]])
plt.grid(True)

```

Observaciones:
- Sistema estable (polos con parte real negativa).
- El polo más lento está en $s=-1$, su constante de tiempo asociada es $\tau = -1/\sigma = 1s$, por lo que el tiempo de estabilización es aproximadamente $3\tau=3s$.
- El error de estado estable ante una entrada escalón es $0.5$.

A partir de las observaciones en lazo abierto, se plantean los requerimientos para el sistema controlado:

- Que el sistema en lazo cerrado reduzca el error de estado estable a $0.1$
- Que el sistema en lazo cerrado reduzca el tiempo de estabilización.

Se plantea una forma para el controlador $G_c(s)$ de manera que sea capaz de eliminar a la dinámica más lenta del proceso.
$$G_c(s) = k_c\frac{s+a}{s+b}$$

El diseño de este controlador consiste en encontrar los valores apropiados para $k_c,a$ y $b$.

Teniendo en cuenta el bucle de control

se obtiene la función de transferencia de lazo directo:

\begin{align}
G(s) &= G_c(s)G_p(s)\\
G(s) &= k_c\frac{s+a}{s+b} \frac{2.5}{s^2+6s+5}\\
G(s) &= k_c\frac{s+a}{s+b} \frac{2.5}{(s+1)(s+5)}
\end{align}

Como la dinámica más lenta del proceso es el polo en $s=-1$, puede definirse que $a=1$.

\begin{align}
G(s) &= k_c\frac{s+1}{s+b} \frac{2.5}{(s+1)(s+5)}\\
G(s) &= k_c\frac{1}{s+b} \frac{2.5}{s+5}\\
G(s) &= k_c\frac{2.5}{(s+b)(s+5)}
\end{align}

Suponga que $b=8$ para introducir una dinámica más rápida en el comportamiento.

$$G_p(s) = \frac{2.5}{(s+1)(s+5)}$$
$$G_c(s) = k_c\frac{s+1}{s+8}$$
$$G(s) = \frac{2.5k_c}{(s+8)(s+5)}$$

```python
Gc = control.tf([1,1],[1,8])
Gc
```

$$\frac{s + 1}{s + 8}$$

```python
_,_ = control.root_locus(Gp,plot = True)
```

```python
_,_ = control.root_locus(Gc*Gp,plot = True)
```

El sistema es tipo 0 por no tener polos integradores, el error de estado estacionario ante una señal escalón es:

\begin{align}
E_{ee} &= \frac{1}{1+\lim_{s \rightarrow 0}G(s)}\\
E_{ee} &= \frac{1}{1+\lim_{s \rightarrow 0}k_c\frac{5}{2(s+8)(s+5)}}\\
E_{ee} &= \frac{1}{1+k_c\frac{5}{80}}\\
E_{ee} &= \frac{80}{80+5k_c}\\
\end{align}

Se desea $E_{ee} = 0.1$

$$E_{ee} =0.1= \frac{80}{80+5k_c}$$
$$k_c= \frac{720}{5}=144$$

El controlador es:

$$G_c(s) = 144 \frac{s+1}{s+8}$$

```python
Gc = 144*control.tf([1,1],[1,8])
Gc
```

$$\frac{144 s + 144}{s + 8}$$

```python
G = Gc*Gp
G
```

$$\frac{360 s + 360}{s^3 + 14 s^2 + 53 s + 40}$$

```python
GLC = control.feedback(G,1)
GLC
```

$$\frac{360 s + 360}{s^3 + 14 s^2 + 413 s + 400}$$

```python
ts = np.linspace(0, 10, 2000)
e = 0.05

_, y = control.step_response(Gp, ts)
_, yLC = control.step_response(GLC, ts)

plt.plot(ts, y,ts,yLC)
plt.plot([ts[0],ts[-1]], [y[-1],y[-1]],':k',
         [ts[0],ts[-1]], [y[-1]*(1-e),y[-1]*(1-e)],':k',
         [ts[0],ts[-1]], [y[-1]*(1+e),y[-1]*(1+e)],':k')
plt.plot([ts[0],ts[-1]], [yLC[-1],yLC[-1]],':k',
         [ts[0],ts[-1]], [yLC[-1]*(1-e),yLC[-1]*(1-e)],':k',
         [ts[0],ts[-1]], [yLC[-1]*(1+e),yLC[-1]*(1+e)],':k')
plt.grid(True)
```

- ¿Es satisfactorio el comportamiento obtenido para el sistema en lazo cerrado?
- ¿Qué propone hacer para mejorar el comportamiento?
- ¿Qué propone hacer para eliminar el error de estado estable?

```python

```


# "Local affine features: useful side product"
> "Why to throw away an important information?"
- toc: false
- image: images/affine_matches.png
- branch: master
- badges: true
- comments: true
- hide: false
- search_exclude: false

## Keypoints are not just points

Wide baseline stereo matching often as perceived as establishing (key-)point correspondences between images. While this might be true for the some local features like SuperPoint \cite{SuperPoint2017}, typically it is more than that.

Specifically, detectors like DoG\cite{Lowe99}, Harris\cite{Harris88}, Hessian\cite{Hessian78}, KeyNet\cite{KeyNet2019}, ORB\cite{ORB2011}, and many others rate on scale-space provide at least 3 parameters: x, y, and scale.

Most of the local descriptors  -- SIFT\cite{Lowe99}, HardNet\cite{HardNet2017} and so on -- are not rotation invariant and those which are - mostly require complex matching function\cite{RIFT2005}, \cite{sGLOH2}, so the patch orientation has to be estimated anyway, in order to match reliably. This can be done by various methods:
corners center of mass (ORB\cite{ORB2011}, dominant gradient orientation (SIFT)\cite{Lowe99} or by some learned estimator (OriNets\cite{OriNet2016},\cite{AffNet2018}). Sometimes it is possible to rely on smartphone camera IMU or photographer and assume that images are upright\cite{PerdochRetrieval2009}.

Thus, we can assume that if local descriptors match, this means the local feature scale and orientation also match, at least approximately -- see Figure below. Possible exceptions are cases, when the patch is symmetrical and orientation is ambiguous up to some symmetry group.

In addition, one could assume that we observe the patch not from the fronto-parallel position and try to estimate local normal, or, more precisely, affine shape of the feature point, modeling it as an ellipse instead of circle. One could also think of affine shape estimation as finding the camera position, from where the patch is seen in some "canonical" view.

This gives us 3 points correspondences from a single local feature match, see an example in Figure below.

Why is it important and how to use it -- see in current post. How to esimate local affine features robustly -- in the next post.

# Benefits of local affine features

## Making descriptor job easier

The most straightforward benefit of using local affine features is that they increase the repeatability of the detector and potentially reduce appearance changes of a local patch caused by viewpoint difference. This makes possible matching more challenging image pairs.

The practice is a little bit more complicated. Our recent benchmark\cite{IMW2020}, which measure the accuracy of the output fundamental matrix, shows that the difference in using affine and similarity-covariant features is quite minor. Specifically, the relative difference between SIFT vs SIFT-Affine features is 0.5% and between Hessian-SIFT and Hessian-AffNet SIFT is 5.1%, see Table below.

Therefore, if the benefit of local features would be to only improve descriptor extraction stage, it would be arguably not worth it. Luckily, there are more benefits, which are more pronounced.

## Making RANSAC job easier

Let's recall how RANSAC works.

1. Randomly sample a minimally required number of tentative correspondences to fit the geometrical model of the scene: 4 for homography, 7 for epipolar geometry and estimate the model.
2. Calculate "support": other correspondeces, which are consistent with the model.
3. Repeat steps (1), (2) and output the model which is supported with the most of correspondences. If you were lucky and have sampled all-inlier sample, meaning that all correspondences used to estimate the model were correct, you would have a correct model.

Reality is more complicated than I have just described, but the principle is the same.
The most important part is the sampling and it is sensitive to inlier ratio $\nu$ - the percentage of the correct correspondences in the set. Lets denote the minimal number of correspondences required to estimate the model as **m**.
To recover the correct model with the confidence **p** one needs to sample the number of correspondences, which is described by formula:

\begin{equation}
N = \frac{\log{(1 - p)}}{\log{(1 - \nu^{m})}}
\end{equation}

Lets plot the how the number of required samples changes with inlier ratio for confidence equal 99%. Note the log scale on Y axis. Different lines are for different minimal sample size **m**.

```python

```

As you can see from the plot above, reducing the minimal sample size required for the model estimation even by 1 saves and order of magnitude of computation.
In reality the benefit is a smaller, as modern RANSACs like GC-RANSAC\cite{gcransac2018} and MAGSAC\cite{magsac2019} could estimate the correct model from the sample containing outliers, but it is still huge, especially for low inlier rate cases.

### Image retrieval
The ideal case would be to estimate a model from just a single sample and that is exactly what is done in spatial reranking paper "**Object retrieval with large vocabularies and fast spatial matching**"  by Philbin et.al \cite{Philbin07}.

Specifically, they are solving a particular object retrieval problem: given an image containing some object, return all the images from the database, which also containg the same object.

The inital list of images is formed by the descriptor distance and then is reranked.
The authors propose to approximate a perspective change between two images as an affine image transformation, and count number of feature points, which are reprojected inside the second image. This number produces better ranking that the original short-list.

### Back to wide baseline stereo

While working for spatial re-ranking, 3-degrees of freedom camera model is too rough for the wide baseline stereo.
Yet, going from 4 point correspondences (PC) to 2 affine correspondences (AC) for homogaphy and from 7 PC to 3 AC for the fundamental matrix would be huge benefit anyway for the robust model estimation.

Various variant of RANSAC working for local features were proposed in the last 15 years: Perdoch et.al\cite{perd2006epipolar}, Pritts et.al.\cite{PrittsRANSAC2013}, Barath and Kukelova \cite{Barath2019ICCV}, Rodríguez et.al\cite{RANSACAffine2020}.

Finally, the systematic study of using is presented by Barath et.al\cite{barath2020making} in "Making Affine Correspondences Work in Camera Geometry Computation" paper. Authors show that if used naively, affine correspondence lead to worse results, because they are more noisy than point correspondences.
However, there is a bag of tricks presented in the paper, which allow to solve the noise issue and make the affine RANSAC working in practice, resulting in orders of magnitude faster computation.

Moreover, for a special cases like autonomous driving, where the motion is mostly horizonal, one could even use 2 affine correspondes for both motion estimation and consistency check, significantly improving the efficiency of the outliers removal compared to the standard RANSAC loop\cite{guan2020relative}.

Besides the special case considerations, additional contraints can also come from running other algorithms, like monocular depth estimation. Such a constraint could reduce the required number of matches from two affine correspondences to a single one for calibrated camera case\cite{OneACMonoDepth2020}.

## Application-specific benefits

Besides the wide baseline stereo, local affine features and correspondences have other applications. I will briefly describe some of them here (to be updated).

### Image rectification

Instead of matching local features between two images one might match them within a single image. Why would someone do it? This allows finding repeated pattern: think about windows, doors and so on. Typically they have the same physical size, therefore the diffrence in local features around them could tell us about the geometry of the scene and lens distortion.

This is the idea of the series of works by Pritts and Chum.

### Surface normals estimation

Ivan Eichhardt and Levente Hajder have a series of works, exploiting the local affine correspondences for surface normals estimation\cite{SurfaceNormals2019}

# Summary

Despite not being popular right now, treating keypoints as local affine features has a lot of advantages over the traditional treatment the local correspondence as the point correspondences. In the next post I will describe a way of estimating the local feature affine shape and orientation.

# References

[SuperPoint2017] Detone D., Malisiewicz T. and Rabinovich A., ``_Superpoint: Self-Supervised Interest Point Detection and Description_'', CVPRW Deep Learning for Visual SLAM, vol. , number , pp. ,  2018.

[Lowe99] D. Lowe, ``_Object Recognition from Local Scale-Invariant Features_'', ICCV,  1999.

[Harris88] C. Harris and M. Stephens, ``_A Combined Corner and Edge Detector_'', Fourth Alvey Vision Conference,  1988.

[Hessian78] P.R. Beaudet, ``_Rotationally invariant image operators_'', Proceedings of the 4th International Joint Conference on Pattern Recognition,  1978.

[KeyNet2019] A. Barroso-Laguna, E. Riba, D. Ponsa et al., ``_Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters_'', ICCV,  2019.

[ORB2011] E. Rublee, V. Rabaud, K. Konolidge et al., ``_ORB: An Efficient Alternative to SIFT or SURF_'', ICCV,  2011.

[HardNet2017] A. Mishchuk, D. Mishkin, F. Radenovic et al., ``_Working Hard to Know Your Neighbor's Margins: Local Descriptor Learning Loss_'', NeurIPS,  2017.

[RIFT2005] {Lazebnik} S., {Schmid} C. and {Ponce} J., ``_A sparse texture representation using local affine regions_'', IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, number 8, pp. 1265-1278,  2005.

[sGLOH2] {Bellavia} F. and {Colombo} C., ``_Rethinking the sGLOH Descriptor_'', IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, number 4, pp. 931-944,  2018.

[OriNet2016] K. M., Y. Verdie, P. Fua et al., ``_Learning to Assign Orientations to Feature Points_'', CVPR,  2016.

[AffNet2018] D. Mishkin, F. Radenovic and J. Matas, ``_Repeatability is Not Enough: Learning Affine Regions via Discriminability_'', ECCV,  2018.

[PerdochRetrieval2009] M. {Perd'och}, O. {Chum} and J. {Matas}, ``_Efficient representation of local geometry for large scale object retrieval_'', CVPR,  2009.

[IMW2020] Jin Yuhe, Mishkin Dmytro, Mishchuk Anastasiia et al., ``_Image Matching across Wide Baselines: From Paper to Practice_'', arXiv preprint arXiv:2003.01587, vol. , number , pp. ,  2020.

[gcransac2018] D. Barath and J. Matas, ``_Graph-Cut RANSAC_'', The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),  2018.

[magsac2019] J.N. Daniel Barath, ``_MAGSAC: marginalizing sample consensus_'', CVPR,  2019.

[Philbin07] J. Philbin, O. Chum, M. Isard et al., ``_Object Retrieval with Large Vocabularies and Fast Spatial Matching_'', CVPR,  2007.

[perd2006epipolar] M. Perd'och, J. Matas and O. Chum, ``_Epipolar geometry from two correspondences_'', ICPR,  2006.

[PrittsRANSAC2013] J. {Pritts}, O. {Chum} and J. {Matas}, ``_Approximate models for fast and accurate epipolar geometry estimation_'', 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013),  2013.

[Barath2019ICCV] D. Barath and Z. Kukelova, ``_Homography From Two Orientation- and Scale-Covariant Features_'', ICCV,  2019.

[RANSACAffine2020] M. {Rodríguez}, G. {Facciolo}, R. G. et al., ``_Robust estimation of local affine maps and its applications to image matching_'', 2020 IEEE Winter Conference on Applications of Computer Vision (WACV),  2020.

[barath2020making] Barath Daniel, Polic Michal, Förstner Wolfgang et al., ``_Making Affine Correspondences Work in Camera Geometry Computation_'', arXiv preprint arXiv:2007.10032, vol. , number , pp. ,  2020.

[guan2020relative] Guan Banglei, Zhao Ji, Barath Daniel et al., ``_Relative Pose Estimation for Multi-Camera Systems from Affine Correspondences_'', arXiv preprint arXiv:2007.10700, vol. , number , pp. ,  2020.

[OneACMonoDepth2020] D.B. Ivan Eichhardt, ``_Relative Pose from Deep Learned Depth and a Single Affine Correspondence_'', ECCV,  2020.

[SurfaceNormals2019] {Baráth} D., {Eichhardt} I. and {Hajder} L., ``_Optimal Multi-View Surface Normal Estimation Using Affine Correspondences_'', IEEE Transactions on Image Processing, vol. 28, number 7, pp. 3301-3311,  2019.


# solveODE

*solveODE* is a function created to compute and solve a differential equations system. It takes the simbolic equations create from *simbODE* and replaces variables with their respective numeric values.

```
#requiered packages
import simsysbio as s2b
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
from scipy.integrate import odeint
```

First, let's define the biological systems and find the differential equations system

```
####### Determine the differential equations system
#molecular species
species = ['mRNA', 'Protein']

#reagent and product matrices
reagents = np.array([[0, 1, 1, 0],[0, 0, 0, 1]])
products = np.array([[1, 0, 1, 0],[0, 0, 1, 0]])

#kinetic parameters
parameters = ['c1', 'c2', 'c3', 'c4']

#system input. It affects first reaction
inp = 'U'
idxR = 1

#gets simbolic differential equations
equations, variables = s2b.simbODE(species, reagents, products, parameters,
                                   inputN=inp)

#muestra las ODEs obtenidas
for s in range(0, len(species)):
    print(f'd{species[s]}/dt:', equations[s])

print(variables)
```

    dmRNA/dt: U*c1 - c2*mRNA
    dProtein/dt: -Protein*c4 + c3*mRNA
    {'species': [mRNA, Protein], 'pars': array([c1, c2, c3, c4], dtype=object), 'nameVar': array([U, mRNA, Protein], dtype=object)}

**Creates System Input**

```
#computes a hog signal as system input
#duration experiment
tend = np.array([100], float)

#pulse start and end
ton = np.array([1], float)
tdur = np.array([3], float)

#calculo de la expresion y sus respectivos perfiles
inputHOG, tog, perfiles = s2b.HOGexpr(ton, tdur, tend)

#Plotting
plt.figure()
plt.plot(tog, perfiles['t_u_Valve'], label='Step Signal (Valve)')
plt.plot(perfiles['t_u_Chamber'][0], perfiles['t_u_Chamber'][1], label='Delayed Step Signal (Camera)')
plt.plot(tog, inputHOG, label='Model Signal (HOG)')
plt.legend(loc='best')
plt.xlabel('Time (min)')
plt.ylabel('Concentration')
plt.grid()
plt.show()

```

**Solves Differential Equations System**

```
#replaces values in the simbolic expressions
Vpars = [4.0, 0.010, 1.0, 0.006]

odePars = []
for expr in equations:
  for i in range(0, len(Vpars)):
    expr = expr.subs(parameters[i], Vpars[i])
  #end for i
  odePars.append(expr)
#end for expr

print(odePars)
```

    [4.0*U - 0.01*mRNA, -0.006*Protein + 1.0*mRNA]

```
def modelODE(z,t,hog):
    #each variable variable changes through iterations
    tempZ = [hog]
    tempZ.extend(z)

    #empty list to store numeric values
    value = []

    #evaluates function
    evalue = exp(tempZ)
    value = evalue

    return value

#defines function that actually evaluates the expression
exp = sp.lambdify([variables['nameVar']], odePars, "numpy")

#species initial conditions
Sn = len(species)
sp0 = np.zeros(Sn)

#array to store output
valuesSp = np.zeros((Sn, len(tog)))

#attaches initial conditions
valuesSp[:,0] = sp0

#loop to solve the differential equation system
for t in range(1, len(tog)):
    #sets a period of time
    tspan = [tog[t-1], tog[t]]

    #solve differential equations system in a period of time
    z = odeint(modelODE,sp0,tspan,args=(inputHOG[t],))

    #stores outcome from solving system
    valuesSp[:,t] = z[1,:]

    #changes initial conditions for the next iteration
    sp0 = z[1]
#end for t
```

**System Output**

```
plt.figure()
for s in range(0,len(species)):
    plt.subplot(len(species),1,s+1)
    plt.plot(tog, valuesSp[s,:], label=str(variables["species"][s]))
    plt.xlabel('Time (min)')
    plt.ylabel('Concentration')
    plt.legend(loc='best')
    plt.grid()
plt.show()
```


# Salt budget of the Atlantic

The aim is to put $M_{ov}$ into context.

The salt budget terms for a given ocean basin are:

\begin{align}
     \mathrm{change \, of \, basin \, salt \, content} & = \mathrm{advection \, into \, basin} + \mathrm{surface \, fluxes} \\
     \partial_t S & = S_{adv} (x) + (E - P - R) \\
     \partial_t S & = S_{adv} (x) + (E - P - R) \\
\end{align}

The salt advection across a zonal section at $x$ is
$$ S_{adv} (x) = \int \int v(x,y,z) S(x,y,z) \, \mathrm{d}y \, \mathrm{d}z$$

Similar to the freshwater budget terms we define the overturning and azonal salt transport components as follows:
$$ S_{ov} (x) = \int \bar{v}(x,z) \langle S(x,z) \rangle \, \mathrm{d}z $$
$$ S_{az} (x) = \int \overline{v' (x,z) S'(x,z)} \, \mathrm{d}z $$
where $\langle x \rangle$ is the zonal average, $\bar{x}$ is the zonal integral, and $x'$ the deviation from the zonal mean.

```python
import os
import sys
import numpy as np
import xesmf as xe
import pickle
import xarray as xr
import cmocean
import cartopy
import cartopy.crs as ccrs
import matplotlib
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
matplotlib.rc_file('rc_file_paper')
%config InlineBackend.print_figure_kwargs={'bbox_inches':None}
%load_ext autoreload
%autoreload 2
```

```python
from maps import add_cyclic_POP
from tqdm import notebook
from paths import path_results, path_prace, file_RMASK_ocn, file_RMASK_ocn_low, file_ex_ocn_ctrl, file_ex_ocn_lpd
from FW_budget import load_obj, lat_bands
from timeseries import IterateOutputCESM
from xr_DataArrays import xr_AREA, xr_DZ
from xr_regression import ocn_field_regression, xr_linear_trend
```

```python
RMASK_ocn = xr.open_dataset(file_ex_ocn_ctrl, decode_times=False).REGION_MASK
RMASK_low = xr.open_dataset(file_RMASK_ocn_low, decode_times=False).REGION_MASK
```

```python
# to convert integral to average, divde by column height
DZT_HIGH_100m  = xr_DZ(domain='ocn').isel(z_t=slice(0,9)).sum('z_t')
DZT_HIGH_1000m = xr_DZ(domain='ocn').isel(z_t=slice(9,22)).sum('z_t')
DZT_LOW_100m   = xr_DZ(domain='ocn_low').isel(z_t=slice(0,10)).sum('z_t')
DZT_LOW_1000m  = xr_DZ(domain='ocn_low').isel(z_t=slice(10,40)).sum('z_t')
```

```python
def regrid_rectangular(da):
    """ regrids to rectangular grid """
    assert 'lat' in da.coords
    assert 'lon' in da.coords
    target = xr.DataArray(coords={'lat':np.arange(89.5,-90,-1), 'lon':np.arange(-179.5,180,1)}, dims=['lat','lon'])
    regridder = xe.Regridder(da, target, 'nearest_s2d', reuse_weights=True, periodic=True)
    da_regridded = regridder(da)
    return da_regridded
```

```python
rn = {'TLAT':'lat', 'TLONG':'lon'}
drop = ['ULONG','ULAT']

f = plt.figure(figsize=(6.4,2.5), constrained_layout=False)
for k, run in enumerate(['rcp', 'lr1']):
#     if k>0:  continue
    RMASK = [RMASK_ocn, RMASK_low][k]
    for d, depth in notebook.tqdm(enumerate(['0-100m', '100-1000m'])):
        maxv = [1,1][d]
        DZT_int = [[DZT_HIGH_100m, DZT_HIGH_1000m], [DZT_LOW_100m, DZT_LOW_1000m]][k][d]
        stats = xr.open_dataset(f'{path_prace}/SALT/SALT_dz_{depth}_trend_stats_{run}.nc')
        stats['p_value'] = stats.p_value.where(RMASK>0)
        stats = stats.assign_coords({'TLAT':RMASK.TLAT, 'TLONG':RMASK.TLONG})
        j = k+2*d
        ax = f.add_axes([j*.22+.01,.01,.21,.85], projection=ccrs.LambertAzimuthalEqualArea(central_longitude=-30, central_latitude=20))
        ax.set_extent((-6e6, 3.5e6, -8.5e6, 1e7), crs=ccrs.LambertAzimuthalEqualArea())
        ax.set_title([f'HR-CESM\n{depth}', f'LR-CESM\n{depth}'][k])
        ax.text(.01,.94, '('+['a','b','c','d'][j]+')', transform=ax.transAxes)

        # trends
        da = add_cyclic_POP(stats.slope.where(RMASK>0)/DZT_int*100*365)
        im = ax.pcolormesh(da.TLONG, da.TLAT, da, transform=ccrs.PlateCarree(),
                           cmap='cmo.balance', vmin=-maxv, vmax=maxv)

        #hatches
        da = xr.where(np.isfinite(stats.p_value),stats.p_value,0).rename(rn).drop(drop)
        pvalue = regrid_rectangular(da)
        ax.contourf(pvalue.lon, pvalue.lat, pvalue, [0,0.05,1],
                    colors='none', hatches=[None,'//////'], transform=ccrs.PlateCarree())

        ax.add_feature(cartopy.feature.LAND, zorder=2, edgecolor='black', facecolor='grey', linewidth=.5)
        gl = ax.gridlines(crs=ccrs.PlateCarree(), draw_labels=False)
        gl.ylocator = matplotlib.ticker.FixedLocator([-90, -60, -30, 0, 30, 60, 90])

# colorbar
ax = f.add_axes([.89,.11,.02,.65])
cbar = f.colorbar(im, cax=ax, extend='both', ticks=np.arange(-1,1.1,.5), label='salinity trend [1/100yr]')
cbar.ax.tick_params(labelsize=8)
# plt.savefig(f'{path_results}/FW-paper/Fig8', dpi=300)

```


# Tutorial 1: Probability Distributions
**Week 0, Day 5: Probability & Statistics**

**By Neuromatch Academy**

__Content creators:__ Ulrik Beierholm

__Content reviewers:__ Natalie Schaworonkow, Keith van Antwerp, Anoop Kulkarni, Pooya Pakarian, Hyosub Kim

__Production editors:__ Ethan Cheng, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

We will cover the basic ideas from probability and statistics, as a reminder of what you have hopefully previously learned. These ideas will be important for almost every one of the following topics covered in the course.

There are many additional topics within probability and statistics that we will not cover as they are not central to the main course. We also do not have time to get into a lot of details, but this should help you recall material you have previously encountered.

By completing the exercises in this tutorial, you should:
* get some intuition about how stochastic randomly generated data can be
* understand how to model data using simple probability distributions
* understand the difference between discrete and continuous probability distributions
* be able to plot a Gaussian distribution

---
# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
from scipy.stats import norm  # the normal probability distribution
```

##  Figure settings

```python
#@title Figure settings
import ipywidgets as widgets    # interactive display
from ipywidgets import interact, fixed, HBox, Layout, VBox, interactive, Label, interact_manual
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
#plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/NMA2020/nma.mplstyle")
```

##  Plotting Functions

```python
#@title Plotting Functions

def plot_random_sample(x, y, figtitle = None):
  """ Plot the random sample between 0 and 1 for both the x and y axes.

    Args:
      x (ndarray): array of x coordinate values across the random sample
      y (ndarray): array of y coordinate values across the random sample
      figtitle (str): title of histogram plot (default is no title)

    Returns:
      Nothing.
  """
  fig, ax = plt.subplots()
  ax.set_xlabel('x')
  ax.set_ylabel('y')
  plt.xlim([-0.25, 1.25]) # set x and y axis range to be a bit less than 0 and greater than 1
  plt.ylim([-0.25, 1.25])
  plt.scatter(dataX, dataY)
  if figtitle is not None:
    fig.suptitle(figtitle, size=16)
  plt.show()

def plot_random_walk(x, y, figtitle = None):
  """ Plots the random walk within the range 0 to 1 for both the x and y axes.

    Args:
      x (ndarray): array of steps in x direction
      y (ndarray): array of steps in y direction
      figtitle (str): title of histogram plot (default is no title)

    Returns:
      Nothing.
  """
  fig, ax = plt.subplots()
  plt.plot(x,y,'b-o', alpha = 0.5)
  plt.xlim(-0.1,1.1)
  plt.ylim(-0.1,1.1)
  ax.set_xlabel('x location')
  ax.set_ylabel('y location')
  plt.plot(x[0], y[0], 'go')
  plt.plot(x[-1], y[-1], 'ro')

  if figtitle is not None:
    fig.suptitle(figtitle, size=16)
  plt.show()

def plot_hist(data, xlabel, figtitle = None, num_bins = None):
  """ Plot the given data as a histogram.

    Args:
      data (ndarray): array with data to plot as histogram
      xlabel (str): label of x-axis
      figtitle (str): title of histogram plot (default is no title)
      num_bins (int): number of bins for histogram (default is 10)

    Returns:
      count (ndarray): number of samples in each histogram bin
      bins (ndarray): center of each histogram bin
  """
  fig, ax = plt.subplots()
  ax.set_xlabel(xlabel)
  ax.set_ylabel('Count')
  if num_bins is not None:
    count, bins, _ = plt.hist(data, bins = num_bins)
  else:
    count, bins, _ = plt.hist(data, bins = np.arange(np.min(data)-.5, np.max(data)+.6)) # 10 bins default
  if figtitle is not None:
    fig.suptitle(figtitle, size=16)
  plt.show()
  return count, bins

def my_plot_single(x, px):
  """
  Plots normalized Gaussian distribution

    Args:
        x (numpy array of floats):     points at which the likelihood has been evaluated
        px (numpy array of floats):    normalized probabilities for prior evaluated at each `x`

    Returns:
        Nothing.
  """
  if px is None:
      px = np.zeros_like(x)

  fig, ax = plt.subplots()
  ax.plot(x, px, '-', color='C2', LineWidth=2, label='Prior')
  ax.legend()
  ax.set_ylabel('Probability')
  ax.set_xlabel('Orientation (Degrees)')

def plot_gaussian_samples_true(samples, xspace, mu, sigma, xlabel, ylabel):
  """ Plot a histogram of the data samples on the same plot as the gaussian
  distribution specified by the give mu and sigma values.

    Args:
      samples (ndarray): data samples for gaussian distribution
      xspace (ndarray): x values to sample from normal distribution
      mu (scalar): mean parameter of normal distribution
      sigma (scalar): variance parameter of normal distribution
      xlabel (str): the label of the x-axis of the histogram
      ylabel (str): the label of the y-axis of the histogram

    Returns:
      Nothing.
  """
  fig, ax = plt.subplots()
  ax.set_xlabel(xlabel)
  ax.set_ylabel(ylabel)
  # num_samples = samples.shape[0]

  count, bins, _ = plt.hist(samples, density=True)
  plt.plot(xspace, norm.pdf(xspace, mu, sigma),'r-')
  plt.show()
```

---

# Section 1: Stochasticity and randomness

## Section 1.1: Intro to Randomness

###  Video 1: Stochastic World

```python
# @title Video 1: Stochastic World
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1sU4y1G7Qt", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="-QwTPDp7-a8", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Before trying out different probability distributions, let's start with the simple uniform distribution, U(a,b), which assigns equal probability to any value between a and b.

To show that we are drawing a random number $x$ from a uniform distribution with lower and upper bounds $a$ and $b$ we will use this notation:
$x \sim U(a,b)$. Alternatively, we can say that all the potential values of $x$ are distributed as a uniform distribution between $a$ and $b$.

### Coding Exercise 1.1: Create randomness

Numpy has many functions and capabilities related to randomness.  We can draw random numbers from various probability distributions. For example, to draw 5 uniform numbers between 0 and 100, you would use `np.random.uniform(0, 100, size = (5,))`.

 We will use `np.random.seed` to set a specific seed for the random number generator. For example, `np.random.seed(0)` sets the seed as 0. By including this, we are actually making the random numbers reproducible, which may seem odd at first. Basically if we do the below code without that 0, we would get different random numbers every time we run it. By setting the seed to 0, we ensure we will get the same random numbers. There are lots of reasons we may want randomness to be reproducible. In NMA-world, it's so your plots will match the solution plots exactly!

```
np.random.seed(0)
random_nums = np.random.uniform(0, 100, size = (5,))
```

Below, you will complete a function `generate_random_sample` that randomly generates `num_points` $x$ and $y$ coordinate values, all within the range 0 to 1. You will then generate 10 points and visualize.

```python
def generate_random_sample(num_points):
  """ Generate a random sample containing a desired number of points (num_points)
  in the range [0, 1] using a random number generator object.

  Args:
    num_points (int): number of points desired in random sample

  Returns:
    dataX, dataY (ndarray, ndarray): arrays of size (num_points,) containing x
    and y coordinates of sampled points

  """

  ###################################################################
  ## TODO for students: Draw the uniform numbers
  ## Fill out the following then remove
  # raise NotImplementedError("Student exercise: need to complete generate_random_sample")
  ###################################################################

  # Generate desired number of points uniformly between 0 and 1 (using uniform) for
  #     both x and y
  dataX = np.random.uniform(0, 1, size = (num_points,))
  dataY = np.random.uniform(0, 1, size = (num_points,))

  return dataX, dataY

# Set a seed
np.random.seed(0)

# Set number of points to draw
num_points = 10

# Draw random points
dataX, dataY = generate_random_sample(num_points)

# Visualize
plot_random_sample(dataX, dataY, "Random sample of 10 points")
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_0e972635.py)

*Example output:*

### Interactive Demo 1.1: Random Sample Generation from Uniform Distribution
In practice this may not look very uniform, although that is of course part of the randomness! Uniform randomness does not mean smoothly uniform. When we have very little data it can be hard to see the distribution.

Below, you can adjust the number of points sampled with a slider. Does it look more uniform now? Try increasingly large numbers of sampled points.

```python
#@markdown Make sure you execute this cell to enable the widget!

def generate_random_sample(num_points):
  """ Generate a random sample containing a desired number of points (num_points)
  in the range [0, 1] using a random number generator object.

  Args:
    num_points (int): number of points desired in random sample

  Returns:
    dataX, dataY (ndarray, ndarray): arrays of size (num_points,) containing x
    and y coordinates of sampled points

  """

  # Generate desired number of points uniformly between 0 and 1 (using uniform) for
  #     both x and y
  dataX = np.random.uniform(0, 1, size = (num_points,))
  dataY = np.random.uniform(0, 1, size = (num_points,))

  return dataX, dataY

@widgets.interact
def gen_and_plot_random_sample(num_points = widgets.SelectionSlider(options=[("%g"%i,i) for i in np.arange(0, 500, 10)])):

  dataX, dataY = generate_random_sample(num_points)
  fig, ax = plt.subplots()
  ax.set_xlabel('x')
  ax.set_ylabel('y')
  plt.xlim([-0.25, 1.25])
  plt.ylim([-0.25, 1.25])
  plt.scatter(dataX, dataY)
  fig.suptitle("Random sample of " + str(num_points) + " points", size=16)
  plt.show()
```

    interactive(children=(SelectionSlider(description='num_points', options=(('0', 0), ('10', 10), ('20', 20), ('3…

## Section 1.2: Random walk

###  Video 2: Random walk

```python
# @title Video 2: Random walk
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV11U4y1G7Bu", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="Tz9gjHcqj5k", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Stochastic models can be used to create models of behaviour. As an example, imagine that a rat is placed inside a novel environment, a box. We could try and model its exploration behaviour by assuming that for each time step it takes a random uniformly sampled step in any direction (simultaneous random step in x direction and random step in y direction)

### Coding Exercise 1.2: Modeling a random walk

Use the `generate_random_sample` function from above to obtain the random steps the rat takes at each time step and complete the generate_random_walk function below. For plotting, the box will be represented graphically as the unit square enclosed by the points (0, 0) and (1, 1).

```python
def generate_random_walk(num_steps, step_size):
  """ Generate the points of a random walk within a 1 X 1 box.

  Args:
    num_steps (int): number of steps in the random walk
    step_size (float): how much each random step size is weighted

  Returns:
    x, y (ndarray, ndarray): the (x, y) locations reached at each time step of the walk

  """
  x = np.zeros(num_steps + 1)
  y = np.zeros(num_steps + 1)

  ###################################################################
  ## TODO for students: Collect random step values with function from before
  ## Fill out the following then remove
  # raise NotImplementedError("Student exercise: need to complete generate_random_walk")
  ###################################################################

  # Generate the uniformly random x, y steps for the walk
  random_x_steps, random_y_steps = generate_random_sample(num_steps)

  # Take steps according to the randomly sampled steps above
  for step in range(num_steps):

    # take a random step in x and y. We remove 0.5 to make it centered around 0
    x[step + 1] = x[step] + (random_x_steps[step] - 0.5)*step_size
    y[step + 1] = y[step] + (random_y_steps[step] - 0.5)*step_size

    # restrict to be within the 1 x 1 unit box
    x[step + 1]= min(max(x[step + 1], 0), 1)
    y[step + 1]= min(max(y[step + 1], 0), 1)

  return x, y

# Set a random seed
np.random.seed(2)

# Select parameters
num_steps = 100   # number of steps in random walk
step_size = 0.5   # size of each step

# Generate the random walk
x, y = generate_random_walk(num_steps, step_size)

# Visualize
plot_random_walk(x, y, "Rat's location throughout random walk")
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_84dec82b.py)

*Example output:*

We put a little green dot for the starting point and a red point for the ending point.

### Interactive Demo 1.2: Varying parameters of a random walk
In the interactive demo below, you can examine random walks with different numbers of steps or step sizes, using the sliders.

1.  What could an increased step size mean for the actual rat's movement we are simulating?
2. For a given number of steps, is the rat more likely to visit all general areas of the arena with a big step size or small step size?

```python
# @markdown Make sure you execute this cell to enable the widget!

@widgets.interact(num_steps = widgets.IntSlider(value=100, min=0, max=500, step=1), step_size = widgets.FloatSlider(value=0.1, min=0.1, max=1, step=0.1))
def gen_and_plot_random_walk(num_steps, step_size):
  x, y = generate_random_walk(num_steps, step_size)
  plot_random_walk(x, y, "Rat's location throughout random walk")
```

    interactive(children=(IntSlider(value=100, description='num_steps', max=500), FloatSlider(value=0.1, descripti…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_6e912c89.py)

1) A larger step size could mean that the rat is moving faster, or that we sample
   the rats location less often.

2) The rat tends to visit more of the arena with a large step size.

In practice a uniform random movement is too simple an assumption. Rats do not move completely randomly; even if you could assume that, you would need to approximate with a more complex probability distribution.

Nevertheless, this example highlights how you can use sampling to approximate behaviour.

**Main course preview:** On day W3D2 we will see how random walk models can be used to also model accumulation of information in decision making.

---
# Section 2: Discrete distributions

## Section 2.1: Binomial distributions

###  Video 3: Binomial distribution

```python
# @title Video 3: Binomial distribution
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Ev411W7mw", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="kOXEQlmzFyw", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers the Bernoulli and binomial distributions.

 Click here for text recap of video

The uniform distribution is very simple, and can only be used in some rare cases. If we only had access to this distribution, our statistical toolbox would be very empty. Thankfully we do have some more advanced distributions!

The uniform distribution that we looked at above is an example of a continuous distribution. The value of $X$ that we draw from this distribution can take **any value** between $a$ and $b$.

However, sometimes we want to be able to look at discrete events. Imagine that the rat from before is now placed in a T-maze, with food placed at the end of both arms. Initially, we would expect the rat to be choosing randomly between the two arms, but after learning it should choose more consistently.

A simple way to model such random behaviour is with a single **Bernoulli trial**, that has two outcomes, {$Left, Right$}, with probability $P(Left)=p$ and $P(Right)=1-p$ as the two mutually exclusive possibilities (whether the rat goes down the left or right arm of the maze).

The binomial distribution simulates $n$ number of binary events, such as the $Left, Right$ choices of the random rat in the T-maze. Imagine that you have done an experiment and found that your rat turned left in 7 out of 10 trials. What is the probability of the rat indeed turning left 7 times ($k = 7$)?

This is given by the binomial probability of $k$, given $n$ trials and probability $p$:

$$ P(k|n,p)= \left( \begin{array} \\n \\ k\end{array} \right) p^k (1-p)^{n-k}$$
$$\binom {n}{k}={\frac {n!}{k!(n-k)!}}$$

In this formula, $p$ is the probability of turning left, $n$ is the number of binary events, or trials, and $k$ is the number of times the rat turned left. The term $\binom {n}{k}$ is the binomial coefficient.

This is an example of a *probability mass function*, which specifies the probability that a discrete random variable is equal to each value. In other words, how large a part of the probability space (mass) is placed at each exact discrete value. We require that all probability adds up to 1, i.e. that

$\sum_k P(k|n,p)=1$.

Essentially, if $k$ can only be one of 10 values, the probabilities of $k$ being equal to each possible value have to sum up to 1 because there is a probability of 1 it will equal one of those 10 values (no other options exist).

If we assume an equal chance of turning left or right, then $p=0.5$. Note that if we only have a single trial $n=1$ this is equivalent to a single Bernoulli trial (feel free to do the math!).

### Think! 2.1: Binomial distribution sampling
We will draw a desired number of random samples from a binomial distribution, with $n = 10$ and $p = 0.5$. Each sample returns the number of trials, $k$, a rat turns left out of $n$ trials.

We will draw 1000 samples of this (so it is as if we are observing 10 trials of the rat, 1000 different times). We can do this using numpy: `np.random.binomial(n, p, size = (n_samples,))`

See below to visualize a histogram of the different values of $k$, or the number of times the rat turned left in each of the 1000 samples. In a histogram all the data is placed into bins and the contents of each bin is counted, to give a visualisation of the distribution of data. Discuss the following questions.

1.   What are the x-axis limits of the histogram and why?
2.   What is the shape of the histogram?
3.  Looking at the histogram, how would you interpret the outcome of the simulation if you didn't know what p was? Would you have guessed p = 0.5?
3.   What do you think the histogram would look like if the probability of turning left is 0.8 ($p = 0.8$)?

```python
# @markdown Execute this cell to see visualization

# Select parameters for conducting binomial trials
n = 10
p = 0.8
n_samples = 1000

# Set random seed
np.random.seed(1)

# Now draw 1000 samples by calling the function again
left_turn_samples_1000 = np.random.binomial(n, p, size = (n_samples,))

# Visualize
count, bins = plot_hist(left_turn_samples_1000, 'Number of left turns in sample')
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_06a79c9b.py)

1) The limits of the histogram at 0 and 10, as these are the minimum and maximum
number of trials that a rat can turn left out of 10 trials.

2) The shape seems symmetric and is centered around 5.

3) An average/mean around 5 left turns out of 10 trials indicates that the rat
chose left and right turns in the maze with equal probability (that p = 0.5)

4) With p = 0.8, the center of the histogram would be at x = 8 and it would not be
   as symmetrical (since it would be cut off at max 10). You can go into the code above
   and run it with p = 0.8 to see this.

When working with the Bernoulli and binomial distributions, there are only 2 possible outcomes (in this case, turn left or turn right). In the more general case where there are $n$ possible outcomes (our rat is an n-armed maze) each with their own associated probability $p_1, p_2, p_3, p_4, ...$ , we use a **categorical distribution**. Draws from this distribution are a simple extension of the Bernoulli trial: we now have a probability for each outcome and draw based on those probabilities. We have to make sure that the probabilities sum to one:

$$\sum_i P(x=i)=\sum_i p_i =1$$

If we sample from this distribution multiple times, we can then describe the distribution of outcomes from each sample as the **multinomial distribution**. Essentially, the categorical distribution is the multiple outcome extension of the Bernoulli, and the multinomial distribution is the multiple outcome extension of the binomial distribution. We'll see a bit more about this in the next tutorial when we look at Markov chains.

---
## Section 2.2 Categorical distributions and Markov

####  Video 4: Categorical distributions

```python
# @title Video 4: Categorical distributions
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Rh41187ZC", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="XjQF13xMpss", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

In the more general case where there are $n$ possible outcomes (our rat is an n-armed maze) each with their own associated probability $p_1, p_2, p_3, p_4, ...$ , we use a **categorical distribution**. Draws from this distribution are a simple extension of the Bernoulli trial. We have to make sure that the probabilities sum to one:

$$\sum_i P(x=i)=\sum_i p_i =1$$

If we sample from this distribution multiple times, we can then describe the distribution of outcomes from each sample as the **multinomial distribution** (generalisation from the binomial distribution).

### Coding exercise 2.2 Markov chains

The **categorical distribution** becomes important when using **Markov chains**. The Markov property specifies that you can fully encapsulate the important properties of a system based on its *current* state, any previous history does not matter

As an example imagine that a rat is able to move freely between 3 areas: a dark rest area
($state=1$), a nesting area ($state=2$) and a bright area for collecting food ($state=3$). Every 5 minutes (timepoint $i$) we record the rat's location. The table below shows the probability of the rat transitioning from one area to another ($state_i$ to $state_{i+1}$).

\begin{array}{|l |  l | l | l |} \hline
state_{i} &P(state_{i+1}=1|state_i=*) &P(state_{i+1}=2|state_i=*) & P(state_{i+1}=3|state=_i*) \\ \hline
state_{i}=1& 0.2  &0.6 &0.2\\
state_{i}=2& .6 &0.3& 0.1\\
state_{i}=3& 0.8 &0.2 &0\\ \hline
\end{array}

We are modeling this as a Markov chain, so the animal is only in one of the states at a time and can transition between the states.

As the transition probabilities are in a matrix form we can use simple linear algebra and multiply this matrix with the vector over the current state, which can be deterministic (e.g. (0,1,0)) or probabilistic (e.g. (0.1, 0.7, 0.2)). Every time we take a further step we can just multiply with the transition matrix again.

If the animal starts in area 2, what is the probability the animal will again be in area 2 when we check on it 20 minutes (4 transitions) later?

Fill in the transition matrix in the code below.

```python
###################################################################
## TODO for student: Fill transition matrix with table values
## Fill out the following then remove
# raise NotImplementedError("Student exercise: need to fill transition matrix")
###################################################################

# Fill out transition matrix with matching table values
# in the form np.array([ [...], [...], [...] ])
transition_matrix = np.array([
    [0.2, 0.6, 0.2],
    [0.6, 0.3, 0.1],
    [0.8, 0.2, 0.0]
])

# Initial state, x0
x0 = np.array([0, 1, 0])

# Then multiply initial state by the matrix 4 times, or equivalently raise the
# matrix to the 4th power before multiplying it to the initial state
x4 = x0 @ np.linalg.matrix_power(transition_matrix, 4)

# The second area is indexed as 1 (Python starts indexing at 0)
print("The probability the rat will be in area 2 after 4 transitions is: " + str(x4[1]))
```

    The probability the rat will be in area 2 after 4 transitions is: 0.4311

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_e52c73c6.py)

You should get a probability of 0.4311, i.e. there is a 43.11% chance that you will find the rat in area 2 in 20 minutes.

What is the average amount of time spent by the rat in each of the states? Implicit in the question is the idea that we can start off with a random initial state and then measure how much relative time is spent in each area. If we make a few assumptions (e.g. ergodic or 'randomly mixing' system) we can instead start with an initial random distribution and see how that changes after 100 time steps/transitions.

```python
# Initialize random initial distribution
x_random = np.ones((1,3))/3

###################################################################
## TODO for student: Fill compute the state matrix after 100 transitions
# raise NotImplementedError("Student exercise: need to complete computation below")
###################################################################

# Fill in the missing line to get the state matrix after 100 transitions, like above
x_average_time_spent = x_random @ np.linalg.matrix_power(transition_matrix, 100)
print("The proportion of time spend by the rat in each of the three states is: "
            + str(x_average_time_spent[0]))
```

    The proportion of time spend by the rat in each of the three states is: [0.44736842 0.42105263 0.13157895]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_f7d895fb.py)

The proportion of time spend in each of the three areas are 0.4473, 0.4211, and 0.1316, respectively.

Imagine now that if the animal is satiated and tired the transitions change to:

\begin{array}{|l |  l | l | l |} \hline
state_{i} &P(state_{i+1}=1|state_i=*) &P(state_{i+1}=2|state_i=*) &P(state_{i+1}=3|state_i=*) \\ \hline
state_{i}=1& 0.2  &0.7 &0.1\\
state_{i}=2& .3 &0.7& 0.\\
state_{i}=3& 0.8 &0.2 &0\\ \hline
\end{array}

Try repeating the questions above for this table of transitions by changing the transition matrix. Based on the probability values, what would you predict? Check how much time the rat spends on average in each area and see if it matches your predictions.

**Main course preview:** The Markov property is extremely important for many models, particularly Hidden Markov Models, discussed on day W3D2, and for methods such as Markov Chain Monte Carlo sampling.

## Section 2.3: Poisson distribution

###  Video 5: Poisson distribution

```python
# @title Video 5: Poisson distribution
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1wV411x7P6", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="E_nvNb596DY", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers the Poisson distribution and how it can be used to describe neural spiking.

 Click here for text recap of  video

For some phenomena there may not be a natural limit on the maximum number of possible events or outcomes.

The Poisson distribution is a '**point-process**', meaning that it determines the number of discrete 'point', or binary, events that happen within a fixed space or time, allowing for the occurence of a potentially infinite number of events. The Poisson distribution is specified by a single parameter $\lambda$ that encapsulates the mean number of events that can occur in a single time or space interval (there will be more on this concept of the 'mean' later!).

Relevant to us, we can model the number of times a neuron spikes within a time interval using a Poisson distribution. In fact, neuroscientists often do! As an example, if we are recording from a neuron that tends to fire at an average rate of 4 spikes per second, then the Poisson distribution specifies the distribution of recorded spikes over one second, where $\lambda=4$.

The formula for a Poisson distribution on $x$ is:

$$P(x)=\frac{\lambda^x e^{-\lambda}}{x!}$$
where $\lambda$ is a parameter corresponding to the mean number of events.

### Coding Exercise 2.2: Poisson distribution sampling

In the exercise below we will draw some samples from the Poisson distribution and see what the histogram looks.

In the code, fill in the missing line so we draw 5 samples from a Poisson distribution with $\lambda = 4$. Use `np.random.poisson`.

```python
# Set random seed
np.random.seed(0)

# Draw 5 samples from a Poisson distribution with lambda = 4
sampled_spike_counts = np.random.poisson(4, 5)

# Print the counts
print("The samples drawn from the Poisson distribution are " +
          str(sampled_spike_counts))
```

    The samples drawn from the Poisson distribution are [6 7 1 8 4]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_90422623.py)

You should see that the neuron spiked 6 times, 7 times, 1 time, 8 times, and 4 times in 5 different intervals.

### Interactive Demo 2.2: Varying parameters of Poisson distribution

Use the interactive demo below to vary $\lambda$ and the number of samples, and then visualize the resulting histogram.

1.  What effect does increasing the number of samples have?
2.  What effect does changing $\lambda$ have?
3.  With a small lambda, why is the distribution asymmetric?

```python
# @markdown Make sure you execute this cell to enable the widget!

@widgets.interact(lambda_value = widgets.FloatSlider(value=4, min=0.1, max=10, step=0.1),
                  n_samples = widgets.IntSlider(value=5, min=0, max=500, step=1))

def gen_and_plot_possion_samples(lambda_value, n_samples):
  sampled_spike_counts = np.random.poisson(lambda_value, n_samples)
  count, bins = plot_hist(sampled_spike_counts, 'Recorded spikes per second')

```

    interactive(children=(FloatSlider(value=4.0, description='lambda_value', max=10.0, min=0.1), IntSlider(value=5…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_81eb1b08.py)

1) More samples means the distribution starts to look more symmetric.

2) Increasing lambda moves the distribution along the x-axis, essentialy changing the
   mean of the distribution. With lambda = 6 for example, we would expect to often see
   6 spike counts per interval.

3) It becomes asymmetric because the Poisson distribution is restricted to non-negative counts
   (you can't have negative numbers of spikes)

---
# Section 3: Continuous distributions

##  Video 6: Continuous distributions

```python
# @title Video 6: Continuous distributions
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1dq4y1L7eC", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="LJ4Zdokb6lc", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

We do not have to restrict ourselves to only probabilistic models of discrete events. While some events in neuroscience are discrete (e.g. number of spikes by a neuron), many others are continuous (e.g. neuroimaging signals in EEG or fMRI, distance traveled by an animal, human pointing in a direction of a stimulus).

With continuous distributions we have to replace the normalising sum
\begin{equation}\sum_i P(x=p_i) =1\end{equation}
over all possible events, with an integral
\begin{equation}\int_a^b P(x) =1\end{equation}

where a and b are the limits of the random variable $x$ (often $-\infty$ and $\infty$).

If we want to make predictions about possible outcomes ("I believe the BOLD signal from the area will be in the range $x_1$ to $ x_2 $") we can use the integral $\int_{x_1}^{x_2} P(x) =1$.
$P(x)$ is now a **probability density function**, sometimes written as $f(x)$ to distinguish it from the probability mass functions.

While for discrete outcomes we can ask about the probability of an specific event ("what is the probability this neuron will fire 4 times in the next second"), this is not defined for a continuous distribution ("what is the probability of the BOLD signal being exactly 4.000000000..."). Hence we need to focus on intervals when calculating probabilities from a continuous distribution.

## Section 3.1: Gaussian Distribution

The most widely used continuous distribution is probably the Gaussian (also known as Normal) distribution. It is extremely common across all kinds of statistical analyses. **Because of the central limit theorem, many quantities are Gaussian distributed.** Gaussians also have some nice mathematical properties that permit simple closed-form solutions to several important problems.

As a working example, imagine that a human participant is asked to point in the direction where they perceived a sound coming from. As an approximation, we can assume that the variability in the direction/orientation they point towards is Gaussian distributed.

### Coding Exercise 3.1A: Gaussian Distribution

In this exercise, you will implement a Gaussian by filling in the missing portions of code for the function `my_gaussian` below. Gaussians have two parameters. The **mean** $\mu$, which sets the location of its center, and its "scale" or spread is controlled by its **standard deviation** $\sigma$, or **variance** $\sigma^2$ (i.e. the square of standard deviation). **Be careful not to use one when the other is required.**

The equation for a Gaussian probability density function is:
$$
f(x;\mu,\sigma^2)=\mathcal{N}(\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(\frac{-(x-\mu)^2}{2\sigma^2}\right)
$$
In Python $\pi$ and $e$ can be written as `np.pi` and `np.exp` respectively.

As a probability distribution this has an integral of one when integrated from $-\infty$ to $\infty$, however in the following your numerical Gaussian will only be computed over a finite number of points (for the cell below we will sample from -8 to 9 in step sizes of 0.1). You therefore need to explicitly normalize it to sum to one yourself.

Test out your implementation with a $\mu = -1$ and $\sigma = 1$.

```python
def my_gaussian(x_points, mu, sigma):
  """ Returns normalized Gaussian estimated at points `x_points`, with
  parameters: mean `mu` and standard deviation `sigma`

  Args:
      x_points (ndarray of floats): points at which the gaussian is evaluated
      mu (scalar): mean of the Gaussian
      sigma (scalar): standard deviation of the gaussian

  Returns:
      (numpy array of floats) : normalized Gaussian evaluated at `x`
  """

  ###################################################################
  ## TODO for students: Implement the formula for a Gaussian
  ## Add code to calculate the gaussian px as a function of mu and sigma,
  ## for every x in x_points
  ## Function Hints: exp -> np.exp()
  ##                 power -> z**2
  ##
  ## Fill out the following then remove
  # raise NotImplementedError("Student exercise: need to implement Gaussian")
  ###################################################################
  px = 1/(2*np.pi*sigma**2)**1/2 *np.exp(-(x_points-mu)**2/(2*sigma**2))

  # as we are doing numerical integration we may have to remember to normalise
  # taking into account the stepsize (0.1)
  px = px/(0.1*sum(px))
  return px

x = np.arange(-8, 9, 0.1)

# Generate Gaussian
px = my_gaussian(x, -1, 1)

# Visualize
my_plot_single(x, px)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D5_Statistics/solutions/W0D5_Tutorial1_Solution_2730515e.py)

*Example output:*

### Interactive Demo 3.1: Sampling from a Gaussian distribution

Now that we have gained a bit of intuition about the shape of the Gaussian, let's imagine that a human participant is asked to point in the direction of a sound source, which we then measure in horizontal degrees. To simulate that we draw samples from a Normal distribution:

$$x \sim \mathcal{N}(\mu,\sigma) $$

We can sample from a Gaussian with mean $\mu$ and standard deviation $\sigma$ using `np.random.normal(mu, sigma, size = (n_samples,))`.

In the demo below, you can change the mean and standard deviation of the Gaussian, and the number of samples, we can compare the histogram of the samples to the true analytical distribution (in red).

1.   With what number of samples would you say that the full distribution (in red) is well approximated by the histogram?
2.   What if you just wanted to approximate the variables that defined the distribution, i.e. mean and variance?

```python
#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(mean = widgets.FloatSlider(value=0, min=-5, max=5, step=0.5),
                  standard_dev = widgets.FloatSlider(value=0.5, min=0, max=10, step=0.1),
                  n_samples = widgets.IntSlider(value=5, min=1, max=300, step=1))
def gen_and_plot_normal_samples(mean, standard_dev, n_samples):
  x = np.random.normal(mean, standard_dev, size = (n_samples,))
  xspace = np.linspace(-20, 20, 100)
  plot_gaussian_samples_true(x, xspace, mean, standard_dev,
                            'orientation (degrees)', 'probability')
```

    interactive(children=(FloatSlider(value=0.0, description='mean', max=5.0, min=-5.0, step=0.5), FloatSlider(val…

**Main course preview:** Gaussian distriutions are everywhere and are critical for filtering, linear systems (W2D2), optimal control (W3D3) and almost any statistical model of continuous data (W3D1, W3D2, etc.).

---
# Summary

Across the different exercises you should now:
* have gotten some intuition about how stochastic randomly generated data can be
* understand how to model data using simple distributions
* understand the difference between discrete and continuous distributions
* be able to plot a Gaussian distribution

For more reading on these topics see just about any statistics textbook, or take a look at the online resources at
https://github.com/NeuromatchAcademy/precourse/blob/master/resources.md