###### Content provided under a Creative Commons Attribution license, CC-BY 4.0; code under MIT license. (c)2014 Lorena A. Barba, Olivier Mesnard. Thanks: NSF for support via CAREER award #1149784.

[@LorenaABarba](https://twitter.com/LorenaABarba)

##### Version 0.4 -- April 2015

# Source panel method

We are now getting close to the finish line with *AeroPython*! Our first few lessons introduced the fundamental flow solutions of potential flow, and we quickly learned that using our superposition powers we could get some useful results in aerodynamics.

The superposition of a [doublet](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/03_Lesson03_doublet.ipynb) and a free stream gave the flow around a circular cylinder, and we learned about the *D'Alembert paradox*: the result of zero drag for potential flow around a cylinder. Adding a [vortex](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/06_Lesson06_vortexLift.ipynb) at the center of the cylinder, we learned about lift and the *Kutta-Joukowski theorem* stating that lift is proporional to circulation: $L=\rho U \Gamma$. A most important result!

Adding together fundamental solutions of potential flow and seeing what we get when interpreting a dividing streamline as a solid body is often called an *indirect method*. This method goes all the way back to Rankine in 1871! But its applicability is limited because we can't stipulate a geometry and find the flow associated to it.

In [Lesson 9](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb), we learned that it is possible to stipulate first the geometry, and then solve for the source strengths on a panel discretization of the body that makes the flow tangent at the boundary. This is called a *direct method* and it took off in the 1960s with the work of Hess and Smith at Douglas Aircraft Company.

A set of panels (line segments in 2D) can represent the surface of any solid body immersed in a potential flow by making the source-sheet strengths such that the normal velocity at each panel is equal to zero. This is a very powerful idea! But you should realize that all the panel strengths are coupled to each other, which is why we end up with a linear system of equations.

For an arbitrary geometry, we need to build a set of panels according to some points that define  the geometry. In this lesson, we will read from a file a geometry definition corresponding to a **NACA0012 airfoil**, create a set of panels, and solve for the source-sheet strengths to get flow around the airfoil.

*Make sure you have studied [Lesson 9](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb) carefully before proceeding!* We will not repeat the full mathematical formulation in this notebook, so refer back as needed.

First, load our favorite Python libraries, and the `integrate` module from SciPy:

```
import math
import numpy
from scipy import integrate
from matplotlib import pyplot
```

Next, we read the body geometry from a file using the NumPy function [`loadtxt()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html). The file comes from the [Airfoil Tools](http://airfoiltools.com/airfoil/details?airfoil=n0012-il) website and it contains a set of coordinates for the standard NACA0012 symmetric profile. We saved the file in the `resources` folder and load it from our local copy.

The geometry points get loaded into one NumPy array, so we separate the data into two arrays: `x,y` (for better code readability). The subsequent code will plot the geometry of the airfoil.

```
# reads of the geometry from a data file
with open ('./resources/naca0012.dat') as file_name:
    x, y = numpy.loadtxt(file_name, dtype=float, delimiter='\t', unpack=True)

# plots the geometry
%matplotlib inline

val_x, val_y = 0.1, 0.2
x_min, x_max = x.min(), x.max()
y_min, y_max = y.min(), y.max()
x_start, x_end = x_min-val_x*(x_max-x_min), x_max+val_x*(x_max-x_min)
y_start, y_end = y_min-val_y*(y_max-y_min), y_max+val_y*(y_max-y_min)

size = 10
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.plot(x, y, color='k', linestyle='-', linewidth=2);
```

## Discretization into panels

Like in [Lesson 9](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb), we will create a discretization of the body geometry into panels (line segments in 2D). A panel's attributes are: its starting point, end point and mid-point, its length and its orientation. See the following figure for the nomenclature used in the code and equations below.

We can modify the `Panel` class from our previous notebook slightly, to work better for our study of flow over an airfoil. The only difference is that we identify points on the top or bottom surfaces with the words `upper` and `lower`, which is only used later for plotting results with different colors for the top and bottom surfaces of the profile.

```
class Panel:
    """Contains information related to a panel."""
    def __init__(self, xa, ya, xb, yb):
        """Creates a panel.

        Arguments
        ---------
        xa, ya -- Cartesian coordinates of the first end-point.
        xb, yb -- Cartesian coordinates of the second end-point.
        """
        self.xa, self.ya = xa, ya
        self.xb, self.yb = xb, yb

        self.xc, self.yc = (xa+xb)/2, (ya+yb)/2       # control-point (center-point)
        self.length = math.sqrt((xb-xa)**2+(yb-ya)**2)     # length of the panel

        # orientation of the panel (angle between x-axis and panel's normal)
        if xb-xa <= 0.:
            self.beta = math.acos((yb-ya)/self.length)
        elif xb-xa > 0.:
            self.beta = math.pi + math.acos(-(yb-ya)/self.length)

        # location of the panel
        if self.beta <= math.pi:
            self.loc = 'upper'
        else:
            self.loc = 'lower'

        self.sigma = 0.                             # source strength
        self.vt = 0.                                # tangential velocity
        self.cp = 0.                                # pressure coefficient
```

For the circular cylinder, the discretization into panels was really easy. This is the part that gets more complicated when you want to compute the flow around a general geometry, while the solution part is effectively the same as in [Lesson 9](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb).

The function below will create the panels from the geometry data that was read from a file. It is better to have small panels near the leading-edge and the trailing edge, where the curvature is large. One method to get a non uniform distribution around the airfoil is to first discretize a circle with diameter equal to the airfoil's chord, with the leading edge and trailing edge touching the circle at a node, as shown in the following sketch.

Then, we store the $x$-coordinates of the circle points, `x_circle`, which will also be the $x$-coordinates of the panel nodes, `x`, and project the $y$-coordinates of the circle points onto the airfoil by interpolation. We end up with a node distribution on the airfoil that is refined near the leading edge and the trailing edge. It will look like this:

With the discretization method just described, the function `definePanels()` returns an array of objects, each an instance of the class `Panel` and containing all information about a panel, given the desired number of panels and the set of body coordinates.

A few remarks about the implementation of the function `definePanels()`:

* we just need to compute the $x$-coordinates of the circle (`x_circle`) since the $y$-coordinates of the panel nodes will be computed by interpolation;
* we create a circle with `N+1` points, but the first and last points coincide;
* we extend our NumPy arrays by adding an extra value that is equal to the first one; thus we don't have to do anything special with the value `x[i+1]` in the different loops;
* the *while*-loop is used to find two consecutive points, (`x[I]`,`y[I]`) and (`x[I+1]`,`y[I+1]`), on the foil such that the interval [`x[I]`,`x[I+1]`] contains the value `x_ends[i]`; we use the keyword `break` to get out of the loop;
* once the two points have been identified, the value `y_ends[i]` is computed by interpolation.

```
def define_panels(x, y, N=40):
    """Discretizes the geometry into panels using the 'cosine' method.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the geometry (1D arrays).
    N - number of panels (default 40).

    Returns
    -------
    panels -- Numpy array of panels.
    """
    R = (x.max()-x.min())/2                                    # radius of the circle
    x_center = (x.max()+x.min())/2                             # x-coord of the center
    x_circle = x_center + R*numpy.cos(numpy.linspace(0, 2*math.pi, N+1))  # x-coord of the circle points

    x_ends = numpy.copy(x_circle)      # projection of the x-coord on the surface
    y_ends = numpy.empty_like(x_ends)  # initialization of the y-coord Numpy array

    x, y = numpy.append(x, x[0]), numpy.append(y, y[0])    # extend arrays using numpy.append

    # computes the y-coordinate of end-points
    I = 0
    for i in range(N):
        while I < len(x)-1:
            if (x[I] <= x_ends[i] <= x[I+1]) or (x[I+1] <= x_ends[i] <= x[I]):
                break
            else:
                I += 1
        a = (y[I+1]-y[I])/(x[I+1]-x[I])
        b = y[I+1] - a*x[I+1]
        y_ends[i] = a*x_ends[i] + b
    y_ends[N] = y_ends[0]

    panels = numpy.empty(N, dtype=object)
    for i in range(N):
        panels[i] = Panel(x_ends[i], y_ends[i], x_ends[i+1], y_ends[i+1])

    return panels
```

Now we can use this function, calling it with a desired number of panels whenever we execute the cell below. We also plot the resulting geometry.

```
N = 40                            # number of panels
panels = define_panels(x, y, N)   # discretizes of the geometry into panels

# plots the geometry and the panels
val_x, val_y = 0.1, 0.2
x_min, x_max = min(panel.xa for panel in panels), max(panel.xa for panel in panels)
y_min, y_max = min(panel.ya for panel in panels), max(panel.ya for panel in panels)
x_start, x_end = x_min-val_x*(x_max-x_min), x_max+val_x*(x_max-x_min)
y_start, y_end = y_min-val_y*(y_max-y_min), y_max+val_y*(y_max-y_min)

size = 10
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.plot(x, y, color='k', linestyle='-', linewidth=2)
pyplot.plot(numpy.append([panel.xa for panel in panels], panels[0].xa),
         numpy.append([panel.ya for panel in panels], panels[0].ya),
         linestyle='-', linewidth=1, marker='o', markersize=6, color='#CD2305');
```

## Freestream conditions

The NACA0012 airfoil will be immersed in a uniform flow with velocity $U_\infty$ and an angle of attack $\alpha=0$. Even though it may seem like overkill to create a class for the freestream, we'll do it anyway. When creating a class, one is expecting to also create several intances of its objects. Here, we just have one freestream, so why define a class? Well, it makes the code more readable and does not block the programer from using the variable names `u_inf` and `alpha` for something else outside of the class.
Also, every time we need the freestream condition as input to a function, we will just have to pass the object as an argument and not all the attributes of the freestream.

```
class Freestream:
    """Freestream conditions."""
    def __init__(self, u_inf=1.0, alpha=0.0):
        """Sets the freestream conditions.

        Arguments
        ---------
        u_inf -- Farfield speed (default 1.0).
        alpha -- Angle of attack in degrees (default 0.0).
        """
        self.u_inf = u_inf
        self.alpha = alpha*math.pi/180          # degrees --> radians
```

```
# defines and creates the object freestream
u_inf = 1.0                                # freestream spee
alpha = 0.0                                # angle of attack (in degrees)
freestream = Freestream(u_inf, alpha)      # instantiation of the object freestream
```

## Flow tangency boundary condition

Enforcing the flow-tangency condition on each *control point* approximately makes the body geometry correspond to a dividing streamline (and the approximation improves if we represented the body with more and more panels). So, for each panel $i$, we make $u_n=0$ at $(x_{c_i},y_{c_i})$, which leads to the equation derived in the previous lesson:

\begin{equation}
u_{n_i} = \frac{\partial}{\partial n_i}\left\lbrace \phi\left(x_{c_i},y_{c_i}\right) \right\rbrace = 0
\end{equation}

i.e.

\begin{equation}
0 = U_\infty \cos\beta_i + \frac{\sigma_i}{2} + \sum_{j=1,j\neq i}^{N_p} \frac{\sigma_j}{2\pi} \int \frac{
\left(x_{c_i}-x_j(s_j)\right) \cos\beta_i
+ \left(y_{c_i}-y_j(s_j)\right) \sin\beta_i
}
{\left(x_{c_i}-x_j(s)\right)^2 + \left(y_{c_i}-y_j(s)\right)^2} {\rm d}s_j
\end{equation}

In the equation above, we calculate the derivative of the potential in the normal direction to enforce the flow tangency condition on each panel. But later, we will have to calculate the derivative in the tangential direction to compute the surface pressure coefficient. And, when we are interested in plotting the velocity field onto a mesh, we will have to calculate the derivative in the $x$- and $y$-direction.

Therefore the function below is similar to the one implemented in [Lesson 9](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb) to obtain the integrals along each panel, but we've generalized it to adapt to the direction of derivation (by means of two new arguments, `dxdz` and `dydz`, which respectively represent the value of $\frac{\partial x_{c_i}}{\partial z_i}$ and $\frac{\partial y_{c_i}}{\partial z_i}$, $z_i$ being the desired direction).

Moreover, the function is also more general in the sense of allowing any evaluation point, not just a control point on a panel (the argument `p_i` has been replaced by the coordinates `x` and `y` of the control-point, and `p_j` has been replaced with `panel`).

```
def integral(x, y, panel, dxdz, dydz):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.
    dxdz -- derivative of x in the z-direction.
    dydz -- derivative of y in the z-direction.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    def func(s):
        return ( ((x - (panel.xa - math.sin(panel.beta)*s))*dxdz
                  +(y - (panel.ya + math.cos(panel.beta)*s))*dydz)
                / ((x - (panel.xa - math.sin(panel.beta)*s))**2
                   +(y - (panel.ya + math.cos(panel.beta)*s))**2) )
    return integrate.quad(lambda s:func(s), 0., panel.length)[0]
```

## Building the linear system

Here, we build and solve the linear system of equations of the form

\begin{equation}[A][\sigma] = [b].\end{equation}

In building the matrix, below, we call the `integral()` function with the correct values for the last parameters: $\cos \beta_i$ and $\sin\beta_i$, corresponding to a derivative in the normal direction.

Finally, we use `linalg.solve()` from NumPy to solve the system and find the strength of each panel.

```
def build_matrix(panels):
    """Builds the source matrix.

    Arguments
    ---------
    panels -- array of panels.

    Returns
    -------
    A -- NxN matrix (N is the number of panels).
    """
    N = len(panels)
    A = numpy.empty((N, N), dtype=float)
    numpy.fill_diagonal(A, 0.5)

    for i, p_i in enumerate(panels):
        for j, p_j in enumerate(panels):
            if i != j:
                A[i,j] = 0.5/math.pi*integral(p_i.xc, p_i.yc, p_j, math.cos(p_i.beta), math.sin(p_i.beta))

    return A

def build_rhs(panels, freestream):
    """Builds the RHS of the linear system.

    Arguments
    ---------
    panels -- array of panels.
    freestream -- farfield conditions.

    Returns
    -------
    b -- 1D array ((N+1)x1, N is the number of panels).
    """
    b = numpy.empty(len(panels), dtype=float)

    for i, panel in enumerate(panels):
        b[i] = -freestream.u_inf * math.cos(freestream.alpha - panel.beta)

    return b
```

```
A = build_matrix(panels)                    # computes the singularity matrix
b = build_rhs(panels, freestream)           # computes the freestream RHS
```

```
# solves the linear system
sigma = numpy.linalg.solve(A, b)

for i, panel in enumerate(panels):
    panel.sigma = sigma[i]
```

## Surface pressure coefficient

From Bernoulli's equation, the pressure coefficient on the $i$-th panel is

\begin{equation}C_{p_i} = 1-\left(\frac{u_{t_i}}{U_\infty}\right)^2
\end{equation}

where $u_{t_i}$ is the tangential component of the velocity at the center point of the $i$-th panel,

\begin{equation}
u_{t_i} = -U_\infty \sin\beta_i + \sum_{j=1}^{N_p} \frac{\sigma_j}{2\pi} \int \frac{
\left(x_{c_i}-x_j(s_j)\right) \frac{\partial x_{c_i}}{\partial t_i}
+ \left(y_{c_i}-y_j(s_j)\right) \frac{\partial y_{c_i}}{\partial t_i}
}
{\left(x_{c_i}-x_j(s)\right)^2 + \left(y_{c_i}-y_j(s)\right)^2} {\rm d}s_j
\end{equation}

with

\begin{equation}
\frac{\partial x_{c_i}}{\partial t_i} = -\sin\beta_i \quad\text{and} \quad \frac{\partial y_{c_i}}{\partial t_i} = \cos\beta_i
\end{equation}

Notice that below we call the function `integral()` with different arguments: $-\sin\beta_i$ and $\cos\beta_i$ to get the derivation in the tangential direction.

```
def get_tangential_velocity(panels, freestream):
    """Computes the tangential velocity on the surface.

    Arguments
    ---------
    panels -- array of panels.
    freestream -- farfield conditions.
    """
    N = len(panels)
    A = numpy.empty((N, N), dtype=float)
    numpy.fill_diagonal(A, 0.0)

    for i, p_i in enumerate(panels):
        for j, p_j in enumerate(panels):
            if i != j:
                A[i,j] = 0.5/math.pi*integral(p_i.xc, p_i.yc, p_j, -math.sin(p_i.beta), math.cos(p_i.beta))

    b = freestream.u_inf * numpy.sin([freestream.alpha - panel.beta for panel in panels])

    sigma = numpy.array([panel.sigma for panel in panels])

    vt = numpy.dot(A, sigma) + b

    for i, panel in enumerate(panels):
        panel.vt = vt[i]
```

```
# computes the tangential velocity at the center-point of each panel
get_tangential_velocity(panels, freestream)
```

```
def get_pressure_coefficient(panels, freestream):
    """Computes the surface pressure coefficients.

    Arguments
    ---------
    panels -- array of panels.
    freestream -- farfield conditions.
    """
    for panel in panels:
        panel.cp = 1.0 - (panel.vt/freestream.u_inf)**2
```

```
# computes the surface pressure coefficients
get_pressure_coefficient(panels, freestream)
```

### Theoretical solution

There is a classical method to obtain the theoretical characteristics of airfoils, known as *Theodorsen's method*. It uses the Joukowski transformation but is able to deal with any airfoil by an additional transformation between a "near circle" and a circle. The method is hairy indeed! But the resulting values of pressure coefficient are provided for some airfoils in table form in the 1945 [NACA Report No.824](http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19930090976.pdf), available from the NASA web server (see p. 71).

The values of $(u/U_{\infty})^2$ are given for several stations along the chord length. We transcribed them here, saving them into an array:

```
voverVsquared=numpy.array([0, 0.64, 1.01, 1.241, 1.378, 1.402, 1.411, 1.411, 1.399, 1.378, 1.35, 1.288, 1.228, 1.166, 1.109, 1.044, 0.956, 0.906, 0])
print voverVsquared
```

    [ 0.     0.64   1.01   1.241  1.378  1.402  1.411  1.411  1.399  1.378
      1.35   1.288  1.228  1.166  1.109  1.044  0.956  0.906  0.   ]

```
xtheo=numpy.array([0, 0.5, 1.25, 2.5, 5.0, 7.5, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 95, 100])
xtheo = xtheo/100
print xtheo
```

    [ 0.      0.005   0.0125  0.025   0.05    0.075   0.1     0.15    0.2     0.25
      0.3     0.4     0.5     0.6     0.7     0.8     0.9     0.95    1.    ]

### And plot the result!

We will use the values from the NACA report (also given in the book by Abbot and von Doenhoff, ["Theory of Wing Sections,"](http://books.google.com/books/about/Theory_of_Wing_Sections_Including_a_Summ.html?id=DPZYUGNyuboC) 1949) to visually compare the pressure distribution with the result of our source panel method. Let's see how it looks!

```
# plots the surface pressure coefficient
val_x, val_y = 0.1, 0.2
x_min, x_max = min( panel.xa for panel in panels ), max( panel.xa for panel in panels )
cp_min, cp_max = min( panel.cp for panel in panels ), max( panel.cp for panel in panels )
x_start, x_end = x_min-val_x*(x_max-x_min), x_max+val_x*(x_max-x_min)
y_start, y_end = cp_min-val_y*(cp_max-cp_min), cp_max+val_y*(cp_max-cp_min)

pyplot.figure(figsize=(10, 6))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('$C_p$', fontsize=16)
pyplot.plot([panel.xc for panel in panels if panel.loc == 'upper'],
         [panel.cp for panel in panels if panel.loc == 'upper'],
         color='r', linewidth=1, marker='x', markersize=8)
pyplot.plot([panel.xc for panel in panels if panel.loc == 'lower'],
         [panel.cp for panel in panels if panel.loc == 'lower'],
         color='b', linewidth=0, marker='d', markersize=6)
pyplot.plot(xtheo, 1-voverVsquared, color='k', linestyle='--',linewidth=2)
pyplot.legend(['upper', 'lower'], loc='best', prop={'size':14})
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.gca().invert_yaxis()
pyplot.title('Number of panels : %d' % N);
```

That looks pretty good! The only place where the panel method doesn't quite match the tabulated data from Theordorsen's method is at the trailing edge. But note that the flow-tangency boundary condition in the panel method is applied at the control point of the panel (not at the endpoints), so this discrepancy is not surprising.

##### Accuracy check

For a closed body, the sum of all the source strengths must be zero. If not, it means the body would be adding or absorbing mass from the flow! Therfore, we should have

$$\sum_{j=1}^{N} \sigma_j l_j = 0$$

where $l_j$ is the length of the $j^{\text{th}}$ panel.

With this, we can get a get an idea of the accuracy of the source panel method.

```
# calculates the accuracy
accuracy = sum([panel.sigma*panel.length for panel in panels])
print '--> sum of source/sink strengths:', accuracy
```

    --> sum of source/sink strengths: 0.00461703117528

## Streamlines onto a mesh grid

To get a streamline plot, we have to create a mesh (like we've done in all *AeroPython* lessons!) and compute the velocity field onto it. Knowing the strength of every panel, we find the $x$-component of the velocity by taking derivative of the velocity potential in the $x$-direction, and the $y$-component by taking derivative in the $y$-direction:

$$u\left(x,y\right) = \frac{\partial}{\partial x}\left\lbrace \phi\left(x,y\right) \right\rbrace$$

$$v\left(x,y\right) = \frac{\partial}{\partial y}\left\lbrace \phi\left(x,y\right) \right\rbrace$$

Notice that here we call the funcion `integral()` with $1,0$ as the final arguments when calculating the derivatives in the $x$-direction, and $0,1$ for the derivatives in th $y$-direction.

```
def get_velocity_field(panels, freestream, X, Y):
    """Returns the velocity field.

    Arguments
    ---------
    panels -- array of panels.
    freestream -- farfield conditions.
    X, Y -- mesh grid.
    """
    Nx, Ny = X.shape
    u, v = numpy.empty((Nx, Ny), dtype=float), numpy.empty((Nx, Ny), dtype=float)

    for i in xrange(Nx):
        for j in xrange(Ny):
            u[i,j] = freestream.u_inf*math.cos(freestream.alpha)\
                     + 0.5/math.pi*sum([p.sigma*integral(X[i,j], Y[i,j], p, 1, 0) for p in panels])
            v[i,j] = freestream.u_inf*math.sin(freestream.alpha)\
                     + 0.5/math.pi*sum([p.sigma*integral(X[i,j], Y[i,j], p, 0, 1) for p in panels])

    return u, v
```

```
# defines a mesh grid
Nx, Ny = 20, 20                  # number of points in the x and y directions
val_x, val_y = 1.0, 2.0
x_min, x_max = min( panel.xa for panel in panels ), max( panel.xa for panel in panels )
y_min, y_max = min( panel.ya for panel in panels ), max( panel.ya for panel in panels )
x_start, x_end = x_min-val_x*(x_max-x_min), x_max+val_x*(x_max-x_min)
y_start, y_end = y_min-val_y*(y_max-y_min), y_max+val_y*(y_max-y_min)

X, Y = numpy.meshgrid(numpy.linspace(x_start, x_end, Nx), numpy.linspace(y_start, y_end, Ny))

# computes the velicity field on the mesh grid
u, v = get_velocity_field(panels, freestream, X, Y)
```

```
# plots the velocity field
size=10
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.streamplot(X, Y, u, v, density=1, linewidth=1, arrowsize=1, arrowstyle='->')
pyplot.fill([panel.xc for panel in panels],
         [panel.yc for panel in panels],
         color='k', linestyle='solid', linewidth=2, zorder=2)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.title('Streamlines around a NACA 0012 airfoil, AoA = %.1f' % alpha);
```

We can now calculate the pressure coefficient. In Lesson 9, we computed the pressure coefficient on the surface of the circular cylinder. That was useful because we have an analytical solution for the surface pressure on a cylinder in potential flow. For an airfoil, we are interested to see how the pressure looks all around it, and we make a contour plot in the flow domain.

```
# computes the pressure field
cp = 1.0 - (u**2+v**2)/freestream.u_inf**2

# plots the pressure field
size=12
pyplot.figure(figsize=(1.1*size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
contf = pyplot.contourf(X, Y, cp, levels=numpy.linspace(-2.0, 1.0, 100), extend='both')
cbar = pyplot.colorbar(contf)
cbar.set_label('$C_p$', fontsize=16)
cbar.set_ticks([-2.0, -1.0, 0.0, 1.0])
pyplot.fill([panel.xc for panel in panels],
         [panel.yc for panel in panels],
         color='k', linestyle='solid', linewidth=2, zorder=2)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.title('Contour of pressure field');
```

### Final words

We've learned to use a source-sheet to represent any solid body: first a [circular cylinder](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/09_Lesson09_flowOverCylinder.ipynb) (which we knew we could get by superposing a doublet and a freestream), and now an airfoil.

But what is the feature of airfoils that makes them interesting? Well, the fact that we can use them to generate lift and make things that fly, of course! But what do we need to generate lift? Think, think ... what is it?

## References

1. [Airfoil Tools](http://airfoiltools.com/index), website providing airfoil data.
1. Ira Herbert Abbott, Albert Edward Von Doenhoff and Louis S. Stivers, Jr. (1945), "Summary of Airfoil Data," NACA Report No.824, [PDF on the NASA web server](http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19930090976.pdf) (see p. 71)
1. Ira Herbert Abbott, Albert Edward Von Doenhoff, "Theory of Wing Sections, Including a Summary of Airfoil Data" (1949), Dover Press.

A further reference on Theodorsen's method is:

* Roland Schinzinger, Patricio A. A. Laura (1991), "Conformal Mapping: Methods and Applications." Dover edition in 2003. [Read on Google Books](https://books.google.com/books?id=qe-7AQAAQBAJ&lpg=PA128&ots=wbg0jLlqq5&dq=method%20theodorsen&pg=PA128#v=onepage&q=%22method%20of%20theodorsen%20and%20garrick%22&f=false)

---

###### Please ignore the cell below. It just loads our style for the notebook.

```
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Alegreya Sans', sans-serif;
    font-style:regular;
    font-weight: 200;
    font-size: 50pt;
    line-height: 100%;
    color:#CD2305;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Fenix', serif;
    font-size: 22pt;
    line-height: 100%;
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Fenix', serif;
    margin-top:12px;
	font-size: 16pt;
    margin-bottom: 3px;
    font-style: regular;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Fenix', serif;
    font-size: 2pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: #CD2305;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

# SciPy – A collection of scientific algorithms and data structures

Much of this lecture is based on the excellent work of  J.R. Johansson who published his work here:
J.R. Johansson (jrjohansson at gmail.com) [http://jrjohansson.github.io](http://jrjohansson.github.io).
Some examples have been pulled from https://github.com/scipy/scipy-cookbook/

The SciPy framework builds on top of the low-level NumPy framework for multidimensional arrays, and provides a large number of higher-level scientific algorithms. Some of the topics that SciPy covers are:

* Special functions ([scipy.special](http://docs.scipy.org/doc/scipy/reference/special.html))
* Integration and ODE  ([scipy.integrate](http://docs.scipy.org/doc/scipy/reference/integrate.html))
* Optimization ([scipy.optimize](http://docs.scipy.org/doc/scipy/reference/optimize.html))
* Interpolation ([scipy.interpolate](http://docs.scipy.org/doc/scipy/reference/interpolate.html))
* Fourier Transforms ([scipy.fftpack](http://docs.scipy.org/doc/scipy/reference/fftpack.html))
* Signal Processing ([scipy.signal](http://docs.scipy.org/doc/scipy/reference/signal.html))
* Linear Algebra ([scipy.linalg](http://docs.scipy.org/doc/scipy/reference/linalg.html))
* Sparse Eigenvalue Problems ([scipy.sparse](http://docs.scipy.org/doc/scipy/reference/sparse.html))
* Statistics ([scipy.stats](http://docs.scipy.org/doc/scipy/reference/stats.html))
* Multi-dimensional image processing ([scipy.ndimage](http://docs.scipy.org/doc/scipy/reference/ndimage.html))
* File IO ([scipy.io](http://docs.scipy.org/doc/scipy/reference/io.html))

Each submodule in SciPy  provides a number of functions and classes that can be used to solve problems in their respective domain.

Table of Contents

```python
#some code to help display things in this notebook
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from mpl_toolkits.mplot3d import Axes3D

sns.set(context='notebook')
```

```python
# plot style so figures are more readable. Hopefully
plt.rcParams['figure.figsize'] = (11, 7.5)
plt.rcParams['font.size'] = 15
plt.rcParams['lines.linewidth'] = 3
plt.rcParams['xtick.labelsize'] = 14
plt.rcParams['ytick.labelsize'] = 14
plt.rcParams['axes.labelsize'] = 15
plt.rcParams['axes.titlesize'] = 15
plt.rcParams['legend.fontsize'] = 14
```

# Linear algebra

The linear algebra module contains a lot of matrix related functions, including linear equation solving, eigenvalue solvers, matrix functions (for example matrix-exponentiation), a number of different decompositions (SVD, LU, cholesky), etc.

Detailed documetation is available at: http://docs.scipy.org/doc/scipy/reference/linalg.html

Here we will look at how to use some of these functions:

## Linear equation systems

Linear equation systems on the matrix form

$A x = b$

where $A$ is a matrix and $x,b$ are vectors can be solved like:

```python
from scipy import linalg
```

```python
A = (np.arange(100) + np.random.normal(0, 10, size=(100))).reshape(10, 10)
b = np.arange(10)
A.shape, b.shape
```

```python
plt.matshow(A)
plt.grid(False) # turn off the grid for this plot
```

```python
x = linalg.solve(A, b)
x
```

```python
# check whether we found the correct x. All entries should be close to zero.
np.dot(A, x) - b
```

We can also do the same with

$A X = B$

where $A, B, X$ are matrices:

```python
A = np.random.rand(5,5)
B = np.random.rand(5,5)
f, [ax1, ax2] = plt.subplots(1, 2)
ax1.matshow(A)
ax2.matshow(B)
# plt.matshow(B)
ax1.grid()
ax2.grid()
```

```python
X = linalg.solve(A, B)
X
```

```python
# check
linalg.norm(np.dot(A, X) - B)
```

## Eigenvalues and eigenvectors

The eigenvalue problem for a matrix $A$:

$\displaystyle A v_n = \lambda_n v_n$

where $v_n$ is the $n$th eigenvector and $\lambda_n$ is the $n$th eigenvalue.

To calculate eigenvalues of a matrix, use the `eigvals` and for calculating both eigenvalues and eigenvectors, use the function `eig`. Both return complex numbers.

```python
A = np.random.rand(4, 4)

eigen_values = linalg.eigvals(A)
eigen_values
```

```python
values, vectors = linalg.eig(A)
values, vectors
```

There are also more specialized solvers, like the `eigh` for Hermitian matrices.

## Exercise:  Simple Linear Algebra

1. Create a 10x10 matrix $A$ with random entries

2. Create a symmetric matrix from $A$ using its transpose (use A.T to transpose) and then plot the new matrix using `plt.matshow`

3. Use `linalg.eigh` to get the eigenvalues and vectors.

```python
# %load simple_linalg_solution.py
```

# Scipy Spatial

Methods and data structures to help deal with spatial data. Spatial data means that we can define some sort of metric on the data.

```python
from scipy import spatial
N = 5000
points = np.random.uniform(0, 1, (N, 3)) # create 5000 points in 3D

fig = plt.figure()
ax = Axes3D(fig)
ax.scatter(points[:, 0], points[:, 1], points[:, 2])
```

## Exercise: SciPy Spatial

Find all points that have a distance of 0.4 to the center point (0.5, 0.5, 0.5).

1. Read the first paragraph of https://en.wikipedia.org/wiki/K-d_tree.
2. Create a KDtree using `scipy.spatial.KDTree(points)`
3. Check the scipy documentation of `scipy.spatial.KDTree` for a method finding all points within a given radius.

```python
# %load solution_spatial.py
```

Now we test the SciPy solution versus the NumPy solution from earlier for finding the nearest neighbour for each point in the point cloud. (See NumPy execrises from Tamas earlier).

```python
#%%timeit
#distances, indices = kdtree.query(points, k=2)
#distances[:, 1]
```

```python
#%%timeit
#distances = np.sum((points.reshape(N, 1, 3) - points)**2, axis=2)
#distances[np.diag_indices_from(distances)] = np.inf
#np.argmin(distances, axis=1)
```

# Special functions

A large number of mathematical special functions are important for many computional physics problems. SciPy provides implementations of a very extensive set of special functions. For details, see the list of functions in the reference documention at http://docs.scipy.org/doc/scipy/reference/special.html#module-scipy.special.

- Airy functions
- Elliptic functions and integrals
- Bessel functions
- Gamma functions
- etc....

## Polynomials

The module contains a large number of functions for evaluating polynomials and calculating their roots.

The following plot shows just a few of them.

```python
from scipy import special

functions = [special.eval_chebys, special.eval_laguerre, special.eval_legendre, special.eval_chebyu]

fig, axs = plt.subplots(2, 2, sharex=True, sharey=True )

x = np.linspace(-2., 2., 200)

for ax, f in zip(np.ravel(axs), functions):
    ax.set_title(f.__name__)
    for n in range(6):
        ax.plot(x, f(n, x))
    ax.set_ylim([-3, 3])
```

## Statistical Functions

The module also contains a host of useful functions for statistical computations. for example the so called 'error function' `scipy.special.erf`

\begin{align}
\operatorname {erf}(x)&={\frac {1}{\sqrt {\pi }}}\int _{-x}^{x}e^{-t^{2}},dt \\ &={\frac {2}{\sqrt {\pi }}}\int _{0}^{x}e^{-t^{2}}\,dt.
\end{align}

Which allows you to get the gaussian probability dsitribution
$$
\phi(z) = \frac 1 2 \left(1 + erf\left(\frac{z}{sqrt(2)}\right) \right)
$$

The module `scipy.stats` contains nicer methods to handle statistical methods. More on that later.

```python
x = np.linspace(-3, 3)
plt.plot(x, special.erf(x), label='$erf$')
plt.plot(x, 0.5 * (1 + special.erf(x/np.sqrt(2))), label='$phi$')
plt.xlabel('$x$')
plt.legend()
```

## Bessel Functions
To demonstrate the typical usage of special functions we will look in more detail at the Bessel functions.
The `scipy.special` module includes a large number of Bessel-functions
Here we will use the functions $j_n$ and $y_n$, which are the Bessel functions
of the first and second kind and real-valued order.

Bessel functions are a family of solutions to Bessel’s differential equation with real or complex order alpha:
$$
x^2 \frac{d^2 y}{dx^2} +x  \frac{dy}{dx} +(x^2−α^2)y=0
$$

```python
from scipy import special
x = np.linspace(0, 10, 100)

for n in range(4):
    plt.plot(x, special.jn(n, x), label=f'$J_{n}(x)$')
plt.legend();
```

Among other uses, these functions arise in wave propagation problems such as the vibrational modes of a thin drum head. Here is an example of a circular drum head anchored at the edge.

https://en.wikipedia.org/wiki/Vibrations_of_a_circular_membrane

```python
# fix the radius of the drum
DRUM_RADIUS = 1

def drumhead_height(n, k, distance, angle, t):
    '''
    Solution to the Drumhead problem with a fixed membrane on the edge of the drum
    See https://en.wikipedia.org/wiki/Vibrations_of_a_circular_membrane
    '''
    kth_zero = special.jn_zeros(n, k)[-1] # fix the drum on the outter edge to be zero.
    return np.cos(t) * np.cos(n*angle) * special.jn(n, distance*kth_zero/DRUM_RADIUS)

# create two lists containing all combinations of radius and theta we want to plot
r = np.linspace(0, DRUM_RADIUS, 50)
t = np.linspace(0, 2*np.pi, 50)
radius, theta = np.meshgrid(r, t) # this creates all combinations we are interested in

x = radius * np.cos(theta)
y = radius * np.sin(theta)
z = drumhead_height(1, 1, radius, theta, 0.5)
```

```python
fig = plt.figure()
ax = Axes3D(fig)
ax.plot_surface(x, y, z, cmap='YlGnBu')
```

```python
# some code to create a nice animation within the notebook.
# this takes a couple of minutes on my machine so I commented it out

# from matplotlib.animation import FuncAnimation
# from functools import partial
# from tqdm import tqdm_notebook

# my_drum = partial(drumhead_height, n=2, k=1, distance=radius, angle=theta)

# frames = 200
# fps = 25

# fig = plt.figure()
# ax = Axes3D(fig)
# ax.plot_surface(x, y, my_drum(t=0), cmap='YlGnBu')

# bar = tqdm_notebook(total=frames)

# def update(i):
#    bar.update()

#    t = i / fps
#    ax.cla()
#    z = my_drum(t=t)
#    ax.plot_surface(x, y, z, cmap='YlGnBu', vmin=-0.5, vmax=0.5)
#    ax.set_zlim(-0.5, 0.5)

# ani = FuncAnimation(fig, update, frames=frames, interval=1000 / fps, blit=False, repeat=False)
# ani.save('animation_membrane.mp4', writer="ffmpeg")
```

```python
from IPython.display import Video
Video('./animation_membrane.mp4')
```

# Ordinary Differential Equations.

To demonstrate the usefulness of SciPy lets try to predict what will happen to humanity in case of a zombie outbreak.  This example comes courtesy of Christopher Campo.

We will show that, in case of a Zombie outbreak, humanity is inevitably doomed.
As shown by [Phillip Munz et al](http://mysite.science.uottawa.ca/rsmith43/Zombies.pdf) we can model a simple outbreak scenario like so.

\begin{align}
\frac{S}{dt} &= B - \beta S Z - \delta S \\
\frac{Z}{dt} &= \beta S Z + \gamma R - \alpha S Z \\
\frac{R}{dt} &= \delta S + \alpha S Z - \gamma R
\end{align}

Where $S$ is the number of susceptible humans, $Z$ is the number of zombies and $R$ is the number of removed persons which have died either by natural causes or zombie attack. The human birthrate is assumed to be constant and modeled by $B$

Susceptibles can become zombies through an encounter with a zombie ($\beta$). Natural causes of human deaths are parameterized by $\delta$. Humans in the removed class can be resurrected and become zombies ($\gamma$).
Zombies can enter the removed class by cutting their heads of or removing their brains ($\alpha$)

The question I'm trying to answer is

> __Can Humanity survive the zombie Apocalypse?__

Since I don't know how to solve differential equations, I'm going to use SciPy instead.

Now we use SciPy to solve this system of differential equations. The function `solve_ivp` does the entire job for us.

If we go to [the SciPy documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) about this function we will read the following.

>This function numerically integrates a system of ordinary differential equations given an initial value:
>
> $y^{\prime} = f(t, y)$
>
>$y(t_0) = y_0$
>
>Here $t$ is a one-dimensional independent variable (time), $y(t)$ is an __n-dimensional vector-valued function__ (state), and an __n-dimensional vector-valued function f(t, y) determines the differential equations__. The goal is to find y(t) approximately satisfying the differential equations, given an initial value y(t0)=y0.

So we start by defining some starting values for the problem and the model function $f(t, y)$.

```python
from scipy.integrate import solve_ivp

# Type `solve_ivp?` to get inline documentation about the method in your notebook.
```

```python
# fixed model parameters
B = 0.1 # birth rate
delta = 0.0001 # rate of natural casues of death
beta = 0.00016 # transmission coefficent
gamma = 0.0001 # resurrection rate
alpha = 0.0001 # rate of destroyed zombies

# initial conditions
S0 = 5000.              # initial population
Z0 = 0                 # initial zombie population
R0 = 0                 # initial dead population
y0 = [S0, Z0, R0]     # initial condition vector

def f_model(t, y):
    Si = y[0]
    Zi = y[1]
    Ri = y[2]
    # the model equations (see Munz et al. 2009)
    f0 = B - beta*Si*Zi - delta*Si
    f1 = beta*Si*Zi + gamma*Ri - alpha*Si*Zi
    f2 = delta*Si + alpha*Si*Zi - gamma*Ri
    return [f0, f1, f2]

```

Now we solve the equation and output the result

```python
t_start, t_end = 0, 150  # 200 days of zobie acopalypse
solution = solve_ivp(f_model, (t_start, t_end), y0, t_eval=np.linspace(t_start, t_end, 200))
```

```python
s = solution.y[0, :]
z = solution.y[1, :]
plt.plot(solution.t, s, label='survivors')
plt.plot(solution.t, z, label='zombies')
plt.xlabel('t / days')
plt.ylabel('Population')
plt.legend()
```

## Exercise: Save the world.

The ODE defined above is a very pessimistic model. There is one unstable equilibrium state if $\beta = 0$.
Your task is to save the world by adding a the notion of a _cure_ to the model above. The cure
can move individuals from the $Z$ to the $S$ group.

1. Introduce a new constant $\rho$ which models the effectivness of the cure.
2. Add a term to the ODE to model the cure.
3. Solve the ODE using `solve_ivp`

```python
# %load solution_zombies.py
```

## Example : Complicated  ODE

We use SciPy `solve_ivp` method to numerically solve the following system of coupled differential euqations

\begin{aligned}{{\dot {\theta }}_{1}}&={\frac {6}{ml^{2}}}{\frac {2p_{\theta _{1}}-3\cos(\theta _{1}-\theta _{2})p_{\theta _{2}}}{16-9\cos ^{2}(\theta _{1}-\theta _{2})}}\\{{\dot {\theta }}_{2}}&={\frac {6}{ml^{2}}}{\frac {8p_{\theta _{2}}-3\cos(\theta _{1}-\theta _{2})p_{\theta _{1}}}{16-9\cos ^{2}(\theta _{1}-\theta _{2})}}.
\end{aligned}

\begin{aligned}{{\dot {p}}_{\theta _{1}}}&={\frac {\partial L}{\partial \theta _{1}}}=-{\tfrac {1}{2}}ml^{2}\left({{\dot {\theta }}_{1}}{{\dot {\theta }}_{2}}\sin(\theta _{1}-\theta _{2})+3{\frac {g}{l}}\sin \theta _{1}\right)\\{{\dot {p}}_{\theta _{2}}}&={\frac {\partial L}{\partial \theta _{2}}}=-{\tfrac {1}{2}}ml^{2}\left(-{{\dot {\theta }}_{1}}{{\dot {\theta }}_{2}}\sin(\theta _{1}-\theta _{2})+{\frac {g}{l}}\sin \theta _{2}\right).
\end{aligned}

Now we plot the results for the $\theta_1$ and $\theta_2$ using

\begin{align}
x_1 &= L \; sin(\theta_1(t))\\
y_1 &= - L\;  cos(\theta_1(t)) \\
x_2 &= x_1 + L \; sin(\theta_2(t)) \\
y_2 &= y_1 - L \; cos(\theta_2(t))
\end{align}

```python
from scipy.constants import g
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt

L = 0.5
m = 0.1

t_start = 0
t_end = 20
fps = 50
frames = (t_end - t_start) * fps

def pendulum_model(t, y):
    """
    The right-hand side of the pendulum ODE
    """
    phi_1, phi_2, p_1, p_4 = y[0], y[1], y[2], y[3]

    c = (m * L**2)
    cos_delta_phi = np.cos(phi_1 - phi_2)
    sin_delta_phi = np.sin(phi_1 - phi_2)
    denominator = 16 - 9 * cos_delta_phi**2

    dphi_1 = 6 / c * (2 * p_1 - 3 * cos_delta_phi * p_4) / denominator
    dphi_2 = 6 / c * (8 * p_4 - 3 * cos_delta_phi * p_1) / denominator

    dp_1 = -c / 2 * ( dphi_1 * dphi_2 * sin_delta_phi + 3 * g / L * np.sin(phi_1))
    dp_4 = -c / 2 * (-dphi_1 * dphi_2 * sin_delta_phi + g / L * np.sin(phi_2))

    return [dphi_1, dphi_2, dp_1, dp_4]

# initial values
y0 = [np.pi / 3, -np.pi / 4, 0, 0.065]

solution = solve_ivp(
    pendulum_model,
    (t_start, t_end),
    y0,
    t_eval=np.linspace(t_start, t_end, frames),
)

x1s = + L * np.sin(solution.y[0])
y1s = - L * np.cos(solution.y[0])

x2s = x1s + L * np.sin(solution.y[1])
y2s = y1s - L * np.cos(solution.y[1])

plt.plot(x1s, y1s, '-', alpha=0.5, color='xkcd:sky blue')
plt.plot(x2s, y2s, '-', alpha=0.5, color='xkcd:dark magenta')
plt.axis('off')
plt.show()

```

```python
from matplotlib import animation
from IPython.display import HTML

fig, ax = plt.subplots(figsize=(6, 5))

ax.set_ylim([-1.1, 0.2])
ax.set_xlim([1, -1])

pendulum1, = ax.plot([], [])
pendulum2, = ax.plot([], [])

bar = tqdm_notebook(total=frames)

def init():
   pendulum1.set_data([], [])
   pendulum2.set_data([], [])
   return (pendulum1, pendulum2)

def update(n):
   bar.update(1)
   # update the line data
   pendulum1.set_data([0, x1s[n]], [0, y1s[n]])
   pendulum2.set_data([x1s[n], x2s[n]], [y1s[n], y2s[n]])
   return (pendulum1, pendulum2)

anim = animation.FuncAnimation(fig, update, init_func=init, frames=frames, blit=True, interval=1000 / fps)
anim.save('animation_pendulum.mp4', writer="ffmpeg")
```

```python
Video("./animation_pendulum.mp4")
```

```python
Video?
```

# Numerical Integration

## Numerical integration: quadrature

Numerical evaluation of a function of the type

$\displaystyle \int_a^b f(x) dx$

is called *numerical quadrature*, or simply *quadature*. SciPy provides a series of functions for different kind of quadrature, for example the `quad`, `dblquad` and `tplquad` for single, double and triple integrals, respectively.

```python
from scipy.integrate import quad, dblquad, tplquad
```

The `quad` function takes a large number of optional arguments, which can be used to fine-tune the behaviour of the function.

The basic usage is as follows:

```python
# define a simple function for the integrand
def f(x):
    return x**2
```

```python
x_lower = 0 # the lower limit of x
x_upper = 1 # the upper limit of x

val, abserr = quad(f, x_lower, x_upper)

print(f'integral value={val} , absolute error={abserr}' )
x = np.linspace(-1.2, 1.2, 200)
plt.plot(x, f(x))
plt.fill_between(x, f(x), alpha=0.3, where=(x > 0) & (x < 1))

```

If we need to pass extra arguments to integrand function we can use the `args` keyword argument:

```python
def integrand(x, n):
    return special.jn(n, x)

x_lower = 0  # the lower limit of x
x_upper = 10 # the upper limit of x

val, abserr = quad(integrand, x_lower, x_upper, args=(3,))
print(val, abserr)
```

For simple functions we can use a lambda function (name-less function) instead of explicitly defining a function for the integrand:

```python
val, abserr = quad(lambda x: np.exp(-x**2), -np.Inf, np.Inf)
print(f'numerical= {val}, {abserr}')
print(f'analytical solution = {np.sqrt(np.pi)}')
```

As show in the example above, we can also use 'Inf' or '-Inf' as integral limits.

Higher-dimensional integration works in the same way:

```python
def integrand(x, y):
    return np.exp(-x**2 - y**2)

x, y = np.meshgrid(np.linspace(-2, 2, 100), np.linspace(-2, 2, 100))
z = integrand(x, y)

fig = plt.figure()
ax = Axes3D(fig)
ax.plot_surface(x, y, z, cmap='YlGnBu')

# the integral boundarys can be functions when integrating in more than one dimensions
val, abserr = dblquad(integrand, -2, 2, lambda x: -2, lambda x: 2)

print(f'numerical solution= {val}, {abserr}')
```

Note how we had to pass lambda functions for the limits for the y integration, since these in general can be functions of x.

## Exercise II:  Integrals and Special Functions

Find the circumference of an ellipse. The formula for calculating circumference $C$ is as follows.

$$
\frac{C}{4a} =  E(\epsilon)=\int _{0}^{\pi /2}{\sqrt {1-\epsilon^{2}\sin ^{2}\theta }}\ d\theta
$$

where $\epsilon$ is the eccentricity of the ellipse given by the length and width of the ellipse.

$$
\epsilon = \sqrt{1 - \frac{b^2}{a^2}}
$$

compare the solution found using `quad` with the appropriate function from the `scipy.special` package.

```python
# %load solution_2.py
```

# Signal Processing

SciPy offers a large amount of utilities to handle time series data. However the methods in the `scipy.signal` are useful for much more than just time series data.

## Convolution

The convolution of two continuos functions is defined by a simple integral transform.

\begin{aligned}
(f*g)(t)&\,{\stackrel {\mathrm {def} }{=}}\ \int _{-\infty }^{\infty }f(\tau )g(t-\tau )\,d\tau \\&=\int _{-\infty }^{\infty }f(t-\tau )g(\tau )\,d\tau .
\end{aligned}

Convolution happens whenever data gets smeared out by a detector or filter of any kind.

```python
from scipy import signal

# create two rectangular signals
f = np.repeat([0., 1., 0.], 120)
g = np.repeat([0., 1., 0.], 20)

result = signal.convolve(f, g, mode='same')  # same output shape as input, see docs
```

```python
fig, (ax_orig, ax_win, ax_filt) = plt.subplots(3, 1, sharex=True)

ax_orig.plot(f, color='gray')
ax_orig.set_title('Original pulse')
ax_orig.margins(0, 0.1)

ax_win.plot(g, color='gray')
ax_win.arrow(70, 0.5, 50.0, 0.0, width=0.08, head_length=8)
ax_win.set_title('Filter impulse response')

ax_filt.plot(result)
ax_filt.set_title('Filtered signal')
None
```

The naive algorithm for producing discrete convolutions is not very efficient. It takes $\mathcal{O}(n^2)$ steps to get a result. SciPy uses a heuristic to automagically find the fastest convolution method. It is based on the *convolution theorem*

> The convolution theorem states that
>
 $$
 \mathcal{F}\{f*g\}=k\cdot {\mathcal {F}}\{f\}\cdot {\mathcal {F}}\{g\}
 $$
>
> where
> $\mathcal {F}\{f\}$ denotes the Fourier transform of $f$, and $\mathcal {F}\{g\}$ the Fourier transform of $g$.
>
> https://en.wikipedia.org/wiki/Convolution

## Signal Windows

SciPy conveniently provides a handful of common signal windows ready to be used for convolution or other filtering operations. All of them can be found in `scipy.signal.window` but their are also available in the `scipy.signal` name space.

```python
for g in [signal.nuttall(150), signal.hann(150), signal.cosine(150), signal.triang(150), signal.blackman(150)]:
    plt.plot(g)
plt.axis('off')
None
```

## Correlation

The correlation of two functions $f$ and $g$ looks similar to a convolution

> For continuous functions $f$ and $g$, the cross-correlation is defined as:
>
$$
(f\star g)(\tau )\ {\stackrel {\mathrm {def} }{=}}\int _{-\infty }^{\infty }f^{*}(t)\ g(t+\tau )\,dt,
$$
>
> where $f^{*}$ denotes the complex conjugate of  $f$, and $\tau$ is the displacement, also known as lag, although a positive value of $\tau$ actually means that $g(t+\tau)$ leads $f(t)$.

> *https://en.wikipedia.org/wiki/Cross-correlation*

The result of the correlation is a function of the delay $\tau$. This is useful for finding the offset in shifted signals.

```python
offset = 0.25
t = np.linspace(-1, 1, 600)
f = signal.gausspulse(t, fc=3)
g = signal.gausspulse( t + offset, fc=3)

result = signal.correlate(f, g, mode='same')
offset = t[np.argmax(result)]
```

```python
grid = plt.GridSpec(2, 2, wspace=0.1, hspace=0.4, width_ratios=[1, 1])
ax1 = plt.subplot(grid[0, 0])
ax2 = plt.subplot(grid[1, 0])
ax3 = plt.subplot(grid[:, 1], )

ax1.plot(t, f, color='gray')
ax1.set_title('$f$')
ax1.set_yticks([])
ax1.set_xlabel('t')

ax2.plot(t, g, color='gray')
ax2.set_title('$g$')
ax2.set_yticks([])
ax2.set_xlabel('t')

ax3.plot(t, result)
ax3.axvline(offset, color='xkcd:tangerine')
ax3.set_title('$(f\star g)(\\tau )$')
ax3.set_yticks([])
ax3.set_xlabel('τ')
None
```

## Exercise III:  Signal Filtering

If we ever find an alien civilization we will have to communicate with them using binary encoded signals recorded by large radio telescopes.

Assuming the aliens use binary signals with 0s and 1s.
Given the clock speed and the sample length with which the signal was created. Can you decode the following signal?

The plot below shows the shape of an example signal without any noise containing the message [0, 1 0]

```python
sample_length = 64
plt.plot(np.repeat([0, 1, 0], sample_length))
plt.text(25, 0.5, '0')
plt.text(100, 0.5, '1')
plt.text(175, 0.5, '0')
```

```python
alien_signal = np.loadtxt('alien_signal.txt')
sample_length = 128

# let's sample the signal in the middle of each period
clock = np.arange(sample_length // 2, len(alien_signal), sample_length)
plt.plot(alien_signal)
for c in clock:
    plt.axvline(c, color='gray', lw=1.5)
```

```python
# %load solution_3.py
```

# Fourier transform

Fourier transforms are one of the universal tools in computational physics, which appear over and over again in different contexts. SciPy provides functions for accessing the classic [FFTPACK](http://www.netlib.org/fftpack/) library, which is an efficient and well tested FFT library written in FORTRAN. The SciPy API has a few additional convenience functions, but overall the API is closely related to the original FORTRAN library.

To use the `fftpack` module in a python program, include it using:

```python
from scipy import fftpack
```

To demonstrate how to do a fast Fourier transform with SciPy, we will create some artificial data first.

```python
# Number of sample points
N = 1000
L = 10
# sample spacing
T = L / N
x = np.linspace(0.0, L, N)

freqs = [8, 3, 32]
amps = [1, 0.5, 0.25]

y = np.zeros(N)
for a, f in zip(amps, freqs):
    y += a * np.sin(2 * np.pi * f * x)

y += np.random.normal(0, 0.75, len(y))
plt.plot(x, y, lw=2)
```

```python
yf = fftpack.fft(y)
xf = np.linspace(0.0, 1.0/(2.0*T), N//2)
plt.plot(xf, 2.0/N * np.abs(yf[0:N//2]))

# plt.yscale('log')
# plt.xscale('log')
```

As expected, we now see a peaks in the spectrum that match the true frequencies in the signal.
Another, maybe more convenient,  way to create frequencies from a transformed signal uses the `fftfreq` function.

# Statistics

The `scipy.stats` module contains a large number of statistical distributions, statistical functions and tests. For a complete documentation of its features, see http://docs.scipy.org/doc/scipy/reference/stats.html.

Below we instantiate an object modeling a random variable following the normal distribution.

```python
from scipy import stats

mean = 0
sigma = 1
X = stats.norm(loc=mean, scale=sigma)
```

All distributions in this submodule extend either `scipy.stats.rv_continuous` or `scipy.stats.rv_discrete`.
Below is a list of functions supported by most continuous distribution objects

```
rvs(*args, **kwds) 	Random variates of given type.
pdf(x, *args, **kwds) 	Probability density function at x of the given RV.
logpdf(x, *args, **kwds) 	Log of the probability density function at x of the given RV.
cdf(x, *args, **kwds) 	Cumulative distribution function of the given RV.
logcdf(x, *args, **kwds) 	Log of the cumulative distribution function at x of the given RV.
sf(x, *args, **kwds) 	Survival function (1 - cdf) at x of the given RV.
logsf(x, *args, **kwds) 	Log of the survival function of the given RV.
ppf(q, *args, **kwds) 	Percent point function (inverse of cdf) at q of the given RV.
isf(q, *args, **kwds) 	Inverse survival function (inverse of sf) at q of the given RV.
moment(n, *args, **kwds) 	n-th order non-central moment of distribution.
stats(*args, **kwds) 	Some statistics of the given RV.
entropy(*args, **kwds) 	Differential entropy of the RV.
expect([func, args, loc, scale, lb, ub, …]) 	Calculate expected value of a function with respect to the distribution.
median(*args, **kwds) 	Median of the distribution.
mean(*args, **kwds) 	Mean of the distribution.
std(*args, **kwds) 	Standard deviation of the distribution.
var(*args, **kwds) 	Variance of the distribution.
interval(alpha, *args, **kwds) 	Confidence interval with equal areas around the median.
__call__(*args, **kwds) 	Freeze the distribution for the given arguments.
fit(data, *args, **kwds) 	Return MLEs for shape (if applicable), location, and scale parameters from data.
fit_loc_scale(data, *args) 	Estimate loc and scale parameters from data using 1st and 2nd moments.
nnlf(theta, x) 	Return negative loglikelihood function.

```

```python
N = 1200
measurments = X.rvs(size=N)
x = np.linspace(-3.5, 3.5, 200)

f, [ax1, ax2] = plt.subplots(1, 2, figsize=(16, 8))
ax1.hist(measurments, bins=40, density=True)
ax1.plot(x, X.pdf(x), color='red')

ax2.hist(measurments, bins=40, cumulative=True, density=True, )

ax2.plot(x, X.cdf(x), color='red')
None
```

Discrete functions work in a very similar manner.

```
rvs(*args, **kwargs) 	Random variates of given type.
pmf(k, *args, **kwds) 	Probability mass function at k of the given RV.
logpmf(k, *args, **kwds) 	Log of the probability mass function at k of the given RV.
cdf(k, *args, **kwds) 	Cumulative distribution function of the given RV.
logcdf(k, *args, **kwds) 	Log of the cumulative distribution function at k of the given RV.
sf(k, *args, **kwds) 	Survival function (1 - cdf) at k of the given RV.
logsf(k, *args, **kwds) 	Log of the survival function of the given RV.
ppf(q, *args, **kwds) 	Percent point function (inverse of cdf) at q of the given RV.
isf(q, *args, **kwds) 	Inverse survival function (inverse of sf) at q of the given RV.
moment(n, *args, **kwds) 	n-th order non-central moment of distribution.
stats(*args, **kwds) 	Some statistics of the given RV.
entropy(*args, **kwds) 	Differential entropy of the RV.
expect([func, args, loc, lb, ub, …]) 	Calculate expected value of a function with respect to the distribution for discrete distribution.
median(*args, **kwds) 	Median of the distribution.
mean(*args, **kwds) 	Mean of the distribution.
std(*args, **kwds) 	Standard deviation of the distribution.
var(*args, **kwds) 	Variance of the distribution.
interval(alpha, *args, **kwds) 	Confidence interval with equal areas around the median.
__call__(*args, **kwds) 	Freeze the distribution for the given arguments.
```

```python
# create a (discreet) random variable with poissionian distribution
Y = stats.poisson(3.5)
```

```python
n = np.arange(0, 12)

fig, axes = plt.subplots(3,1, sharex=True)

# plot the probability mass function (PMF)
axes[0].plot(n, Y.pmf(n), 'o')

# plot the commulative distribution function (CDF)
axes[1].step(n, Y.cdf(n))

# plot histogram of 1000 random realizations of the stochastic variable X
axes[2].hist(Y.rvs(size=200), bins=np.arange(13) - 0.5, density=True, lw=2, edgecolor='w');

axes[2].set_xticks(n)
None
```

The objects support easy access to some of its statistical properties:

```python
print(X.mean(), X.std(), X.var()) # normal distribution
print(Y.mean(), Y.std(), Y.var()) # poission distribution
```

## Statistical tests

The module contains some usefull methods to perform statistical tests.

Below we tests whether two sets of (independent) random data comes from the same distribution:

```python
# create two sets of data
N = 500
a = X.rvs(size=N)
b = X.rvs(size=N)

plt.hist(a, bins=50, histtype='step', lw=4, density=True)
plt.hist(b, bins=50, histtype='step', lw=4, density=True)
t_statistic, p_value = stats.ttest_ind(a, b)

print(f't-statistic = {t_statistic}, p-value = {p_value}')
```

Since the p value is very large we cannot reject the hypothesis that the two sets of random data have *different* means.

# Optimization

Optimization (finding minima or maxima of a function) is a large field in mathematics, and optimization of complicated functions or in many variables can be rather complicated. Here we will only look at a few very simple cases. For a more detailed introduction to optimization with SciPy see: http://scipy-lectures.github.com/advanced/mathematical_optimization/index.html

To use the optimization module in scipy first include the `optimize` module:

```python
from scipy import optimize
```

## Finding Minima

Let's first look at how to find the minima of a simple function of a single variable:

```python
def f(x):
    return 4.5*x**3 + (x-4)**2 + 0.75*x**4 - 20

x = np.linspace(-6, 3, 100)
plt.plot(x, f(x));
```

We can use the `minimize_scalar` function to find the minima of a scalar function. It provides wrapper around many optimization algorithms you can choose from using the `method` keyword.

```python
result = optimize.minimize_scalar(f)

plt.plot(x, f(x));
plt.axvline(result.x, color='xkcd:orange')

# add some boundaries
result = optimize.minimize_scalar(f, bracket=[-6, -2])
plt.axvline(result.x, color='xkcd:dark red')
```

## Exercise: Likelihood minimization (might be a bit advanced)

Imagine you have a sensor with 400 pixels. The pixel measure the number of photons in in a fixed length of time. Fit the mean of the photon cunts using the negative log likelihood of the poisson distribution

$$
X(k, \lambda) \; \sim \; \frac{\lambda^k}{k!} e^{-\lambda}
$$

```python
true_lambda=2
data = np.random.poisson(true_lambda, (20,20))
plt.imshow(data)
plt.grid()
```

```python
# %load solution_7.py
```

## Minimzation in more dimensions.

Below you'll find the infamous *Rosenbrock* function in two dimensions. It's very popular for benchmarking global optimization problems. Its solutions are well known and easily parametrized.

```python
from mpl_toolkits.mplot3d import Axes3D

def rosenbrock_2d(x, y, a=1, b=1):
    return (a - x)**2 + b*(y - x**2)**2

fig = plt.figure()
ax = Axes3D(fig)

x, y = np.meshgrid(np.linspace(-2 ,2, 100), np.linspace(-3. ,5, 100))
ax.plot_surface(x, y, rosenbrock_2d(x, y), cmap='magma_r')
```

```python
result = optimize.minimize(lambda x: rosenbrock_2d(x[0], x[1]), x0=[5, 2])
fig = plt.figure()
ax = Axes3D(fig)
ax.view_init(20, 30)

x, y = np.meshgrid(np.linspace(-2 ,2, 100), np.linspace(-3. ,5, 100))
ax.plot_surface(x, y, rosenbrock_2d(x, y), cmap='magma_r', alpha=0.6)

x, y, z = result.x[0], result.x[1], 20
ax.quiver(x, y, z, 0, 0, -1, length=20, arrow_length_ratio=0.04)

result
```

## Least Squares Minimization

Lets presume a linear weighted combination of variables:

$$
f(x)= \hat{y} =  \hat{\beta}_0 + \sum_{j=1}^p x_j \hat{\beta}_j
$$
where $f:\mathbb{R}^{p} \to \mathbb{R}$.

When we include a 1 as the first entry into our sample $x$ e.g. $x = (1, x_1, x_2, \ldots, x_p)$ we can rewrite
$f$ in matrix form

$$
f(x)= \hat{y} =  x^T \mathbf{\beta}
$$

where $\beta = (\beta_0, \beta_1, \beta_2, \ldots, \beta_p)$.

How do you find those weights? Like before we choose a loss function and try to opimize it.
In this case we choose a loss function called the residual sum of squares (RSS).
We calculate it over all samples $x_i$ in a matrix $\mathbf{X}$.

$$L(\beta) = RSS(\mathbf{\beta}) = \sum_{i=1}^N (y_i - x_i^T \beta)^2 $$

Here $x_i$ is a row in $\mathbf{X}$, hence the transpose.

We can now rewrite the loss function in matrix form:

$$
RSS(\beta) = (\mathbf{y} - \mathbf{X} \beta)^T (\mathbf{y} - \mathbf{X} \beta )
$$

Now we optimize the loss function just like we would any other function, by differentiating with respect to $\beta$ and setting the result equals to zero.

$$
 \mathbf{X}^T (\mathbf{y} - \mathbf{X} \beta ) \stackrel{!}{=} 0
$$

Solving for $\beta$ leads to

$$
\hat{\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}
$$

We just performed  __Linear Least Squares__ regression. Below we fit some noisy a polynomial data.

```python
def f(x, beta=[4.5, 1, 0.75]):
    return beta[0]*x**3 + beta[1]*(x-4)**2 + beta[2]*x**4

x = np.linspace(-6, 3, 100)
data = f(x) + np.random.normal(0, 35, size=x.shape)

# build the answer matrix
A = np.vstack([x**3, (x - 4)**2, x**4]).T

# find the linear least squares solution for the parameters
result = optimize.lsq_linear(A, data)

plt.plot(x, data, '.')
plt.plot(x, f(x), color='gray', label='truth')
plt.plot(x, f(x, beta=result.x), color='crimson', label='fit')
plt.legend()
```

## Curve Fitting

For more general cases where the parameters are not linear there is the more convenient function `curve_fit`

```python
from scipy import stats
def f(x, scale=5, loc=4, ):
    return (1/(scale))**2*x**3 + (1/(2*loc))*x**2

x = np.linspace(-3,  1.5, 100)
data = f(x) + np.random.normal(0, 0.15, size=x.shape)

popt, pcov = optimize.curve_fit(f, x, data, p0=[5, 4, ])

plt.plot(x, data, '.')
plt.plot(x, f(x), color='gray', label='truth')
plt.plot(x, f(x, *popt), color='crimson', label='fit')

pcov
```

## Exercise VI:  Curve Fitting

Use `numpy.loadtxt` to load the file called `munich_temperatures_average.txt` and fit a model to it using `optimize.curve_fit`.

```python
# %load solution_6.py
```

# Interpolation

Interpolation is simple and convenient in scipy: The `interp1d` function, when given arrays describing X and Y data, returns and object that behaves like a function that can be called for an arbitrary value of x (in the range covered by X), and it returns the corresponding interpolated y value:

```python
from scipy import interpolate

def f(x):
    return np.sin(x)
```

```python
n = [0, 1, 2, 3, 3.5, 4, 6, 6.5, 7, 9]
x = np.linspace(0, 9, 100)

y_measured = f(n)

linear_interpolation = interpolate.interp1d(n, y_measured)

cubic_interpolation = interpolate.interp1d(n, y_measured, kind='cubic')
# this returns a new function we can call for values of x
cubic_interpolation
```

```python
plt.plot(n, y_measured, 's', label='noisy data', color='black')
plt.plot(x, f(x), c='gray', lw=2, label='true function')
plt.plot(x, linear_interpolation(x), label='linear interpolation')
plt.plot(x, cubic_interpolation(x), label='cubic interpolation')
plt.legend();
```

# SciPy IO Module

## IO and the sound of hydrogen

SciPy provides some useful methods for reading file from MatLab, NetCDF or WAV data. Below is an example which uses the `scipy.io.wavfile` module to write a file containing the sound of hydrogen atoms.

```python
from scipy.io import wavfile
from scipy import signal

rate = 44100 #44.1 khz
duration = 10 # in sec

def rydberg(n, m):
    return (1/(n**2) -1/(m**2))

def lyman(m):
    return rydberg(1,m)

def balmer(m):
    return rydberg(2,m)

def sound(frequency, time):
    return 2**12*np.sin(2*np.pi*440*frequency*time)

t = np.linspace(0, duration, rate*duration)
hydrogen_sound = np.sum([sound(lyman(i), t) + sound(balmer(i), t) for i in range(2, 12)], axis=0)/6

plt.plot(t[:10000], hydrogen_sound[:10000])

wavfile.write('hydrogen.wav', rate, hydrogen_sound.astype(np.int16))
```

```python
%%HTML
<audio controls="controls" style="width:600px" >
  <source src="hydrogen.wav" type="audio/wav" />
</audio>
```

## Exercise IV:  Signal Processing and IO.

Load the file `synth_sound.wav` and plot a the frequency spectrum vs time.

```python
# %load solution_4.py
```

# Root Finding

To find the solution for a function of the form $f(x) = 0$ we can use the `fsolve` or the `root` function. It requires one or more initial guesses:

```python
def f(x):
    return 4.5*x**3 + (x-4)**2 + 0.75*x**4 - 20

x = np.linspace(-6, 3, 100)

plt.plot(x, f(x))
plt.axhline(color='black')

result = optimize.root(f, [-6, -3, 0, 2])
for r in result.x:
    plt.axvline(r, color='xkcd:tangerine')

```

# Sparse matrices

Sparse matrices are often useful in numerical simulations dealing with large systems, if the problem can be described in matrix form where the matrices or vectors mostly contains zeros. Scipy has a good support for sparse matrices, with basic linear algebra operations (such as equation solving, eigenvalue calculations, etc).

There are many possible strategies for storing sparse matrices in an efficient way. Some of the most common are the so-called coordinate form (COO), list of list (LIL) form,  and compressed-sparse column CSC (and row, CSR). Each format has some advantanges and disadvantages. Most computational algorithms (equation solving, matrix-matrix multiplication, etc) can be efficiently implemented using CSR or CSC formats, but they are not so intuitive and not so easy to initialize. So often a sparse matrix is initially created in COO or LIL format (where we can efficiently add elements to the sparse matrix data), and then converted to CSC or CSR before used in real calcalations.

For more information about these sparse formats, see e.g. http://en.wikipedia.org/wiki/Sparse_matrix

When we create a sparse matrix we have to choose which format it should be stored in. For example,

```python
from scipy import sparse
```

```python
# dense matrix
M = np.array([[1,0,0,0], [0,3,0,0], [0,1,1,0], [1,0,0,1]]); M
```

```python
# convert from dense to sparse
A = sparse.csr_matrix(M)
A
```

```python
# convert from sparse to dense
A.todense()
```

More efficient way to create sparse matrices: create an empty matrix and populate with using matrix indexing (avoids creating a potentially large dense matrix)

```python
A = sparse.lil_matrix((4,4)) # empty 4x4 sparse matrix
A[0,0] = 1
A[1,1] = 3
A[2,2] = A[2,1] = 1
A[3,3] = A[3,0] = 1
A
```

```python
A.todense()
```

Converting between different sparse matrix formats:

```python
A
```

```python
A = sparse.csr_matrix(A); A
```

```python
A = sparse.csc_matrix(A); A
```

We can compute with sparse matrices like with dense matrices:

```python
A.todense()
```

```python
(A * A).todense()
```

```python
A.todense()
```

```python
A.dot(A).todense()
```

```python
v = np.array([1,2,3,4])[:,np.newaxis]
v
```

```python
# sparse matrix - dense vector multiplication
A * v
```

```python
# same result with dense matrix - dense vector multiplcation
A.todense() * v
```


# Finite Rate of Innovation (FRI) introduction

In this notebook, we provide an introductory tutorial on the Finite Rate of Innovation (FRI), based on the following paper ([IEEE link](https://ieeexplore.ieee.org/document/1003065/)):

    M. Vetterli, P. Marziliano and T. Blu, "Sampling signals with finite rate of innovation," in IEEE Transactions
    on Signal Processing, vol. 50, no. 6, pp. 1417-1428, Jun 2002.

The purpose is to breakdown the underlying concept and sampling approach in an intuitive fashion, i.e. with code and clear explanations. In this notebook, we will consider the **_noiseless baseband_** case as to develop the intuition behind FRI. In another notebook, we will tackle _modulated_ and _noisy_ situations. For a more rigorous treatment, please refer to the paper!

```python
import numpy as np
import sys
sys.path.append('..')
import plot_settings
from frius import create_pulse_param, sample_ideal_project, estimate_fourier_coeff, compute_ann_filt
```

# 1. Projecting to bandlimited space prior to sampling

The fundamental Nyquist-Shannon sampling theorem sets a lower bound on the sampling rate in order to perfectly recover **_bandlimited signals_**, i.e. signals of finite frequency support: "if a function $ f(t) $ contains no frequencies higher than $ f_m $ Hz, it is completely determined by giving its ordinates at a series of points spaced $ (1/2f_m) $ seconds apart" [1].

If the signal we would like to sample is not bandlimited, we can first **_project_** it to a bandlimited space by applying a lowpass filter with the appropriate cut-off frequency. Assume that we are able to use a very simple (but very ideal) low-pass filter $ s_B(t) $, whose frequency response can be written as:

\begin{align}
	\hat{s}_B(f) =
	\begin{cases}
	1/B, & f\in[-B/2, B/2]   \\[0.5em]
	0,      & \text{otherwise},  \\
	\end{cases} \nonumber
\end{align}

where the amplitude is $ (1/B) $ so that we have unit norm, $ \|\hat{s}_B(f)\|_2 = 1 $. An example of such a filter can be seen below. The purpose of this filter is to prevent aliasing, which is why it is often called an **_anti-aliasing_** filter.

```python
# create ideal low-pass filter
cutoff = 24000
n_vals = 1000
freqs = np.linspace(-1e5, 1e5, 1000)
lpf = np.ones(n_vals)
lpf[freqs<-cutoff] = 0
lpf[freqs>cutoff] = 0

plt.title("Ideal low-pass filter")
plt.plot(freqs, lpf)
plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.xlabel("Frequencies [Hz]");
```

By taking the inverse Fourier Transform (FT), we can express $ \hat{s}_B(f) $ in the time-domain as:

\begin{align}
s_B(t) = \text{sinc}(Bt),
\end{align}

where $ \text{sinc}(t)=\sin(\pi t)/\pi t $.

After applying our ideal projection/filter to our input signal, the maximum frequency has been set to $ f_m = B/2 $. So according to Nyquist-Shannon, in order to sample the _now bandlimited signal_ so that it can be perfectly reconstructed from the measured samples, it is necessary to obtain samples at every $T_s = (1/B)$ seconds (or less), i.e. sample it at a rate $ f_s \geq B$~Hz. We will refer to **_critical sampling_** as the case where we obtain just enough samples for perfect reconstruction. We can combine our filtering with critical sampling as such:

\begin{align}
	y_{BL}[n] &= y_{BL}(nT_s) \nonumber\\[0.05cm]
	&= (y \ast s_B)(u)|_{u = nT_s} \nonumber \\
	&= \int_{-\infty}^{\infty} y(t) \hspace{0.05cm}  s_B(nT_s-t) dt \nonumber\\
	&= \int_{-\infty}^{\infty} y(t) \hspace{0.05cm} \text{sinc}\Big(B(nT_s-t)\Big) dt \nonumber\\
	&= \int_{-\infty}^{\infty} y(t) \hspace{0.05cm} \text{sinc}\Big(\dfrac{nT_s-t}{T_s}\Big)  dt,\nonumber
\end{align}

where $ n \in \mathbb{Z} $ and we use that $ T_s=(1/B) $ for our critical sampling case. We can equally express the above critical sampling in a single line as an inner product:

\begin{align}
y_{BL}[n] = \langle y, \tilde{\varphi }_n \rangle, \nonumber
\end{align}

if we set $ \tilde{\varphi}_n(t) = \text{sinc}\big((nT_s-t)/T_s\big) $.

# 2. Bandlimited reconstruction

 We can then perfectly reconstruct the best bandlimited approximation $ y_{BL}(t) $, in a least-squares sense, of our original non-bandlimited signal from our samples $ y_{BL}[n] $ as such:

\begin{align}
 y_{BL}(t) &= \sum_{n\in\mathbb{Z}} y_{BL}[n] \hspace{0.05em} \text{sinc}\Big(\dfrac{t - nT_s}{T_s}\Big). \nonumber
 \end{align}

Equivalently, if we let $ \varphi_n(t) = \tilde{\varphi}_n(-t) = \text{sinc}\big((t - nT_s)/T_s\big)$ and use our inner product notation from before, we can write the whole sampling _and_ reconstruction process in one shot!

 \begin{align}
 	y_{BL}(t) &=\sum_{n\in\mathbb{Z}} \langle y, \tilde{\varphi }_n \rangle \hspace{0.05em} \varphi_n. \nonumber
 \end{align}

If $ y(t) $ was already bandlimited to $ [-B/2,B/2] $, then $ y_{BL}(t) =y(t) $, reinforcing our idea of sampling and reconstruction as a projection, due to the process being idempotent. We can also observe that our optimal reconstruction filter $ \varphi_n $ and the anti-aliasing operation $ \tilde{\varphi}_n $ are flipped versions of each other for perfect recovery. This is due to $ \varphi_n $ being orthonormal to its integer shifts of $ T_s $: $ \langle \varphi_n, \varphi_k \rangle = \delta_{n-k} $.

# 3. Generalizing the projection operation

We can interpret $ \varphi(t) = \text{sinc}(Bt) $ as a **_generating function_** for the bandlimited space $ [-B, B] $ as it allows us to construct any signal in this space with the appropriate shifts and weights of $ \varphi(t) = \text{sinc}(Bt) $. In [2], this generating function idea was extended to other functions that allow for sampling and reconstruction with more _practical_ choices of $ \varphi(t) $, i.e. those with a much faster decay than $ \text{sinc}(t) $ and with compact support. One idea was to relax the orthonormal requirement, e.g. with splines which can approximate polynomials up until a certain degree very well. In ou bandlimited reconstruction, we specifically saw the best approximation in the bandlimited space, which corresponds to a basis of shifted sinc functions. This idea can be extended to a different generating function at integer shifts of $ T_s $, i.e. uniform samples, for which we can denote the approximation in this $ \varphi_n(t)$ basis as $y_{\varphi}(t)$, namely:

\begin{align}
 	y_{\varphi}(t) &=\sum_{n\in\mathbb{Z}} \langle y, \tilde{\varphi }_n \rangle \hspace{0.05em} \varphi_n. \nonumber
\end{align}

# 4. The "rate of innovation" perspective

Another extension made by [3] is to consider the **_degrees of freedom per unit of time_**, for which the authors coined the term **_rate of innovation_**. For our bandlimited reconstruction, Nyquist-Shannon required us to have samples spaced $ (1/B) $ seconds apart, if the bandwidth of our input signal to limited to $ B $. This implies a **_finite_** rate of innovation (FRI) of $ \rho = B = (1/T_s) $, or $ \rho = (1/2f_m) $ when expressed in terms of the maximum frequency. On the other hand, an arbitrary non-bandlimited signal could have an _infinite_ rate of innovation.

Continuing with the narrative of [3], we can write our sampling and reconstruction for at _arbitrary_ shifts $ t_n $ instead of integer shifts of $ T_s $:

\begin{align}
    y_{\varphi}(t)  &= \sum_{n\in\mathbb{Z}} \langle y, \tilde{\varphi }_n \rangle \hspace{0.05em} \varphi_n \nonumber
\end{align}

\begin{align}
\Rightarrow y_{\varphi}(t) &= \sum_{n\in\mathbb{Z}} c_n \hspace{0.05em} \varphi\Big(\dfrac{t - t_n}{T_s}\Big), \nonumber
\end{align}

where we now express $ \langle y, \tilde{\varphi }_n \rangle $ as coefficients $ c_n $ for our shifted generating functions (as was also done in [2]). From the more general expression above, we are interested in the conditions for which the approximation $ y_{\varphi}(t)  $ perfectly matches the original signal $ y(t) $. For the bandlimited case, we have the following "ingredients":

1. $ y(t) $ is bandlimited to $ B $; if not we make it bandlimited and are able to perfectly reconstruct this bandlimited version.
2. $ c_n $ are samples taken at every $ T_s = (1/B) $ seconds.
3. $ \varphi(t) = \text{sinc}(Bt) $, which will be "placed" at integer shifts of $ T_s $.

However, in certain cases can we do better than a bandlimited approximation? In other words, can we achieve perfect reconstruction of $ y(t) $ in a situation deemed "impossible" according to the Nyquist-Shannon sampling theorem? Consider a single Dirac which has **_infinite bandwidth_**:

\begin{align}
y(t) = \delta(t-t_0) \xrightarrow{\mathscr{F}} \hat{y}(f) = \exp(-j2\pi f t_0). \nonumber
\end{align}

We would certainly be able to sample and reconstruct a bandlimited version of this signal according to Nyquist-Shannon, where an anti-aliasing filter/generating function pair with larger and larger bandwidth would allow us to obtain an increasingly better bandlimited approximation. But determining the precise location $ t_0 $ of the Dirac would certainly be a much more faithful representation! Similarly, we can consider a $ T $-periodic sum of $ K $ known pulses $ h(t) $:

\begin{align}
y(t) = \sum_{n\in\mathbb{Z}}\sum_{k=0}^{K-1}c_k \hspace{0.05cm} h(t-t_k-nT), \nonumber
\end{align}

where in the single Dirac case we simply had $ h(t)=\delta(t) $ and $ K=1 $. If we instead have a pulse stream over a finite duration $ T $, we can define it as $ T $-periodic as by simply repeating the signal at every $ T $ seconds. It does not make a significant difference for the recovery process that will be presented below, as it works on one period of the signal.

A noteworthy observation we can make about the above **_pulse stream_** signal is that given we know the pulse shape $ h(t) $, there are only $2K$ degrees of freedom per period: the $K$ locations $\{t_k\}_{k=0}^{K-1}$ and the $K$ amplitudes $\{c_k\}_{k=0}^{K-1}$. We have, however, made a few __very strong__ assumptions:

1. Our signal consists of exactly $ K $ pulses.
2. We know the exact shape of the pulse.
3. There is only one pulse shape.

In the general bandlimited case, our only assumption was that our signal had (essentially) no frequency content above a certain maximum frequency $ f_m$. If not, we apply an anti-aliasing filter to project the signal onto a known bandlimited space. Nevertheless, if our pulse stream model assumption is correct, we can safely say that there are $ 2K $ degrees of freedom per period $ T $, implying an FRI of $ \rho = (2K/T) $.

In [3], a method for recovering the unknown finite parameters of a pulse stream is presented in the noiseless scenario. In the remainder of this notebook, we look at how this recovery is performed.

# 5. Pulse stream recovery

As in the bandlimited case, we will obtain samples of our pulse stream after it has gone through a sampling kernel $ s_B(t) $, which may or may not be of our choosing.

## Sampling requirements

Moreover, for pulse stream recovery, we require:

1. The bandwidth $ B $ of our sampling kernel $ s_B(t) $ to be greater than and or equal to the rate of innovation $ \rho $.
2. $ N \geq (2M+1) $ uniform samples where $ M = \lfloor BT/2 \rfloor $. Moreover, the sampling period should be $ T_s = T/N $.

As for the bandlimited case, we will require a sampling kernel with bandwidth $ B \geq \rho $. For critical sampling of a pulse stream signal, this corresponds to $ B = (2K/T) $, where $ K $ is the number of pulses and the $ T $ is the period of the signal.

In the bandlimited scenario, we then require samples at every $ T_s = (1/B) $ seconds. For pulse stream recovery, in order to use real-valued sampling kernels, i.e. $ \hat{s}_B(f) $ being conjugate symmetric, we will need to meet the second condition above, yielding the following samples:

\begin{align}
y_{BL}[n] &= y(nT_s)  =\langle y(t), s_B(nT_s - t) \rangle \nonumber
\end{align}

and if we again assume an ideal sampling kernel as in the bandlimited case:

\begin{align}
y_{BL}[n] = \langle y(t), \text{sinc}\big((nT_s-t)B\big)  \rangle  \text{ for }  n\in[0, 1, \ldots, \mathrel{{N}{-}{1}}].\nonumber
\end{align}

## Recovery in code and visuals

With these samples, the recovery boils to determining a **_mask_** whose zeros coincide precisely with the location of the pulses.

Below we consider a pulse stream with six _Diracs_ that goes through a sampling kernel that meets the above requirements. In other words, are signal has the following form:

\begin{align}
y(t) = \sum_{n\in\mathbb{Z}}\sum_{k=0}^{K-1}c_k \hspace{0.05cm} \delta(t-t_k-nT), \nonumber
\end{align}

where $K=6$ and for this example we will set $T=1$. Consequently $y(t)$ goes through a sampling kernel with the following frequency response.

\begin{align}
	\hat{s}_B(f) =
	\begin{cases}
	1/B, & f\in[-B/2, B/2]   \\[0.5em]
	0,      & \text{otherwise},  \\
	\end{cases} \nonumber
\end{align}

where $B=2K/T$, i.e. a filter with the minimum bandwidth possible for recovery. Finally, we obtain $N=2K+1$ samples, corresponding to a sampling rate of $f_s = (2K+1)/T$.

Below, we start off by creating and signal parameters and synthesize the lowpass filtering and sampling. See 'frius/fri_utils.py' for the definitions of the used functions.

```python
K = 6
period = 1
ck, tk = create_pulse_param(K=K, period=period)
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, viz=True, figsize=(10,5))
```

The novelty in the pulse stream recovery algorithm of [3] is in identifying the "mapping" between our measured samples and values with which we can determine this **_mask_** that will lead us to the pulse stream parameters. This idea can be generalized to models different than the pulse stream we are considering here, as are also discussed in [3] and [4].

For the pulse stream signal, the FT of our samples yield us the samples that will allow us to determine the mask, which is also referred to as the **_annihilating filter_**.

```python
fs_coeff_hat = estimate_fourier_coeff(y_samp, t_samp)
ann_filt = compute_ann_filt(fs_coeff_hat, K)
```

The term "annihilation" arises from the fact that when we convolve our annihilating filter with the Fourier coefficients of the measured samples, the resulting output will be zero, i.e. annihilated!

```python
print("ann_filt conv fs_coeff_hat = ", np.abs(np.convolve(ann_filt, fs_coeff_hat, mode='valid')))
```

    ann_filt conv fs_coeff_hat =  [  3.12639119e-13   2.64074429e-14   1.18856725e-13   4.61886137e-14
       1.15667977e-13   2.89169943e-14   3.79429987e-15]

Moreover, we can observe that the zeros of our mask (the time domain representation of the annihilating filter) coincide exactly with the zeros of our pulse stream!

```python
# annihilating filter in time domain
t_vals = np.linspace(0., period, num=1000, endpoint=True)
z = np.exp(-1j*2*np.pi*t_vals/period)
mask = np.abs(np.polyval(ann_filt, z))
mask /= max(mask)

# plot
plt.figure(figsize=(12,5))
baseline = plt.stem(tk, ck/max(abs(ck)), 'g', markerfmt='go', label="Pulse parameters")[2]
plt.setp(baseline, color='g')
baseline.set_xdata([0, period])
plt.xlim([0,period])
plt.grid()
plt.xlabel("Time [seconds]")
plt.plot(t_vals, mask, label="Mask")
plt.legend(loc='lower left');
```

Therefore, recovering the locations of our pulses, $\{t_k\}_{k=0}^{K-1}$, corresponds precisely to determining the zeros of this mask!

```python
uk = np.roots(ann_filt)
uk = uk / np.abs(uk)                                         # project to unit circle
tk_hat = np.real(1j * period * np.log(uk) / (2 * np.pi))     # invert "mapping" to get to parameters
tk_hat = tk_hat - np.floor(tk_hat / period) * period         # ensure location within [0, period]
tk_hat = np.sort(tk_hat)                                     # sort for comparison as `tk` are sorted
print("||tk - tk_hat||_2 = %f " % np.linalg.norm(tk-tk_hat))
```

    ||tk - tk_hat||_2 = 0.000000

The function 'estimate_time_param' in 'frius/fri_utils.py' implements the above code snippet.

Once we know the pulse locations, determining the amplitudes of the pulses, $\{c_k\}_{k=0}^{K-1}$, can be cast as a least-squares problem by "placing" the pulses at the recovered positions, forming the forward matrix between the amplitudes and the measured samples.

We can also perform the "fitting" in the frequency domain as is done below. Moreover, the Fourier coefficient indices divided by the period yields the corresponding frequency values in Hertz.

```python
tk_grid, freqs_grid = np.meshgrid(tk_hat, fs_ind_base/period)
Phi = np.exp(-1j*2*np.pi*freqs_grid*tk_grid)/period             # forward matrix A in y=Ax
ck_hat = np.real(lstsq(Phi, fs_coeff_hat)[0])                   # 'fs_coeff_hat' corresponds to 'y'
print("||ck - ck_hat||_2 = %f " % np.linalg.norm(ck-ck_hat))
```

    ||ck - ck_hat||_2 = 0.000000

The function 'estimate_amplitudes' in 'frius/fri_utils.py' implements the above code snippet.

## Recovery steps

Assuming the above sampling requirements have been met, we can use the following procedure to recover the $2K$ parameters of an (ideal) lowpass filtered **_Dirac_** stream with $K$ pulses [3]:

1. With the samples, estimate the Fourier coefficients with the DFT, as done with the function 'estimate_fourier_coeff' in 'frius/fri_utils.py'.
2. Obtain the annihilating filter for the Fourier coefficients. See [3] for a mathematical description and 'compute_ann_filt' in 'frius/fri_utils.py' for implementation details.
3. Find the $K$ zeros of the mask corresponding to the annihilating filter. This will yield the locations $\{t_k\}_{k=0}^{K-1}$.
4. With the locations and measured samples, solve a least-squares for the pulse amplitudes $\{c_k\}_{k=0}^{K-1}$.

We place an emphasis on **_Dirac_**, as an additional step needs to be performed **_before_** determining the annihilating filter in order to work with arbitrary pulse shapes. This will be discussed in the next notebook!

# References

    [1] C. E. Shannon. "Communication in the Presence of Noise," In: Proceedings of the IRE 37 (1949), pp. 10–21.
    [2] M. Unser. "Sampling-50 years after Shannon". In: Proceedings of the IEEE 88 (2000), pp. 569–587.
    [3] M. Vetterli, P. Marziliano and T. Blu, "Sampling signals with finite rate of innovation," in IEEE Transactions on Signal Processing, vol. 50, no. 6, pp. 1417-1428, Jun 2002.
    [4] P. L. Dragotti, M. Vetterli and T. Blu, "Sampling Moments and Reconstructing Signals of Finite Rate of Innovation: Shannon Meets Strang–Fix," in IEEE Transactions on Signal Processing, vol. 55, no. 5, pp. 1741-1757, May 2007.


```python
%matplotlib inline
%config IPython.matplotlib.backend = 'retina'
%config InlineBackend.figure_format = 'retina'
```

```python
import matplotlib
import matplotlib.pyplot as plt
from cycler import cycler
matplotlib.rc("font", family="serif", size=14)
#matplotlib.rc("figure", figsize="10, 5")
colors = ['k', 'c', 'm', 'y']
matplotlib.rc('axes', prop_cycle=cycler("color", colors))

import scipy.optimize
import numpy as np
```

# Example: Fitting a line to data
Our model will consist of a set of  NN  i.i.d. observations, including: a coordinate (fixed parameter)  xixi , a noise level  σiσi , and observed variable  ŷ iy^i  drawn from a Gaussian with mean  mxi+bmxi+b  and variance  σ2iσi2 , i.e.  ŷ i∼(mxi+b;σ2i)y^i∼N(mxi+b;σi2) .

## generate a model:

```python
slope_true = np.random.uniform(0, 1)
intercept_true = np.random.uniform(0, 1)
print('Slopes:', slope_true)
print('Intercepts:', intercept_true)
# This notebook is ready for you to play with 2+ components and more complicated models.
```

    Slopes: 0.760292197390898
    Intercepts: 0.22579349625822198

## generate some data drawn from that model:

```python
ndatapoints = 20
xis_true = np.random.uniform(0, 1, ndatapoints)
x_grid = np.linspace(0, 1, 100)

def model_linear(xs, slope, intercept): return xs * slope + intercept
yis_true = model_linear(xis_true, slope_true, intercept_true)
```

```python
ndatapoints = 20
xis_true = np.random.uniform(0, 1, ndatapoints)
x_grid = np.linspace(0, 1, 100)

def model_linear(xs, slope, intercept): return xs * slope + intercept
yis_true = model_linear(xis_true, slope_true, intercept_true)

sigma_yis = np.repeat(0.1, ndatapoints) * np.random.uniform(0.5, 2.0, ndatapoints)
yis_noisy = yis_true + np.random.randn(ndatapoints) * sigma_yis

y_min, y_max = np.min(yis_noisy - sigma_yis), np.max(yis_noisy + sigma_yis)
y_min = np.min([y_min, np.min(model_linear(x_grid, slope_true, intercept_true))])
y_max = np.max([y_max, np.max(model_linear(x_grid, slope_true, intercept_true))])

plt.figure(figsize=(10, 5))
plt.plot(x_grid, model_linear(x_grid, slope_true, intercept_true), c=colors[0])
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', c=colors[0])
plt.xlabel('$x$'); plt.ylabel('$y$');# plt.ylim([y_min, y_max])
```

## Pretend we don't know the model

```python
y_min, y_max = 0, 1
plt.figure(figsize=(10, 5))
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
plt.xlabel('$x$'); plt.ylabel('$y$'); #plt.ylim([y_min, y_max])
```

## What do we need to use MCMC to find?

### Bayes Theorem

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) \  p\bigl(m, b\bigr) $$

We want to know how the parameters m and b are distributed given the data

### Likelihood

$$\begin{align}
p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) &\ = \prod_{i=1}^{N} p(\hat{y}_i \vert x_i, \sigma_i, b, m)\\
&\ = \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \\
&\ = \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi}\sigma_i}\exp\left( - \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right)  \\
&\ \propto \ \exp\left( - \sum_{i=1}^{N} \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right) \
\end{align}
$$

Since the data points are independent and the noise is Gaussian.

use log posterior, increases in same way and is numerically more managable

```python
def ln_like(params, xs, observed_yis, yi_uncertainties):
    model_yis = model_linear(xs, params[0], params[1])
    chi2s = ((observed_yis - model_yis) / yi_uncertainties)**2
    return np.sum(-0.5 * chi2s - 0.5*np.log(2*np.pi) - np.log(yi_uncertainties))

def ln_prior(params):
    if np.any(params < 0) or np.any(params > 1):
        return - np.inf
    return 0.

def ln_post(params, xs, observed_yis, yi_uncertainties):
    lnprior_val = ln_prior(params)
    if ~np.isfinite(lnprior_val):
        return lnprior_val
    else:
        lnlike_val = ln_like(params, xs, observed_yis, yi_uncertainties)
        return lnprior_val + lnlike_val
```

```python
x0 = np.array([0.5, 0.5])
print('Likelihood:', ln_like(x0, xis_true, yis_noisy, sigma_yis))
print('Prior:', ln_prior(x0))
print('Posterior:', ln_post(x0, xis_true, yis_noisy, sigma_yis))
```

    Likelihood: 0.718904638822
    Prior: 0.0
    Posterior: 0.718904638822

## The Metropolis-Hastings algorithm

For a given target probability $p(\theta)$ and a (symmetric) proposal distribution $p(\theta_{i+1}|\theta_i)$. We repeat the following:
- draw a sample $\theta_{i+1}$ given $\theta_i$ from the proposal distribution,
- compute the acceptance probability ratio $a={p(\theta_{i+1})}/{p(\theta_i)}$,
- draw a random uniform number $r$ in $[0, 1]$ and accept $\theta_{i+1}$ if $r < a$.

```python
num_draws = 1000
params_drawn = np.zeros((num_draws, 2))
i_draw = 1
num_draws_tot = 0
params_drawn[0, :] = np.random.uniform(0, 1, 2)
while i_draw < num_draws:
    num_draws_tot += 1
    params_drawn[i_draw, :] = params_drawn[i_draw-1, :] \
        + 0.05 * np.random.randn(2)
    a = np.exp(ln_post(params_drawn[i_draw, :], xis_true, yis_noisy, sigma_yis)\
                   - ln_post(params_drawn[i_draw-1, :], xis_true, yis_noisy, sigma_yis))
    if a >= 1 or np.random.uniform(0, 1, 1) < a:
        i_draw += 1
print('Acceptance rate:', num_draws/num_draws_tot)
```

    Acceptance rate: 0.39635354736424894

```python
fig, axs = plt.subplots(1, 1, sharex=True, sharey=True, figsize=(8,6))
axs.hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs.set_title('Metropolis Hastings');
axs.set_xlabel('slope'); axs.set_ylabel('intercept');
axs.set_xlim(0.0,1.0)
axs.set_ylim(0.0, 1.0)
```

Visualise the chains:

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(params_drawn[:, i]);
```

Has the chain converged?
What is the acceptance fraction?

Choices: proposal distribution, starting point, number of steps

```python

```