# Numerically solving the Schrodinger Equation

This python notebook will show a simple method to find solutions to the Schrodinger equation using what is called the shooting method. Recall that the Schrodinger equation can be written as:

$$ -\frac{\hbar^2}{2 m} \frac{\partial^2 \psi}{\partial x^2} +(V(x)- E) \psi =0 $$

where $V(x)$ is the potential and both the energy $E$ and the wavefunction $\psi$ are unknown.
The fact that we do not know the eigenvalue $E$ means that we have to use trial and error to guess a value for $E$ and the numerically solve Schrodinger's equation to find $\psi(x)$. We will know if our solution is correct if the wavefunction approaches 0 for large $x$.

To proceed we first change our units so that $\hbar= m=1$ and then add in a new variable $\phi$ which represents the derivative of $\psi$ to give us the set of first order differential equations:

$$ \begin{align}
    \frac{d \psi}{d x} &= \phi(x) \\
    \frac{d \phi}{d x} &= 2(V(x)-E) \psi(x)
    \end{align} $$

Recall that we did the same trick in the Cyclotron Motion notebook to convert the 2nd order differential equations into a set of first order differential equations. This means that we can use the inbuilt functions such as 'odeint' to automatically solve the equations for us.

So first as usual we import the required libraries

```python
from modules.schrodinger import splitdf

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.integrate import odeint
import pandas as pd
```

Next we define the range over which we want to find the solutions. For the example below this is taken to be between $-20< x <20$ and we split it up into 2000 points.

```python
start_position = -20.0
end_position   =  20   # set this to be the correct end point.
space_steps    =  2000
step           =  (end_position-start_position)/(space_steps-1)
space          =  np.linspace(start_position, end_position, space_steps)
```

Next we define our potential function. This can be an arbitrary function but here I have taken it to be a square well with a negative depth and lying symmetrically about $x=0$.

```python
def potential(x, depth, width):  # square well potential
    pot = 0.0
    if abs(x) <= width/2:
        pot = depth
    else:
        pot = 0.0
    return pot
```

Lets plot the potential so that we know what it looks like.

```python
potfun = np.zeros(space_steps)
for ii in range(0, space_steps):
    potfun[ii] = potential(space[ii], -4, 30)

plt.figure()
plt.plot(space, potfun, linewidth=3, label='potential')
plt.legend()
plt.show()
```

Now we define the derivatives functions for the equations above. These are the equations that we want to solve.

```python
def shoot(yin, position, depth, width, energy):
    return np.array([yin[1], 2.0 * (potential(position, depth, width) - energy) * yin[0]])
```

Now we still don't know the initial values for $\psi(-20)$ and $\phi(-20)$ since we don't know the energy eigenvalue. However the Schrodinger equation is linear which means that we are free to set $\psi(-20)=1$ since if $\psi(x)$ is a solution so is $\alpha \psi(x)$ for any complex number $\alpha$. We also know that $\psi(x)$ decays exponentially if the energy is less than the potential and that the rate of decay is given by $V(x)-E$. Thus we can approximate the derivative of $psi(x)$ (i.e. $\phi(x)$) as $\phi(x) \approx (V(x)-E) \psi(x)$.

We then guess an energy level $E_g$ which means that the initially
$\phi(-20)=(V(-20)-E_g)$. Since for the square well potential $V(-20)=0$ this just becomes $\phi(-20)=-E_g$.

The Shooting method thus involves guessing an initial value of the energy and integrating the equations to see if it that value is correct. To see this in action let us use a potential well with a depth of -5 and a width of 30. This ensures that there are plenty of solutions to the Schrodinger equation. Our first guess will be that E=-4.9896.

```python
energy_guess = -3.999  # this is our initial guess.
start        = np.array([1.0, -np.sqrt(-energy_guess)])

xout     = odeint(shoot, start, space, args=(-4.0, 30, energy_guess))
xt       = np.transpose(xout)
psi      = xt[0]
psiprime = xt[1]

# Create the matplotlib figure and plot functions
plt.figure()
plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
plt.legend()
plt.show()
```

Notice that the solution is clearly wrong since it heads towards $\infty$ for large $x$. So
we can amend our guess of the energy for something larger and try again.

```python
energy_guess = -3.99 # this is our second guess.
start        = np.array([1.0,-np.sqrt(-energy_guess)])

xout     = odeint(shoot, start, space, args=(-4.0, 30, energy_guess))
xt       = np.transpose(xout)
psi      = xt[0]
psiprime = xt[1]

# Create the matplotlib figure and plot functions
plt.figure()
plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
plt.legend()
plt.show()
```

Now the solution is still wrong but it heads towards $-\infty$ for large $x$. We can thus say that the correct value for the energy must lie between our two solutions and all we have to do is find it. Trial and error leads to the solution below.

```python
energy_guess = -3.9947665559297138316  # this is ground state
start        =  np.array([1.0, np.sqrt(-energy_guess)])

xout     = odeint(shoot, start, space, args=(-4.0, 30, energy_guess))
xt       = np.transpose(xout)
psi      = xt[0]
psiprime = xt[1]

# Create the matplotlib figure and plot functions
plt.figure()
plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
plt.legend()
plt.show()
```

Note that as expected the ground state wavefunction has a single maximum and goes to zero as the distance from the well increases. It is clear however that the wavefunction is not normalised i.e. the integral of $|\psi(x)|^2 \neq 1$. However since this is just a number we can easily calculate the integral of the solution above and divide through by that to normalise everything.

```python
def normalise(y, step):  # a quick and dirty integrator that is good enough.
    out = 0.0
    for ii in range(0, len(y)):
        out = out + y[ii] ** 2
    return np.sqrt(out * step)
```

```python
result   = normalise(psi, step)

psi      = psi / result
psiprime = psiprime / result

# Create the matplotlib figure and plot functions
plt.figure()
plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
# Show the potential well as well
plt.plot(space, potfun/8+0.5, linewidth = 1, label = 'scaled potential')
plt.legend()
plt.show()
```

A bit more trial and error leads to the next guess for the energy of the first higher order state.

```python
energy_guess = -3.9790668560912517  # this is first higher order state
start        =  np.array([1.0, np.sqrt(-energy_guess)])

xout     = odeint(shoot, start, space, args=(-4.0, 30, energy_guess))
xt       = np.transpose(xout)
psi      = xt[0]
psiprime = xt[1]
result   = normalise(psi, step)

psi      = psi / result
psiprime = psiprime / result

# Create the matplotlib figure and plot functions
plt.figure(figsize = [5,5])
plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
# Show the potential well as well
plt.plot(space, potfun/8+0.5, linewidth = 1, label = 'scaled potential')
plt.legend()
plt.show()
```

It very quickly gets complicated to continually bisect the interval to find new
solutions. However it is fairly easy to write a program that does that for us. There are lots of different ways of doing so since it is the same as finding the zero of a function but this is perhaps the simplest. We start with two values $E_1$ and $E_2$ which have the property that
the solution to the equation goes to $\pm \infty$ for the two values and then we look at the
sign of the solution at the midpoint.

If the sign of the solution at the midpoint is the same value as $E_1$ we replace $E_1$ by
$(E_1+E_2)/2$ and start again. Otherwise we replace $E_2$ by the midpoint and start again.
Each time we do this the size of the interval decreases by a factor of 2 so we can continue doing it for as long as we like. In practice we stop when we run out of numerical precision.

```python
def find(start,stop):
    """Finds where our system of equations are satisfied by a wavefunction
        with an energy between start and stop through the bisection method.

    Make sure that the solution goes in different directions for the two values of start and stop.

    Parameters:
    -----------
        start : float64
            A lower bound for energy.
        stop  : float64
            An upper bound for energy.

    Prints:
    -----------
        output : pandas dataframe
            This stores the values of out1, out2 and out3 as the code iterates,
            The pandas dataframe allows for aesthetic printing of values.

    Returns:
    -----------
        guess : float64
            A guess as to what value the function shoot will return zero.
            This will be between the values of "start" and "stop".
    """
    # out1 is the final value of our trial solution for the guess Energy = start
    # the element [-1][0] picks the last element in the array.
    # out2 is the final value of our trial solution for the guess Energy = stop.
    # guess is the midpoint of the interval.
    # out3 is the final value of the trial solution for the guess Energy = guess.

    def step(start, stop, output, count):
        """Runs for each step of this method"""
        out1  = odeint(shoot, np.array([1.0, np.sqrt(-start)]), space, args = (-4.0, 30, start))[-1][0]
        out2  = odeint(shoot, np.array([1.0, np.sqrt(-stop)]),  space, args = (-4.0, 30, stop ))[-1][0]
        guess = (stop + start) / 2.0
        out3  = odeint(shoot, np.array([1.0, np.sqrt(-guess)]), space, args = (-4.0, 30, guess))[-1][0]

        output.iloc[count] = [out1, out2, out3, start, stop, guess]

        # Assigning the new start and stop values
        if np.sign(out1) == np.sign(out3):
            start = guess
        else:
            stop  = guess

        return start, stop, output

    output = pd.DataFrame(columns = ['out1','out2','out3', 'start', 'stop', 'guess'], index = range(50))

    for count in range(50):
        start, stop, output = step(start, stop, output, count)

    return output
```

Let us see it in action

```python
splitdf(find(-3.95, -3.979), n=2)
```

      out1
      out2
      out3
      start
      stop
      guess

      0
      -1.74497e+12
      1.05633e+11
      7.29901e+12
      -3.95
      -3.979
      -3.9645

      1
      -1.74497e+12
      7.29901e+12
      2.95391e+12
      -3.95
      -3.9645
      -3.95725

      2
      -1.74497e+12
      2.95391e+12
      4.71129e+11
      -3.95
      -3.95725
      -3.95363

      3
      -1.74497e+12
      4.71129e+11
      -6.86147e+11
      -3.95
      -3.95363
      -3.95181

      4
      -6.86147e+11
      4.71129e+11
      -1.18073e+11
      -3.95181
      -3.95363
      -3.95272

      5
      -1.18073e+11
      4.71129e+11
      1.74115e+11
      -3.95272
      -3.95363
      -3.95317

      6
      -1.18073e+11
      1.74115e+11
      2.73871e+10
      -3.95272
      -3.95317
      -3.95295

      7
      -1.18073e+11
      2.73871e+10
      -4.55053e+10
      -3.95272
      -3.95295
      -3.95283

      8
      -4.55053e+10
      2.73871e+10
      -9.0993e+09
      -3.95283
      -3.95295
      -3.95289

      9
      -9.0993e+09
      2.73871e+10
      9.13389e+09
      -3.95289
      -3.95295
      -3.95292

      10
      -9.0993e+09
      9.13389e+09
      1.47511e+07
      -3.95289
      -3.95292
      -3.9529

      11
      -9.0993e+09
      1.47511e+07
      -4.54302e+09
      -3.95289
      -3.9529
      -3.9529

      12
      -4.54302e+09
      1.47511e+07
      -2.26418e+09
      -3.9529
      -3.9529
      -3.9529

      13
      -2.26418e+09
      1.47511e+07
      -1.12479e+09
      -3.9529
      -3.9529
      -3.9529

      14
      -1.12479e+09
      1.47511e+07
      -5.54889e+08
      -3.9529
      -3.9529
      -3.9529

      15
      -5.54889e+08
      1.47511e+07
      -2.7001e+08
      -3.9529
      -3.9529
      -3.9529

      16
      -2.7001e+08
      1.47511e+07
      -1.27639e+08
      -3.9529
      -3.9529
      -3.9529

      17
      -1.27639e+08
      1.47511e+07
      -5.6375e+07
      -3.9529
      -3.9529
      -3.9529

      18
      -5.6375e+07
      1.47511e+07
      -2.08418e+07
      -3.9529
      -3.9529
      -3.9529

      19
      -2.08418e+07
      1.47511e+07
      -2.95322e+06
      -3.9529
      -3.9529
      -3.9529

      20
      -2.95322e+06
      1.47511e+07
      5.85131e+06
      -3.9529
      -3.9529
      -3.9529

      21
      -2.95322e+06
      5.85131e+06
      1.4098e+06
      -3.9529
      -3.9529
      -3.9529

      22
      -2.95322e+06
      1.4098e+06
      -829766
      -3.9529
      -3.9529
      -3.9529

      23
      -829766
      1.4098e+06
      291807
      -3.9529
      -3.9529
      -3.9529

      24
      -829766
      291807
      -242102
      -3.9529
      -3.9529
      -3.9529

      out1
      out2
      out3
      start
      stop
      guess

      25
      -242102
      291807
      -11372.5
      -3.9529
      -3.9529
      -3.9529

      26
      -11372.5
      291807
      114727
      -3.9529
      -3.9529
      -3.9529

      27
      -11372.5
      114727
      71947.8
      -3.9529
      -3.9529
      -3.9529

      28
      -11372.5
      71947.8
      -39667.4
      -3.9529
      -3.9529
      -3.9529

      29
      -39667.4
      71947.8
      108635
      -3.9529
      -3.9529
      -3.9529

      30
      -39667.4
      108635
      74172.2
      -3.9529
      -3.9529
      -3.9529

      31
      -39667.4
      74172.2
      60265.4
      -3.9529
      -3.9529
      -3.9529

      32
      -39667.4
      60265.4
      3203.8
      -3.9529
      -3.9529
      -3.9529

      33
      -39667.4
      3203.8
      53773.8
      -3.9529
      -3.9529
      -3.9529

      34
      -39667.4
      53773.8
      84760.7
      -3.9529
      -3.9529
      -3.9529

      35
      -39667.4
      84760.7
      41847.2
      -3.9529
      -3.9529
      -3.9529

      36
      -39667.4
      41847.2
      5387.08
      -3.9529
      -3.9529
      -3.9529

      37
      -39667.4
      5387.08
      82474.2
      -3.9529
      -3.9529
      -3.9529

      38
      -39667.4
      82474.2
      51537.8
      -3.9529
      -3.9529
      -3.9529

      39
      -39667.4
      51537.8
      41437.4
      -3.9529
      -3.9529
      -3.9529

      40
      -39667.4
      41437.4
      107459
      -3.9529
      -3.9529
      -3.9529

      41
      -39667.4
      107459
      137273
      -3.9529
      -3.9529
      -3.9529

      42
      -39667.4
      137273
      -7134.26
      -3.9529
      -3.9529
      -3.9529

      43
      -7134.26
      137273
      51572.7
      -3.9529
      -3.9529
      -3.9529

      44
      -7134.26
      51572.7
      87407.5
      -3.9529
      -3.9529
      -3.9529

      45
      -7134.26
      87407.5
      48176.3
      -3.9529
      -3.9529
      -3.9529

      46
      -7134.26
      48176.3
      -7134.26
      -3.9529
      -3.9529
      -3.9529

      47
      -7134.26
      48176.3
      -7134.26
      -3.9529
      -3.9529
      -3.9529

      48
      -7134.26
      48176.3
      -7134.26
      -3.9529
      -3.9529
      -3.9529

      49
      -7134.26
      48176.3
      -7134.26
      -3.9529
      -3.9529
      -3.9529

```python
def plot_wavefunctions(energy_guess):
    start = np.array([1.0, np.sqrt(-energy_guess)])

    xout     = odeint(shoot, start, space, args=(-4.0, 30, energy_guess))
    xt       = np.transpose(xout)
    psi      = xt[0]
    psiprime = xt[1]
    result   = normalise(psi, step)

    psi      = psi / result
    psiprime = psiprime / result

    # Plotting
    plt.figure()
    plt.plot(space, psi,      linewidth = 3, label = 'wavefunction')
    plt.plot(space, psiprime, linewidth = 3, label = 'derivative')
    plt.plot(space, potfun/10 + 0.5, linewidth = 1, label = 'scaled potential') # show the potential well as well
    plt.legend()
    plt.show()

energy_guess = -3.952902809182185  # this is 2nd higher order state
plot_wavefunctions(energy_guess)
```

```python
splitdf(find(-2.80, -2.90), n=2)
```

      out1
      out2
      out3
      start
      stop
      guess

      0
      -8.62729e+09
      2.74826e+10
      1.04639e+10
      -2.8
      -2.9
      -2.85

      1
      -8.62729e+09
      1.04639e+10
      -2.13682e+08
      -2.8
      -2.85
      -2.825

      2
      -2.13682e+08
      1.04639e+10
      4.98109e+09
      -2.825
      -2.85
      -2.8375

      3
      -2.13682e+08
      4.98109e+09
      2.32748e+09
      -2.825
      -2.8375
      -2.83125

      4
      -2.13682e+08
      2.32748e+09
      1.04055e+09
      -2.825
      -2.83125
      -2.82812

      5
      -2.13682e+08
      1.04055e+09
      4.09075e+08
      -2.825
      -2.82812
      -2.82656

      6
      -2.13682e+08
      4.09075e+08
      9.65752e+07
      -2.825
      -2.82656
      -2.82578

      7
      -2.13682e+08
      9.65752e+07
      -5.88381e+07
      -2.825
      -2.82578
      -2.82539

      8
      -5.88381e+07
      9.65752e+07
      1.87975e+07
      -2.82539
      -2.82578
      -2.82559

      9
      -5.88381e+07
      1.87975e+07
      -2.0039e+07
      -2.82539
      -2.82559
      -2.82549

      10
      -2.0039e+07
      1.87975e+07
      -624096
      -2.82549
      -2.82559
      -2.82554

      11
      -624096
      1.87975e+07
      9.08421e+06
      -2.82554
      -2.82559
      -2.82556

      12
      -624096
      9.08421e+06
      4.23032e+06
      -2.82554
      -2.82556
      -2.82555

      13
      -624096
      4.23032e+06
      1.80274e+06
      -2.82554
      -2.82555
      -2.82554

      14
      -624096
      1.80274e+06
      589213
      -2.82554
      -2.82554
      -2.82554

      15
      -624096
      589213
      -18639.8
      -2.82554
      -2.82554
      -2.82554

      16
      -18639.8
      589213
      285735
      -2.82554
      -2.82554
      -2.82554

      17
      -18639.8
      285735
      134035
      -2.82554
      -2.82554
      -2.82554

      18
      -18639.8
      134035
      57845.7
      -2.82554
      -2.82554
      -2.82554

      19
      -18639.8
      57845.7
      20734.2
      -2.82554
      -2.82554
      -2.82554

      20
      -18639.8
      20734.2
      188.17
      -2.82554
      -2.82554
      -2.82554

      21
      -18639.8
      188.17
      -8644.17
      -2.82554
      -2.82554
      -2.82554

      22
      -8644.17
      188.17
      -4285.05
      -2.82554
      -2.82554
      -2.82554

      23
      -4285.05
      188.17
      -969.274
      -2.82554
      -2.82554
      -2.82554

      24
      -969.274
      188.17
      847.168
      -2.82554
      -2.82554
      -2.82554

      out1
      out2
      out3
      start
      stop
      guess

      25
      -969.274
      847.168
      -1451.03
      -2.82554
      -2.82554
      -2.82554

      26
      -1451.03
      847.168
      990.846
      -2.82554
      -2.82554
      -2.82554

      27
      -1451.03
      990.846
      -463.534
      -2.82554
      -2.82554
      -2.82554

      28
      -463.534
      990.846
      -877.195
      -2.82554
      -2.82554
      -2.82554

      29
      -877.195
      990.846
      -862.498
      -2.82554
      -2.82554
      -2.82554

      30
      -862.498
      990.846
      -274.765
      -2.82554
      -2.82554
      -2.82554

      31
      -274.765
      990.846
      -166.624
      -2.82554
      -2.82554
      -2.82554

      32
      -166.624
      990.846
      -660.485
      -2.82554
      -2.82554
      -2.82554

      33
      -660.485
      990.846
      -116.454
      -2.82554
      -2.82554
      -2.82554

      34
      -116.454
      990.846
      -770.484
      -2.82554
      -2.82554
      -2.82554

      35
      -770.484
      990.846
      -723.286
      -2.82554
      -2.82554
      -2.82554

      36
      -723.286
      990.846
      -975.331
      -2.82554
      -2.82554
      -2.82554

      37
      -975.331
      990.846
      -327.776
      -2.82554
      -2.82554
      -2.82554

      38
      -327.776
      990.846
      -101.524
      -2.82554
      -2.82554
      -2.82554

      39
      -101.524
      990.846
      -825.575
      -2.82554
      -2.82554
      -2.82554

      40
      -825.575
      990.846
      -912.438
      -2.82554
      -2.82554
      -2.82554

      41
      -912.438
      990.846
      -1300.87
      -2.82554
      -2.82554
      -2.82554

      42
      -1300.87
      990.846
      556.133
      -2.82554
      -2.82554
      -2.82554

      43
      -1300.87
      556.133
      -630.457
      -2.82554
      -2.82554
      -2.82554

      44
      -630.457
      556.133
      -571.854
      -2.82554
      -2.82554
      -2.82554

      45
      -571.854
      556.133
      19.0422
      -2.82554
      -2.82554
      -2.82554

      46
      -571.854
      19.0422
      -74.267
      -2.82554
      -2.82554
      -2.82554

      47
      -74.267
      19.0422
      -74.267
      -2.82554
      -2.82554
      -2.82554

      48
      -74.267
      19.0422
      -74.267
      -2.82554
      -2.82554
      -2.82554

      49
      -74.267
      19.0422
      -74.267
      -2.82554
      -2.82554
      -2.82554

```python
energy_guess = -2.825538679175513  # which state is this?
plot_wavefunctions(energy_guess)
```

## Simple Harmonic Potential

In class we have also looked at a harmonic potential well defined by

$$ V(x) =\frac{x^2}{w^2} $$

where $w$ defines the steepness of the well. We can easily adapt our code to find the wavefunctions for this case as well.

```python
def potential2(x, width):  # harmonic potential well
    return x ** 2 / (width ** 2)

potfun2 = np.zeros(space_steps)

for ii in range(0, space_steps):
    potfun2[ii] = potential2(space[ii], 10)

def shoot2(yin, position, width, energy):
    return np.array([yin[1], 2.0 * (potential2(position, width) - energy) * yin[0]])
```

```python
def find_harmonic(start, stop, width):
    """Finds where the function shoot has the value 0 through the bisection method.

    Make sure that the solution goes in different directions for the two values of start and stop.

    Parameters:
    -----------
        start : float64
            A lower bound.
        stop  : float64
            An upper bound.

    Prints:
    -----------
        output : pandas dataframe
            This stores the values of out1, out2 and out3 as the code iterates,
            The pandas dataframe allows for aesthetic printing of values.

    Returns:
    -----------
        guess : float64
            A guess as to what value the function shoot will return zero.
            This will be between the values of "start" and "stop".
    """
    # out1 is the final value of our trial solution for the guess Energy = start
    # the element [-1][0] picks the last element in the array.
    # out2 is the final value of our trial solution for the guess Energy = stop.
    # guess is the midpoint of the interval.
    # out3 is the final value of the trial solution for the guess Energy=guess.

    def step(start, stop, width, output, count):
        out1  = odeint(shoot2, np.array([1.0, np.sqrt(potential2(-20, width) - start)]), space, args=(width, start))[-1][0]
        out2  = odeint(shoot2, np.array([1.0, np.sqrt(potential2(-20, width) - stop )]), space, args=(width, stop ))[-1][0]
        guess = (stop + start) / 2.0
        out3  = odeint(shoot2, np.array([1.0, np.sqrt(potential2(-20, width) - guess)]), space, args=(width, guess))[-1][0]

        output.iloc[count] = [out1, out2, out3, start, stop, guess]

        if np.sign(out1) == np.sign(out3):
            start = guess
        else:
            stop = guess
        return start, stop, width, output, count

    output = pd.DataFrame(columns = ['out1','out2','out3', 'start', 'stop', 'guess'], index = range(50))

    for count in range(50):
        start, stop, width, output, count = step(start, stop, width, output, count)

    return output
```

```python
splitdf(find_harmonic(0, 0.2, 10), n = 2)
```

      out1
      out2
      out3
      start
      stop
      guess

      0
      4.46476e+24
      -8.23049e+20
      -4.95874e+22
      0
      0.2
      0.1

      1
      4.46476e+24
      -4.95874e+22
      2.33367e+23
      0
      0.1
      0.05

      2
      2.33367e+23
      -4.95874e+22
      -1.92456e+22
      0.05
      0.1
      0.075

      3
      2.33367e+23
      -1.92456e+22
      5.87244e+22
      0.05
      0.075
      0.0625

      4
      5.87244e+22
      -1.92456e+22
      1.11243e+22
      0.0625
      0.075
      0.06875

      5
      1.11243e+22
      -1.92456e+22
      -5.87695e+21
      0.06875
      0.075
      0.071875

      6
      1.11243e+22
      -5.87695e+21
      2.13105e+21
      0.06875
      0.071875
      0.0703125

      7
      2.13105e+21
      -5.87695e+21
      -1.99109e+21
      0.0703125
      0.071875
      0.0710938

      8
      2.13105e+21
      -1.99109e+21
      3.98347e+19
      0.0703125
      0.0710938
      0.0707031

      9
      3.98347e+19
      -1.99109e+21
      -9.83084e+20
      0.0707031
      0.0710938
      0.0708984

      10
      3.98347e+19
      -9.83084e+20
      -4.73499e+20
      0.0707031
      0.0708984
      0.0708008

      11
      3.98347e+19
      -4.73499e+20
      -2.17302e+20
      0.0707031
      0.0708008
      0.070752

      12
      3.98347e+19
      -2.17302e+20
      -8.88513e+19
      0.0707031
      0.070752
      0.0707275

      13
      3.98347e+19
      -8.88513e+19
      -2.45378e+19
      0.0707031
      0.0707275
      0.0707153

      14
      3.98347e+19
      -2.45378e+19
      7.6411e+18
      0.0707031
      0.0707153
      0.0707092

      15
      7.6411e+18
      -2.45378e+19
      -8.45018e+18
      0.0707092
      0.0707153
      0.0707123

      16
      7.6411e+18
      -8.45018e+18
      -4.04997e+17
      0.0707092
      0.0707123
      0.0707108

      17
      7.6411e+18
      -4.04997e+17
      3.61794e+18
      0.0707092
      0.0707108
      0.07071

      18
      3.61794e+18
      -4.04997e+17
      1.60644e+18
      0.07071
      0.0707108
      0.0707104

      19
      1.60644e+18
      -4.04997e+17
      6.00715e+17
      0.0707104
      0.0707108
      0.0707106

      20
      6.00715e+17
      -4.04997e+17
      9.78573e+16
      0.0707106
      0.0707108
      0.0707107

      21
      9.78573e+16
      -4.04997e+17
      -1.53571e+17
      0.0707107
      0.0707108
      0.0707107

      22
      9.78573e+16
      -1.53571e+17
      -2.78567e+16
      0.0707107
      0.0707107
      0.0707107

      23
      9.78573e+16
      -2.78567e+16
      3.50003e+16
      0.0707107
      0.0707107
      0.0707107

      24
      3.50003e+16
      -2.78567e+16
      3.57176e+15
      0.0707107
      0.0707107
      0.0707107

      out1
      out2
      out3
      start
      stop
      guess

      25
      3.57176e+15
      -2.78567e+16
      -1.21425e+16
      0.0707107
      0.0707107
      0.0707107

      26
      3.57176e+15
      -1.21425e+16
      -4.28537e+15
      0.0707107
      0.0707107
      0.0707107

      27
      3.57176e+15
      -4.28537e+15
      -3.56803e+14
      0.0707107
      0.0707107
      0.0707107

      28
      3.57176e+15
      -3.56803e+14
      1.60748e+15
      0.0707107
      0.0707107
      0.0707107

      29
      1.60748e+15
      -3.56803e+14
      6.25338e+14
      0.0707107
      0.0707107
      0.0707107

      30
      6.25338e+14
      -3.56803e+14
      1.34268e+14
      0.0707107
      0.0707107
      0.0707107

      31
      1.34268e+14
      -3.56803e+14
      -1.11268e+14
      0.0707107
      0.0707107
      0.0707107

      32
      1.34268e+14
      -1.11268e+14
      1.15001e+13
      0.0707107
      0.0707107
      0.0707107

      33
      1.15001e+13
      -1.11268e+14
      -4.98838e+13
      0.0707107
      0.0707107
      0.0707107

      34
      1.15001e+13
      -4.98838e+13
      -1.91921e+13
      0.0707107
      0.0707107
      0.0707107

      35
      1.15001e+13
      -1.91921e+13
      -3.84581e+12
      0.0707107
      0.0707107
      0.0707107

      36
      1.15001e+13
      -3.84581e+12
      3.82744e+12
      0.0707107
      0.0707107
      0.0707107

      37
      3.82744e+12
      -3.84581e+12
      -9.11549e+09
      0.0707107
      0.0707107
      0.0707107

      38
      3.82744e+12
      -9.11549e+09
      1.90913e+12
      0.0707107
      0.0707107
      0.0707107

      39
      1.90913e+12
      -9.11549e+09
      9.49859e+11
      0.0707107
      0.0707107
      0.0707107

      40
      9.49859e+11
      -9.11549e+09
      4.70165e+11
      0.0707107
      0.0707107
      0.0707107

      41
      4.70165e+11
      -9.11549e+09
      2.30581e+11
      0.0707107
      0.0707107
      0.0707107

      42
      2.30581e+11
      -9.11549e+09
      1.10669e+11
      0.0707107
      0.0707107
      0.0707107

      43
      1.10669e+11
      -9.11549e+09
      5.07153e+10
      0.0707107
      0.0707107
      0.0707107

      44
      5.07153e+10
      -9.11549e+09
      2.08232e+10
      0.0707107
      0.0707107
      0.0707107

      45
      2.08232e+10
      -9.11549e+09
      5.77114e+09
      0.0707107
      0.0707107
      0.0707107

      46
      5.77114e+09
      -9.11549e+09
      -2.06435e+09
      0.0707107
      0.0707107
      0.0707107

      47
      5.77114e+09
      -2.06435e+09
      1.75553e+09
      0.0707107
      0.0707107
      0.0707107

      48
      1.75553e+09
      -2.06435e+09
      -7.30428e+07
      0.0707107
      0.0707107
      0.0707107

      49
      1.75553e+09
      -7.30428e+07
      8.57382e+08
      0.0707107
      0.0707107
      0.0707107

```python
def plot_wavefunctions2(energy_guess, scale = 10, ylims = None):
    start    = np.array([1.0, np.sqrt(potential2(-20, 10) - energy_guess)])
    xout     = odeint(shoot2, start, space, args = (10, energy_guess))
    xt       = np.transpose(xout)
    psi      = xt[0]
    psiprime = xt[1]
    result   = normalise(psi, step)
    psi      = psi / result       # again we normalise the solution.
    psiprime = psiprime / result

    plt.figure()
    plt.plot(space, psi,       linewidth=3, label = 'wavefunction')
    plt.plot(space, psiprime,  linewidth=3, label = 'derivative')
    plt.plot(space, potfun2/scale, linewidth=1, label = 'scaled potential') #show the potential well as well
    if ylims != None:
        plt.ylim(*ylims)
    plt.legend()
    plt.show()

energy_guess = 0.07071067758588272  #this is a higher order state
plot_wavefunctions2(energy_guess)
```

It can be seen that this looks like the ground state solution to Schrodinger's equation. It is symmetric about $x=0$ and has a single maximum. Note that as before we normalise the solution.

If our method works then we should be able to find the next solution which will be anti-symmetric with $\psi(0)=0$ and have two extrema. So lets try a new guess.

```python
splitdf(find_harmonic(0.1, 0.5, 10), n = 2)
```

      out1
      out2
      out3
      start
      stop
      guess

      0
      -4.95874e+22
      1.08366e+17
      1.55763e+20
      0.1
      0.5
      0.3

      1
      -4.95874e+22
      1.55763e+20
      -8.23049e+20
      0.1
      0.3
      0.2

      2
      -8.23049e+20
      1.55763e+20
      5.17976e+20
      0.2
      0.3
      0.25

      3
      -8.23049e+20
      5.17976e+20
      4.09213e+20
      0.2
      0.25
      0.225

      4
      -8.23049e+20
      4.09213e+20
      1.72681e+19
      0.2
      0.225
      0.2125

      5
      -8.23049e+20
      1.72681e+19
      -3.3274e+20
      0.2
      0.2125
      0.20625

      6
      -3.3274e+20
      1.72681e+19
      -1.42144e+20
      0.20625
      0.2125
      0.209375

      7
      -1.42144e+20
      1.72681e+19
      -5.87665e+19
      0.209375
      0.2125
      0.210938

      8
      -5.87665e+19
      1.72681e+19
      -1.98587e+19
      0.210938
      0.2125
      0.211719

      9
      -1.98587e+19
      1.72681e+19
      -1.07606e+18
      0.211719
      0.2125
      0.212109

      10
      -1.07606e+18
      1.72681e+19
      8.15041e+18
      0.212109
      0.2125
      0.212305

      11
      -1.07606e+18
      8.15041e+18
      3.55084e+18
      0.212109
      0.212305
      0.212207

      12
      -1.07606e+18
      3.55084e+18
      1.24078e+18
      0.212109
      0.212207
      0.212158

      13
      -1.07606e+18
      1.24078e+18
      8.32673e+16
      0.212109
      0.212158
      0.212134

      14
      -1.07606e+18
      8.32673e+16
      -4.96224e+17
      0.212109
      0.212134
      0.212122

      15
      -4.96224e+17
      8.32673e+16
      -2.06402e+17
      0.212122
      0.212134
      0.212128

      16
      -2.06402e+17
      8.32673e+16
      -6.15561e+16
      0.212128
      0.212134
      0.212131

      17
      -6.15561e+16
      8.32673e+16
      1.08616e+16
      0.212131
      0.212134
      0.212132

      18
      -6.15561e+16
      1.08616e+16
      -2.53437e+16
      0.212131
      0.212132
      0.212132

      19
      -2.53437e+16
      1.08616e+16
      -7.24087e+15
      0.212132
      0.212132
      0.212132

      20
      -7.24087e+15
      1.08616e+16
      1.81042e+15
      0.212132
      0.212132
      0.212132

      21
      -7.24087e+15
      1.81042e+15
      -2.71521e+15
      0.212132
      0.212132
      0.212132

      22
      -2.71521e+15
      1.81042e+15
      -4.52397e+14
      0.212132
      0.212132
      0.212132

      23
      -4.52397e+14
      1.81042e+15
      6.79011e+14
      0.212132
      0.212132
      0.212132

      24
      -4.52397e+14
      6.79011e+14
      1.13308e+14
      0.212132
      0.212132
      0.212132

      out1
      out2
      out3
      start
      stop
      guess

      25
      -4.52397e+14
      1.13308e+14
      -1.69544e+14
      0.212132
      0.212132
      0.212132

      26
      -1.69544e+14
      1.13308e+14
      -2.81168e+13
      0.212132
      0.212132
      0.212132

      27
      -2.81168e+13
      1.13308e+14
      4.25956e+13
      0.212132
      0.212132
      0.212132

      28
      -2.81168e+13
      4.25956e+13
      7.23914e+12
      0.212132
      0.212132
      0.212132

      29
      -2.81168e+13
      7.23914e+12
      -1.04401e+13
      0.212132
      0.212132
      0.212132

      30
      -1.04401e+13
      7.23914e+12
      -1.60083e+12
      0.212132
      0.212132
      0.212132

      31
      -1.60083e+12
      7.23914e+12
      2.81914e+12
      0.212132
      0.212132
      0.212132

      32
      -1.60083e+12
      2.81914e+12
      6.08192e+11
      0.212132
      0.212132
      0.212132

      33
      -1.60083e+12
      6.08192e+11
      -4.95989e+11
      0.212132
      0.212132
      0.212132

      34
      -4.95989e+11
      6.08192e+11
      5.53889e+10
      0.212132
      0.212132
      0.212132

      35
      -4.95989e+11
      5.53889e+10
      -2.19661e+11
      0.212132
      0.212132
      0.212132

      36
      -2.19661e+11
      5.53889e+10
      -8.08824e+10
      0.212132
      0.212132
      0.212132

      37
      -8.08824e+10
      5.53889e+10
      -1.18613e+10
      0.212132
      0.212132
      0.212132

      38
      -1.18613e+10
      5.53889e+10
      2.27059e+10
      0.212132
      0.212132
      0.212132

      39
      -1.18613e+10
      2.27059e+10
      4.90908e+09
      0.212132
      0.212132
      0.212132

      40
      -1.18613e+10
      4.90908e+09
      -3.17055e+09
      0.212132
      0.212132
      0.212132

      41
      -3.17055e+09
      4.90908e+09
      8.30787e+08
      0.212132
      0.212132
      0.212132

      42
      -3.17055e+09
      8.30787e+08
      -2.54283e+09
      0.212132
      0.212132
      0.212132

      43
      -2.54283e+09
      8.30787e+08
      -3.64425e+08
      0.212132
      0.212132
      0.212132

      44
      -3.64425e+08
      8.30787e+08
      -5.25134e+08
      0.212132
      0.212132
      0.212132

      45
      -5.25134e+08
      8.30787e+08
      1.13961e+09
      0.212132
      0.212132
      0.212132

      46
      -5.25134e+08
      1.13961e+09
      7.3855e+08
      0.212132
      0.212132
      0.212132

      47
      -5.25134e+08
      7.3855e+08
      1.15497e+09
      0.212132
      0.212132
      0.212132

      48
      -5.25134e+08
      1.15497e+09
      -9.77219e+08
      0.212132
      0.212132
      0.212132

      49
      -9.77219e+08
      1.15497e+09
      6.49231e+08
      0.212132
      0.212132
      0.212132

```python
energy_guess = 0.21213203429823696  # this is the second eigenvalue
plot_wavefunctions2(energy_guess)
```

Again our code has found the solution that we expect. We can see that it is quite a bit wider than the fundamental solution. We have scaled the potential by a factor of 4 so that it fits nicely on the graph. We can carry on and find the next solution.

```python
splitdf(find_harmonic(0.25, 0.6, 10), n = 2)
```

      out1
      out2
      out3
      start
      stop
      guess

      0
      5.17976e+20
      6.83957e+16
      -6.15917e+18
      0.25
      0.6
      0.425

      1
      5.17976e+20
      -6.15917e+18
      2.10088e+19
      0.25
      0.425
      0.3375

      2
      2.10088e+19
      -6.15917e+18
      -1.09089e+19
      0.3375
      0.425
      0.38125

      3
      2.10088e+19
      -1.09089e+19
      -4.3282e+18
      0.3375
      0.38125
      0.359375

      4
      2.10088e+19
      -4.3282e+18
      5.09e+18
      0.3375
      0.359375
      0.348438

      5
      5.09e+18
      -4.3282e+18
      -3.04135e+17
      0.348438
      0.359375
      0.353906

      6
      5.09e+18
      -3.04135e+17
      2.20657e+18
      0.348438
      0.353906
      0.351172

      7
      2.20657e+18
      -3.04135e+17
      9.06571e+17
      0.351172
      0.353906
      0.352539

      8
      9.06571e+17
      -3.04135e+17
      2.90294e+17
      0.352539
      0.353906
      0.353223

      9
      2.90294e+17
      -3.04135e+17
      -9.62245e+15
      0.353223
      0.353906
      0.353564

      10
      2.90294e+17
      -9.62245e+15
      1.39656e+17
      0.353223
      0.353564
      0.353394

      11
      1.39656e+17
      -9.62245e+15
      6.48484e+16
      0.353394
      0.353564
      0.353479

      12
      6.48484e+16
      -9.62245e+15
      2.75704e+16
      0.353479
      0.353564
      0.353522

      13
      2.75704e+16
      -9.62245e+15
      8.96379e+15
      0.353522
      0.353564
      0.353543

      14
      8.96379e+15
      -9.62245e+15
      -3.32243e+14
      0.353543
      0.353564
      0.353554

      15
      8.96379e+15
      -3.32243e+14
      4.31532e+15
      0.353543
      0.353554
      0.353548

      16
      4.31532e+15
      -3.32243e+14
      1.99161e+15
      0.353548
      0.353554
      0.353551

      17
      1.99161e+15
      -3.32243e+14
      8.2944e+14
      0.353551
      0.353554
      0.353552

      18
      8.2944e+14
      -3.32243e+14
      2.48581e+14
      0.353552
      0.353554
      0.353553

      19
      2.48581e+14
      -3.32243e+14
      -4.18333e+13
      0.353553
      0.353554
      0.353553

      20
      2.48581e+14
      -4.18333e+13
      1.03413e+14
      0.353553
      0.353553
      0.353553

      21
      1.03413e+14
      -4.18333e+13
      3.07758e+13
      0.353553
      0.353553
      0.353553

      22
      3.07758e+13
      -4.18333e+13
      -5.54686e+12
      0.353553
      0.353553
      0.353553

      23
      3.07758e+13
      -5.54686e+12
      1.30673e+13
      0.353553
      0.353553
      0.353553

      24
      1.30673e+13
      -5.54686e+12
      3.99446e+12
      0.353553
      0.353553
      0.353553

      out1
      out2
      out3
      start
      stop
      guess

      25
      3.99446e+12
      -5.54686e+12
      -9.97978e+11
      0.353553
      0.353553
      0.353553

      26
      3.99446e+12
      -9.97978e+11
      1.26339e+12
      0.353553
      0.353553
      0.353553

      27
      1.26339e+12
      -9.97978e+11
      1.62552e+11
      0.353553
      0.353553
      0.353553

      28
      1.62552e+11
      -9.97978e+11
      -4.38427e+11
      0.353553
      0.353553
      0.353553

      29
      1.62552e+11
      -4.38427e+11
      -1.42616e+11
      0.353553
      0.353553
      0.353553

      30
      1.62552e+11
      -1.42616e+11
      3.40973e+09
      0.353553
      0.353553
      0.353553

      31
      3.40973e+09
      -1.42616e+11
      -6.50238e+10
      0.353553
      0.353553
      0.353553

      32
      3.40973e+09
      -6.50238e+10
      2.84252e+11
      0.353553
      0.353553
      0.353553

      33
      2.84252e+11
      -6.50238e+10
      -4.70616e+10
      0.353553
      0.353553
      0.353553

      34
      2.84252e+11
      -4.70616e+10
      -2.40256e+10
      0.353553
      0.353553
      0.353553

      35
      2.84252e+11
      -2.40256e+10
      -3.80363e+10
      0.353553
      0.353553
      0.353553

      36
      2.84252e+11
      -3.80363e+10
      4.11181e+11
      0.353553
      0.353553
      0.353553

      37
      4.11181e+11
      -3.80363e+10
      4.11557e+11
      0.353553
      0.353553
      0.353553

      38
      4.11557e+11
      -3.80363e+10
      -1.90051e+10
      0.353553
      0.353553
      0.353553

      39
      4.11557e+11
      -1.90051e+10
      4.07655e+11
      0.353553
      0.353553
      0.353553

      40
      4.07655e+11
      -1.90051e+10
      -5.00739e+10
      0.353553
      0.353553
      0.353553

      41
      4.07655e+11
      -5.00739e+10
      -4.13917e+10
      0.353553
      0.353553
      0.353553

      42
      4.07655e+11
      -4.13917e+10
      4.0781e+11
      0.353553
      0.353553
      0.353553

      43
      4.0781e+11
      -4.13917e+10
      -3.89362e+10
      0.353553
      0.353553
      0.353553

      44
      4.0781e+11
      -3.89362e+10
      -4.40134e+10
      0.353553
      0.353553
      0.353553

      45
      4.0781e+11
      -4.40134e+10
      -1.90452e+10
      0.353553
      0.353553
      0.353553

      46
      4.0781e+11
      -1.90452e+10
      2.95727e+09
      0.353553
      0.353553
      0.353553

      47
      2.95727e+09
      -1.90452e+10
      4.07951e+11
      0.353553
      0.353553
      0.353553

      48
      4.07951e+11
      -1.90452e+10
      2.76342e+11
      0.353553
      0.353553
      0.353553

      49
      2.76342e+11
      -1.90452e+10
      2.62854e+11
      0.353553
      0.353553
      0.353553

```python
energy_guess = 0.35355339015204545  # this is a higher order state
plot_wavefunctions2(energy_guess, ylims = [-0.4,0.4])
```

We know for the harmonic potential the eigenvalues are evenly spaced. In our case the spacing is about $\sqrt{2}/10$ so we expect the next solution to be at 0.49497474683058326708.

```python
energy_guess = 0.49497474683058326708  #no need to use find now since we know the energy spacing.
plot_wavefunctions2(energy_guess, ylims = [-0.3,0.3])
```

```python

```


# Gaussian Quadrature

## The idea behind Gaussian Quadrature
So far we focused on uniformly spaced grid points (nodes). Gaussian Quadrature is an algorithm
that chooses the nodes such that in the approximation
$$\int_a^b f(x)dx =\sum_{i=1}^{n}c_i f(x_i) \label{GQ}\tag{1}$$
the error is minimized in the sense that the result is _exact_ for the class of polynomials of degree $2n-1$.
This is because Gaussian Quadrature treats both $c_i$ and $x_i$ as parameters, i.e., there are $2n$ parameters which
is the number of parameters in a polynomial of degree $2n-1$.

### Example: n=2 Gaussian Quadrature
Let's consider the case $n=2$ for $x\in [-1,1]$. We wish to determine $c_1, c_2, x_1, x_2$ such that
\begin{align}
\int_{-1}^{1} f(x)dx = &\sum_{i=1}^{n}c_i f(x_i) = c_1 f(x_1) + c_2 f(x_2) \label{condition}\tag{2}
\end{align}
is exact  whenever $f(x)$ is a polynomial of degree $2n-1=2\cdot 2-1=3$ or less, i.e.,
$$
f(x) = a_0 + a_1 x+a_2 x^2 + a_3 x^3.
$$

To achieve this we will first integrate this polynomial

$$
\int_{-1}^{1}f(x)dx = a_0 \int_{-1}^{1}dx + a_1 \int_{-1}^{1}x dx+ a_2 \int_{-1}^{1}x^2dx + a_3 \int_{-1}^{1}x^3dx.$$

We can now consider the function $f(x)$ as a linear combination of the
4 functions $f_0(x)=1$, $f_1(x)=x$, $f_2(x)=x^2$, $f_3(x)=x^3$, i.e.,
$f(x)=a_0 f_0(x)+a_1 f_1(x)+a_2 f_2(x)+a_3 f_3(x)=\sum_{i=0}^3
a_if(x_i)$.  Thus, if for each $f_i(x),\ i=0,1,2,3$ we satisfy (\ref{condition}) because
\begin{split}
  \int_{-1}^{1}f(x)dx = & \int_{-1}^{1}\sum_i a_i f_i(x)dx =
  \sum_i a_i \int_{-1}^{1} f_i(x)dx = \sum_i a_i (c_1 f_i(x_1) + c_2
  f_i(x_2)) = \sum_i a_i c_1 f_i(x_1) + \sum_i a_i c_2 f_i(x_2)\\
   = &\  c_1 \sum_i a_i f_i(x_1) + c_2\sum_i a_i f_i(x_2)
  = c_1 f(x_1) + c_2
  f(x_2).
\end{split}

Thus, for the 4 $f_i(x)$ polynomials we need to satisfy $\int_{-1}^1
f_i(x) dx = c_1 f_i(x_1) + c_2 f_i(x_2)$, which will provide 4
equations for the 4 parameters $c_1,c_2$, $x_1,x_2$:
  \begin{split}
    c_1 f_0(x_1) + c_2 f_0(x_2) = & c_1 \cdot 1 + c_2 \cdot 1 = \int_{-1}^1 1 dx = 2\\
    c_1 f_1(x_1) + c_2 f_1(x_2) = & c_1 \cdot x_1 + c_2 \cdot x_2 = \int_{-1}^1 x dx = 0\\
    c_1 f_2(x_1) + c_2 f_2(x_2) = & c_1 \cdot x_1^2 + c_2 \cdot x_2^2 = \int_{-1}^1 x^2 dx = \frac{2}{3}\\
    c_1 f_3(x_1) + c_2 f_3(x_2) = & c_1 \cdot x_1^3 + c_2 \cdot x_2^3 = \int_{-1}^1 x^3 dx = 0.
  \end{split}

This system can be solved easily by eliminating $c_1$ and $c_2$ from the first two equations, e.g.,
solve the first w.r.t $c_1$ and then replace the result in the second equation to express $c_2$ in
terms of $x_1$ and $x_2$. Then substitute $c_1$ and $c_2$ in terms of $x_1$ and $x_2$ in the last
two equations that will provide a system of 2 equations for the 2 unknowns $x_1$ and $x_2$.

The solution
of the system is $c_1=1$, $c_2=1$, $x_1=-\sqrt{3}/3=-x_2$.

**Therefore, the $n=2$ Gaussian quadrature
is
$$
\int_{-1}^{1} f(x)dx =  f(-\frac{\sqrt{3}}{3}) + f(\frac{\sqrt{3}}{3}),
$$
which yields an _exact_ result for any $f(x)$ that is a polynomial up to degree $n=3$!**

###  Hands-on example: n=2 Gaussian quadrature on the [-1,1] interval

```python
import numpy as np
#simple version of Gaussian Quadrature with n = 2
def GaussianQ_n2_unit_interval(f,*P):
    n = 2
    #define nodes (derived above)
    x = np.array([-np.sqrt(3)/3.,np.sqrt(3)/3.])
    #define coefficients (derived above)
    c = np.array([1.,1.])
    I = 0
    #now sum up the integral based on the n=2 Gaussian Quadrature formula
    for j in range(n):
        I += c[j]*f(x[j],*P)
    return I
```

```python
#define a fifth-order polynomial
def p_n5(x,a0,a1,a2,a3,a4,a5):
    return a0+a1*x+a2*x**2+a3*x**3+a4*x**4+a5*x**5

#integral of fifth-order polynomial over arbitrary integral
def int_p_n5(xmin,xmax,a0,a1,a2,a3,a4,a5):
    I = a0*(xmax-xmin) + a1*(xmax**2-xmin**2)/2. + a2*(xmax**3-xmin**3)/3. + a3*(xmax**4-xmin**4)/4. + a4*(xmax**5-xmin**5)/5. +a5*(xmax**6-xmin**6)/6.
    return I
#wrapper function for calling integral over [-1,1] interval
def int_p_n5_unit_interval(*P):
    return int_p_n5(-1,1,*P)
```

```python
#now test this by defining the coefficients of p_n5, our integrand
P = (1,1,1,0,0,0)
GQ_n2 = GaussianQ_n2_unit_interval(p_n5,*P)
exact = int_p_n5_unit_interval(*P)

print("exact integral: %e"%(exact))
print("Gaussian Quadrature: %e"%(GQ_n2))
```

    exact integral: 2.666667e+00
    Gaussian Quadrature: 2.666667e+00

 Convince yourself that this is exact for `p_n5` with degree n<=3

### Arbitrary intervals
The general procedure for Gaussian quadrature outlined earlier applies
to the interval $[-1,1]$. However, we will often have general
integration domains $[a,b]$. We can still apply $\int_a^b f(x)dx =\sum_{i=1}^{n}c_i f(x_i)$ by a simple
transformation of the integration variable. An integral of the form
$$
\int_a^b f(x) dx
$$
can easily be transformed to the form
$$\int_{-1}^1 f(t) dt$$

if we let

$$
t=\frac{2 x -a -b}{b-a},
$$

which implies
$$x=\frac{1}{2}[(b-a)t+a+b].
$$
Thus, when $x=a$, $t=-1$, and when $x=b$, $t=1$. We then have $dx=\frac{b-a}{2}dt$, and hence
$$
\int_a^b f(x) dx = \frac{b-a}{2}\int_{-1}^1 f\left(\frac{(b-a)t+(b+a)}{2} \right)dt.
$$
The last expression allows the application of our Gaussian Quadrature expression to
any function in any integration interval.

 Now rewrite `GaussianQ_n2_unit_interval` to perform Gaussian Quadrature over any integration interval:
- write a function `tranform_variable(t, a,b)`
- use `tranform_variable(t, a,b)` within the n = 2 Gaussian Quadrature formula
- remember to apply the rescaling (b-a)/2 to the result obtained on the interval [-1,1]

```python
def transform_variable(t, a,b):
    return ((b-a)*t+a+b)/2
```

```python
def GaussianQ_n2_any_interval(f,a,b,*P):
    n = 2
    #define nodes on [-1,1] interval(derived above)
    t = np.array([-np.sqrt(3)/3.,np.sqrt(3)/3.])
    #define coefficients (derived above)
    c = np.array([1.,1.])
    I = 0
    #now sum up the integral based on the n=2 Gaussian Quadrature formula
    #evaluate f(x(t[j])) with x on the interval [a,b]
    for j in range(n):
        I += (b-a)/2.*c[j]*f(transform_variable(t[j],a,b),*P)
    return I
```

```python
P = (1,0,1,1,0,0)
a = 0
b = 2
GQ_n2 = GaussianQ_n2_any_interval(p_n5,a,b,*P)
exact = int_p_n5(a,b,*P)

print("exact integral: %e"%(exact))
print("Gaussian Quadrature: %e"%(GQ_n2))
```

    exact integral: 8.666667e+00
    Gaussian Quadrature: 8.666667e+00

### Gaussian Quadrature with n>2
The same approach can yield results for degree $2n-1$ polynomials, but
this turns out to be more involved.  By using Legendre polynomials,
the process can be expedited. So, let's introduce the Legendre polynomials first.

**Legendre polynomials** ($P_n(x)$) are orthogonal ($\int_{-1}^1 P_n(x) P_m(x) = \delta_{nm}$) and form a complete basis.

Some of theirproperties are:
-  For each $n$, $P_n(x)$ is a monic polynomial of degree $n$, i.e., the coeficient of the smallest power of $x$
  that appears in the polynomial is $1$.

- $\int_{-1}^1 p(x) P_n(x) = 0$, when $p(x)$ is a polynomial of degree $<n$.

- The first 5 Legendre polynomials are: $P_0(x)$=1, $P_1(x)$=x, $P_2(x)=x^2-1/3$, $P_3(x)=x^3-3x/5$, \\ $P_4(x)=x^4-6x^2/7 +3/35$.

- **Theorem**
  Assume $x_1, x_2, \ldots x_n$ are the roots of the $n$-th Legendre
  polynomial $P_n(x)$ and for each $i=1,2,\ldots n$, consider the
  numbers
  $$
    c_i = \int_{-1}^{1} \prod_{j=1,j\neq i}^n \frac{x-x_j}{x_i-x_j} dx \label{eqci}\tag{3}
  $$
  If $p(x)$ is _any_ polynomial of degree less than $2n-1$, then
  $$
    \int_{-1}^1 p(x) dx = \sum_{i=1}^n c_i p(x_i).
  $$

 ### Example: n=3 Gaussian Quadrature
Let's consider the integral
$$
  I=\int_{-1}^1 e^x cos(x) dx = 1.933421496200713,
$$
and lets approximate it using Gaussian quadrature
with $n=3$, i.e.,

$$\tilde I = c_1 f(x_1) + c_2 f(x_2) + c_3 f(x_3).$$

where $x_1, x_2, x_3$ are the roots of the Legendre polynomial
$P_3(x)$, i.e., $x^3 - 3x/5 =0 \Rightarrow$ $x=0$ or $x^2-3/5=0
\Rightarrow$ $x=\pm \sqrt{3/5}$. Thus, $x_1=-\sqrt{3/5}$, $x_2=0$, and
$x_3=\sqrt{3/5}$. Next, we calculate the coefficients $c_i$ using Eq.(\ref{eqci}):
  \begin{split}
    c_1 = & \int_{-1}^{1} \prod_{j=1,j\neq 1}^n \frac{x-x_j}{x_1-x_j} dx = \int_{-1}^{1} \frac{x-x_2}{x_1-x_2}\frac{x-x_3}{x_1-x_3} dx = \frac{5}{9} \\
    c_2 = & \int_{-1}^{1} \prod_{j=1,j\neq 2}^n \frac{x-x_j}{x_2-x_j} dx = \int_{-1}^{1} \frac{x-x_1}{x_2-x_1}\frac{x-x_3}{x_2-x_3} dx = \frac{8}{9} \\
    c_3 = & \int_{-1}^{1} \prod_{j=1,j\neq 3}^n \frac{x-x_j}{x_3-x_j} dx = \int_{-1}^{1} \frac{x-x_1}{x_3-x_1}\frac{x-x_2}{x_3-x_2} dx = \frac{5}{9} \\
  \end{split}

Thus,
$$
\tilde I = \frac{5}{9} f\left(-\sqrt{\frac{3}{5}}\right) + \frac{8}{9} f(0) + \frac{5}{9} f\left(\sqrt{\frac{3}{5}}\right) = 1.933390469264298. \label{Gaussn4}\tag{4}
$$

Thus, the relative error is $|(\tilde I-I)/I|=1.6\times 10^{-5}$!!!
**Notice that Eq.(\ref{Gaussn4}) applies to any function, and is _exact_ for any polynomial of degree 5 or less.**

###  Hands-on example: n=3 Gaussian quadrature of $
I = \int_0^\pi x\cos(x) d x$
- following the n = 2 example above, implement the n = 3 Gaussian quadrature expression on an arbitraty interval
- calculate the relative error by comparing to the exact value $I = 2$.

```python
def GaussianQ_n3_any_interval(f,a,b,*P):
    n = 3
    #define nodes on [-1,1] interval(derived above)
    t = np.array([-np.sqrt(3/5.),0,np.sqrt(3/5.)])
    #define coefficients (derived above)
    c = np.array([5./9,8./9,5./9])
    I = 0
    #now sum up the integral based on the n=3 Gaussian Quadrature formula
    for j in range(n):
        I += (b-a)/2.*c[j]*f(transform_variable(t[j],a,b),*P)
    return I
```

```python
def func(x,*P):
    return x*np.cos(x)
```

```python
exact_value = -2.
GQ_n3_approximation = GaussianQ_n3_any_interval(func,0,np.pi,*P)
fractional_error = np.abs((GQ_n3_approximation-exact_value)/exact_value)
print(fractional_error )
```

    0.0040611225242603854

```python

```


#
Límites

```python
from sympy import *
init_printing()
x, y, z, h = symbols('x, y, z, h')
```

Para calcular límites se utiliza la función **limit**. Esta función puede admitir 4 argumentos, siendo el último opcional. El primer argumento debe ser la función a que le queremos calcular el límite. El segundo es la variable que haremos variar en el límite. El tercero es el punto hacia el que tiende la variable (para escribir infinito podemos escribir **oo**). El cuarto sirve para los límites laterales: si escribimos **dir = '+'** calculamos el límite por la derecha, y escribiendo **dir 0 '-'** calculamos el límite por la izquierda. Por defecto Sympy calcula los límites por la derecha.

###
Calcula la expresión $\displaystyle\frac{\sin(x)}{x}$ para números cercanos a cero, dibuja la función y calcula:

 $$\lim_{x\rightarrow 0}\frac{\sin(x)}{x}$$

```python
plot(sin(x)/x)
```

###
Calcula los límites laterales en $x=0$ de la función:

$$\displaystyle f(x)=\frac{1}{x}$$

```python
pl
```

Si escribimos **Limit** con mayúscula, Sympy no calcula el límite. Simplemente lo escribe. Si queremos que verdaderamente calcule el límite debemos utilizar el método **doit**.

###
Calcula el límite en el infinito:

*  $\displaystyle \lim_{x\rightarrow \infty} \left(1+\frac{1}{x}\right)^x$

```python
limit((1+1/x)**x,x,oo)
```

Para calcular la derivada de una función $f(x)$ en un punto $x$ debemos realizar el siguiente límite:

$$\lim_{h\rightarrow 0} \frac{f(x+h)-f(x)}{h}$$

###
Calcular la derivada de la función $f(x)=\cos(x)$ utilizando la definición con límites.

```python
limit((2+x)**(1/x),x,0, dir='-')
```

###
Comprueba la continuidad de la función en  $x=0$.

$$f(x)=(2+x)^\frac{1}{x}$$

```python

```


```
%pylab inline
import numpy as np
import scipy.optimize
import scipy.sparse
import matplotlib.pyplot as plt
from IPython.display import clear_output
import time
```

    Populating the interactive namespace from numpy and matplotlib

# The model

In this notebook we'll solve a system of reaction-diffusion PDEs in two dimensions:

\begin{align}
u_t & = \delta D_1 \nabla^2 u + f(u,v) \\\\
v_t & = \delta D_2 \nabla^2 v + g(u,v)
\end{align}
where $\nabla^2 u = u_{xx} + u_{yy}$ denotes the Laplacian and $f,g$ represent reaction terms.

For simplicity, we'll consider the square domain $[-1,1]\times[-1,1]$ with periodic boundary conditions; i.e., the conditions on $u(x,y,t)$ are
\begin{align}
u(-1,y,t) & = u(1,y,t) \\\\
u(x,-1,t) & = u(x,1,t)
\end{align}
with corresponding conditions on $v$.

# Reaction equations
The reaction terms we will use are
\begin{align}
f(u,v) & = \alpha u (1-\tau_1 v^2) + v(1-\tau_2 u) \\\\
g(u,v) & = \beta v + \alpha \tau_1 u v^2 + u (\gamma + \tau_2 v).
\end{align}
Once we have things working, we'll investigate how the constants $D_1, D_2, \alpha, \beta, \tau_1, \tau_2, \gamma$ change the kinds of solutions obtained.

```
delta=0.0045; tau1=0.02; tau2=0.2; alpha=0.899; beta=-0.91; gamma=-alpha;

def f(u,v):
    return alpha*u*(1-tau1*v**2) + v*(1-tau2*u);

def g(u,v):
    return beta*v*(1+alpha*tau1/beta*u*v) + u*(gamma+tau2*v);
```

# Diffusion terms
Next let's write a discretization of the diffusion terms.  We'll use the 5-point Laplacian approximation:

\begin{align}
\nabla^2 u(x_i,y_j) & \approx \frac{U_{i-1,j} - 2U_{ij} + U_{i+1,j}}{\Delta x}
                           +  \frac{U_{i,j-1} - 2U_{ij} + U_{i,j+1}}{\Delta y}
\end{align}
For simplicity we'll assume $\Delta x = \Delta y = h$.  Examine the code below carefully, noting how the periodic boundary conditions are imposed.

```
def five_pt_laplacian_sparse_periodic(m,a,b):
    """Construct a sparse matrix that applies the 5-point laplacian discretization
       with periodic BCs on all sides."""
    e=np.ones(m**2)
    e2=([1]*(m-1)+[0])*m
    e3=([0]+[1]*(m-1))*m
    h=(b-a)/(m+1)
    A=scipy.sparse.spdiags([-4*e,e2,e3,e,e],[0,-1,1,-m,m],m**2,m**2)
    # Top & bottom BCs:
    A_periodic = scipy.sparse.spdiags([e,e],[m-m**2,m**2-m],m**2,m**2).tolil()
    # Left & right BCs:
    for i in range(m):
        A_periodic[i*m,(i+1)*m-1] = 1.
        A_periodic[(i+1)*m-1,i*m] = 1.
    A = A + A_periodic
    A/=h**2
    A = A.todia()
    return A
```

```
A = five_pt_laplacian_sparse_periodic(4,-1.,1.)
plt.spy(A)
```

# Forward Euler time stepping
Since we are primarily interested in the final, steady-state solution, we can use a first-order accurate method in time.  Let's start by trying Euler's method:
$$\begin{align}
U^{n+1} & = U^n + k ( \delta D_1 \nabla^2_h U^n  + f(U^n,V^n)) \\\\
V^{n+1} & = V^n + k ( \delta D_2 \nabla^2_h V^n  + g(U^n,V^n)).
\end{align}$$
In the cell below, write a function that advances the solution one step using this approach.

```
def one_step(u,v,k,A,delta,D1=0.5,D2=1.0):
    u_new = u + k * (delta*D1*A*u + f(u,v))
    v_new = v + k * (delta*D2*A*v + g(u,v))

    return u_new, v_new
```

Now we'll use the function one_step() to solve the problem.  It turns out that to observe the behavior we're interested in, we can take an initial condition composed of random values in the interval $[-1/2, 1/2]$.

What time step should we use?  Since the diffusion terms are stiff, we may guess that it's sufficient to consider only them when selecting a time step size.  Recall that for Euler's method and centered differences, the 2D diffusion equation requires a step size
$$k \le \frac{h^2}{4\kappa}$$
where $\kappa$ is the diffusion coefficient.

Here is the code:

```
delta=0.0021; tau1=3.5; tau2=0; alpha=0.899; beta=-0.91; gamma=-alpha;

def step_size(h,delta):
    return h**2/(5.*delta)

def pattern_formation(m=10,T=1000):
    r"""Model pattern formation by solving a reaction-diffusion PDE on a periodic
        square domain with an m x m grid."""
    D1 = 0.5
    D2 = 1.0

    # Set up the grid
    a=-1.; b=1.
    h=(b-a)/m;                 # Grid spacing
    x = np.linspace(a,b,m)     # Coordinates
    y = np.linspace(a,b,m)
    Y,X = np.meshgrid(y,x)

    # Initial data
    u=np.random.randn(m,m)/2.;
    v=np.random.randn(m,m)/2.;

    plt.clf(); plt.hold(False)
    plt.pcolormesh(x,y,u); plt.colorbar(); plt.axis('image');
    plt.draw()

    u=u.reshape(-1)
    v=v.reshape(-1)

    A=five_pt_laplacian_sparse_periodic(m,-1.,1.)

    t=0.                      # Initial time
    k = step_size(h,delta)    # Time step size
    N = int(round(T/k))      # Number of steps to take

    #Now step forward in time
    next_plot = 0
    for j in range(N):

        u,v = one_step(u,v,k,A,delta)

        t = t+k;

        #Plot every t=5 units
        if t>next_plot:
            next_plot = next_plot + 5
            U=u.reshape((m,m))
            plt.clf()
            plt.pcolormesh(x,y,U)
            plt.colorbar()
            plt.axis('image')
            plt.title(str(t))
            time.sleep(0.2)
            clear_output()
            fi=plt.gcf()
            display(fi)
            plt.clf()
```

Now let's use this code to solve the problem.  Try it first with small values of $m$, then for larger values.

```
pattern_formation(m=150,T=300)
```

There is a problem with our approach, of course.  For larger values of $m$, the stiff diffusion term necessitates a very small time step, so the solution advances very slowly in time, so we would need to wait hours to get to the steady state.  Perhaps we can this problem by using a method that allows a larger step size.

# Implicit-explicit splitting
If we used the backward Euler method (or another A-stable method), we could use a much larger step size, but we would need to solve a large, nonlinear system of equations at each step.  To avoid this, we will split the equation into nonlinear parts and solve each part separately, i.e. first solve
$$\begin{align}
u_t & = D_1 \nabla^2 u  \\\\
v_t & = D_2 \nabla^2 v
\end{align}$$
and then solve
$$\begin{align}
u_t & = f(u,v)  \\\\
v_t & = g(u,v).
\end{align}$$

This is referred to as first-order operator splitting, or Godunov splitting.  It will reduce the temporal accuracy of our solution to first-order, but we are primarily interested in the final, steady-state solution, so that's acceptable.  We can use the backward Euler method for the diffusion terms -- to deal with the stiffness -- and the forward Euler method for the reaction terms:
$$\begin{align}
U^{n+1/2} & = U^n + k \delta D_1 \nabla^2_h U^{n+1/2}  \\\\
V^{n+1/2} & = V^n + k \delta D_2 \nabla^2_h V^{n+1/2} \\\\
U^{n+1} & = U^{n+1/2} + k f(U^{n+1/2},V^{n+1/2})  \\\\
V^{n+1} & = V^{n+1/2} + k g(U^{n+1/2},V^{n+1/2})
\end{align}$$

In the cell below, write a function that advances the solution one step using this approach.

```
def one_step(u,v,k,A,delta,D1=0.5,D2=1.0):
    # Your code here
    return u_new, v_new
```

```
def step_size(h,delta):
    return h/(10.*delta)
```

```
pattern_formation(m=150,T=300)
```

How fast is this code?  Why might that be?  What could we do to make it faster?  Can you implement something that is faster than either of these approaches, based on what you have learned in the course?

Using either code, try some of the following parameter values.  What kind of patterns can you generate?

```
delta=0.0045; tau1=2.02; tau2=0.; alpha=2.0; beta=-0.91; gamma=-alpha;
delta=0.0005; tau1=2.02; tau2=0.; alpha=2.0; beta=-0.91; gamma=-alpha;
delta=0.0021; tau1=3.5; tau2=0; alpha=0.899; beta=-0.91; gamma=-alpha;
delta=0.0045; tau1=0.02; tau2=0.2; alpha=1.9; beta=-0.85; gamma=-alpha;
delta=0.0001; tau1=0.02; tau2=0.2; alpha=0.899; beta=-0.91; gamma=-alpha;
delta=0.0045; tau1=0.02; tau2=0.2; alpha=1.9; beta=-0.91; gamma=-alpha;
```

```
print scipy.sparse.eye(4,4)
```

```
scipy.sparse.linalg.spsolve
```

ipython/A reaction-diffusion model for pattern formation.ipynb


# Bayesian Model Fitting in Python with emcee
## *Computational Research Day 2018 Workshop*

***
By AM Geller and AA Miller <br/>
April 10, 2018

Today we will focus on a seemingly simple problem: fitting a straight line to data.

Though we've all done this before, we've probably at some point oversimplified the analysis or at least not fully understood all the assumptions that went into our analysis.

This workshop is inspired by the paper [Data Analysis Recipes: Fitting a Model to Data](https://arxiv.org/abs/1008.4686) by Hogg, Bovy, & Lang. If you haven't already read this paper, I ***highly*** recommend it.

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse
from scipy.optimize import minimize

import emcee
import corner

#%matplotlib notebook
%matplotlib inline

np.random.seed(seed=2222)
ncores = 6 # adjust this to match your machine
```

The first thing we need is the data.  Let's start with a prepared data set from that paper.  Later we can move to a real data set (maybe your own).

In this data set, you'll notice that each $\left(x,y\right)$ point has an uncertainty characterized by a guassian with variance $\left(\sigma_x^, \sigma_y^2\right)$, while there is also covariance between the $x$ and $y$ measurements described by a correlation coefficient $\rho_{xy}$.  I found [this nice 1-page info sheet online](http://www.cs.utah.edu/~tch/CS4300/resources/refs/ErrorEllipses.pdf) if you want a further reference.

The full uncertainty covariance matrix for each data point is given by:

$$S_i = \left[ {\begin{array}{cc}
        \sigma_{xi}^2 & \rho_{xyi}\sigma_{xi}\sigma_{yi} \\
        \rho_{xyi}\sigma_{xi}\sigma_{yi} & \sigma_{yi}^2  \\
        \end{array}
       }
       \right]
       $$

```python
obs_x = np.array([203., 58., 210., 202., 198., 158.,
                165., 201., 157., 131., 166., 160.,
                186., 125., 218., 146.])
obs_y = np.array([495., 173., 479., 504., 510., 416.,
                393., 442., 317., 311., 400., 337.,
                423., 334., 533., 344.])
sigma_x = np.array([5., 9., 4., 4., 11., 7.,
                    5., 5., 5., 6., 6., 5.,
                    9., 8., 6., 5.])
sigma_y = np.array([21., 15., 27., 14., 30., 16.,
                    14., 25., 52., 16., 34., 31.,
                    42., 26., 16., 22.])
rho_xy = np.array([-0.33, 0.67, -0.02, -0.05, -0.84, -0.69,
                    0.30, -0.46, -0.03, 0.50, 0.73, -0.52,
                    0.90, 0.40, -0.78, -0.56])
```

Let's plot the data.

```python
def plotData(ax, x, y, xerr, yerr, rhoxy, zorder = 10):
    ax.scatter(x, y, color = 'black', zorder = 10)
    for (xx, yy, sx, sy, rxy) in zip(x,y,xerr, yerr, rhoxy):
        cov = np.array([[ sx**2, rxy*sx*sy],
                        [rxy*sx*sy, sy**2.]])
        w, v  = np.linalg.eig(cov)
        theta = np.arctan(2. * rxy * sx * sy / (sx**2. - sy**2.))/2.
        ax.add_artist(Ellipse((xx, yy), 2.*w[0]**0.5, 2.*w[1]**0.5, \
                              angle = np.rad2deg(theta),\
                              facecolor="none", edgecolor="black", zorder = zorder))

f, ax = plt.subplots(1, figsize = (5,5))
plotData(ax, obs_x, obs_y, sigma_x, sigma_y, rho_xy)
ax.set_xlim(0,300)
ax.set_ylim(0,700)
ax.set_xlabel('x')
ax.set_ylabel('y')
```

Now, let's fit the line :

$y = mx + b$.

How would you do this?

Probably the first attempt would be some kind of least squares optimization ... but ...

A standard assumption of the least-squares method is that the independent variable (typically $x$) is measured with perfect precision. Thus, standard least squares cannot account for the uncertainties on both $x$ and $y$ (simultaneously).  As a demonstration of ***WHAT NOT TO DO*** (but is sometimes attempted), here's an example of a "forward-reverse" fitting procedure, where you first fit by ignoring $\sigma_x$ then do the inverse and ignore $\sigma_y$.

I repeat ***DO NOT DO THIS*** for a real publication.  This is only to illustrate a point.

```python
f, ax = plt.subplots(1, figsize = (5,5))
plotData(ax, obs_x, obs_y, sigma_x, sigma_y, rho_xy)
ax.set_xlim(0,300)
ax.set_ylim(0,700)
ax.set_xlabel('x')
ax.set_ylabel('y')

#"abominable" forward–reverse fitting procedure
#DON'T DO THIS!
#polyfit with uncertainties on y ("forward")
p_xy, pcov = np.polyfit(obs_x, obs_y, 1, w = 1./sigma_y, cov = True)
err_xy = np.sqrt(np.diag(pcov))
p_xy_eval = np.poly1d(p_xy)
print("The best-fit value for the ('forward') slope and intercept are: {:.4f} +/- {:.4f} and {:.4f} +/- {:.4f}"\
      .format(p_xy[0], err_xy[0], p_xy[1], err_xy[1]))
ax.plot([0,300],p_xy_eval([0,300]),color='C0')

#polyfit with uncertainties on x ("reverse")
p_yx, pcov = np.polyfit(obs_y, obs_x, 1, w = 1./sigma_x, cov = True)
err_yx = np.sqrt(np.diag(pcov))
p_yx_eval = np.poly1d(p_yx)
inv = [1./p_yx[0], -p_yx[1]/p_yx[0]]
inv_err = [inv[0]**2.*err_yx[0],  (p_yx[0]**-2 * err_yx[1]**2. + (p_yx[1]/p_yx[0]**2.)**2. * err_yx[0]**2.)**0.5]
print("The best-fit value for the ('reverse') slope and intercept are: {:.4f} +/- {:.4f} and {:.4f} +/- {:.4f}"\
      .format(inv[0], inv_err[0], inv[1], inv_err[1]))
ax.plot(p_yx_eval([0.,700.]), [0.,700.],'--',color='C1')

mFreq = p_xy[0]
bFreq = p_xy[1]

```

Again ***DO NOT DO THIS IN A PUBLICATION***.

It should be fairly obvious why you don't want to do this.  For instance,
- The results (at least for the intercept) are not in agreement, given the uncertainties on the parameters
- How would you combine these results to give some meaningful answer?
- Do you actually trust the uncertainties on these parameters?  Are the parameter uncertainties actually symmetric?
- Most importantly, neither method actually accounts for both uncertainties or their correlation coefficients ($\rho_{xy}$).

Least-squares fitting reduces to some linear algebra. [See the Hogg, Bovy, & Lang paper referenced above for the derivation of these relations.]

Below we demonstrate the same results as `np.polyfit()` above.

```python
#a little bit more thorough and returns the same result as the "forward" fit
Y = obs_y.reshape(-1,1)
A = np.vstack((np.ones_like(obs_x), obs_x)).T
C = np.diag(sigma_y**2)

X = np.linalg.inv(A.transpose()@np.linalg.inv(C)@A) @ (A.transpose()@np.linalg.inv(C)@Y)

best_fit = np.poly1d(X[::-1,0])
print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format(X[1][0], X[0][0]))
```

OK.  The naive first attempt is not appropriate for this problem.  That least squares fitting makes certain assumptions (e.g., that the $x$ uncertainties are negligible).  Let's relax those assumptions and frame the problem more generally.

First, some statistics... (Some of you may enjoy this, others may not.  It's OK if you don't fully understand all of this today.  We will get to the `emcee` fitter very soon.)

## Minimizing $\chi^2$

A common procedure taught in undergraduate lab classes is to minimize the "chi squared" in order to identify the "best-fit" parameters $m$ and $b$:

$$ \chi^2 = \Sigma_{i = 1}^{N} \frac{\left[y_i - f(x_i)\right]^2}{\sigma_{y_i}^2},$$

where $f(x) = mx + b$ is the model for the data.

A least-squares fit, such as the ones used above, minimizes the value of $\chi^2$, which in turn provides a "best-fit" estimate of $m$ and $b$.

## Maximizing the Likelihood

Alternatively, we could consider a generative model, in which case the probability of any observation $y_i$ is dependent on $x_i$, our model $f(x)$, and some scatter associated with the measurement itself $\sigma_{yi}$:

$$ p\left(y_i\, | \, x_i, \sigma_{yi}, f(x_i) \right) = \frac{1}{\sqrt{2\pi\sigma_{y_i}^2}}\,\exp\left(-\frac{\left[y_i - f(x_i)\right]^2}{2\,\sigma_{y_i}^2} \right).$$

An alternative to $\chi^2$ minimization is to maximize the probability of the observed data given the model, which is to say we want to maximize the *likelihood of the observed data given the model*. We can write the likelihood function $\mathscr{L}$ as the product of the conditional probability of every observation:

$$ \mathscr{L} = \prod_{i=1}^{N} p\left(y_i\, |\, x_i, \sigma_i, f(x_i) \right)$$

This product is often difficult to evaluate, so we often take the logarithm of the likelihood, and maximize that. In this case we get:

$$ \ln \mathscr{L} = K - \sum_{i=1}^{N} \frac{\left[y_i - f(x_i) \right]^2.}{2\sigma_{yi}^2}
   = K - \frac{1}{2}\chi^2 $$

where $K$ is some constant. As you can see, maximizing the likelihood (in this case) is equivalent to minimizing $\chi^2$.

Note: an important assumption in all of this is that the uncertainties, $\left(\sigma_x, \sigma_y\right)$, are Gaussian.  This is usually a reasonable assumption.

When maximizing the likelihood we can get a "pretty good" estimate of the model parameters, $m$ and $b$. This estimate is only "pretty good" because Thomas Bayes is rolling in his grave. Bayes' theorem tells us about conditional probability distributions:

$$ p\left(\theta\, |\, X, I\right) = \frac{p\left(X\, |\, \theta, I\right) p\left(\theta\, |\, I\right)}{p\left(X\, |\, I\right)} $$

In words, we want to know the (*posterior*) probability distribution for the parameters $\theta$, which $= (m,b)$ in this case, given the data $X$ and any prior knowledge $I$, $p\left(\theta\, |\, X, I\right)$.

$p\left(X\, |\, \theta, I\right)$ is the *likelihood*, and has been defined above.

$p\left(\theta\, |\, I\right)$ is the "*prior*" probability distribution for the parameters, that contains all of our knowledge of what those parameters should be (this can come from physical models or previous experiments).

The denominator can be thought of as a normalization constant that most people ignore (and we will also ignore it).

Ok.  That's very nice.

But how do we actually use all this math??!!

And, hey, you said this was a workshop about emcee!

```python
# compute the log likelihood
def lnlike(theta, x, y, yerr):
    m, b = theta
    model = m * x + b
    ln_l = np.sum( np.log( 1./np.sqrt(2*np.pi *sigma_y**2.) * np.exp(-0.5*((y - model)/sigma_y)**2) ) )

    return ln_l.squeeze()

# compute the log prior
def lnprior(theta):
    m, b = theta
    if ( (0 < m < 10) and (-200 < b < 200) ):
        return 0.0
    return -np.inf

# compute the log of the likelihood multiplied by the prior
def lnprob(theta, x, y, yerr):
    lnp = lnprior(theta)
    lnl = lnlike(theta,  x, y, yerr)
    if (not np.isfinite(lnp) or not np.isfinite(lnl)):
        return -np.inf
    return lnp + lnl

```

Now, as a check, let's try to maximize this likelihood with scipy and confirm that we get the same results as the least-squares procedure described above (actually we'll minimize the negative log likelihood as this is an easier computational task).

```python
def ilnlike(theta, x, y, yerr):
    lnl = -lnlike(theta, x, y, yerr)
    return lnl.squeeze()

guess0 = np.array([2.*np.random.random() * mFreq, 2.*np.random.random() * bFreq])
scFit = minimize(ilnlike, guess0, args=(obs_x, obs_y, sigma_y))
print(scFit.message)
print("The initial guess for the slope and intercept are: {:.4f} and {:.4f}"\
      .format(guess0[0], guess0[1]))
print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format(scFit.x[0], scFit.x[1]))

```

## *emcee*

Now, let's use *[emcee](http://dfm.io/emcee/current/)*, a pure-Python implementation of [Goodman & Weare’s Affine Invariant Markov chain Monte Carlo (MCMC) Ensemble sampler](https://msp.org/camcos/2010/5-1/p04.xhtml) written by [Dan Foreman-Mackey](http://dfm.io/).

We've already spent enough time on background, so I will let you read about MCMC on your own (e.g., [here's the wikipedia entry](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo)).  You may also want to read [this journal article about emcee](https://arxiv.org/abs/1202.3665).

In short, we can use *emcee* to sample the posterior distribution that is defined by a Bayesian likelihood and priors (using MCMC).  *emcee* will use a number of "walkers" to explore this parameter space, each sampling around the maximum of the likelihood function (while accounting for the priors).  The combination of the paths that each of these walkers take, i.e. their "chains", define the posterior distribution, and provide us with probability distributions for each of the model parameters.

```python
nwalkers = 100
nfac = [1e-2, 1e-2]
ndim = len(guess0)
pos = [guess0 + nfac * np.random.randn(ndim) for i in range(nwalkers)]

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(obs_x, obs_y, sigma_y), threads = ncores)

nsamples = 2000
foo = sampler.run_mcmc(pos, nsamples)

```

And now let's explore the output

```python
# set a "burn-in" limit
nburn = 200
m_samples = np.array(sampler.chain[:,nburn:,0]).flatten()
b_samples = np.array(sampler.chain[:,nburn:,1]).flatten()

#print the results taking 1-sigma confidence intervals
samples = np.vstack([m_samples, b_samples]).T
m_mcmc, b_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]), zip(*np.percentile(samples, [16, 50, 84], axis=0)))
print("emcee results with 1-sigma uncertainties\n \
      m = {:.4f} +{:.4f} -{:.4f}\n \
      b = {:.4f} +{:.4f} -{:.4f}\n" \
      .format(m_mcmc[0], m_mcmc[1], m_mcmc[2], b_mcmc[0], b_mcmc[1], b_mcmc[2]))
```

```python
#some function to make plots (will re-use them later)
#to plot the chains
def plotChains(sampler, nburn, paramsNames):
    Nparams = len(paramsNames)
    fig, ax = plt.subplots(Nparams,1, figsize = (8,2*Nparams), sharex = True)
    fig.subplots_adjust(hspace = 0)
    ax[0].set_title('Chains')
    xplot = range(len(sampler.chain[0,:,0]))

    for i,p in enumerate(paramsNames):
        for w in range(sampler.chain.shape[0]):
            ax[i].plot(xplot[:nburn], sampler.chain[w,:nburn,i], color="0.5", alpha = 0.4, lw = 0.7, zorder = 1)
            ax[i].plot(xplot[nburn:], sampler.chain[w,nburn:,i], color="k", alpha = 0.4, lw = 0.7, zorder = 1)

            ax[i].set_ylabel(p)

    return ax

paramsNames = ['m','b']

axC = plotChains(sampler, nburn, paramsNames)
```

```python
#corner plot
def makeCorner(sampler, nburn, paramsNames, quantiles=[0.16, 0.5, 0.84]):
    samples = sampler.chain[:, nburn:, :].reshape((-1, len(paramsNames)))
    f = corner.corner(samples, labels = paramsNames, quantiles = quantiles)

makeCorner(sampler, nburn, paramsNames)
```

```python
#Adam's plots
def prettyPlot(x, y, xerr, yerr, rhoxy, m_samples, b_samples, m, b, nburn, ndraw = 20, extent = None):
    fig, ax = plt.subplots(1,2, figsize = (8,4))
    ax[0].set_xlabel("m")
    ax[0].set_ylabel("b")
    ax[1].set_xlabel("x")
    ax[1].set_ylabel("y")

    ax[0].hexbin(m_samples[nburn:], b_samples[nburn:], gridsize = 250,  mincnt = 1, bins = "log", extent = extent)

    plotData(ax[1], x, y, xerr, yerr, rhoxy)
    ax[1].plot([0,300], [b + 0*m, b + 300*m], lw = 2, zorder = 2)
    if (ndraw > 0):
        for rand_draw in np.random.randint(nburn, len(m_samples), ndraw):
            m_draw = m_samples[rand_draw]
            b_draw = b_samples[rand_draw]
            ax[1].plot([-1e10,1e10], [b_draw + -1e10*m_draw, b_draw + 1e10*m_draw],
                     color = "0.5", alpha = 0.4, lw = 0.7, zorder = 1)

    return ax

axA = prettyPlot(obs_x, obs_y, sigma_x, sigma_y, rho_xy, m_samples, b_samples, m_mcmc[0], b_mcmc[0], nburn, ndraw = 20)
axA[1].set_xlim(0,300)
axA[1].set_ylim(0,700)
```

OK.  But... didn't we just get the same result as before?

And, didn't you start this workshop talking about how we need to account for the uncertainties on $x$ as well?

## Uncertainties on both axes

To also account for the uncertainties on $x$, we need do define a new likelihood function.  We've gone through a lot of statistics already, so, for the sake of time, we'll breeze through this.  A more complete derivation is included in the Hogg, Bovy & Lang paper.

If we assume the uncertainties are Gaussian (as we are doing), we can write the probability for obtaining a measurement,

$$ Z_i = \left[ {\begin{array}{c}
        x_i \\
        y_i \\
        \end{array}
       }
       \right]
$$

when the "true value" (the value you would have this data point if it had been observed with negligible noise) is:

$$ Z = \left[ {\begin{array}{c}
        x \\
        y \\
        \end{array}
       }
       \right]
$$

as

$$ p\left(x_i, y_i\, |\, S_i, x,y\right) = \frac{1}{2\pi \sqrt{\det(S_i)}} \exp\left(-\frac{1}{2}\left[Z_i - Z\right]^T S_i^{-1} \left[Z_i - Z\right]\right).$$

But how do we use this to fit a line?  (What should we use for Z?)  The "trick" is to project the 2D uncertainties onto the sub-space that is orthogonal to the line and to evaluate the projected displacements.  This will involve a change in variables from $(m, b)$ to $(\theta, b_\perp)$, as follows:

$$
\begin{align}
\theta &= \arctan m \\
b_\perp &= b \cos \theta \\
\hat{v} &= \left[ {\begin{array}{c}
        -\sin \theta \\
        \cos \theta \\
        \end{array}
       }
       \right] \\
\Delta_i &= \hat{v}\, Z_i - b_\perp \\
\Sigma_i^2 &= \hat{v}^T\, S_i\, \hat{v} \\
\ln \mathscr{L} &= K - \sum_{i=1}^{N} \frac{\Delta_i^2}{2\Sigma_i^2} \\
\end{align}
$$

In short, $\Delta_i$ is the orthogonal displacement of each data point $(x_i, y_i)$from the line, and $\Sigma^2$ is the orthogonal variance (by projecting the covariance matrix $S_i$, which we defined previously).

Whew!  Let's see that in action.

```python
def lnlike2(theta, x, y, xerr, yerr, rhoxy):
    th, bperp = theta
    lnl = 0.
    v = np.array([[-np.sin(th)], [np.cos(th)]])
    vT = v.transpose()
    for (xx, yy, sx, sy, rxy) in zip(x, y, xerr, yerr, rhoxy):
        S = np.array([[ sx**2, rxy*sx*sy],
                    [rxy*sx*sy, sy**2.]])
        Z = np.array([[xx],[yy]])
        Delta = vT @ Z - bperp
        Sigma2 = vT @ S @ v
        lnl -= Delta**2. / (2. * Sigma2)      #this ignores K (a constant), which doesn't change the likelihood maximum

    return lnl#.squeeze()

def lnprior2(theta):
    th, bperp = theta
    if ( (0 < np.abs(th) < 2.*np.pi) and (-200 < bperp < 200)):
        return 0.0
    return -np.inf

def lnprob2(theta, x, y, xerr, yerr, rhoxy):
    lnp = lnprior2(theta)
    lnl = lnlike2(theta,  x, y, xerr, yerr, rhoxy)
    if (not np.isfinite(lnp) or not np.isfinite(lnl)):
        return -np.inf
    return lnp + lnl

def thFromM(m):
    return np.arctan(m)
def bpFromBth(b, th):
    return b * np.cos(th)
def mFromTh(th):
    return np.tan(th)
def bFromThBp(th, bp):
    return bp/np.cos(th)
```

## Exercise:

Run this probability distribution through *emcee*, then print and plot the results.

```python
#Hint: find the related cell(s) above; copy the contents and paste it here.  Then make any necessary edits.
```

```python
# set a "burn-in" limit
nburn = 200
th_samples = np.array(sampler.chain[:,nburn:,0]).flatten()
bp_samples = np.array(sampler.chain[:,nburn:,1]).flatten()
m_samples = mFromTh(th_samples)
b_samples = bFromThBp(th_samples, bp_samples)

#print the results taking 1-sigma confidence intervals
samples = np.vstack([m_samples, b_samples]).T
m_mcmc, b_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]), zip(*np.percentile(samples, [16, 50, 84], axis=0)))
print("emcee results with 1-sigma uncertainties\n \
      m = {:.4f} +{:.4f} -{:.4f}\n \
      b = {:.4f} +{:.4f} -{:.4f}\n" \
      .format(m_mcmc[0], m_mcmc[1], m_mcmc[2], b_mcmc[0], b_mcmc[1], b_mcmc[2]))
```

```python
paramsNames = [r'$\theta$', r'$b_\perp$']

#chain plot
axC = plotChains(sampler, nburn, paramsNames)
```

```python
#corner plot
makeCorner(sampler, nburn, paramsNames)
```

```python
#Adam's plots
axA = prettyPlot(obs_x, obs_y, sigma_x, sigma_y, rho_xy, m_samples, b_samples, m_mcmc[0], b_mcmc[0], nburn, ndraw = 20)
axA[1].set_xlim(0,300)
axA[1].set_ylim(0,700)
```

So far we haven't really showcased the power of Bayesian model fitting.  One example of when you would really want to use a Baysian tool is when you have model parameters that you want to *marginalize over*.  The following examples dive into this.

We will first increase the scatter of our data by adding a few outliers.

## Instrinsic Scatter

Let's assume that the data is still best described by a line, and we think the best fit solution is what we found above (without the extra scatter).

```python
obs_x = np.append(obs_x, np.array([201., 244., 287.]))
obs_y = np.append(obs_y, np.array([592., 401., 402.]))
sigma_x = np.append(sigma_x, np.array([9., 4., 7.]))
sigma_y = np.append(sigma_y, np.array([61., 25., 15.]))
rho_xy = np.append(rho_xy, np.array([-0.84, 0.31, -0.27]))
```

## Exercise:

Use our previous likelihood function with scipy.minimize to fit these data.  Print and plot the results.

```python
#Hint: find the related cell(s) above; copy the contents and paste it here.  Then make any necessary edits.
```

The fit is different, clearly pulled by the outliers.  Both fit parameters are $>3\sigma$ away from what we found previously.   Also, notice that (at least on my notebook) the minimize function is giving an ominous warning.  (I would not want to publish the result of that fit!)

But we can deal with this using a slightly different likelihood function, and using *emcee*.  We will introduce one additional variable, $V$ : Gaussian variance orthogonal to the line.  Hogg et al. give us the likelihood:

$$
\ln \mathscr{L} = K - \sum_{i=1}^{N}\frac{1}{2}\ln\left(\Sigma_i^2 + V\right) - \sum_{i=1}^{N} \frac{\Delta_i^2}{2\left[\Sigma_i^2 + V\right]}
$$

## Exercise:

Code this up into a new likelihood, prior and posterior probability distribution.  Run this through *emcee*.  Print and plot the results.  In your plot that includes the fit line(s), also show the "$1\sigma$" range in the scatter.

```python
#Hint: find the right cell above; copy the contents and paste it here.  Then make any necessary edits.
```

```python
nwalkers = 100
nfac = [1e-2, 1e-2, 1.]
th = thFromM(mFreq)
bp = bpFromBth(bFreq, th)
guess0 = np.array([th, bp, 30.])
ndim = len(guess0)
pos = [guess0 + nfac * np.random.randn(ndim) for i in range(nwalkers)]

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprobS, args=(obs_x, obs_y, sigma_x, sigma_y, rho_xy), threads = ncores)

nsamples = 2000
foo = sampler.run_mcmc(pos, nsamples)
```

```python
# set a "burn-in" limit
nburn = 200
th_samples = np.array(sampler.chain[:,nburn:,0]).flatten()
bp_samples = np.array(sampler.chain[:,nburn:,1]).flatten()
s_samples = np.array(sampler.chain[:,nburn:,2]).flatten()
m_samples = mFromTh(th_samples)
b_samples = bFromThBp(th_samples, bp_samples)

#print the results taking 1-sigma confidence intervals
samples = np.vstack([m_samples, b_samples, s_samples]).T
m_mcmc, b_mcmc, s_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]), zip(*np.percentile(samples, [16, 50, 84], axis=0)))
print("emcee results with 1-sigma uncertainties\n \
      m = {:.4f} +{:.4f} -{:.4f}\n \
      b = {:.4f} +{:.4f} -{:.4f}\n \
      s = {:.4f} +{:.4f} -{:.4f}\n" \
      .format(m_mcmc[0], m_mcmc[1], m_mcmc[2], b_mcmc[0], b_mcmc[1], b_mcmc[2], s_mcmc[0], s_mcmc[1], s_mcmc[2]))
```

```python
paramsNames = [r'$\theta$', r'$b_\perp$', r'$\sqrt{V}$']

#chain plot
axC = plotChains(sampler, nburn, paramsNames)
```

```python
#corner plot
makeCorner(sampler, nburn, paramsNames)
```

```python
#Adam's plots
fac = 10.
axA = prettyPlot(obs_x, obs_y, sigma_x, sigma_y, rho_xy, m_samples, b_samples, m_mcmc[0], b_mcmc[0], nburn, ndraw = 0, \
                 extent = [m_mcmc[0] - fac*m_mcmc[2], m_mcmc[0] + fac*m_mcmc[1], b_mcmc[0] - fac*b_mcmc[2], b_mcmc[0] + fac*b_mcmc[1]])
axA[0].set_xlim(m_mcmc[0] - fac*m_mcmc[2], m_mcmc[0] + fac*m_mcmc[1])
axA[0].set_ylim(b_mcmc[0] - fac*b_mcmc[2], b_mcmc[0] + fac*b_mcmc[1])
axA[1].set_xlim(0,300)
axA[1].set_ylim(0,700)

smax = s_mcmc[0] + s_mcmc[1]
x0 = -b_mcmc[0]/m_mcmc[0] #x intercept
beta = np.arctan2(x0, b_mcmc[0]) #angle between line and y axis
alpha = np.pi/2. - beta #angle between perpendicular offset (s) and line
d = smax / np.cos(alpha) #distance to perpendicular offset in y direction
axA[1].plot([-1e10,1e10], [d + b_mcmc[0] + -1e10*m_mcmc[0], d + b_mcmc[0] + 1e10*m_mcmc[0]], ':k')
axA[1].plot([-1e10,1e10], [-d + b_mcmc[0] + -1e10*m_mcmc[0], -d + b_mcmc[0] + 1e10*m_mcmc[0]], ':k')
```

## Pruning Outliers

We will now append an additional outlier to our data.

Again, let's assume that the data is still best described by a line, and we think the best fit solution is what we found above (without the extra scatter or outliers).

```python
obs_x = np.append(obs_x, 47.)
obs_y = np.append(obs_y, 583.)
sigma_x = np.append(sigma_x, 11.)
sigma_y = np.append(sigma_y, 38.)
rho_xy = np.append(rho_xy, 0.64)
```

## Exercise:

Use our previous likelihood function with scipy.minimize to fit these data.  Print and plot the results.

```python
#Hint: find the related cell(s) above; copy the contents and paste it here.  Then make any necessary edits.
```

Clearly this result is different from what we found before, and again we see the ominous warning from scipy.

Given just the data, and thus without any direct knowledge which observations are "outliers", how could we verify this fit and deal with what we may think "by eye" are outliers?

An ad hoc solution would be to try some type of sigma clipping algorithm.  This is very popular, but also rather subjective.  We don't want to introduce unnecesary biases into our fit.  So we want to ***avoid sigma clipping***.

Instead we will alter the likelihood function to describe a gaussian mixture model where some of the data that we have observed come from a distribution described by a line and the remaining observations come from a gaussian model. Ultimately, we don't care about which points are outliers, we are primarily interested in determining the values of $m$ and $b$. This is where we will truly take advantage of the Bayesian framework that we have been developing as we are going to *marginalize* over the outlier parameters to determine confidence regions on the values of $m$ and $b$.

With math: Recall our likelihood function that describes the fit line with uncertainties on both axes:

$$
\ln \mathscr{L} = K - \sum_{i=1}^{N} \frac{\Delta_i^2}{2\Sigma_i^2}
$$

This is the "good" model that we care about.  Let's assume that some fraction of our data, $p_g$, comes from this model.  For this step, we need the constant, $K$, to be defined explicitly. So let's be a bit more thorough:

$$
\mathscr{L}_g = \sum_{i=1}^{N} \left( \frac{p_g}{\sqrt{2\pi\Sigma_i^2}} exp\left(  - \frac{\Delta_i^2}{2\Sigma_i^2}\right) \right)
$$

We now assume that the remaining fraction, $1-p_g$, of our data comes from a "bad" model that we *don't* care about, but is described by some variance, $V_b$, and mean value, $\Delta_b$, relative to the line:

$$
\mathscr{L}_b = \sum_{i=1}^{N} \left( \frac{1 - p_g}{\sqrt{2\pi\left(\Sigma_i^2 + V_b\right)}} exp\left(  - \frac{\left(\Delta_i - \Delta_b\right)^2}{2\left(\Sigma_i^2 + V_b\right)}\right) \right)
$$

The total log likelihood is the sum of these two log likelihoods written above"

$$
\ln \mathscr{L} = \ln \left(\mathscr{L}_g + \mathscr{L}_b \right)
$$

In principle we could also use this model to evaluate if any given data point is drawn from the outlier distribution ("bad") or the line ("good").  There's an interesting discussion on this on [Dan Foreman-Mackey's blog](http://dfm.io/posts/mixture-models/).

```python
#writing things slightly differently here so that I can return both "blobs" from emcee
def lnlikeOg(theta, x, y, xerr, yerr, rhoxy):
    th, bperp, _, _, _ = theta
    lnlg = np.array([])
    v = np.array([[-np.sin(th)], [np.cos(th)]])
    vT = v.transpose()
    for (xx, yy, sx, sy, rxy) in zip(x, y, xerr, yerr, rhoxy):
        S = np.array([[ sx**2, rxy*sx*sy],
                    [rxy*sx*sy, sy**2.]])
        Z = np.array([[xx],[yy]])
        Delta = vT @ Z - bperp
        Sigma2 = vT @ S @ v

        lnlg = np.append(lnlg, -np.log(np.sqrt(2*np.pi*Sigma2)) - Delta**2/(2*Sigma2))

    return lnlg

def lnlikeOb(theta, x, y, xerr, yerr, rhoxy):
    th, bperp, p_g, sigma_b, delta_b = theta
    sigma_b = np.clip(sigma_b, 0, 2000)         # prevent errors in likelihood eval
    lnlb = np.array([])
    v = np.array([[-np.sin(th)], [np.cos(th)]])
    vT = v.transpose()
    for (xx, yy, sx, sy, rxy) in zip(x, y, xerr, yerr, rhoxy):
        S = np.array([[ sx**2, rxy*sx*sy],
                    [rxy*sx*sy, sy**2.]])
        Z = np.array([[xx],[yy]])
        Delta = vT @ Z - bperp
        Sigma2 = vT @ S @ v

        lnlb = np.append(lnlb, -np.log(np.sqrt(2*np.pi*(Sigma2 + sigma_b**2.))) - (Delta - delta_b)**2/(2*(Sigma2 + sigma_b**2.)))

    return lnlb

#priors
def lnpriorO(theta):
    th, bperp, p_g, sigma_b, delta_b = theta
    if ( (0 < np.abs(th) < 2.*np.pi) and (-200 < bperp < 200) and (0 <= sigma_b < 2000)
        and (0 < p_g < 1) and (-2000 < delta_b < 2000)):
        return 0.0
    return -np.inf

def lnprobO(theta, x, y, xerr, yerr, rhoxy):
    th, bperp, p_g, sigma_b, delta_b = theta

    lnp = lnpriorO(theta)
    p_g = np.clip(p_g, 1e-20, 1 - 1e-20)         # prevent errors in likelihood eval

    lnlg = lnlikeOg(theta,  x, y, xerr, yerr, rhoxy)
    arg1 = lnlg + np.log(p_g)

    lnlb = lnlikeOb(theta,  x, y, xerr, yerr, rhoxy)
    arg2 = lnlb + np.log(1. - p_g)

    lnl = np.sum(np.logaddexp(arg1, arg2))

    if (not np.isfinite(lnp) or not np.isfinite(lnl)):
        return -np.inf, None

    # We're using emcee's "blobs" feature in order to keep track of the
    # "good" and "bad" likelihoods for reasons that will become
    # clear soon.

    return lnp + lnl, (arg1, arg2)
```

## Exercise:

Run this probability distribution through *emcee*, then print and plot the results. (Hint, you may need to increase the number of samples and the burn-in limit because we are now fitting to more parameters.)

```python
#Hint: find the related cell(s) above; copy the contents and paste it here.  Then make any necessary edits.
```

# Now let's work with real data

I pulled Table 1 from [this paper on the "M-$\sigma$" relation](http://adsabs.harvard.edu/abs/2009ApJ...698..198G)

I will use *pandas* to read in and reformat the data, but feel free to do this instead with your own favorite method.

```python
import pandas as pd
from IPython.display import display
```

```python
#data from : http://adsabs.harvard.edu/abs/2009ApJ...698..198G
#They find log(MBH/M) = α + β log(σ/200 km s−1) with (α,β,0) = (8.12 ± 0.08, 4.24 ± 0.41, 0.44 ± 0.06)
#But they include some with upper limits that I don't have here
mSigma = pd.read_csv('apj303542t1_ascii.txt', quotechar='"', sep='\t')
#print(mSigma.columns.values)
display(mSigma)

def fixSci(series):
    x = series.str.split(' ', expand = True)
    x2 = x[:][2].str.split('^', expand = True)
    return(x[:][0].astype("float") * 10.**x2[:][1].astype("float"))

def fixSig(series):
    x = series.str.split(' ', expand = True)
    return(x[:][0].astype("float"))
def fixSige(series):
    x = series.str.split(' ', expand = True)
    x2 = x[:][2].str.split('^', expand = True)
    return(x2[:][0].astype("float"))

obs_x = np.array(fixSig(mSigma['sigma_e (km s^-1)']))
obs_lx = np.log10(obs_x)
sigma_lx = 1./(obs_x * np.log(10.)) * np.array(fixSige(mSigma['sigma_e (km s^-1)']))

obs_y = np.array(fixSci(mSigma['M_BH (M_sun)']))
obs_ly = np.log10(obs_y)

#this is probably not what we would want for publication, but we can first try to take a mean uncertainty in x
y_low = np.array(fixSci(mSigma['M_low (M_sun)']))
y_high = np.array(fixSci(mSigma['M_high (M_sun)']))
ye_low = obs_y - y_low
ye_high = y_high - obs_y
ye = np.array([ (y1 + y2)/2. for (y1, y2) in zip(ye_low, ye_high)])
sigma_ly = 1./(obs_y * np.log(10.)) * ye

rho_lxy = np.full_like(obs_x, 0.0)

```

## Exercise:

Plot these data ("$x$" is $\log_{10}\sigma$,  and "$y$" is $\log_{10} M$).  Then run the data through *emcee*, using the likelihood that accounts for intrinsic scatter.  Print and plot the results.

```python
#Hint: find the related cell(s) above; copy the contents and paste it here.  Then make any necessary edits.
```


# Forward and Reverse mode gradients in TFF

```python
#@title Licensed under the Apache License, Version 2.0 (the "License"); { display-mode: "form" }
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

```python
import tensorflow as tf
import functools
import tf_quant_finance.math.gradient as tff_gradient
import tf_quant_finance.math.optimizer as tff_optimizer
import numpy as np
```

Consider a simple function in two variables $x$ and $y$ but in three dimensions such that the function along each axes is [$x^2$, $y^2$, $x y$]

$
\begin{align}
& f = [f_1, f_2, f_3] \\
& where \\
\end{align}
$

$
\begin{align}
f_1 &= x^2 \\
f_2 &= y^2 \\
f_3 &= x + y \\
\end{align}
$

```python
def func_grad(x):
    func = tf.stack([x[0]**2, x[1]**2, x[0]*x[1]])
    grad = tf.stack([2.0*x[0]+x[1], 2.0*x[1]+x[0]])
    return func, grad

def func(x):
    func = tf.stack([x[0]**2, x[1]**2, x[0]*x[1]])
    return func

start = tf.constant([1,1], dtype=tf.float64)

```

# Reverse mode

$
\begin{align}
\frac {\partial f}{\partial x} &= u_1 \frac{\partial f_1}{\partial x} + u_2 \frac{\partial f_2}{\partial x} + u_3 \frac{\partial f_3}{\partial x} \\ \\
&= u_1 2 x + u_3 y\\ \\
\frac {\partial f}{\partial y} &= u_1 \frac{\partial f_1}{\partial y} + u_2 \frac{\partial f_2}{\partial y} + u_3 \frac{\partial f_3}{\partial y} \\ \\
&= u_1 2 y + u_3 x
\end{align}
$

In Tensorflow, [$u_1$, $u_2$, $u_3$] is normally set to [1, 1, 1]

Setting [$x$, $y$] to [1,1]

Reverse mode returns the gradients summed up by components

```python
func_grad(start)[1]
```

    <tf.Tensor: shape=(2,), dtype=float64, numpy=array([3., 3.])>

# Forward mode

$
\begin{align}
{\partial f_1} &= w_1 \frac{\partial f_1}{\partial x} + w_2 \frac{\partial f_1}{\partial y} \\ \\
&= w_1 2 x \\ \\
{\partial f_2} &= w_1 \frac{\partial f_2}{\partial x} + w_2 \frac{\partial f_2}{\partial y} \\ \\
&= w_2 2 y \\ \\
{\partial f_3} &= w_1 \frac{\partial f_3}{\partial x} + w_2 \frac{\partial f_3}{\partial y} \\ \\
&= w_1 x + w_2 y \\ \\
\end{align}
$

In Tensorflow, [$w_1$, $w_2$] is normally set to [1, 1]

Setting [$x$, $y$] to [1,1]

Forward mode returns the gradients by components

```python
tff_gradient.fwd_gradient(func, start)
```

    <tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 2., 2.])>

Remember, Tensorflow is the tool used in Machine Learning. In Machine Learning, the aim is to minmise the scalar loss function, the loss function being the sum of the gradient with respect to the feature set. This lowest loss is the loss summed up over all training examples.

However, lets take the use case where we are valuing a set of Options, say ten, against a single spot price $S_0$. We now have ten price functions and we need their gradients against spot $S_0$ (ten delta's).

Using the forward gradients with respect to $S_0$ would give us the ten delta's in a single pass.

Using the backward gradients would result in the sum of the ten delta's, which may not be that useful.

It is useful to note that varying the weights would also give you individual components of the gradients (in other words [1,0] and [0,1] as values of [$w_1$, $w_2$], instead of the default [1,1], similarly for backward. This is, of course, at the expense of more compute.

```python

```