## Introduction

The WBK method is an approximation technique for approximating solutions to the time-independent Schrödinger equation
with non-constant potential.
It is named after Wentzel, Kramers, and Brillouin. In 1923, three years prior to the development of the WKB method, a
British mathematician Harold Jeffreys also developed it. But it was not very popular back then. In England, people
also call it the JWKB method for the contribution made by Jeffreys.

## Derivation of WKB

The Schrödinger equation with non-constant potential is,

\begin{equation}
    -\frac{\hbar^2}{2m} \frac{d^2 \psi}{dx^2} + V(x)\psi = E\psi.
\end{equation}

We assume $E>V(x)$ and rewrite it as

\begin{equation}
    \frac{d^2 \psi}{dx^2}= -\frac{p^2}{\hbar^2} \psi,
\end{equation}

where $p(x)=\sqrt{2m[E-V(x)]}$ is the classical momentum of a particle with total energy E and potential $V(x)$.

The general solution of this equation has the form

\begin{equation}
    \psi(x) = A(x)e^{i\phi(x)},
\end{equation}

where $A(x)$ is the amplitude and $\phi(x)$ is the phase, both as a function of x. Then we can calculate the first and
the second derivative of $\psi$ and plug them back to the Schrödinger equation to get

\begin{equation}
    A'' + 2i A' \phi' + iA\phi'' - A(\phi')^2 = - \frac{p^2}{\hbar^2}A.
\end{equation}

The real and imaginary part each gives us an equation.

The imaginary part is

\begin{equation}
    2A'\phi' + A\phi'' = 0.
\end{equation}

Its solution is

\begin{equation}
    A = \frac{C}{\sqrt{\phi'}},
\end{equation}

where $C$ is a real number.

The real part is

\begin{equation}
    A'' - A (\phi')^2 = -\frac{p^2}{\hbar^2}A.
\end{equation}

Unfortunately this equation cannot be solved in most cases. Here, the approximation comes in. By assuming $A(x)$ varies
slowly as a function of $x$, we can make $A''=0$. Then we can solve it to get

\begin{equation}
    \phi(x) = \pm \frac{1}{\hbar} \int p(x)dx.
\end{equation}

Therefore, the general solution is given by

\begin{equation}
    \psi(x) = \frac{C}{\sqrt{p(x)}} e^{\pm \frac{i}{\hbar} \int p(x)dx}.
\end{equation}

For $E<V$, we need to take $p(x)$ to absolute value to keep it a real function.

Please see Chapter 8 of Griffiths' Introduction to Quantum Mechanics for a more detailed discussion. Professor
Barton Zwiebach's lecture presented another derivation. It's very interesting as well.

## Validity of WKB

The argument we used in the derivation for the approximation is rather mathematical. A more physical way to look it is
that the probability of finding a particle at location $x$ is inversely proportional to its momentum. It agrees with our
intuition. If a particle moves fast at one location, it's less likely to be found by us. Professor Zwiebach derived some
more formal inequalities to justify the validity of the WKB approximation. Dropping the second derivative term leads
to the inequality.

\begin{equation}
    \lambda \frac{d \lambda(x)}{dx} << \lambda.
\end{equation}

There are several others, but this one makes the most sense physically. It says that the wavelength change over one
de Broglie wavelength is much smaller than the de Broglie wavelength itself. This is the physical assumption of this
approximation.

For a potential barrier with sloping walls, if the energy of the incoming particle is less than the potential of the
barrier in one portion of it. The points where $E=V$ is called the turning points. The WKB method is only valid away from
those turning points. As we move close to the turning points, the derivative of the local de Broglie wavelength blows up.

## The Connection Formulas

To approximate the solution near the turning points, people developed the connection formulas. Its derivation is rather
involved. So I decide to not talk about it here.

The idea is that we use a straight line to approximate the potential near the turning points. Then find a "patching"
wave function to connect both sides. The connection formulas that joining the WKB solutions at a turning point $x_1$ is

\begin{equation}
    \psi(x) =
    \begin{cases}
    & \frac{2D}{\sqrt{p(x)}} sin[ \frac{1}{\hbar} \int_{x}^{x_1}p(x')dx' + \frac{\pi}{4} ], \quad \text{if } x<x_1; \\
    & \frac{D}{\sqrt{|p(x)|}} exp[ -\frac{1}{\hbar} \int_{x}^{x_1}p(x')dx' + \frac{\pi}{4} ], \quad \text{if } x>x_1. \\
    \end{cases}
\end{equation}

Using WKB with connection formulas, we can calculate the bound state energies and transmission coefficient without actually
solving the Schrödinger equation.

## Analytical Solution of a Finite Rectangular Potential Barrier

We derived the expressions for the transmission coefficient the conditions that the wave function and its first
derivative must be continues everywhere. We wrote down the boundary conditions

\begin{aligned}
\psi_1 (0) &= \psi_2 (0) \\
\psi_1' (0) &= \psi_2' (0) \\
\psi_2 (L) &= \psi_3 (L) \\
\psi_2' (L) &= \psi_3' (L).
\end{aligned}

Then we solved for the transmission coefficient $T$ for three cases: $E<V_0$, $E>V_0$, and $E=V_0$. We have done one
of those three cases in homework 4. The other two are similar.

### Case 1: $E<V_0$
We have
\begin{equation}
T = \bigg[ 1 + \frac{V_0^2 sinh^2(k L)}{4E(V_0-E)} \bigg]^{-1},
\end{equation}

where $k=\sqrt{2m(V_0 - E)} / \hbar$. This is the quantum tunneling case. Classically, particles with energy less than
the potential barrier have zero chance going through. But in a quantum system, the particle has some probability
tunneling through. This probability is exponentially decayed as the width of of the barrier increases.

### Case 2: $E>V_0$
\begin{equation}
T = \bigg[ 1 + \frac{V_0^2 sin^2(k L)}{4E(E-V_0)} \bigg]^{-1},
\end{equation}

where $k=\sqrt{2m(E - V_0)} / \hbar$. Classically, particles with energy greater than
the potential barrier would go through certainly. But the quantum scenario, the transmission coefficient actually
oscillates as a function of the width and potential of the barrier and the energy of the particle.

### Case 3: $E=V_0$
\begin{equation}
T = \bigg[ 1 + mL^2 V_0 / 2\hbar^2 \bigg]^{-1}.
\end{equation}

In the case $E=V_0$ the transmission coefficient depends on the width and potential of the barrier.

### Plot of the Transmission Coefficient T

In the following graph, I plotted the three cases separately and combined them to form a single graph. We can see the
exponential decay of the transmission coefficient as the width of the barrier increases. And in the region that $E>V_0$,
$T$ oscillates and trends to 1.

```python
import numpy as np
import scipy.constants as sc
import matplotlib.pyplot as plt

# suppress divide by zero warnings
import warnings; warnings.simplefilter('ignore')
```

```python
# constants are imported from the scipy physical constants library
hbar = sc.physical_constants['reduced Planck constant in eV s'][0]
m_e = sc.physical_constants['electron mass energy equivalent'][0]
```

```python
T_l = lambda e_, l_, v_: (1 + (v_**2 * np.sinh(l_*np.sqrt(2*m_e*(v_-e_))/hbar)**2)/(4*e_*(v_-e_)))**-1
T_eq = lambda e_, l_, v_: (1 + m_e*l_**2 * v_ / (2*hbar**2))**-1
T_g = lambda e_, l_, v_: (1 + (v_**2 * np.sin(l_*np.sqrt(2*m_e*(e_-v_))/hbar)**2)/(4*e_*(e_-v_)))**-1
```

```python
n = 100
l = 10E-9 # m
vs = [0.05, 0.1, 0.5, 1] # eV

plt.figure()
for v in vs:
    es_lv = np.linspace(0, v*((n-1)/n), n-1)
    es_gv = np.linspace(v*((n+1)/n), v*3, n*2)
    es = np.append(es_lv, v)
    es = np.concatenate((es, es_gv))
    leq = np.append(T_l(es_lv, l, v), T_eq(v, l, v))
    Ts = np.concatenate((leq, T_g(es_gv, l, v)))
    es_v = es / v
    plt.plot(es_v, Ts, label=f'v={v} eV')

plt.vlines(1, 0, 1, linestyles='dashed')
plt.xlabel('E / V_0')
plt.ylabel('T')
plt.legend()
plt.show()

```

## The WKB approximation of a Finite Rectangular Potential Barrier

### WKB inside of the Barrier
Since the wave function of an electron passing through a finite rectangular potential barrier is known. We can compare
the exact analytical solution with the one getting from the WKB approximation. The first thing I tried is to solve for the
transmission coefficient using the WKB wave function inside the barrier.

\begin{equation}
    \psi_2(x) = \frac{C}{\sqrt{p}} e^{\frac{p}{\hbar}x} + \frac{D}{\sqrt{p}} e^{-\frac{p}{\hbar}x}, \\
\end{equation}

where $p=\sqrt{2m(V_0 - E)}$. The phase part is the result of the integral $\int_{0}^{L}|p(x')|dx'$. Then we use the
boundary conditions to solve for the transmission coefficient T.

\begin{align}
    T = \bigg\vert \frac{F}{A} \bigg\vert^2
      = \bigg\vert \bigg[
      \frac{1}{2} e^{ikL-\frac{pL}{\hbar}} \big(1+\frac{p}{ik\hbar} \big) \big(1+\frac{ik\hbar}{p} \big)
         + \frac{1}{2} e^{ikL+\frac{pL}{\hbar}} \big(1-\frac{p}{ik\hbar} \big) \big(1-\frac{ik\hbar}{p} \big)
         \bigg]^{-1} \bigg\vert^2.
\end{align}

This equation could be farther simplified. But since we are plotting it with a program. This form works perfectly fine.

For a rectangular potential barrier with a flat top. The second derivative term of the amplitude of the wave equation is
zero. So dropping it in the approximation would not affect the result. Any of its deviation from the exact analytical
solution should be because WKB doesn't work very well near the boundaries.

### WKB for Potential with Sloping Walls

The way of getting the exponential decay part of the transmission coefficient was presented in both homework 4 and
the Griffiths book. It's give by

\begin{equation}
    T \approx e^{-2\gamma}, \\
\end{equation}
where $\gamma = \frac{1}{\hbar}\int_0^L |p(x)|dx$. The problem 8.10 of the Griffiths book also talks about the WKB solution of a scattering from
barrier with sloping walls. It uses the connection formula. The expression is

\begin{equation}
    T = \frac{e^{-2\gamma}}{ [1-(e^{-\gamma /2})^2]^2 }, \\
\end{equation}
where $\gamma = \frac{1}{\hbar}\int_0^L |p(x)|dx$.
I'm not sure if it applies to a finite rectangular potential barrier with vertical walls. But it would be interesting to
see how it looks.

```python
gamma = lambda e_, l_, v_: np.exp(l_/hbar*np.sqrt(2*m_e*(v_-e_)))
T_wkb = lambda e_, l_, v_: np.exp(-2*gamma(e_, l_, v_)) / np.power(1 - np.power(np.exp(-gamma(e_, l_, v_)/2), 2), 2)

p = lambda e_, v_: np.sqrt(2*m_e*(v_ - e_))
k = lambda e_: np.sqrt(2*m_e*e_)/hbar
T_test = lambda e_, v_, l_: np.power(np.absolute( ((1+p(e_, v_)/(k(e_)*hbar*1j))*(0.5*np.exp(k(e_)*l_*1j - p(e_, v_)*l_/hbar)*(1+(k(e_)*hbar*1j)/p(e_, v_))) + (1-p(e_, v_)/(k(e_)*hbar*1j))*(0.5*np.exp(k(e_)*l_*1j + p(e_, v_)*l_/hbar)*(1-(k(e_)*hbar*1j)/p(e_, v_))))**-1 ) , 2)
```

```python
l = 10E-9
v = 1
es_lv = np.linspace(0, v*((n-1)/n), n-1)
es_vv = es_lv / v
plt.plot(es_vv, T_test(es_lv, v, l), label='WKB')
plt.plot(es_vv, T_wkb(es_lv, l, v), label='WKB-connection')
plt.plot(es_vv, T_l(es_lv, l, v), label='Analytical')
plt.xlabel('E / V_0')
plt.ylabel('T')
plt.legend()
plt.show()
```

The results agree with the approximation. As the energy of the particle approaches the potential of the barrier, the
transmission coefficient starts to deviate from the exact solution.

```python
l = 5E-9
v = 1
es_lv = np.linspace(0, v*((n-1)/n), n-1)
es_vv = es_lv / v
plt.plot(es_vv, T_test(es_lv, v, l), label='WKB')
plt.plot(es_vv, T_wkb(es_lv, l, v), label='WKB-connection')
plt.plot(es_vv, T_l(es_lv, l, v), label='Analytical')
plt.xlabel('E / V_0')
plt.ylabel('T')
plt.legend()
plt.show()
```

Another assumption of the WKB approximation is that the size of the aperture is much larger than the de Broglie wavelength of the incoming particle. So I reduced the width of the potential barrier by half and tested how it affects the
transmission coefficient T. For the exact solution, T increases overall. It agrees with our intuition since a particle
is more likely to tunnel through a narrower potential barrier. But in this case, the WKB solution deviates even more
from the exact solution.

## References

- Griffiths, D. J. (2004). Introduction to Quantum Mechanics (2nd Edition). Pearson Prentice Hall. ISBN: 0131118927
- Barton Zwiebach. 8.06 Quantum Physics III. Spring 2018. Massachusetts Institute of Technology: MIT OpenCourseWare,
https://ocw.mit.edu. License: Creative Commons BY-NC-SA.


# Limits
You can use algebraic methods to calculate the rate of change over a function interval by joining two points on the function with a secant line and measuring its slope. For example, a function might return the distance travelled by a cyclist in a period of time, and you can use a secant line to measure the average velocity between two points in time. However, this doesn't tell you the cyclist's vecolcity at any single point in time - just the average speed over an interval.

To find the cyclist's velocity at a specific point in time, you need the ability to find the slope of a curve at a given point. *Differential Calculus* enables us to do through the use of *derivatives*. We can use derivatives to find the slope at a specific *x* value by calculating a delta for *x1* and *x2* values that are infinitesimally close together - so you can think of it as measuring the slope of a tiny straight line that comprises part of the curve.

## Introduction to Limits
However, before we can jump straight into derivatives, we need to examine another aspect of differential calculus - the *limit* of a function; which helps us measure how a function's value changes as the *x2* value approaches *x1*

To better understand limits, let's take a closer look at our function, and note that although we graph the function as a line, it is in fact made up of individual points. Run the following cell to show the points that we've plotted for integer values of ***x*** - the line is created by interpolating the points in between:

```python
%matplotlib inline

# Here's the function
def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Get the corresponding y values from the function
y = [f(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='lightgrey', marker='o', markeredgecolor='green', markerfacecolor='red')

plt.show()
```

We know from the function that the ***f(x)*** values are calculated by squaring the ***x*** value and adding ***x***, so we can easily calculate points in between and show them - run the following code to see this:

```python
%matplotlib inline

# Here's the function
def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0,5))
x.append(4.25)
x.append(4.5)
x.append(4.75)
x.append(5)
x.append(5.25)
x.append(5.5)
x.append(5.75)
x = x + list(range(6,11))

# Get the corresponding y values from the function
y = [f(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='lightgrey', marker='o', markeredgecolor='green', markerfacecolor='green')

plt.show()
```

Now we can see more clearly that this function line is formed of a continuous series of points, so theoretically for any given value of ***x*** there is a point on the line, and there is an adjacent point on either side with a value that is as close to ***x*** as possible, but not actually ***x***.

Run the following code to visualize a specific point for *x = 5*, and try to identify the closest point either side of it:

```python
%matplotlib inline

# Here's the function
def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0,5))
x.append(4.25)
x.append(4.5)
x.append(4.75)
x.append(5)
x.append(5.25)
x.append(5.5)
x.append(5.75)
x = x + list(range(6,11))

# Get the corresponding y values from the function
y = [f(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='lightgrey', marker='o', markeredgecolor='green', markerfacecolor='green')

zx = 5
zy = f(zx)
plt.plot(zx, zy, color='red', marker='o', markersize=10)
plt.annotate('x=' + str(zx),(zx, zy), xytext=(zx - 0.5, zy + 5))

# Plot f(x) when x = 5.1
posx = 5.25
posy = f(posx)
plt.plot(posx, posy, color='blue', marker='<', markersize=10)
plt.annotate('x=' + str(posx),(posx, posy), xytext=(posx + 0.5, posy - 1))

# Plot f(x) when x = 4.9
negx = 4.75
negy = f(negx)
plt.plot(negx, negy, color='orange', marker='>', markersize=10)
plt.annotate('x=' + str(negx),(negx, negy), xytext=(negx - 1.5, negy - 1))

plt.show()
```

You can see the point where ***x*** is 5, and you can see that there are points shown on the graph that appear to be right next to this point (at *x=4.75* and *x=5.25*). However, if we zoomed in we'd see that there are still gaps that could be filled by other values of ***x*** that are even closer to 5; for example, 4.9 and 5.1, or 4.999 and 5.001. If we could zoom infinitely close to the line we'd see that no matter how close a value you use (for example, 4.999999999999), there is always a value that's fractionally closer (for example, 4.9999999999999).

So what we can say is that there is a hypothetical number that's as close as possible to our desired value of *x* without actually being *x*, but we can't express it as a real number. Instead, we express its symbolically as a *limit*, like this:

\begin{equation}\lim_{x \to 5} f(x)\end{equation}

This is interpreted as *the limit of function f(x) as *x* approaches 5*.

##  Limits and Continuity
The function ***f(x)*** is *continuous* for all real numbered values of ***x***. Put simply, this means that you can draw the line created by the function without lifting your pen (we'll look at a more formal definition later in this course).

However, this isn't necessarily true of all functions. Consider function ***g(x)*** below:

\begin{equation}g(x) = -(\frac{12}{2x})^{2}\end{equation}

This function is a little more complex than the previous one, but the key thing to note is that it requires a division by *2x*. Now, ask yourself; what would happen if you applied this function to an *x* value of **0**?

Well, 2 • 2 is 0, and anything divided by 0 is *undefined*. So the *domain* of this function does not include 0; in other words, the function is defined when *x* is any real number such that *x is not equal to 0*. The function should therefore be written like this:

\begin{equation}g(x) = -(\frac{12}{2x})^{2},\;\; x \ne 0\end{equation}

So why is this important? Let's investigate by running the following Python code to define the function and plot it for a set of arbitrary of values:

```python
%matplotlib inline

# Define function g
def g(x):
    if x != 0:
        return -(12/(2*x))**2

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-20, 21)

# Get the corresponding y values from the function
y = [g(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='green')

plt.show()
```

Look closely at the plot, and note the gap the line where *x* = 0. This indicates that the function is not defined here.The *domain* of the function (it's set of possible input values) not include 0, and it's *range* (the set of possible output values) does not include a value for x=0.

This is a *non-continuous* function - in other words, it includes at least one gap when plotted (so you couldn't plot it by hand without lifting your pen). Specifically, the function is non-continuous at x=0.

By convention, when a non-continuous function is plotted, the points that form a continuous line (or *interval*) are shown as a line, and the end of each line where there is a discontinuity is shown as a circle, which is filled if the value at that point is included in the line and empty if the value is not included in the line.

In this case, the function produces two intervals with a gap between them where the function is not defined, so we can show the discontinuous point as an unfilled circle - run the following code to visualize this with Python:

```python
%matplotlib inline

# Define function g
def g(x):
    if x != 0:
        return -(12/(2*x))**2

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-20, 21)

# Get the corresponding y values from the function
y = [g(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='green')

# plot a circle at the gap (or close enough anyway!)
xy = (0,g(1))
plt.annotate('O',xy, xytext=(-0.7, -37),fontsize=14,color='green')

plt.show()
```

There are a number of reasons a function might be non-continuous. For example, consider the following function:

\begin{equation}h(x) = 2\sqrt{x},\;\; x \ge 0\end{equation}

Applying this function to a non-negative ***x*** value returns a valid output; but for any value where ***x*** is negative, the output is undefined, because the square root of a negative value is not a real number.

Here's the Python to plot function ***h***:

```python
%matplotlib inline

def h(x):
    if x >= 0:
        import numpy as np
        return 2 * np.sqrt(x)

# Plot output from function h
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-20, 21)

# Get the corresponding y values from the function
y = [h(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='green')

# plot a circle close enough to the h(-x) limit for our purposes!
plt.plot(0, h(0), color='green', marker='o', markerfacecolor='green', markersize=10)

plt.show()
```

Now, suppose we have a function like this:

\begin{equation}
k(x) = \begin{cases}
  x + 20, & \text{if } x \le 0, \\
  x - 100, & \text{otherwise }
\end{cases}
\end{equation}

In this case, the function's domain includes all real numbers, but its output is still non-continuous because of the way different values are returned depending on the value of *x*. The *range* of possible outputs for *k(x ≤ 0)* is ≤ 20, and the range of output values for *k(x > 0)* is x > -100.

Let's use Python to plot function ***k***:

```python
%matplotlib inline

def k(x):
    import numpy as np
    if x <= 0:
        return x + 20
    else:
        return x - 100

# Plot output from function h
from matplotlib import pyplot as plt

# Create an array of x values for each non-contonuous interval
x1 = range(-20, 1)
x2 = range(1, 20)

# Get the corresponding y values from the function
y1 = [k(i) for i in x1]
y2 = [k(i) for i in x2]

# Set up the graph
plt.xlabel('x')
plt.ylabel('k(x)')
plt.grid()

# Plot x against k(x)
plt.plot(x1,y1, color='green')
plt.plot(x2,y2, color='green')

# plot a circle at the interval ends
plt.plot(0, k(0), color='green', marker='o', markerfacecolor='green', markersize=10)
plt.plot(0, k(0.0001), color='green', marker='o', markerfacecolor='w', markersize=10)

plt.show()
```

## Finding Limits of Functions Graphically
So the question arises, how do we find a value for the limit of a function at a specific point?

Let's explore this function, ***a***:

\begin{equation}a(x) = x^{2} + 1\end{equation}

We can start by plotting it:

```python
%matplotlib inline

# Define function a
def a(x):
    return x**2 + 1

# Plot output from function a
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [a(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('a(x)')
plt.grid()

# Plot x against a(x)
plt.plot(x,y, color='purple')

plt.show()
```

Note that this function is continuous at all points, there are no gaps in its range. However, the range of the function is *{a(x) ≥ 1}* (in other words, all real numbers that are greater than or equal to 1). For negative values of ***x***, the function appears to return ever-decreasing values as ***x*** gets closer to 0, and for positive values of ***x***, the function appears to return ever-increasing values as ***x*** gets further from 0; but it never returns 0.

Let's plot the function for an ***x*** value of 0 and find out what the ***a(0)*** value is returned:

```python
%matplotlib inline

# Define function a
def a(x):
    return x**2 + 1

# Plot output from function a
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [a(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('a(x)')
plt.grid()

# Plot x against a(x)
plt.plot(x,y, color='purple')

# Plot a(x) when x = 0
zx = 0
zy = a(zx)
plt.plot(zx, zy, color='red', marker='o', markersize=10)
plt.annotate(str(zy),(zx, zy), xytext=(zx, zy + 5))

plt.show()
```

OK, so ***a(0)*** returns **1**.

What happens if we use ***x*** values that are very slightly higher or lower than 0?

```python
%matplotlib inline

# Define function a
def a(x):
    return x**2 + 1

# Plot output from function a
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [a(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('a(x)')
plt.grid()

# Plot x against a(x)
plt.plot(x,y, color='purple')

# Plot a(x) when x = 0.1
posx = 0.1
posy = a(posx)
plt.plot(posx, posy, color='blue', marker='<', markersize=10)
plt.annotate(str(posy),(posx, posy), xytext=(posx + 1, posy))

# Plot a(x) when x = -0.1
negx = -0.1
negy = a(negx)
plt.plot(negx, negy, color='orange', marker='>', markersize=10)
plt.annotate(str(negy),(negx, negy), xytext=(negx - 2, negy))

plt.show()
```

These ***x*** values return ***a(x)*** values that are just slightly above 1, and if we were to keep plotting numbers that are increasingly close to 0, for example 0.0000000001 or -0.0000000001, the function would still return a value that is just slightly greater than 1. The limit of function *a(x)* as *x* approaches 0, is 1; and the notation to indicate this is:

\begin{equation}\lim_{x \to 0} a(x) = 1 \end{equation}

This reflects a more formal definition of function continuity. Previously, we stated that a function is continuous at a point if you can draw it at that point without lifting your pen. The more mathematical definition is that a function is continuous at a point if the limit of the function as it approaches that point from both directions is equal to the function's value at that point. In this case, as we approach x = 0 from both sides, the limit is 1; and the value of *a(0)* is also 1; so the function is continuous at x = 0.

### Limits at Non-Continuous Points
Let's try another function, which we'll call ***b***:

\begin{equation}b(x) = -2x^{2} \cdot \frac{1}{x},\;\;x\ne0\end{equation}

Note that this function has a domain that includes all real number values of *x* such that *x* does not equal 0. In other words, the function will return a valid output for any number other than 0.

Let's create it and plot it with Python:

```python
%matplotlib inline

# Define function b
def b(x):
    if x != 0:
        return (-2*x**2) * 1/x

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [b(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('b(x)')
plt.grid()

# Plot x against b(x)
plt.plot(x,y, color='purple')

plt.show()
```

The output from this function contains a gap in the line where x = 0. It seems that not only does the *domain* of the function (the values that can be passed in as *x*) exclude 0; but the *range* of the function (the set of values that can be returned from it) also excludes 0.

We can't evaluate the function for an *x* value of 0, but we can see what it returns for a value that is just very slightly less than 0:

```python
%matplotlib inline

# Define function b
def b(x):
    if x != 0:
        return (-2*x**2) * 1/x

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [b(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('b(x)')
plt.grid()

# Plot x against b(x)
plt.plot(x,y, color='purple')

# Plot b(x) for x = -0.1
negx = -0.1
negy = b(negx)
plt.plot(negx, negy, color='orange', marker='>', markersize=10)
plt.annotate(str(negy),(negx, negy), xytext=(negx + 1, negy))

plt.show()
```

We can even try a negative *x* value that's a little closer to 0.

```python
%matplotlib inline

# Define function b
def b(x):
    if x != 0:
        return (-2*x**2) * 1/x

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [b(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('b(x)')
plt.grid()

# Plot x against b(x)
plt.plot(x,y, color='purple')

# Plot b(x) for x = -0.0001
negx = -0.0001
negy = b(negx)
plt.plot(negx, negy, color='orange', marker='>', markersize=10)
plt.annotate(str(negy),(negx, negy), xytext=(negx + 1, negy))

plt.show()
```

So as the value of *x* gets closer to 0 from the left (negative), the value of *b(x)* is decreasing towards 0. We can show this with the following notation:

\begin{equation}\lim_{x \to 0^{-}} b(x) = 0 \end{equation}

Note that the arrow points to 0- (with a minus sign) to indicate that we're describing the limit as we approach 0 from the negative side.

So what about the positive side?

Let's see what the function value is when *x* is 0.1:

```python
%matplotlib inline

# Define function b
def b(x):
    if x != 0:
        return (-2*x**2) * 1/x

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [b(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('b(x)')
plt.grid()

# Plot x against b(x)
plt.plot(x,y, color='purple')

# Plot b(x) for x = 0.1
posx = 0.1
posy = b(posx)
plt.plot(posx, posy, color='blue', marker='<', markersize=10)
plt.annotate(str(posy),(posx, posy), xytext=(posx + 1, posy))

plt.show()
```

What happens if we decrease the value of *x* so that it's even closer to 0?

```python
%matplotlib inline

# Define function b
def b(x):
    if x != 0:
        return (-2*x**2) * 1/x

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the function
y = [b(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('b(x)')
plt.grid()

# Plot x against b(x)
plt.plot(x,y, color='purple')

# Plot b(x) for x = 0.0001
posx = 0.0001
posy = b(posx)
plt.plot(posx, posy, color='blue', marker='<', markersize=10)
plt.annotate(str(posy),(posx, posy), xytext=(posx + 1, posy))

plt.show()
```

As with the negative side, as *x* approaches 0 from the positive side, the value of *b(x)* gets closer to 0; and we can show that like this:

\begin{equation}\lim_{x \to 0^{+}} b(x) = 0 \end{equation}

Now, even although the function is not defined at x = 0; since the limit as we approach x = 0 from the negative side is 0, and the limit when we approach x = 0 from the positive side is also 0; we can say that the overall, or *two-sided* limit for the function at x = 0 is 0:

\begin{equation}\lim_{x \to 0} b(x) = 0 \end{equation}

So can we therefore just ignore the gap and say that the function is *continuous* at x = 0? Well, recall that the formal definition for continuity is that to be continuous at a point, the function's limit as we approach the point in both directions must be equal to the function's value at that point. In this case, the two-sided limit as we approach x = 0 is 0, but *b(0)* is not defined; so the function is ***non-continuous*** at x = 0.

### One-Sided Limits
Let's take a look at a different function. We'll call this one ***c***:

\begin{equation}
c(x) = \begin{cases}
  x + 20, & \text{if } x \le 0, \\
  x - 100, & \text{otherwise }
\end{cases}
\end{equation}

In this case, the function's domain includes all real numbers, but its range is still non-continuous because of the way different values are returned depending on the value of *x*. The range of possible outputs for *c(x ≤ 0)* is ≤ 20, and the range of output values for *c(x > 0)* is x ≥ -100.

Let's use Python to plot function ***c*** with some values for *c(x)* marked on the line

```python
%matplotlib inline

def c(x):
    import numpy as np
    if x <= 0:
        return x + 20
    else:
        return x - 100

# Plot output from function h
from matplotlib import pyplot as plt

# Create arrays of x values
x1 = range(-20, 6)
x2 = range(6, 21)

# Get the corresponding y values from the function
y1 = [c(i) for i in x1]
y2 = [c(i) for i in x2]

# Set up the graph
plt.xlabel('x')
plt.ylabel('c(x)')
plt.grid()

# Plot x against c(x)
plt.plot(x1,y1, color='purple')
plt.plot(x2,y2, color='purple')

# plot a circle close enough to the c limits for our purposes!
plt.plot(5, c(5), color='purple', marker='o', markerfacecolor='purple', markersize=10)
plt.plot(5, c(5.001), color='purple', marker='o', markerfacecolor='w', markersize=10)

# plot some points from the +ve direction
posx = [20, 15, 10, 6]
posy = [c(i) for i in posx]
plt.scatter(posx, posy, color='blue', marker='<', s=70)
for p in posx:
    plt.annotate(str(c(p)),(p, c(p)),xytext=(p, c(p) + 5))

# plot some points from the -ve direction
negx = [-15, -10, -5, 0, 4]
negy = [c(i) for i in negx]
plt.scatter(negx, negy, color='orange', marker='>', s=70)
for n in negx:
    plt.annotate(str(c(n)),(n, c(n)),xytext=(n, c(n) + 5))

plt.show()
```

The plot of the function shows a line in which the *c(x)* value increases towards 25 as *x* approaches 5 from the negative side:

\begin{equation}\lim_{x \to 5^{-}} c(x) = 25 \end{equation}

However, the *c(x)* value decreases towards -95 as *x* approaches 5 from the positive side:

\begin{equation}\lim_{x \to 5^{+}} c(x) = -95 \end{equation}

So what can we say about the two-sided limit of this function at x = 5?

The limit as we approach x = 5 from the negative side is *not* equal to the limit as we approach x = 5 from the positive side, so no two-sided limit exists for this function at that point:

\begin{equation}\lim_{x \to 5} \text{does not exist} \end{equation}

### Asymptotes and Infinity
OK, time to look at another function:

\begin{equation}d(x) = \frac{4}{x - 25},\;\; x \ne 25\end{equation}

```python
%matplotlib inline

# Define function d
def d(x):
    if x != 25:
        return 4 / (x - 25)

# Plot output from function d
from matplotlib import pyplot as plt

# Create an array of x values
x = list(range(-100, 24))
x.append(24.9) # Add some fractional x
x.append(25)   # values around
x.append(25.1) # 25 for finer-grain results
x = x + list(range(26, 101))
# Get the corresponding y values from the function
y = [d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('d(x)')
plt.grid()

# Plot x against d(x)
plt.plot(x,y, color='purple')

plt.show()
```

What's the limit of *d* as *x* approaches 25?

We can plot a few points to help us:

```python
%matplotlib inline

# Define function d
def d(x):
    if x != 25:
        return 4 / (x - 25)

# Plot output from function d
from matplotlib import pyplot as plt

# Create an array of x values
x = list(range(-100, 24))
x.append(24.9) # Add some fractional x
x.append(25)   # values around
x.append(25.1) # 25 for finer-grain results
x = x + list(range(26, 101))
# Get the corresponding y values from the function
y = [d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('d(x)')
plt.grid()

# Plot x against d(x)
plt.plot(x,y, color='purple')

# plot some points from the +ve direction
posx = [75, 50, 30, 25.5, 25.2, 25.1]
posy = [d(i) for i in posx]
plt.scatter(posx, posy, color='blue', marker='<')
for p in posx:
    plt.annotate(str(d(p)),(p, d(p)))

# plot some points from the -ve direction
negx = [-55, 0, 23, 24.5, 24.8, 24.9]
negy = [d(i) for i in negx]
plt.scatter(negx, negy, color='orange', marker='>')
for n in negx:
    plt.annotate(str(d(n)),(n, d(n)))

plt.show()
```

From these plotted values, we can see that as *x* approaches 25 from the negative side, *d(x)* is decreasing, and as *x* approaches 25 from the positive side, *d(x)* is increasing. As *x* gets closer to 25, *d(x)* increases or decreases more significantly.

If we were to plot every fractional value of *d(x)* for *x* values between 24.9 and 25, we'd see a line that decreases indefintely, getting closer and closer to the x = 25 vertical line, but never actually reaching it. Similarly, plotting every *x* value between 25 and 25.1 would result in a line going up indefinitely, but always staying to the right of the vertical x = 25 line.

The x = 25 line in this case is an *asymptote* - a line to which a curve moves ever closer but never actually reaches. The positive limit for x = 25 in this case in not a real numbered value, but *infinity*:

\begin{equation}\lim_{x \to 25^{+}} d(x) = \infty \end{equation}

Conversely, the negative limit for x = 25 is negative infinity:

\begin{equation}\lim_{x \to 25^{-}} d(x) = -\infty \end{equation}

## Finding Limits Numerically Using a Table
Up to now, we've estimated limits for a point graphically by examining a graph of a function. You can also approximate limits by creating a table of x values and the corresponding function values either side of the point for which you want to find the limits.

For example, let's return to our ***a*** function:

\begin{equation}a(x) = x^{2} + 1\end{equation}

If we want to find the limits as x is approaching 0, we can apply the function to some values either side of 0 and view them as a table. Here's some Python code to do that:

```python
# Define function a
def a(x):
    return x**2 + 1

import pandas as pd

# Create a dataframe with an x column containing values either side of 0
df = pd.DataFrame ({'x': [-1, -0.5, -0.2, -0.1, -0.01, 0, 0.01, 0.1, 0.2, 0.5, 1]})

# Add an a(x) column by applying the function to x
df['a(x)'] = a(df['x'])

#Display the dataframe
df
```

      x
      a(x)

      0
      -1.00
      2.0000

      1
      -0.50
      1.2500

      2
      -0.20
      1.0400

      3
      -0.10
      1.0100

      4
      -0.01
      1.0001

      5
      0.00
      1.0000

      6
      0.01
      1.0001

      7
      0.10
      1.0100

      8
      0.20
      1.0400

      9
      0.50
      1.2500

      10
      1.00
      2.0000

Looking at the output, you can see that the function values are getting closer to 1 as x approaches 0 from both sides, so:

\begin{equation}\lim_{x \to 0} a(x) = 1 \end{equation}

Additionally, you can see that the actual value of the function when x = 0 is also 1, so:

\begin{equation}\lim_{x \to 0} a(x) = a(0) \end{equation}

Which according to our earlier definition, means that the function is continuous at 0.

However, you should be careful not to assume that the limit when x is approaching 0 will always be the same as the value when x = 0; even when the function is defined for x = 0.

For example, consider the following function:

\begin{equation}
e(x) = \begin{cases}
  5, & \text{if } x = 0, \\
  1 + x^{2}, & \text{otherwise }
\end{cases}
\end{equation}

Let's see what the function returns for *x* values either side of 0 in a table:

```python
# Define function e
def e(x):
    if x == 0:
        return 5
    else:
        return 1 + x**2

import pandas as pd
# Create a dataframe with an x column containing values either side of 0
x= [-1, -0.5, -0.2, -0.1, -0.01, 0, 0.01, 0.1, 0.2, 0.5, 1]
y =[e(i) for i in x]
df = pd.DataFrame ({' x':x, 'e(x)': y })
df
```

      x
      e(x)

      0
      -1.00
      2.0000

      1
      -0.50
      1.2500

      2
      -0.20
      1.0400

      3
      -0.10
      1.0100

      4
      -0.01
      1.0001

      5
      0.00
      5.0000

      6
      0.01
      1.0001

      7
      0.10
      1.0100

      8
      0.20
      1.0400

      9
      0.50
      1.2500

      10
      1.00
      2.0000

As before, you can see that as the *x* values approach 0 from both sides, the value of the function gets closer to 1, so:

\begin{equation}\lim_{x \to 0} e(x) = 1 \end{equation}

However the actual value of the function when x = 0 is 5, not 1; so:

\begin{equation}\lim_{x \to 0} e(x) \ne e(0) \end{equation}

Which according to our earlier definition, means that the function is non-continuous at 0.

Run the following cell to see what this looks like as a graph:

```python
%matplotlib inline

# Define function e
def e(x):
    if x == 0:
        return 5
    else:
        return 1 + x**2

from matplotlib import pyplot as plt

x= [-1, -0.5, -0.2, -0.1, -0.01, 0.01, 0.1, 0.2, 0.5, 1]
y =[e(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('e(x)')
plt.grid()

# Plot x against e(x)
plt.plot(x, y, color='purple')
# (we're cheating slightly - we'll manually plot the discontinous point...)
plt.scatter(0, e(0), color='purple')
# (... and overplot the gap)
plt.plot(0, 1, color='purple', marker='o', markerfacecolor='w', markersize=10)
plt.show()
```

## Determining Limits Analytically
We've seen how to estimate limits visually on a graph, and by creating a table of *x* and *f(x)* values either side of a point. There are also some mathematical techniques we can use to calculate limits.

### Direct Substitution
Recall that our definition for a function to be continuous at a point is that the two-directional limit must exist and that it must be equal to the function value at that point. It therefore follows, that if we know that a function is continuous at a given point, we can determine the limit simply by evaluating the function for that point.

For example, let's consider the following function ***g***:

\begin{equation}g(x) = \frac{x^{2} - 1}{x - 1}, x \ne 1\end{equation}

Run the following code to see this function as a graph:

```python
%matplotlib inline

# Define function f
def g(x):
    if x != 1:
        return (x**2 - 1) / (x - 1)

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x= range(-20, 21)
y =[g(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

plt.show()
```

Now, suppose we need to find the limit of ***g(x)*** as ***x*** approaches **4**. We can try to find this by simply substituting 4 for the *x* values in the function:

\begin{equation}g(4) = \frac{4^{2} - 1}{4 - 1}\end{equation}

This simplifies to:

\begin{equation}g(4) = \frac{15}{3}\end{equation}

So:

\begin{equation}\lim_{x \to 4} g(x) = 5\end{equation}

Let's take a look:

```python
%matplotlib inline

# Define function g
def g(x):
    if x != 1:
        return (x**2 - 1) / (x - 1)

# Plot output from function f
from matplotlib import pyplot as plt

# Create an array of x values
x= range(-20, 21)
y =[g(i) for i in x]

# Set the x point we're interested in
zx = 4

plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

# Plot g(x) when x = 0
zy = g(zx)
plt.plot(zx, zy, color='red', marker='o', markersize=10)
plt.annotate(str(zy),(zx, zy), xytext=(zx - 2, zy + 1))

plt.show()

print ('Limit as x -> ' + str(zx) + ' = ' + str(zy))
```

### Factorization
OK, now let's try to find the limit of ***g(x)*** as ***x*** approaches **1**.

We know from the function definition that the function is not defined at x = 1, but we're not trying to find the *value* of ***g(x)*** when x *equals* 1; we're trying to find the *limit* of ***g(x)*** as x *approaches* 1.

The direct substitution approach won't work in this case:

\begin{equation}g(1) = \frac{1^{2} - 1}{1 - 1}\end{equation}

Simplifies to:

\begin{equation}g(1) = \frac{0}{0}\end{equation}

Anything divided by 0 is undefined; so all we've done is to confirm that the function is not defined at this point. You might be tempted to assume that this means the limit does not exist, but 0/0 is a special case; it's what's known as the *indeterminate form*; and there may be a way to solve this problem another way.

We can factor the *x2 - 1* numerator in the definition of ***g*** as as *(x - 1)(x + 1)*, so the limit equation can we rewritten like this:

\begin{equation}\lim_{x \to a} g(x) = \frac{(x-1)(x+1)}{x - 1}\end{equation}

The ***x - 1*** in the numerator and the ***x - 1*** in the denominator cancel each other out:

\begin{equation}\lim_{x \to a} g(x)= x+1\end{equation}

So we can now use substitution for *x = 1* to calculate the limit as *1 + 1*:

\begin{equation}\lim_{x \to 1} g(x) = 2\end{equation}

Let's see what that looks like:

```python
%matplotlib inline

# Define function g
def f(x):
    if x != 1:
        return (x**2 - 1) / (x - 1)

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x= range(-20, 21)
y =[g(i) for i in x]

# Set the x point we're interested in
zx = 1

# Calculate the limit of g(x) when x->zx using the factored equation
zy = zx + 1

plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

# Plot the limit of g(x)
zy = zx + 1
plt.plot(zx, zy, color='red', marker='o', markersize=10)
plt.annotate(str(zy),(zx, zy), xytext=(zx - 2, zy + 1))

plt.show()

print ('Limit as x -> ' + str(zx) + ' = ' + str(zy))
```

### Rationalization
Let's look at another function:

\begin{equation}h(x) = \frac{\sqrt{x} - 2}{x - 4}, x \ne 4 \text{ and } x \ge 0\end{equation}

Run the following cell to plot this function as a graph:

```python
%matplotlib inline

# Define function h
def h(x):
    import math
    if x >= 0 and x != 4:
        return (math.sqrt(x) - 2) / (x - 4)

# Plot output from function h
from matplotlib import pyplot as plt

# Create an array of x values
x= range(-20, 21)
y =[h(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='purple')

plt.show()
```

To find the limit of ***h(x)*** as ***x*** approaches **4**, we can't use the direct substitution method because the function is not defined at that point. However, we can take an alternative approach by multiplying both the numerator and denominator in the function by the *conjugate* of the numerator to *rationalize* the square root term (a conjugate is a binomial formed by reversing the sign of the second term of a binomial):

\begin{equation}\lim_{x \to a}h(x) = \frac{\sqrt{x} - 2}{x - 4}\cdot\frac{\sqrt{x} + 2}{\sqrt{x} + 2}\end{equation}

This simplifies to:

\begin{equation}\lim_{x \to a}h(x) = \frac{(\sqrt{x})^{2} - 2^{2}}{(x - 4)({\sqrt{x} + 2})}\end{equation}

The √x2 is x, and 22 is 4, so we can simplify the numerator as follows:

\begin{equation}\lim_{x \to a}h(x) = \frac{x - 4}{(x - 4)({\sqrt{x} + 2})}\end{equation}

Now we can cancel out the *x - 4* in both the numerator and denominator:

\begin{equation}\lim_{x \to a}h(x) = \frac{1}{{\sqrt{x} + 2}}\end{equation}

So for x approaching 4, this is:

\begin{equation}\lim_{x \to 4}h(x) = \frac{1}{{\sqrt{4} + 2}}\end{equation}

This simplifies to:

\begin{equation}\lim_{x \to 4}h(x) = \frac{1}{2 + 2}\end{equation}

Which is of course:

\begin{equation}\lim_{x \to 4}h(x) = \frac{1}{4}\end{equation}

So the limit of ***h(x)*** as ***x*** approaches **4** is 1/4 or 0.25.

Let's calculate and plot this with Python:

```python
%matplotlib inline

# Define function h
def h(x):
    import math
    if x >= 0 and x != 4:
        return (math.sqrt(x) - 2) / (x - 4)

# Plot output from function h
from matplotlib import pyplot as plt

# Create an array of x values
x= range(-20, 21)
y =[h(i) for i in x]

# Specify the point we're interested in
zx = 4

# Calculate the limit of f(x) when x->zx using factored equation
import math
zy = 1 / ((math.sqrt(zx)) + 2)

plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='purple')

# Plot the limit of h(x) when x->zx
plt.plot(zx, zy, color='red', marker='o', markersize=10)
plt.annotate(str(zy),(zx, zy), xytext=(zx + 2, zy))

plt.show()

print ('Limit as x -> ' + str(zx) + ' = ' + str(zy))
```

## Rules for Limit Operations
When you are working with functions and limits, you may want to combine limits using arithmetic operations. There are some intuitive rules for doing this.

Let's define two simple functions, ***j***:

\begin{equation}j(x) = 2x - 2\end{equation}

and ***l***:

\begin{equation}l(x) = -2x + 4\end{equation}

Run the cell below to plot these functions:

```python
%matplotlib inline

# Define function j
def j(x):
    return x * 2 - 2

# Define function l
def l(x):
    return -x * 2 + 4

# Plot output from functions j and l
from matplotlib import pyplot as plt

# Create an array of x values
x = range(-10, 11)

# Get the corresponding y values from the functions
jy = [j(i) for i in x]
ly = [l(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.xticks(range(-10,11, 1))
plt.ylabel('y')
plt.yticks(range(-30,30, 2))
plt.grid()

# Plot x against j(x)
plt.plot(x,jy, color='green', label='j(x)')

# Plot x against l(x)
plt.plot(x,ly, color='magenta', label='l(x)')

plt.legend()

plt.show()

```

### Addition of Limits

First, let's look at the rule for addition:

\begin{equation}\lim_{x \to a} (j(x) + l(x)) = \lim_{x \to a} j(x) + \lim_{x \to a} l(x)\end{equation}

What we're saying here, is that the limit of *j(x)* + *l(x)* as *x* approaches *a*, is the same as the limit of *j(x)* as *x* approaches *a* added to the limit of *l(x)* as *x* approaches *a*.

Looking at the graph for our functions ***j*** and ***l***, let's apply this rule to an *a* value of **8**.

By visually inspecting the graph, you can see that as *x* approaches 8 from either direction, *j(x)* gets closer to 14, so:

\begin{equation}\lim_{x \to 8} j(x) = 14\end{equation}

Similarly, as *x* approaches 8 from either direction, *l(x)* gets closer to -12, so:

\begin{equation}\lim_{x \to 8} l(x) = -12\end{equation}

So based on the addition rule:

\begin{equation}\lim_{x \to 8} (j(x) + l(x)) = 14 + -12 = 2\end{equation}

### Subtraction of Limits
Here's the rule for subtraction:

\begin{equation}\lim_{x \to a} (j(x) - l(x)) = \lim_{x \to a} j(x) - \lim_{x \to a} l(x)\end{equation}

As you've probably noticed, this is consistent with the rule of addition. Based on an *a* value of 8 (and the limits we identified for this *a* value above), we can apply this rule like this:

\begin{equation}\lim_{x \to 8} (j(x) - l(x)) = 14 - -12 = 26\end{equation}

### Multiplication of Limits
Here's the rule for multiplication:

\begin{equation}\lim_{x \to a} (j(x) \cdot l(x)) = \lim_{x \to a} j(x) \cdot \lim_{x \to a} l(x)\end{equation}

Again, you can apply this to the limits as x approached an *a* value of 8 we identified previously:

\begin{equation}\lim_{x \to 8} (j(x) \cdot l(x)) = 14 \cdot -12 = -168\end{equation}

This rule also applies to multipying a limit by a constant:

\begin{equation}\lim_{x \to a} c \cdot l(x) = c \cdot \lim_{x \to a} l(x)\end{equation}

So for an *a* value of 8 and a constant *c* value of 3, this equates to:

\begin{equation}\lim_{x \to 8} 3 \cdot l(x) = 3 \cdot -12 = -36\end{equation}

### Division of Limits
For division, assuming the limit of *l(x)* when x is approaching *a* is not 0:

\begin{equation}\lim_{x \to a} \frac{j(x)}{l(x)} = \frac{\lim_{x \to a} j(x)}{\lim_{x \to a} l(x)}\end{equation}

So, based on our limits for *j(x)* and *l(x*) when *x* approaches 8:

\begin{equation}\lim_{x \to 8} \frac{j(x)}{l(x)} = \frac{14}{-12}= \frac{7}{-6}\end{equation}

### Limit Exponentials and Roots

Assuming *n* is an integer:

\begin{equation}\lim_{x \to a} (j(x))^{n} = \Big(\lim_{x \to a} j(x)\Big)^{n}\end{equation}

So for example:

\begin{equation}\lim_{x \to 8} (j(x))^{2} = \Big(\lim_{x \to 8} j(x)\Big)^{2} = 14^{2} = 196\end{equation}

For roots, again assuming *n* is an integer:

\begin{equation}\lim_{x \to a} \sqrt[n]{j(x)} = \sqrt[n]{\lim_{x \to a} j(x)}\end{equation}

So:

\begin{equation}\lim_{x \to 8} \sqrt[2]{j(x)} = \sqrt[2]{\lim_{x \to 8} j(x)} = \sqrt[2]{14} \approx 3.74\end{equation}


# Bayesian Statistics

Bayes rule
$$
p(\theta| D) = \frac{p(D|\theta)p(\theta)}{p(D)}
$$
However, in the samples that we will soon see it is far more convenient to write the above as
$$
p(\theta| D) \propto p(D|\theta)p(\theta)
$$
since $p(D)$ is a constant **that does not depend on $\theta$**.

However, we Bayesian analysis is far more involved in solving the following integral:
$$
p(D^*|D) = \int p(D^*|\theta)p(\theta|D) d\theta
$$
We denote $D^*$ to be test data, $D$ to be the train set and $\theta$ to be a hidden parameter. This can often be thought of as doing model averaging if $p(\theta|D)$ is thought of as a weighting function.

## Coin Tossing Example

Suppose we have a biased coin where the true probability of landing heads is 0.6. We assume however, (for some odd reason that it is biased towards landing tails). A suitable prior for this would be a beta distribution which has the following functional form:
$$
p(\theta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}\theta^{\alpha - 1}(1-\theta)^{\beta - 1}
$$
and the distribution looks like (https://en.wikipedia.org/wiki/Beta_distribution):

Since we believe it is left skewed we will choose $\alpha < \beta$. Let $\alpha = 2$, $\beta = 3$ and hence, **this forms our prior distribution**. Assume that after 10 coin tosses we observe 7 heads and 3 tails. Each (individual) coin toss is distributed as a bernoulli distribution, conditional on $\theta$, the probability of achieving a heads. At this point a frequentist method (I don't mean this in a derogative fashion) would state that the probability of receiving a heads is 0.7.

The likelihood of the data can be states as follows:
$$ p(y_1, \cdots, y_{10}| \theta) = \prod_{i=1}^{10} p(y_i|\theta)$$ since each draw is independent **given $\theta$** where, $p(y_i|\theta) = \theta^{1(y_i=1)}(1-\theta)^{1(y_i=0)}$. Hence in our example, the likelihood $ p(y_1, \cdots, y_{10}| \theta) = \theta^7(1-\theta)^3$. **Keep in mind that $\theta$ is considered fixed as far as the likelihood is concerned**, hence it is a bernoullii distribution, **not a beta distribution**.

### The posterior
\begin{align}
p(\theta | y) \propto & \quad p(y|\theta) p(\theta) \\
\propto & \quad \theta^7(1-\theta)^3 \theta^2(1-\theta)^3 = \theta^{9}(1-\theta)^6
\end{align}
The normalising constant is irrelevant as we can see that the posterior $p(\theta | y)$ is a beta distribution simply by looking at its functional form. In fact the normalising constant (also known as the partition function) is simply,
$$\int_0^1 \theta^{9}(1-\theta)^6 d\theta$$ since all probability distributions integrate out to one. In this case this turns out to be $\frac{\Gamma(9)\Gamma(6)}{\Gamma(15)}$. What is far more important to recognise than the normalising constant is that $\theta \sim Be(9, 6)$.

### The probability of $p(y^*=1|y)$
\begin{align}
p(y^*=1|y) &= \int p(y^*=1|\theta)p(\theta|y) d\theta
\end{align}

A frequentist method would have chosen the Maximum-A-Posteriori (MAP) estimate and plugged into $\theta$ without integrating it out.
$$
\begin{align}
p(y^*=1|y) &= \int p(y^*=1|\theta)p(\theta|y) d\theta \\
&= \int \theta \frac{\Gamma(15)}{\Gamma(9)\Gamma(6)} \theta^{9}(1-\theta)^6 d\theta\\
&= \frac{\Gamma(15)}{\Gamma(9)\Gamma(6)} \int\theta^{10}(1-\theta)^6 d\theta\\
&= \frac{\Gamma(15)}{\Gamma(9)\Gamma(6)} \frac{\Gamma(10)\Gamma(6)}{\Gamma(16)} \int\frac{\Gamma(16)}{\Gamma(10)\Gamma(6)}\theta^{10}(1-\theta)^6 d\theta \\
&= \frac{\Gamma(15)}{\Gamma(9)\Gamma(6)} \frac{\Gamma(10)\Gamma(6)}{\Gamma(16)}  \quad 1 \\
&= \frac{9}{15}
\end{align}
$$
Note the rule $\Gamma(n) = (n-1)\Gamma(n-1)$ was used.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import pymc3 as pm

from scipy.special import gamma

%matplotlib inline
```

```python
alpha = 2
beta = 3
y_obs = np.zeros(10)
y_obs[:7] = 1
niter = 10000

with pm.Model() as model:
    θ = pm.Beta('θ', alpha=alpha, beta=beta)
    y = pm.Bernoulli('y', p=θ, observed=y_obs)

    trace = pm.sample(niter)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using ADVI...
    Average Loss = 8.1227:   1%|          | 1292/200000 [00:00<00:15, 12915.73it/s]
    Convergence archived at 2500
    Interrupted at 2,500 [1%]: Average Loss = 7.9508
    100%|██████████| 100500/100500 [00:37<00:00, 2691.01it/s]

```python
# norm_const = gamma(9)*gamma(6)/gamma(15)
theta = np.linspace(0,1,1000)
post_theta = (theta**9)*(1-theta)**6
C = np.sum(post_theta*(theta[1]-theta[0]))
post_theta = post_theta/C
```

```python
plt.hist(trace['θ'][1000:],100, normed='true')
plt.plot(theta, post_theta)
plt.show()
```

$$\int \theta p(\theta | y) d\theta \approx \sum_{\theta\sim p(\theta|y)} \theta$$

```python
np.mean(trace['θ'][1000:])
```

    0.59970305353900688

```python

```

```python

```

```python

```

```python

```

Bayesian_stats/Lesson 0 - Coin Toss.ipynb


# AdaBoost

Given a dataset $\left\{(x_{1}, y_{1}),...,({x_{N},y_{N}})\right\}$ where $y_{i} \in \left\{-1, 1\right\}$.

Initial sample distribution $D_{1} = (\frac{1}{N},...,\frac{1}{N}) = (w_{1,1},...,w_{1,N})$

AdaBoost train $G_{m}$ on $D_{m}$, it's misclassification error is $e_{m}$, update rule:

$$
\begin{equation}
w_{m+1, i} =
\begin{cases}
\frac{w_{m,i}}{Z_{m}}& \text{correct classified}\\
{\frac{1 - e_{m}}{e_{m}}}\frac{w_{m,i}}{Z_{m}}& \text{misclassified}
\end{cases}
\end{equation}
$$

where $Z_{m}$ is to make $D_{m+1}$ a valid distribution.

Finally:

$$f(x) = \sum_{m=1}^{M}\log\frac{1 - e_{m}}{e_{m}}G_{m}(x)$$

```python
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

```python
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),
                             n_estimators=100,
                             # SAMME is a multiclass version of AdaBoost
                             # use SAMME.R if predictor can estimate class probability
                             algorithm="SAMME.R",
                             # learning_date grows with n_estimator
                             learning_rate=0.5)
ada_clf.fit(X_train, y_train)
```

    AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=1),
                       learning_rate=0.5, n_estimators=100)

```python
from sklearn.metrics import accuracy_score, roc_auc_score

y_pred = ada_clf.predict(X_test)
y_prob = ada_clf.predict_proba(X_test)

accuracy_score(y_test, y_pred), roc_auc_score(y_test, y_prob[:, 1])
```

    (0.912, 0.9732325819672131)

## Boosting Tree

Boosting Tree model:

$$f(x) = \sum_{m=1}^{M}T(x; \theta_{m})$$

For binary classification, loss function ($\Leftrightarrow$ AdaBoost + DecisionTreeClassifier):

$$L(y, f(x)) = \exp(-yf(x))$$

For regression problem, loss function:

$$L(y, f(x)) = (y - f(x))^2$$

By forward stagewise algorithm:

$$\hat\theta_{m} = \underset{\theta_{m}}{argmin}\sum_{i=1}^{N}L(y_{i}; f_{m-1}(x_{i}) + T(x_{i};\theta_{m}))$$

$$L(y_{i}; f_{m-1}(x_{i}) + T(x_{i};\theta_{m})) = [y_{i} - f_{m-1}(x_{i}) - T(x_{i};\theta_{m})]^{2} = [r_{i} - T(x_{i};\theta_{m})]^{2}$$

So we just need to fit the residual.


# Fitting a line to data - a full tutorial

_Boris Leistedt, September 2017_

This notebook is available at [this location on Github](https://github.com/ixkael/Prob-tools/blob/master/notebooks/Fitting%20a%20line%20to%20data%20-%20a%20full%20tutorial.ipynb).

This notebook It assumes some basic knowledge about Bayesian inference and data analysis. It is accompanied with a set of slides.

```python
%matplotlib inline
%config IPython.matplotlib.backend = 'retina'
%config InlineBackend.figure_format = 'retina'

from IPython.display import HTML
```

Play with the code! Try and do the exercises.

Please interrupt me if you are lost or if you disagree with what I say.

All questions are welcome, especially the ones that you find "simple" (they are often very good, and not simple!)

If you haven't done it, install those packages using conda and/or pip:

``conda install numpy scipy pandas matplotlib jupyter pip``

``pip install emcee corner``

start a jupyter kernel: ``jupyter notebook``

and open a copy of this notebook.

```python
import matplotlib
import matplotlib.pyplot as plt
from cycler import cycler
matplotlib.rc("font", family="serif", size=14)
matplotlib.rc("figure", figsize="10, 5")
colors = ['k', 'c', 'm', 'y']
matplotlib.rc('axes', prop_cycle=cycler("color", colors))

import scipy.optimize
import numpy as np
```

# Why Bayesian inference?

Constrain model parameters with data.

# Fitting a line to data: basic setup

Our model will consist of a set of $N$ i.i.d. observations, including: a coordinate (fixed parameter) $x_i$, a noise level  $\sigma_i$, and observed variable $\hat{y}_i$ drawn from a Gaussian with mean $mx_i + b$ and variance $\sigma^2_i$, i.e. $\hat{y}_i \sim \mathcal{N}(mx_i+b;\sigma_i^2)$.

Let's generate a model:

```python
slope_true = np.random.uniform(0, 1)
intercept_true = np.random.uniform(0, 1)
print('Slopes:', slope_true)
print('Intercepts:', intercept_true)
# This notebook is ready for you to play with 2+ components and more complicated models.
```

Let's generate some data drawn from that model:

```python
ndatapoints = 20
xis_true = np.random.uniform(0, 1, ndatapoints)
x_grid = np.linspace(0, 1, 100)

def model_linear(xs, slope, intercept): return xs * slope + intercept
yis_true = model_linear(xis_true, slope_true, intercept_true)
```

```python
sigma_yis = np.repeat(0.1, ndatapoints) * np.random.uniform(0.5, 2.0, ndatapoints)
yis_noisy = yis_true + np.random.randn(ndatapoints) * sigma_yis
```

```python
y_min, y_max = np.min(yis_noisy - sigma_yis), np.max(yis_noisy + sigma_yis)
y_min = np.min([y_min, np.min(model_linear(x_grid, slope_true, intercept_true))])
y_max = np.max([y_max, np.max(model_linear(x_grid, slope_true, intercept_true))])
```

```python
plt.plot(x_grid, model_linear(x_grid, slope_true, intercept_true), c=colors[0])
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', c=colors[0])
plt.xlabel('$x$'); plt.ylabel('$y$'); plt.ylim([y_min, y_max])
```

We are going to pretend we don't know the true model.

Forget what you saw (please).

Here is the noisy data to be analyzed. Can you (mentally) fit a line through it?

```python
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
plt.xlabel('$x$'); plt.ylabel('$y$'); plt.ylim([y_min, y_max])
```

Let's define a loss/cost function: the total weighted squared error, also called chi-squared:
$$ \chi^2 = \sum_i \left( \frac{ \hat{y}_i - y_i^\mathrm{mod}(x_i, s, m) }{\sigma_i} \right)^2 $$

```python
def loss(observed_yis, yi_uncertainties, model_yis):
    scaled_differences = (observed_yis - model_yis) / yi_uncertainties
    return np.sum(scaled_differences**2, axis=0)
```

We want to minimize this chi-squared to obtain the best possible fit to the data.

Let us look at the fit for a couple of (random) sets of parameters.

```python
random_slopes = np.array([0.25, 0.25, 0.75, 0.75])
random_intercepts = np.array([0.25, 0.75, 0.25, 0.75])
```

```python
fig, axs = plt.subplots(1, 2, sharex=True)
axs[0].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
for i, (slope, intercept) in enumerate(zip(random_slopes, random_intercepts)):
    axs[0].plot(x_grid, model_linear(x_grid, slope, intercept), c=colors[i])
    axs[1].scatter(slope, intercept,marker='x', c=colors[i])
    chi2 = loss(yis_noisy[:, None], sigma_yis[:, None],
                 model_linear(xis_true[:, None], slope, intercept))
    axs[1].text(slope, intercept+0.05, '$\chi^2 = %.1f$'% chi2,
                horizontalalignment='center')
axs[0].set_xlabel('$x$'); axs[0].set_ylabel('$y$')
axs[0].set_ylim([0, y_max]); axs[1].set_ylim([0, 1]);
axs[1].set_xlabel('slope'); axs[1].set_ylabel('intercept')
fig.tight_layout()
```

Let us try a brute-force search, and grid our 2D parameter space.

EXERCISE

Create a 100 x 100 grid covering our parameter space.

Evaluate the loss function on the grid, and plot exp(-0.5*loss).

Also find the point that has the minimal loss value.

```python
# SOLUTION
```

```python
fig, axs = plt.subplots(1, 2, sharex=False, sharey=False)
axs[0].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
axs[0].plot(x_grid, model_linear(x_grid, slope_ml, intercept_ml))
axs[0].set_xlabel('$x$'); axs[0].set_ylabel('$y$')
axs[0].set_ylim([y_min, y_max])
axs[1].set_xlabel('slope'); axs[1].set_ylabel('intercept')
axs[1].axvline(slope_ml, c=colors[1]); axs[1].axhline(intercept_ml, c=colors[1])
axs[1].pcolormesh(slope_grid, intercept_grid, np.exp(-0.5*loss_grid), cmap='ocean_r')
fig.tight_layout()
```

Why visualize $exp(-\frac{1}{2}\chi^2)$ and not simply the $\chi^2$?

Because the former is proportional to our likelihood:

$$\begin{align}
p(D| P, M) &= p(\{ \hat{y}_i \} \vert \{\sigma_i, x_i\}, \textrm{intercept}, \textrm{slope}) \\
&= \prod_{i=1}^{N} p(\hat{y}_i \vert x_i, \sigma_i, b, m)\\
&= \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - y^\mathrm{mod}(x_i; m, b); \sigma^2_i \right)
\ = \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \\
&= \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi}\sigma_i}\exp\left( - \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right)  \\
&\propto \ \exp\left( - \sum_{i=1}^{N} \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right) \ = \ \exp\left(-\frac{1}{2}\chi^2\right)
\end{align}
$$

Since the data points are independent and the noise is Gaussian.

Let's visualize the $\chi^2$ for individual objects

```python
model_yis = model_linear(xis_true, slope_ml, intercept_ml)
object_chi2s = 0.5*((yis_noisy - model_yis) / sigma_yis)**2
```

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x_grid, model_linear(x_grid, slope_ml, intercept_ml))
v = ax.scatter(xis_true, yis_noisy, c=object_chi2s, cmap='coolwarm', zorder=0)
ax.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', zorder=-1)
ax.set_xlabel('$x$'); ax.set_ylabel('$y$'); ax.set_ylim([y_min, y_max])
plt.colorbar(v); fig.tight_layout()
```

## Digression: the limits of maximum likelihood

Is a line a good model?

Should we aiming at maximizing the likelihood only?

Here is a danger of Maximum Likelihood: there is always of model that perfectly fits all of the data.

This model does not have to be complicated...

EXERCISE (5 min): can you try to write a very flexible model that fits the data perfectly, i.e. go through every single point? What $\chi^2$ does it lead to?

NOTE: this might not be trivial, so just look for a model that goes through *most* of the data points.

HINT: numpy has good infrastructure for constructing and fitting polynomials... (try `?np.polyfit`).

If you pick a more complicated model you might need to use `scipy.optimize.minimize`.

```python
# SOLUTION
```

```python
plt.plot(x_grid, bestfit_polynomial(x_grid))
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
plt.ylim([y_min, y_max])
```

##  Bayes' theorem
with explicit Model and Fixed parameters conditioned on:

$$p(P | D, M, F) = \frac{p(D | P, M, F)\ p(P | M, F)}{p(D | M, F)}$$

In our case, if we omit the explicit dependence on a linear model:

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) \  p\bigl(m, b\bigr) \ = \ \exp\bigl(-\frac{1}{2}\chi^2\bigr)\ p\bigl(m, b\bigr) $$

```python
# Let us play with Bayes theorem and pick some un-motivated prior:
prior_grid = np.exp(-slope_grid**-1) * np.exp(-intercept_grid**-1)
likelihood_grid = np.exp(-0.5*loss_grid)
posterior_grid = likelihood_grid * prior_grid
```

```python
fig, axs = plt.subplots(1, 3)
for i in range(3):
    axs[i].set_ylabel('intercept'); axs[i].set_xlabel('slope');
axs[0].set_title('Prior'); axs[1].set_title('Likelihood'); axs[2].set_title('Posterior')
axs[1].axvline(slope_ml, c=colors[1]); axs[1].axhline(intercept_ml, c=colors[1])
axs[0].pcolormesh(slope_grid, intercept_grid, prior_grid, cmap='ocean_r')
axs[1].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[2].pcolormesh(slope_grid, intercept_grid, posterior_grid, cmap='ocean_r')
fig.tight_layout()
```

Discussion: what priors are adequate here?

Four common types of priors are:
- __Flat priors__: uniform probability in $[0, 1]$ for both the slope and intercept.
- __Conjugate priors__: Gaussians or inverse Gamma...
- __Empirical priors__: previous experiments told me that $p(m) = \mathcal{N}(0.5, 0.125)$ and $p(b) = \mathcal{N}(0.5, 0.125)$.
- __Non-informative priors__: rotationally invariance for the line: $p(m) \propto (1 + m^2)^{-1.5}$

```python
## The impact of priors

prior_grid3 = (1 + slope_grid**2)**-1.5
prior_grid2 = np.exp(-0.5*((slope_grid-0.5)/0.125)**2.0) * np.exp(-0.5*((intercept_grid-0.5)/0.125)**2.0)

fig, axs = plt.subplots(2, 3, figsize=(9, 6))
for i in range(3):
    axs[1, i].set_xlabel('slope');
    for j in range(2):
        axs[j, 0].set_ylabel('intercept');
        axs[j, i].axvline(slope_ml, c=colors[1]);
        axs[j, i].axhline(intercept_ml, c=colors[1])
    intercept_vals = intercept_grid[likelihood_grid > likelihood_grid.max() / 10.]
    slope_vals = slope_grid[likelihood_grid > likelihood_grid.max() / 10.]
    axs[1, i].set_xlim([slope_vals.min(), slope_vals.max()])
    axs[1, i].set_ylim([intercept_vals.min(), intercept_vals.max()])

axs[1, 0].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[0, 0].set_title('Flat prior')

axs[0, 1].pcolormesh(slope_grid, intercept_grid, prior_grid2, cmap='ocean_r')
axs[1, 1].pcolormesh(slope_grid, intercept_grid, likelihood_grid*prior_grid2, cmap='ocean_r')
axs[0, 1].set_title('Experimental prior')

axs[0, 2].pcolormesh(slope_grid, intercept_grid, prior_grid3, cmap='ocean_r')
axs[1, 2].pcolormesh(slope_grid, intercept_grid, likelihood_grid*prior_grid3, cmap='ocean_r')
axs[0, 2].set_title('Non-informative prior')

fig.tight_layout()
```

```python
normalization2 = (prior_grid2).max()
normalization3 = (prior_grid3).max()

num_draws = 200
params_drawn1 = np.random.uniform(0, 1, num_draws*2).reshape((num_draws, 2))
params_drawn2 = np.zeros((num_draws, 2))
params_drawn3 = np.zeros((num_draws, 2))

i_draw = 0
num_tot = 0
while i_draw < num_draws:
    params_drawn2[i_draw, :] = np.random.uniform(0, 1, 2)
    pri = np.exp(-0.5*((params_drawn2[i_draw, 0]-0.5)/0.125)**2.0) * np.exp(-0.5*((params_drawn2[i_draw, 1]-0.5)/0.125)**2.0)
    num_tot += 1
    if np.random.uniform(0, 1, 1) < pri / normalization2:
        i_draw += 1
print(num_tot, num_draws)

i_draw = 0
while i_draw < num_draws:
    params_drawn3[i_draw, :] = np.random.uniform(0, 1, 2)
    pri = (1 + params_drawn3[i_draw, 0]**2)**-1.5
    num_tot += 1
    if np.random.uniform(0, 1, 1) < pri / normalization3:
        i_draw += 1
print(num_tot, num_draws)

fig, axs = plt.subplots(2, 3, figsize=(9, 6))
for i in range(3):
    axs[0, i].set_xlabel('slope');
    axs[0, i].set_ylabel('intercept');
    axs[1, i].set_xlabel('$x$');
    axs[1, i].set_ylabel('$y$');

axs[0, 0].scatter(params_drawn1[:, 0], params_drawn1[:, 1], cmap='ocean_r')
axs[0, 0].set_title('Flat prior')

axs[0, 1].pcolormesh(slope_grid, intercept_grid, prior_grid2, cmap='ocean_r')
axs[0, 1].scatter(params_drawn2[:, 0], params_drawn2[:, 1], cmap='ocean_r')
axs[0, 1].set_title('Experimental prior')

axs[0, 2].pcolormesh(slope_grid, intercept_grid, prior_grid3, cmap='ocean_r')
axs[0, 2].scatter(params_drawn3[:, 0], params_drawn3[:, 1], cmap='ocean_r')
axs[0, 2].set_title('Non-informative prior')

for i_draw in range(num_draws):
    axs[1, 0].plot(x_grid, model_linear(x_grid, params_drawn1[i_draw, 0], params_drawn1[i_draw, 1]), c='grey', alpha=0.1)
    axs[1, 1].plot(x_grid, model_linear(x_grid, params_drawn2[i_draw, 0], params_drawn2[i_draw, 1]), c='grey', alpha=0.1)
    axs[1, 2].plot(x_grid, model_linear(x_grid, params_drawn3[i_draw, 0], params_drawn3[i_draw, 1]), c='grey', alpha=0.1)
fig.tight_layout()
```

## The Curse of Dimensionality (v1)

Problems with 'gridding': number of likelihood evaluations, resolution of the grids, etc

```python
fig, ax = plt.subplots(1, 1, figsize=(5, 4))
ax.set_xlabel('slope'); ax.set_ylabel('intercept');
ax.scatter(slope_grid.ravel(), intercept_grid.ravel(), marker='.', s=1)
ax.set_ylim([0, 1])
ax.set_xlim([0, 1])
fig.tight_layout()
print('Number of point/evaluations of the likelihood:', slope_grid.size)
```

Note to Boris: Go back to slides!

## Sampling posterior distributions with MCMC

We are going to approximate the posterior distribution with a set of samples (see slides).

EXERCISE

Write three functions returning:

- the log of the likelihood `ln_like(params, args...)`.

- the log of the prior `ln_prior(params, args...)`.

- the log of the posterior `ln_post(params, args...)`.

The likelihood is pretty much our previous loss function.

The prior should return `-np.inf` outside of our parameter space of interest. At this stage use a uniform prior in $[0, 1] \times [0, 1]$.

Think about what other priors could be used. Include the correct normalization in the prior and the likelihood if possible.

```python
def ln_like(params, xs, observed_yis, yi_uncertainties):
    model_yis = model_linear(xs, params[0], params[1])
    chi2s = ((observed_yis - model_yis) / yi_uncertainties)**2
    return np.sum(-0.5 * chi2s - 0.5*np.log(2*np.pi) - np.log(yi_uncertainties))

def ln_prior(params):
    if np.any(params < 0) or np.any(params > 1):
        return - np.inf
    return 0.

def ln_post(params, xs, observed_yis, yi_uncertainties):
    lnprior_val = ln_prior(params)
    if ~np.isfinite(lnprior_val):
        return lnprior_val
    else:
        lnlike_val = ln_like(params, xs, observed_yis, yi_uncertainties)
        return lnprior_val + lnlike_val
```

```python
x0 = np.array([0.5, 0.5])
print('Likelihood:', ln_like(x0, xis_true, yis_noisy, sigma_yis))
print('Prior:', ln_prior(x0))
print('Posterior:', ln_post(x0, xis_true, yis_noisy, sigma_yis))
```

EXERCISE (2 min)

Find the maximum of the log posterior. Try different optimizers in `scipy.optimize.minimize`. Be careful about the sign of the objective function (is it plus or minus the log posterior?)

```python
# SOLUTION
```

## Sampling strategy 1: Rejection Sampling

EXERCISE

Implement rejection sampling. Randomly draw points in our 2D parameter space. Keep each point with a probability proportional to the posterior distribution.

HINT: you will find that you need to normalize the posterior distribution in some way to make the sampling possible. Use the MAP solution we just found!

```python
# SOLUTION
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_title('Gridding'); axs[1].set_title('Rejection sampling');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
```

## Sampling strategy 2: Metropolis-Hastings

The Metropolis-Hastings algorithm

For a given target probability $p(\theta)$ and a (symmetric) proposal density $p(\theta_{i+1}|\theta_i)$. We repeat the following:
- draw a sample $\theta_{i+1}$ given $\theta_i$ from the proposal density,
- compute the acceptance probability ratio $a={p(\theta_{i+1})}/{p(\theta_i)}$,
- draw a random uniform number $r$ in $[0, 1]$ and accept $\theta_{i+1}$ if $r < a$.

EXERCISE

Use your implementation of the Metropolis-Hastings algorithm to draw samples from our 2D posterior distribution of interest.

Measure the proportion of parameter draws that are accepted: the acceptance rate.

Plot the chain and visualize the burn-in phase.

Compare the sampling to our previous gridded version.

Estimate the mean and standard deviation of the distribution from the samples. Are they accurate?

```python
# SOLUTION
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_title('Gridding'); axs[1].set_title('Metropolis Hastings');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
```

Let's visualize the chains:

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(params_drawn[:, i]);
```

## Validation

MCMC is approximate and is only valid if it has converged. But we can't prove that a chain has converget - we can only show it hasn't.

What to do? ___Be paranoïd.__

Is it crucial to 1) run many chains in various setups, and 2) check that the results are stable, and 3) look at the auto-correlation time:

$$\rho_k = \frac{\mathrm{Covar}[X_t, X_{t+k}]}{\mathrm{Var}[X_t]\mathrm{Var}[X_{t+k}]]}$$

See http://rstudio-pubs-static.s3.amazonaws.com/258436_5c7f6f9a84bd47aeaa33ee763e57a531.html and  www.astrostatistics.psu.edu/RLectures/diagnosticsMCMC.pdf

EXERCISE

Visualize chains, autocorrelation time, etc, for short and long chains with different proposal distributions in the Metropolis Hastings algorithm.

```python
# SOLUTION
```

```python
for i in range(2):
    plt.plot(autocorr_naive(params_drawn[:, i], 500))
plt.xscale('log'); plt.xlabel('$\Delta$'); plt.ylabel('Autocorrelation');
```

## Sampling strategy 3: affine-invariant ensemble sampler

EXERCISE

Let's use a more advanced sampler. Look at the documentation of the `emcee` package and use it to (again) draw samples from our 2D posterior distribution of interest. Make 2D plots with both `plt.hist2d` or `plt.contourf`. For the latter, add 68% and 95% confidence contours.

```python
# SOLUTION
```

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(sampler.chain[:, :, i].T, '-k', alpha=0.2);
```

```python
fig, axs = plt.subplots(1, 3, sharex=True, sharey=True)
for i in range(axs.size):
    axs[i].errorbar(sampler.chain[:, i, 0], sampler.chain[:, i, 1], fmt="-o", alpha=0.5, c='k');
```

```python
num_steps = 1000
sampler.reset()
pos, prob, state = sampler.run_mcmc(pos, num_steps)
```

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(sampler.chain[:, :, i].T, '-k', alpha=0.2);
```

```python
fig, axs = plt.subplots(1, 3, sharex=True, sharey=True)
for i in range(axs.size):
    axs[i].errorbar(sampler.chain[:, i, 0], sampler.chain[:, i, 1], fmt="-o", alpha=0.5, c='k');
```

```python
from corner import hist2d
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], ax=axs[1])
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
fig.tight_layout()
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist(sampler.flatchain[:, 0], histtype='step');
axs[1].hist(sampler.flatchain[:, 1], histtype='step');
axs[0].set_xlabel('slope'); axs[1].set_xlabel('intercept'); axs[0].set_ylabel('Marginal distribution');
fig.tight_layout()
```

It is extremely useful to plot the model in data space!

EXERCISE

Loop through the posterior samples (a random subset of them?) and over-plot them with the data, with some transparency.

```python
# SOLUTION
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist(sampler.flatchain[:, 0], histtype='step');
axs[1].hist(sampler.flatchain[:, 1], histtype='step');
axs[0].set_xlabel('slope'); axs[1].set_xlabel('intercept'); axs[0].set_ylabel('Marginal distribution');
fig.tight_layout()
```

## Parameter estimation

Often, we want to report summary statistics on our parameters, e.g. in a paper.

EXERCISE

Compute some useful summary statistics for our two parameters from the MCMC chains: mean, confidence intervals, etc

```python
# SOLUTION
```

NOTE: for any subsequent analysis, don't use the summary statistics, use the full MCMC chains if you can!

CONTROVERSIAL: if you are only ever going to report and use the mean of a parameter, maybe you don't need MCMC...

# Gibbs sampling

Because of the factorization $p(m, b) = p(m | b)p(b) = p(b|m) p(m)$ we can actually design an algorithm to create an MCMC chain where there is __no need to reject points__!

The procedure is simple: __given__ $(m_i, b_i)$, __draw__ $m_{i+1}$ from $p(m | b=b_i)$, then __draw__ $b_{i+1}$ from $p(b | m=m_{i+1})$.

The only __strong condition__ is to be able to draw directly from the conditional distributions, here $p(m | b)$ and $p(b|m)$.

Generalized to 3+ variables and blocks: draw from (block) conditional distributions in a sequence.

I will draw the way Gibbs sampling works on the board.

What conditional distributions do we need here? Recall that our full posterior distribution is

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) \  p\bigl(m, b\bigr) $$
$$= \ \exp\left( - \sum_{i=1}^{N} \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right) \ p\bigl(m, b\bigr) $$

If we take uniform priors, $p\bigl(m, b\bigr) = C$, then this posterior distribution is Gaussian:

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ \mathcal{N}\left( [X^TX]^{-1}X^T Y ; [X^TX]^{-1} \Sigma \right) $$

$$= \mathcal{N}\left(
\left[\begin{matrix} \hat{m} \\ \hat{b} \end{matrix}\right];
\left[\begin{matrix} \Sigma_{mm} & \Sigma_{mb}\\ \Sigma_{mb} & \Sigma_{bb} \end{matrix}\right]
\right) $$

where I have put the $x_i$'s in a $N x 2$ vector $X$ (the first column is ones, and the second contains the $x_i$'s),  the $\hat{y}_i$'s in a vector $Y$, and the $\sigma_i$'s in a diagonal matrix $\Sigma$. This matches the classic maximum likelihood results for linear regression. The variables $(\hat{m}, \hat{b}, \Sigma_{mm}, \Sigma_{mb}, \Sigma_{mb}, \Sigma_{bb})$ are a convenient notation.

By using Gaussian identities (which can for example be found [here](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)), we can write the conditional distributions

$$p\bigl(m \ \bigl\vert \ b, \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \mathcal{N}\left(
\hat{m} + \Sigma_{mb}\Sigma_{bb}^{-1}(b - \hat{b});
\Sigma_{mm} - \Sigma_{mb}\Sigma_{bb}^{-1}\Sigma_{mb}
\right) $$

$$p\bigl(b \ \bigl\vert \ m, \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \mathcal{N}\left(
\hat{b} + \Sigma_{mb}\Sigma_{mm}^{-1}(m - \hat{m});
\Sigma_{bb} - \Sigma_{mb}\Sigma_{mm}^{-1}\Sigma_{mb}
\right) $$

__Important__: It is because we know how to draw random numbers from Gaussian distributions that we can use Gibbs sampling.

(I lied to you ; there is an analytic solution for this problem!)

BONUS question: look up online what "conjugate priors" are appropriate for linear regression models.

```python
X = np.vstack((xis_true, np.repeat(1, xis_true.size))).T
datacov = np.diag(sigma_yis**2.0)
A = np.dot(X.T, np.linalg.solve(datacov, X))
posterior_mean = np.linalg.solve(A, np.dot(X.T, np.linalg.solve(datacov, yis_noisy[:, None]))).ravel()
print('Posterior mean:', posterior_mean)
posterior_covariance = np.linalg.inv(np.dot(X.T,  np.linalg.solve(datacov, X)))
print('Posterior covariance:', posterior_covariance)
```

Important note: you might think sampling is useless here because we know the analytic posterior distribution...

But there is a very wide class of problems and models where you cannot find an analytic solution for the full posterior distribution, but __you can write the conditional posterior distributions on the parameters__!

In fact, in hierarchical models, in 99% of cases, we automatically have the conditional distributions, but it is how we construct the model. For this reason, and because Gibbs sampling has an acceptance rate of 1, it is very powerful and popular, and often a default solution!

EXERCISE: using those results, implement Gibbs sampling!

```python
# SOLUTION
num_draws = 10000
i_draw = 0
params_drawn = np.zeros((num_draws, 2))
params_drawn[0, :] = posterior_mean + 0.01*np.random.uniform(0, 1, 2)
for i_draw in range(1, num_draws):
    mu = posterior_mean[0] + posterior_covariance[1, 0] *\
        (params_drawn[i_draw-1, 1] - posterior_mean[1]) / posterior_covariance[1, 1]
    cov = posterior_covariance[0, 0] - posterior_covariance[1, 0]**2 / posterior_covariance[1, 1]
    params_drawn[i_draw, 0] = mu + np.random.randn() * cov**0.5

    mu = posterior_mean[1] + posterior_covariance[1, 0] *\
        (params_drawn[i_draw, 0] - posterior_mean[0]) / posterior_covariance[0, 0]
    cov = posterior_covariance[1, 1] - posterior_covariance[1, 0]**2 / posterior_covariance[0, 0]
    params_drawn[i_draw, 1] = mu + np.random.randn() * cov**0.5
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
fig.tight_layout()
```

### Final words on Gibbs sampling
It is great to have an acceptance rate of 1, but we critically need nice conditional distributions, and no nasty degeneracies between the parameters.

Note to Boris: Go back to slides!

## Fitting data with both x and y errors

We observe a set of $\hat{x}_i$ which are noisified versions of the true $x_i$, with Gaussian noise $\gamma_i$.

```python
sigma_xis = np.repeat(0.1, ndatapoints) * np.random.uniform(0.2, 1.0, ndatapoints)
xis_noisy = xis_true + sigma_xis * np.random.randn(xis_true.size)
```

```python
plt.errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o')
plt.xlabel('$x$'); plt.ylabel('$y$'); plt.ylim([y_min, y_max])
```

Our likelihood is now:

$$\begin{align}
p(D| P, M) &= p(\{ \hat{y}_i, \hat{x}_i \} \vert \{\sigma_i, \gamma_i, x_i\}, \textrm{intercept}, \textrm{slope}) \\
&= \prod_{i=1}^{N} p(\hat{y}_i \vert x_i, \sigma_i, b, m) \ p(\hat{x}_i \vert x_i, \gamma_i) \\
& = \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \mathcal{N}\left(\hat{x}_i - x_i; \gamma^2_i \right)
\end{align}
$$

We now have $N$ extra parameters, the $x_i$'s!

The full posterior distribution:

$$  p\bigl( m, s, \{  x_i \} \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \ \propto  \
p\bigl(\{ \hat{y}_i, \hat{x}_i \} \bigl\vert \{\sigma_i, \gamma_i, x_i\}, m, s\bigr) \ \ p\bigl(\{ x_i \}, m, s\bigr) $$

## This is the Curse of Dimensionality v2!

## One solution : Hamiltonian Monte Carlo

Neal's book chapter is a good starting point: https://arxiv.org/abs/1206.1901

Demo: https://chi-feng.github.io/mcmc-demo/app.html

Gradients (and hessians) needed! Three strategies:
- pen and paper, then home-made implementation
- automatic symbolic differentiation
- automatic numerical differentition

Always try auto-diff first (e.g., with `autograd`).

Large-scale inference (gazilion parameters): try `tensorflow`

## Analytic marginalization of latent variables

We are only truly interested in the marginalized posterior distribution:

$$p\bigl( m, s \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \ = \ \int\mathrm{d}\{x_i\} p\bigl( m, s, \{  x_i \} \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \\
 \propto \  \prod_{i=1}^{N} \int \mathrm{d}x_i \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \mathcal{N}\left(\hat{x}_i - x_i; \gamma^2_i \right) \ \ p\bigl(\{ x_i \}, m, s\bigr) \\
 \propto \  \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m \hat{x}_i - b; \sigma^2_i + \gamma^2_i\right)  \ p(s, m) $$

with flat uninformative priors on $x_i$'s $p\bigl(x_i)$.

We have eliminated the $x_i$'s!

Let us do a run with the x's fixed to their noisy values (which is wrong! This is ignoring the x noise).

```python
ndim = 2
nwalkers = 50

starting_params = np.random.uniform(0, 1, ndim*nwalkers).reshape((nwalkers, ndim))
sampler2 = emcee.EnsembleSampler(nwalkers, ndim, ln_post,
                                args=[yis_noisy, xis_noisy, sigma_yis])

num_steps = 100
pos, prob, state = sampler2.run_mcmc(starting_params, num_steps)
num_steps = 1000
sampler2.reset()
pos, prob, state = sampler2.run_mcmc(pos, num_steps)
```

```python
def ln_like(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties):
    xyi_uncertainties = np.sqrt(xi_uncertainties**2. + yi_uncertainties**2.)
    model_yis = model_linear(observed_xis, params[0], params[1])
    return np.sum(-0.5 * ((observed_yis - model_yis) / xyi_uncertainties)**2
                   - 0.5*np.log(2*np.pi) - np.log(xyi_uncertainties))

def ln_prior(params):
    if np.any(params < 0) or np.any(params > 1):
        return - np.inf
    return 0.

def ln_post(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties):
    lnprior_val = ln_prior(params)
    if ~np.isfinite(lnprior_val):
        return lnprior_val
    else:
        lnlike_val = ln_like(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties)
        return lnprior_val + lnlike_val
```

```python
x0 = np.repeat(0.5, 2)
print('Likelihood:', ln_like(x0, yis_noisy, xis_noisy, sigma_yis, sigma_xis))
print('Prior:', ln_prior(x0))
print('Posterior:', ln_post(x0, yis_noisy, xis_noisy, sigma_yis, sigma_xis))
```

```python
ndim = 2
nwalkers = 50

starting_params = np.random.uniform(0, 1, ndim*nwalkers).reshape((nwalkers, ndim))
sampler3 = emcee.EnsembleSampler(nwalkers, ndim, ln_post,
                                args=[yis_noisy, xis_noisy, sigma_yis, sigma_xis])

num_steps = 100
pos, prob, state = sampler3.run_mcmc(starting_params, num_steps)
num_steps = 1000
sampler3.reset()
pos, prob, state = sampler3.run_mcmc(pos, num_steps)
```

```python
fig, axs = plt.subplots(1, 3, sharex=False, sharey=False)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
axs[1].hist2d(sampler2.flatchain[:, 0], sampler2.flatchain[:, 1], 30, cmap="ocean_r");
axs[2].hist2d(sampler3.flatchain[:, 0], sampler3.flatchain[:, 1], 30, cmap="ocean_r");
axs[0].set_title('x true');
axs[1].set_title('x wrongly fixed');
axs[2].set_title('with x errors');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept');
axs[1].set_xlabel('slope');
axs[2].set_xlabel('slope');
for i in range(3):
    axs[i].axvline(slope_true)
    axs[i].axhline(intercept_true)
    axs[i].set_ylim([0, 1])
    axs[i].set_xlim([0, 1])
fig.tight_layout()
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].set_xlabel('$x$'); axs[1].set_xlabel('$x$'); axs[0].set_ylabel('$y$');
num = 1000
y_models = np.zeros((x_grid.size, num))
for j, i in enumerate(np.random.choice(np.arange(sampler3.flatchain.shape[0]), num, replace=False)):
    y_models[:, j] = model_linear(x_grid, sampler3.flatchain[i, 0], sampler3.flatchain[i, 1])
    axs[0].plot(x_grid, y_models[:, j], c='gray', alpha=0.01, zorder=0)
axs[1].plot(x_grid, np.mean(y_models, axis=1), c='gray', alpha=1, zorder=0)
axs[1].fill_between(x_grid, np.mean(y_models, axis=1)-np.std(y_models, axis=1),
            np.mean(y_models, axis=1)+np.std(y_models, axis=1), color='gray', alpha=0.5, zorder=0)
axs[0].errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o', zorder=1)
axs[1].errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o', zorder=1)
```

We got around the number of parameters by analytically marginalizing the ones we don't really care about. Sometimes this is not possible!

# Extensions

- __Automatic differentiation__ with autograd, tensorflow, etc.
- __Nested Sampling for nasty distributions and model comparison__. Application: fitting multiple components/lines to a data set.
- __Model testing, model comparison__. I have multiple models. Which is the best? Example: fit multiple lines to data.
- __Hamiltonian Monte Carlo with quasi-auto-tuning for millions of parameters.__ Application: fitting a line with many latent parameters (x noise, outliers, etc).
- __Multi-stage hybrid sampling__: Application: non-linear models with many parameters and complicated gradients.
- __Connections to deep machine learning__: Bayesian interpretation of Convolution networks, Adversarial training, deep forward models, etc. TensorFlow.

Let me know if you are interested and we will organize a session. A few notebooks and mode advanced examples available on https://ixkael.github.io

# Final thoughts
With the right method you can solve problems/models that seem intractable. Don't underestimate yourself! Start small, but be ambitious.

```python

```

notebooks/Fitting a line to data - a full tutorial.ipynb