```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gs
```

# Effect of a ramp stimulus on a leaky capacitor

To develop a toy model of the peak population firing rate evoked by a ramp stimulus with a given derivative, we'll begin by considering the voltage created when this stimulus is applied to a non-spiking neuron (aka, a leaky capacitor). The membrane filter of our non-spiking neuron is $\kappa(t) = \alpha e^{-t/\tau_m}$, where $\tau_m$ is the membrane time constant and $\alpha = 1/C$ is a scaling factor. The voltage created by passing a time-dependent current $I(t)$ through our membrane filter is $$V(t) = (\kappa * I)(t) =  \int_{-\infty}^t \kappa(t - \tau) I(\tau) d\tau,$$ where $\tau$ is a lag.

Consider the time-dependent input $I$ to be a ramp stimulus of a fixed amplitude $A$ starting at time 0 and lasting until time $t^\prime$. The derivative of $I$ is defined as follows
\begin{equation}
    \frac{dI}{dt} =
    \begin{cases}
        0 & -\infty < t < 0 \\
        A/t^\prime & 0 < t < t^\prime.
    \end{cases}
\end{equation}
Using this definition of $\frac{dI}{dt}$, we can integrate the definition of $V(t)$ above and obtain the following as the voltage at the end of the ramp
$$V(t^\prime) = R_m A \left[1 - \frac{\tau_m}{t^\prime}(1 - e^{-t^\prime/\tau_m})\right],$$
where $R_m = \alpha \tau_m$. Notice that this equation meets a couple of basic intuitions about the voltage evoked by the ramp:

- when the ramp is very long ($t^\prime \gg \tau_m$), this reduces to Ohm's law $V = A R_m$
- when the ramp is short ($t^\prime < \tau_m$), the final voltage $V(t^\prime)$ is less than what you'd get from Ohm's law because the membrane filter hasn't fully integrated the stimulus.

Notice also that since $\frac{dI}{dt}$ is inversely proportional to $t^\prime$, this equation also tells us that increasing $\frac{dI}{dt}$ (aka decreasing $t^\prime$) reduces the voltage.

If we plot this equation in terms of $\tau_m/t^\prime$, we can see how much ramps with different durations (aka derivatives) affect the membrane voltage.

```python
def v_final(tau_duration_ratio):
    return (
        1 - tau_duration_ratio * (1 - np.exp(-1./tau_duration_ratio))
    )
```

```python
tau_duration_ratio = np.logspace(-2, 2)
plt.semilogx(tau_duration_ratio, v_final(tau_duration_ratio))
plt.annotate('DC', (0.01, 0.97), (0.01, 0.8), arrowprops={'arrowstyle': '->'})
plt.annotate('Single EPSC',
             (100, 0.03), (50, 0.3), arrowprops={'arrowstyle': '->'},
            ha='right')
plt.xlabel(r'$\tau_m/t^\prime$')
plt.ylabel('Peak voltage (AU)')
```

A sigmoid on a log scale! As expected, DC currents affect the voltage very much, but short ramps are filtered out.

To help with getting an intuition for this curve, consider a 5-HT neuron with $\tau_m = 50$ms. Ramps longer than 0.5s are basically DC, and ramps shorter than 5ms are completely filtered out.

The ramp simulations I ran all fall on the left hand side of this plot, since the shortest ramps there are 100ms. Here's what that part of the curve looks like using the same parameters as my simulations.

```python
derivatives = np.linspace(10, 1000)  # pA/s
ramp_amplitude = 100.  # pA
durations = ramp_amplitude / derivatives * 1e3  # ms

dur_ax = plt.subplot(121)
plt.ylabel('Peak voltage (AU)')
plt.xlabel('Ramp duration (ms)')
plt.ylim(-0.05, 1.05)

deriv_ax = plt.subplot(122)
plt.xlabel('$dI/dt$ (pA/s)')
plt.ylabel('Peak voltage (AU)')
plt.ylim(-0.05, 1.05)

for label, time_constant in {'SOM': 20., '5-HT': 50.}.items():
    dur_ax.semilogx(
        durations,
        v_final(time_constant / durations),
        label='{}, $\\tau_m = {}$ms'.format(label, time_constant)
    )

    deriv_ax.plot(derivatives, v_final(time_constant / durations))

dur_ax.legend()
plt.tight_layout()
```

Notice that if we look at the same range of values for SOM neurons, which have a shorter time constant, the curve looks flatter.

You can see this effect over a wide range of choices of $\tau_m$ in the heatmap below.

```python
time_constants = np.linspace(10, 100)
final_voltages = [
    v_final(time_constant/durations)
    for time_constant in time_constants
]

plt.subplot(111)
mappable = plt.pcolormesh(
    derivatives,
    time_constants,
    final_voltages
)
plt.ylabel('Membrane time constant (ms)')
plt.xlabel('$dI/dt$ (pA/s)')

cbar = plt.gcf().colorbar(mappable)
cbar.set_label('Peak voltage (AU)')
```

## Summary of effect of ramp stimulus on voltage

Ramps shorter than the membrane time constant are filtered out. This probably shows up more in my simulations of 5-HT neurons than SOM cells because their time constants are longer.

# Translating voltage into a firing rate

## Mathematical details

A LIF neuron with spike threshold $\theta$ and reset potential 0mV receiving a DC input $I$ spikes at the following rate
$$ \rho = \frac{-1}{\tau_m \log (1 - \frac{\theta}{IR_m})}, $$
which is the reciprocal of the ISI. I'll use this as a zero-order approximation of the population firing rate associated with the voltages calculated above.

First, though, notice that this rate is mainly a function of the strength of the input relative to spike threshold $\theta/(IR_m)$. What does that look function look like?

```python
def rate(time_constant, threshold_ratio):
    denominator = time_constant * np.log(1. - threshold_ratio)
    return -1. / denominator
```

```python
plt.figure(figsize=(6, 3))

plt.subplot(121)
threshold_ratios = np.logspace(-2, -0.1)
plt.semilogx(
    threshold_ratios,
    rate(1., threshold_ratios),
)
plt.ylabel('Firing rate (AU)')
plt.xlabel(r'Input strength $\frac{\theta}{IR_m}$')

plt.subplot(122)
threshold_ratios = np.logspace(-4, -0.1)
plt.loglog(
    threshold_ratios,
    rate(1., threshold_ratios),
)
plt.ylabel('Firing rate (AU)')
plt.xlabel(r'Input strength $\frac{\theta}{IR_m}$')

plt.tight_layout()
```

Power law, more or less!

So, what happens when we substitute in the $V(t^\prime)$ we calculated earlier?
$$\frac{\theta}{IR_m} = \frac{\theta}{V(t^\prime)} = \frac{\theta}{AR_mx},$$
where $x$ is the output of the `v_final()` function I used above. I'll call $\frac{\theta}{AR_m}$ the `threshold_ratio` below.

```python
time_constant = 50.
threshold_ratios = np.linspace(0.1, 0.6)  # Small numbers mean big input.
voltage_multipliers = v_final(time_constant / durations)
firing_rates = [
    rate(time_constant, threshold_ratio / voltage_multipliers)
    for threshold_ratio in threshold_ratios
]

mappable = plt.pcolormesh(derivatives, threshold_ratios, firing_rates)
plt.contour(derivatives, threshold_ratios, firing_rates, colors='k')
plt.gca().invert_yaxis()
plt.ylabel('Threshold ratio\n(AU; smaller numbers mean stronger input)')
plt.xlabel('$dI/dt$ (pA/s)')

cbar = plt.gcf().colorbar(mappable)
cbar.set_label('Firing rate (AU)')
```

Notice that we're seeing the diagonal structure of the firing rate contour plot for the first time!

A few squares are missing in the bottom right because these inputs don't quite reach threshold.

Since the firing rate formula is scale-free/power law, this diagonal structure shows up for any choice of `threshold_ratio`.

```python
threshold_ratios = np.linspace(0.001, 0.006)  # Small numbers mean big input.
voltage_multipliers = v_final(time_constant / durations)
firing_rates = [
    rate(time_constant, threshold_ratio / voltage_multipliers)
    for threshold_ratio in threshold_ratios
]

plt.pcolormesh(derivatives, threshold_ratios, firing_rates)
plt.contour(derivatives, threshold_ratios, firing_rates, colors='k')
plt.gca().invert_yaxis()
```

## Membrane time constant controls diagonal structure

The diagonal structure of the firing rate vs. "threshold ratio" (can be interpreted roughly as **distance to threshold**) and input derivative plots above depends very much on the membrane time constant: there's very little diagonal structure when the membrane time constant is low (ie SOM neurons).

```python
plt.figure(figsize=(8, 3))
for i, (label, time_constant) in enumerate({'5-HT': 70., 'SOM': 20.}.items()):
    input_strengths = np.linspace(2, 10)
    voltage_multipliers = v_final(time_constant / durations)
    firing_rates = [
        rate(time_constant, 1. / (input_strength * voltage_multipliers))
        for input_strength in input_strengths
    ]

    plt.subplot(1, 2, i+1)
    plt.title('$\\tau_m = {}$ms\n({})'.format(time_constant, label))
    mappable = plt.pcolormesh(derivatives, input_strengths, firing_rates)
    plt.contour(derivatives, input_strengths, firing_rates, colors='k')
    plt.ylabel('Input strength (AU)')
    plt.xlabel('$dI/dt$ (pA/s)')

    cbar = plt.gcf().colorbar(mappable)
    cbar.set_label('Firing rate (AU)')

plt.tight_layout()
```

This shows how the toy model can explain the diagonal structure of the contour plots from the simulations.

# Capturing the effect of adaptation

Adaptation divisively reduces the firing rate. This is derivative-dependent. We can capture this phenomenologically by multiplying the firing rates generated above with a sigmoid function of $dI/dt$.

```python
def get_sigmoid(slope, offset):
    def sigmoid(arr):
        return 1. / (1 + np.exp(-slope * (arr - offset)))
    return sigmoid
```

```python
cell_params = {
    '5-HT': {
        'time_constant': 70.,
        'sigmoid': get_sigmoid(0.01, 250.),
    },
    'SOM': {
        'time_constant': 20.,
        'sigmoid': get_sigmoid(0.001, 250.)
    }
}

plt.figure(figsize=(8, 3))
for i, (label, params) in enumerate(cell_params.items()):
    input_strengths = np.linspace(2, 10)
    voltage_multipliers = v_final(params['time_constant'] / durations)
    firing_rates = np.array([
        rate(params['time_constant'], 1. / (input_strength * voltage_multipliers))
        for input_strength in input_strengths
    ])
    firing_rates *= params['sigmoid'](derivatives)

    plt.subplot(1, 2, i+1)
    plt.title('$\\tau_m = {}$ms\n({})'.format(params['time_constant'], label))
    mappable = plt.pcolormesh(derivatives, input_strengths, firing_rates)
    plt.contour(derivatives, input_strengths, firing_rates, colors='k')
    plt.ylabel('Input strength (AU)')
    plt.xlabel('$dI/dt$ (pA/s)')

    cbar = plt.gcf().colorbar(mappable)
    cbar.set_label('Firing rate (AU)')

plt.tight_layout()
```


## S-I-R model

System of differential equations:

$$
\begin{equation}
    \left\{
    \begin{array}{lll}
    \frac{dS}{dt} = -\frac{\beta SI}{N}   \\
    \frac{dI}{dt} = \frac{\beta SI}{N} - \gamma I \\
    \frac{dR}{dt} = \gamma I
    \end{array} \right.
\end{equation}
$$

Where
$S = $ number of susceptible people
$I = $ number of currently infected people
$R = $ number of people that have recovered or died from the disease

$N =$ "population size"
$\beta =$ "infection rate"
$\gamma =$ "recovery rate" (This should also include deaths)

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
```

### Pull population and initial conditions from data

N = Population
S0 = population - total cases
I0 = total cases - deaths - recovered
R0 = deaths + recovered

```python
# Total population, N
N = country_population
deaths = country_deaths
#recovered = total_confirmed_cases - deaths

# Initial conditions
R0 = recovered + deaths
I0 = total_confirmed_cases - R0
S0 = N - I0 - R0 #note that S0 is also our upper limit
```

### Estimate parameters from Ryan's model

```python
# Contact rate, beta, and mean recovery rate, gamma, (in 1/days)
beta = 0.2
gamma = 1./14 #assume 14days is the recovery time

# A grid of time points (in days)
t = np.linspace(0, 160, 160)

# The SIR model differential equations
def deriv(y, t, N, beta, gamma):
    S, I, R = y #initial conditions

    #model = system of differential equations
    dSdt = -beta * S * I / N
    dIdt = beta * S * I / N - gamma * I
    dRdt = gamma * I

    return dSdt, dIdt, dRdt

# Initial conditions vector
y0 = S0, I0, R0

# Integrate the SIR equations over the time grid, t
ret = odeint(deriv, y0, t, args=(N, beta, gamma))
S, I, R = ret.T
```

### Plot

```python
# Plot the data on three separate curves for S(t), I(t) and R(t)
fig = plt.figure(facecolor='w')
ax = fig.add_subplot(111, facecolor='#dddddd', axisbelow=True)
ax.plot(t, S/N, 'b', alpha=0.5, lw=2, label='Susceptible')
ax.plot(t, I/N, 'r', alpha=0.5, lw=2, label='Infected')
ax.plot(t, R/N, 'g', alpha=0.5, lw=2, label='Recovered with immunity')
ax.set_xlabel('Time /days')
ax.set_ylabel('Number (1000s)')
ax.set_ylim(0,1.2)
ax.yaxis.set_tick_params(length=0)
ax.xaxis.set_tick_params(length=0)
ax.grid(b=True, which='major', c='w', lw=2, ls='-')
legend = ax.legend()
legend.get_frame().set_alpha(0.5)
for spine in ('top', 'right', 'bottom', 'left'):
    ax.spines[spine].set_visible(False)
plt.show()
```

#### Example

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# Total population, N.
N = 10000
# Initial number of infected and recovered individuals, I0 and R0.
I0, R0 = 50, 20
# Everyone else, S0, is susceptible to infection initially.
S0 = N - I0 - R0

# Contact rate, beta, and mean recovery rate, gamma, (in 1/days).
beta, gamma = 0.2, 1./14
# A grid of time points (in days)
t = np.linspace(0, 160, 160)

# The SIR model differential equations.
def deriv(y, t, N, beta, gamma):
    S, I, R = y
    dSdt = -beta * S * I / N
    dIdt = beta * S * I / N - gamma * I
    dRdt = gamma * I
    return dSdt, dIdt, dRdt

# Initial conditions vector
y0 = S0, I0, R0
# Integrate the SIR equations over the time grid, t.
ret = odeint(deriv, y0, t, args=(N, beta, gamma))
S, I, R = ret.T

# Plot the data on three separate curves for S(t), I(t) and R(t)
fig = plt.figure(facecolor='w')
ax = fig.add_subplot(111, facecolor='#dddddd', axisbelow=True)
ax.plot(t, S/10000, 'b', alpha=0.5, lw=2, label='Susceptible')
ax.plot(t, I/10000, 'r', alpha=0.5, lw=2, label='Infected')
ax.plot(t, R/10000, 'g', alpha=0.5, lw=2, label='Recovered with immunity')
ax.set_xlabel('Time /days')
ax.set_ylabel('Number (10000s)')
ax.set_ylim(0,1.2)
ax.yaxis.set_tick_params(length=0)
ax.xaxis.set_tick_params(length=0)
ax.grid(b=True, which='major', c='w', lw=2, ls='-')
legend = ax.legend()
legend.get_frame().set_alpha(0.5)
for spine in ('top', 'right', 'bottom', 'left'):
    ax.spines[spine].set_visible(False)
plt.show()
```

```python

```


SymPy: Open Source Symbolic Mathematics

```
%load_ext sympyprinting
```

```
from __future__ import division
from sympy import *
x, y, z = symbols("x y z")
k, m, n = symbols("k m n", integer=True)
f, g, h = map(Function, 'fgh')
```

Elementary operations

```
Rational(3,2)*pi + exp(I*x) / (x**2 + y)

```

```
exp(I*x).subs(x,pi).evalf()

```

```
e = x + 2*y
```

```
srepr(e)
```

    Add(Symbol('x'), Mul(Integer(2), Symbol('y')))

```
exp(pi * sqrt(163)).evalf(50)
```

Algebra

```
((x+y)**2 * (x+1)).expand()
```

```
a = 1/x + (x*sin(x) - 1)/x
display(a)
simplify(a)
```

```
solve(Eq(x**3 + 2*x**2 + 4*x + 8, 0), x)
```

    [-2⋅ⅈ, 2⋅ⅈ, -2]

```
a, b = symbols('a b')
Sum(6*n**2 + 2**n, (n, a, b))
```

Calculus

```
limit((sin(x)-x)/x**3, x, 0)
```

```
(1/cos(x)).series(x, 0, 6)
```

```
diff(cos(x**2)**2 / (1+x), x)
```

```
integrate(x**2 * cos(x), (x, 0, pi/2))
```

```
eqn = Eq(Derivative(f(x),x,x) + 9*f(x), 1)
display(eqn)
dsolve(eqn, f(x))
```


# Pendulum on $\sin (x)$.

## System definition

Consider a pendulum with mass $m_2$ hanging from a rod of length $l$. The support point has a mass $m_1$, and can move without friction along a  curve given by formula $y=f(x)$

We will need some helpers for the algebra:

```python
load("cas_utils.sage")
```

```python
var('t')
var('l g m1 m2')
xy_wsp = [('x1','x_1'),('y1','y_1'),('x2','x_2'),('y2','y_2')]

uv_wsp = [('phi','\phi'),('x','x')]

to_fun, to_var = make_symbols(xy_wsp + uv_wsp)

uv = [vars()[v] for v,lv in uv_wsp]
xy = [vars()[v] for v,lv in xy_wsp]

```

We introduce generalized coordinates compliant with contraints: $\varphi$, and $x$.

```python
f(x) = sin(x)
x2u = {x1:x,x2:x+l*sin(phi),y2:-l*cos(phi)+sin(x),y1:f(x)}
showmath(x2u)
```

We express virtual displacement:$\delta x_1,...$ as function of virtual displacements of new coordinates: $\delta x,\delta \phi.$

```python
for w in xy:
    vars()['d'+repr(w)+'_polar']=sum([w.subs(x2u).diff(w2)*vars()['d'+repr(w2)] for w2 in uv])
    showmath([vars()['d'+repr(w)],vars()['d'+repr(w)+'_polar']])
```

Now we can write d'Alembert principle:

$$\sum_{i} ( \mathbf {F}_{i} - m_i \mathbf{a}_i )\cdot \delta \mathbf r_i = 0,$$

```python
dAlemb = (m1*x1.subs(x2u).subs(to_fun).diff(t,2)  )*dx1_polar + \
         (m1*y1.subs(x2u).subs(to_fun).diff(t,2)+m1*g)*dy1_polar + \
         (m2*x2.subs(x2u).subs(to_fun).diff(t,2)  )*dx2_polar + \
         (m2*y2.subs(x2u).subs(to_fun).diff(t,2)+m2*g)*dy2_polar
dAlemb = dAlemb.subs(to_var)
showmath(dAlemb.collect(dx).collect(dphi))
```

and derive  equations of motion in new coordintes:

```python
r1 = dAlemb.coefficient(dx)
r2 = dAlemb.coefficient(dphi)
w1,w2 =  solve([r1,r2],[xdd,phidd])[0]
showmath(w1.trig_simplify())
```

```python
showmath(w2)
```

#### Special case

$m_1\to\infty$, $x=-\frac{\pi}{2}$

```python
showmath( limit(w1.rhs(),m1=oo).subs({xdd:0,xd:0,x:-pi/2})  )
```

```python
showmath( limit(w2.rhs(),m1=oo).subs({xdd:0,xd:0,x:-pi/2}).trig_reduce() )
```

We obtain mathematical pendulum if

```python
showmath( limit(w1.rhs(),m1=0).trig_reduce() )
```

```python
showmath( limit(w2.rhs(),m1=0).trig_reduce() )
```

## Numerical analysis of the system
Initial conditions are four numbers: $x,\phi,\dot x,\dot \phi$.

```python
import numpy as np
```

```python
%%time
pars = {l:1,g:9.81,m1:1.,m2:1}
ode = [xd,phid,w1.rhs().subs(pars),w2.rhs().subs(pars)]
times = srange(0,10.25,0.01)
ics = [1, 0, 0, 1]
sol = desolve_odeint(ode, ics, times, [x,phi,xd,phid])
```

```python
line( zip(times,sol[::1,0]),figsize=(8,2) )+\
 line( zip(times,sol[::1,1]),color='red')
```

```python
line( zip(times,sol[:,0]),figsize=4 )
line( zip(np.sin(sol[:,1])+sol[:,0],-np.cos(sol[:,1])),figsize=4 )+\
 line( zip(np.sin(sol[:,1])+sol[:,0],-np.cos(sol[:,1])),figsize=4 )
```

### Visualization

It is helpfull to write simple function displaying configuration of the system for given set of variables.

```python
def draw_system(ith=0,l=1):
    x,phi = sol[ith,:2]
    x1,y1,x2,y2 =x, f(x),  l*sin(phi) + x,f(x)-l*cos(phi)

    p = point( (x1,y1), size=40) +\
     point( (x2,y2), size=40,color='red',figsize=3) +\
     line( [(x1,y1),(x2,y2)],aspect_ratio=1)
    n=40
    i0 = max(0,ith-n)

    trace = sum([point((l*sin(phi) + x,f(x)-l*cos(phi)),hue=(0,1-(i)/n,1)) for i,(x,phi) in enumerate(sol[ith:i0:-1,:2])])
    trace2 = sum([point((x,f(x)),hue=(.51,(i)/n,1)) for i,(x,phi) in enumerate(sol[i0:ith,:2])])

    p += trace+trace2
    var('x_')
    p += plot(f(x_),(x_,-4.5,1),figsize=6 )
    p.set_axes_range(-4.5,1,-2,1)
    p.set_aspect_ratio(1)
    return p
```

Let's try:

```python
draw_system(120)
```

We can animate:

```python
from IPython.display import clear_output
import time
for ith in range(0,len(sol),20):
    plt = draw_system(ith=ith,l=1)
    clear_output(wait=True)
    plt.show(figsize=3)
    time.sleep(0.021)

```

Alternatively one can use slider:

```python
@interact
def _(ith=slider(range(len(sol)))):
    plt = draw_system(ith=ith,l=2)
    plt.show(figsize=6)
```

### Chaotic properties of the solution

#### Spectrum

The solution looks chaotic, but it can be caused by mixing of few frequencies. We can, however, calculate the Fourier transform of one of the system variables and see.

```python
%%time
pars = {l:1,g:9.81,m1:1.1,m2:1}
ode = [xd,phid,w1.rhs().subs(pars),w2.rhs().subs(pars)]
times = srange(0,5000.25,0.2)
ics = [1,0,0,0]
sol = desolve_odeint(ode,ics,times,[x,phi,xd,phid])
xfft = np.fft.fft(sol[:,0])
n1 = xfft.shape[0]
```

```python
plt = line(enumerate(np.abs(np.fft.fftshift(xfft))),ymax=2500,thickness=0.1)
plt.show(figsize=(8,2))
```

Let us check for comparison that sum of two signals with different frequencies would not cause such effect

```python
expr = sin(1.2*t)+sin(sqrt(1.2)*t)
```

```python
import numpy as np
import sympy
expr_np = np.vectorize( sympy.lambdify(t, sympy.sympify( expr ) ) )
```

```python
nonchaotic = expr_np(np.linspace(0,5000,10000))
```

```python
%time nonchaotic_fft = np.fft.fft(nonchaotic)
```

```python
line(enumerate(nonchaotic[:234])).show(figsize=(7,2))
```

```python
plt2 = line(enumerate(np.abs(np.fft.fftshift(nonchaotic_fft))),alpha=1)
plt2.show(figsize=(8,2))
```

#### Sensitivity to initial conditions

Let us compare two solution which differ by $\frac{1}{1000000}$ in initial velocity.

```python
%%time
pars = {l:1,g:9.81,m1:1.1,m2:1}
ode = [xd,phid,w1.rhs().subs(pars),w2.rhs().subs(pars)]
times = srange(0,35.,0.1)
ics = [1,0,0,0]
sol = desolve_odeint(ode,ics,times,[x,phi,xd,phid])
ics2 = [1+1e-6,0,0,0]
sol2 = desolve_odeint(ode,ics2,times,[x,phi,xd,phid])

```

```python
line(zip(times,sol[:,0]))+line(zip(times,sol2[:,0]),color='red',figsize=(6,3))
```

We can have a look how the error propagetes in log-scale:

```python
line(zip(times[1:],log(abs(sol[1:,0]-sol2[1:,0]))),figsize=(6,3))
```

\newpage


+ This notebook is part of lecture 1 *The geometry of linear equations* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

+ In this series of notebooks I will make use of a custom cascading style sheet
+ The file *style.css* must be in the same folder as the notebook file
+ The first block of code executes the stylesheet

```python
# from IPython.core.display import HTML, Image
# css_file = 'style.css'
# HTML(open(css_file, 'r').read())
```

+ All the modules and function will be imported here

```python
import numpy as np # Using namespace abbreviation to import numerical python
from sympy import init_printing, symbols, Matrix, Eq # Imporint only the
# required functions in the sympy module
import matplotlib.pyplot as plt # Using namespace abbreviation to import
# the pyplot submodule of matplotlib
import seaborn as sns # Using namespace abbreviation to import
# the seaborn plotting library
from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax') # Used to print Latex to the screen
%matplotlib inline
filterwarnings('ignore') # Ignore those ugly pink warning boxes
```

```python
# Comments will be in this form
# Comments are not executed
```

```python
x, y, z = symbols('x y z') # Creating symbolic mathematical variables as opposed to computer variables
# These symbols can no longer be used as computer variable names
```

# Geometrical view

## System of linear equations

* A set of variables (each of power one and not transcendental)
* Example
$$ 2{x}-{y}={0} $$
$$ -{x} + 2{y} = {3} $$

* This can be represented as an augmented matrix

```python
A_augm = Matrix([[2, -1, 0], [-1, 2, 3]]) # Note the placement of ()'s and []'s
A_augm # A_augm is a computer variable that contains the matrix
```

```python
# We can ask python what type of computer variable A_augm holds
type(A_augm)  # We see that it is a mutable dense matrix
```

* The matrix of coefficients:

```python
A = Matrix([[2, -1], [-1, 2]])
A
```

* The variable vector:

```python
x_vect = Matrix([x, y])
x_vect
```

* The solution vector:

```python
b_vect = Matrix([0, 3])
b_vect
```

```python
Eq(A * x_vect, b_vect) # From Ax = b
# The Eq function takes the arguments left-hand-side (LHS), right-handside (RHS) of the equation
```

## The row picture

```python
# Don't be too concerned about the code for plotting
# It does not form part of this series of notebooks

x_vals = np.linspace(-3, 3, 100) # Create 100 values between -3 and 3
# Note that we cannot use the computer variable x, because it has been reserved above as a mathematical variable in
# the symbols function

plt.figure(figsize = (10,8)) # Create a graph of size 10 by 8
plt.plot(x_vals, 2 * x_vals) # Plot every single value created above with 2 times that values
# Taken from the first equation which was y = 2x or f(x) = 2x
# The plot takes the arguments (code between parentheses) of x,y
plt.plot(x_vals, ((x_vals / 2) + (3 / 2))) # Also plot the second equation
plt.show; # Draw the plot on screen
```

## The column picture

* In the column picture we look at the column vector associate with the variables:
$$ x\begin{bmatrix} 2 \\ -1 \end{bmatrix}+y\begin{bmatrix} -1 \\ 2 \end{bmatrix}=\begin{bmatrix} 0 \\ 3 \end{bmatrix} $$
* It asks us to look at the linear combination of the columns

* Performing this multiplication results in the same equation

```python
Eq(x * Matrix([2, -1]) + y * Matrix([-1, 2]), Matrix([0, 3]))
```

```python
from mpl_toolkits.mplot3d import proj3d

fig = plt.figure(figsize = (10, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot([0, 2], [0, -1], zs=[0, 0])
# The three sets of square bracket contain as first element the starting
# point, i.e. 0, 0, 0 (as in  x, y ,z coordinates)
# The second element in each square bracket represents the end-point, i.e. 2, -1, 0
ax.plot([0, -1], [0, 2], zs=[0, 0])

plt.show();
```

```python
# Method adding arrow heads (very complicated)
from mpl_toolkits.mplot3d import Axes3D
from itertools import product, combinations
fig = plt.figure(figsize = (10, 8))
ax = fig.gca(projection='3d')
ax.set_aspect("equal")

#draw a vector
from matplotlib.patches import FancyArrowPatch
from mpl_toolkits.mplot3d import proj3d

class Arrow3D(FancyArrowPatch):
    def __init__(self, xs, ys, zs, *args, **kwargs):
        FancyArrowPatch.__init__(self, (0,0), (0,0), *args, **kwargs)
        self._verts3d = xs, ys, zs

    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj3d.proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.set_positions((xs[0],ys[0]),(xs[1],ys[1]))
        FancyArrowPatch.draw(self, renderer)

a = Arrow3D([0, 2],[0, -1],[0, 0], mutation_scale=20, lw=1, arrowstyle="-|>", color="k")
b = Arrow3D([0, -1],[0, 2],[0, 0], mutation_scale=20, lw=1, arrowstyle="-|>", color="b")
ax.add_artist(a)
ax.add_artist(b)
plt.show();
```

* The column view suggest that we need one of the first vector to be added to two times the second vector to get to point (0,3)

* Note that we are working in the *xy*-plane
* Forgetting for now the solution (0,3), if we took all the possible values (on the real line) for *x* and *y*, we would fill the whole plane
* Linear combinations of the two (column) vectors...
$$ \begin{bmatrix} 2 \\ -1 \end{bmatrix} $$
* ...and...
$$ \begin{bmatrix} -1 \\ 2 \end{bmatrix} $$
* ...fill ℝ2

* It's easy to see that these two vectors are not linear combinations of each other (they don't lie on the same line)
* If this is so (they are linearly independent) and linear combinations of them fill the plane we say they span the plane (ℝ2)

* It's also easy to imagine that the *xy*-plane is filled with (all the points are filled with) vectors, i.e. I can find any coordinate by drawing a vector to it
* All these vectors together can be called a **set**
* Let's call this set *W* and is equals ℝ2
* Later we will see that this vector space is a subspace of *V* = ℝ3
* We will also see that the vectors above span *W*, i.e. *W* = span(set of two vectors above)
* It will also be shown that this set of two vectors is a **basis** of *W* (they are linearly independent and they span *W*)
* ℝ2 is of **dimension** two (2) as the whole space can be represented by a linear combination of just two vectors
    * The basis vectors for ℝ2 are actually...
$$ \begin{bmatrix} 1 \\ 0 \end{bmatrix},\begin{bmatrix} 0 \\ 1 \end{bmatrix} $$
    * ...which we commonly call
$$ \hat { i } ,\hat { j }  $$

## The 3-space picture

$$ {3x}+{2y}-{z}=2 $$
$$ {x}-{2y}-{z}=3 $$
$$ {2x}+{y}-{z}={1} $$

```python
A_augm = Matrix([[3, 2, -1, 2], [1, -2, -1, 3], [2, 1, -1, 1]])
A_augm
```

```python
A_augm.rref()
```

```python
Image(filename = '3d.png')
```

```python

```


# Facility Location

**Originally Contributed by**: Mathieu Tanneau and Alexis Montoison

Benchmark instances: https://resources.mpi-inf.mpg.de/departments/d1/projects/benchmarks/UflLib/

```julia
using Random
using LinearAlgebra

using JuMP
import GLPK
using Plots
```

## Uncapacitated facility location

### Problem description

We are given
* $M=\{1, \dots, m\}$ clients
* $N=\{ 1, \dots, n\}$ sites where a facility can be built

**Decision variables**
Decision variables are split into two categories:
* Binary variable $y_{j}$ indicates whether facility $j$ is built or not
* Binary variable $x_{i, j}$ indicates whether client $i$ is assigned to facility $j$

**Objective**
The objective is to minimize the total cost of serving all clients.
This costs breaks down into two components:
* Fixed cost of building a facility.
In this example, this cost is $f_{j} = 1, \ \forall j$.
* Cost of serving clients from the assigned facility.
In this example, the cost $c_{i, j}$
of serving client $i$ from facility $j$
is the Euclidean distance between the two.

**Constraints**
* Each customer must be served by exactly one facility
* A facility cannot serve any client unless it is open

### MILP formulation

The problem can be formulated as the following MILP:

\begin{align}
\min_{x, y} \ \ \ &
\sum_{i, j} c_{i, j} x_{i, j} +
\sum_{j} f_{j} y_{j}\\
s.t. &
\sum_{j} x_{i, j} = 1, && \forall i \in M\\
& x_{i, j} \leq y_{j}, && \forall i \in M, j \in N\\
& x_{i, j}, y_{j} \in \{0, 1\}, && \forall i \in M, j \in N
\end{align}

where the first set of constraints ensures
that each client is served exactly once,
and the second set of constraints ensures
that no client is served from an unopened facility.

### Problem data

```julia
Random.seed!(314)

m = 12  # number of clients
n = 5  # number of facility locations

# Clients' locations
Xc = rand(m)
Yc = rand(m)

# Facilities' potential locations
Xf = rand(n)
Yf = rand(n)

# Fixed costs
f = ones(n);

# Distance
c = zeros(m, n)
for i in 1:m
    for j in 1:n
        c[i, j] = norm([Xc[i] - Xf[j], Yc[i] - Yf[j]], 2)
    end
end
```

```julia
# Display the data
scatter(Xc, Yc, label = "Clients", markershape=:circle, markercolor=:blue)
scatter!(Xf, Yf, label="Facility",
    markershape=:square, markercolor=:white, markersize=6,
    markerstrokecolor=:red, markerstrokewidth=2
)
```

### JuMP implementation

```julia
# Create a JuMP model
ufl = Model(GLPK.Optimizer)

# Variables
@variable(ufl, y[1:n], Bin);
@variable(ufl, x[1:m, 1:n], Bin);

# Each client is served exactly once
@constraint(ufl, client_service[i in 1:m],
    sum(x[i, j] for j in 1:n) == 1
);

# A facility must be open to serve a client
@constraint(ufl, open_facility[i in 1:m, j in 1:n],
    x[i, j] <= y[j]
)

# Objective
@objective(ufl, Min, f'y + sum(c .* x));
```

```julia
# Solve the uncapacitated facility location problem with GLPK
optimize!(ufl)
println("Optimal value: ", objective_value(ufl))
```

    Optimal value: 5.226417970467934

### Visualizing the solution

```julia
# The threshold 1e-5 ensure that edges between clients and facilities are drawn when x[i, j] ≈ 1.
x_ = value.(x) .> 1 - 1e-5
y_ = value.(y) .> 1 - 1e-5

# Display clients
p = scatter(Xc, Yc, markershape=:circle, markercolor=:blue, label=nothing)

# Show open facility
mc = [(y_[j] ? :red : :white) for j in 1:n]
scatter!(Xf, Yf,
    markershape=:square, markercolor=mc, markersize=6,
    markerstrokecolor=:red, markerstrokewidth=2,
    label=nothing
)

# Show client-facility assignment
for i in 1:m
    for j in 1:n
        if x_[i, j] == 1
           plot!([Xc[i], Xf[j]], [Yc[i], Yf[j]], color=:black, label=nothing)
        end
    end
end

display(p)
```

## Capacitated Facility location

### Problem formulation

The capacitated variant introduces a capacity constraint on each facility, i.e., clients have a certain level of demand to be served, while each facility only has finite capacity which cannot be exceeded.

Specifically, let
* $a_{i} \geq 0$ denote the demand of client $i$
* $q_{j} \geq 0$ denote the capacity of facility $j$

The capacity constraints then write
\begin{align}
\sum_{i} a_{i} x_{i, j} &\leq q_{j} y_{j} && \forall j \in N
\end{align}

Note that, if $y_{j}$ is set to $0$, the capacity constraint above automatically forces $x_{i, j}$ to $0$.

Thus, the capacitated facility location can be formulated as follows

\begin{align}
\min_{x, y} \ \ \ &
\sum_{i, j} c_{i, j} x_{i, j} +
\sum_{j} f_{j} y_{j}\\
s.t. &
\sum_{j} x_{i, j} = 1, && \forall i \in M\\
& \sum_{i} a_{i} x_{i, j} \leq q_{j} y_{j}, && \forall j \in N\\
& x_{i, j}, y_{j} \in \{0, 1\}, && \forall i \in M, j \in N
\end{align}

For simplicity, we will assume that there is enough capacity to serve the demand,
 i.e., there exists at least one feasible solution.

```julia
# Demands
a = rand(1:3, m);

# Capacities
q = rand(5:10, n);
```

```julia
# Display the data
scatter(Xc, Yc, label=nothing,
    markershape=:circle, markercolor=:blue, markersize= 2 .*(2 .+ a)
)

scatter!(Xf, Yf, label=nothing,
    markershape=:rect, markercolor=:white, markersize= q,
    markerstrokecolor=:red, markerstrokewidth=2
)
```

### JuMP implementation

```julia
# Create a JuMP model
cfl = Model(GLPK.Optimizer)

# Variables
@variable(cfl, y[1:n], Bin);
@variable(cfl, x[1:m, 1:n], Bin);

# Each client is served exactly once
@constraint(cfl, client_service[i in 1:m], sum(x[i, :]) == 1)

# Capacity constraint
@constraint(cfl, capacity, x'a .<= (q .* y))

# Objective
@objective(cfl, Min, f'y + sum(c .* x));
```

```julia
# Solve the problem
optimize!(cfl)
println("Optimal value: ", objective_value(cfl))
```

    Optimal value: 6.481890820500534

### Visualizing the solution

```julia
# The threshold 1e-5 ensure that edges between clients and facilities are drawn when x[i, j] ≈ 1.
x_ = value.(x) .> 1 - 1e-5
y_ = value.(y) .> 1 - 1e-5

# Display the solution
p = scatter(Xc, Yc, label=nothing,
    markershape=:circle, markercolor=:blue, markersize= 2 .*(2 .+ a)
)

mc = [(y_[j] ? :red : :white) for j in 1:n]
scatter!(Xf, Yf, label=nothing,
    markershape=:rect, markercolor=mc, markersize=q,
    markerstrokecolor=:red, markerstrokewidth=2
)

# Show client-facility assignment
for i in 1:m
    for j in 1:n
        if x_[i, j] == 1
            plot!([Xc[i], Xf[j]], [Yc[i], Yf[j]], color=:black, label=nothing)
            break
        end
    end
end

display(p)
```

## Further
* [Benders decomposition](https://github.com/JuliaOpt/JuMPTutorials.jl/blob/master/script/optimization_concepts/benders_decomposition.jl)
is a method of choice for solving facility location problems.
* Benchmark instances can be found
[here](https://resources.mpi-inf.mpg.de/departments/d1/projects/benchmarks/UflLib/).


## Differentiation and Derivatives
So far in this course, you've learned how to evaluate limits for points on a line. Now you're going to build on that knowledge and look at a calculus technique called *differentiation*. In differentiation, we use our knowledge of limits to calculate the *derivative* of a function in order to determine the rate of change at an individual point on a line.

Let's remind ourselves of the problem we're trying to solve, here's a function:

\begin{equation}f(x) = x^{2} + x\end{equation}

We can visualize part of the line that this function defines using the folllowing Python code:

```python
%matplotlib inline

# Here's the function
def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

Now, we know that we can calculate the average rate of change for a given interval on the line by calculating the slope for a secant line that connects two points on the line. For example, we can calculate the average change for the interval between x=4 and x=6 by dividing the change (or *delta*, indicated as Δ) in the value of *f(x)* by the change in the value of *x*:

\begin{equation}m = \frac{\Delta{f(x)}}{\Delta{x}} \end{equation}

The delta for *f(x)* is calculated by subtracting the *f(x)* values of our points, and the delta for *x* is calculated by subtracting the *x* values of our points; like this:

\begin{equation}m = \frac{f(x)_{2} - f(x)_{1}}{x_{2} - x_{1}} \end{equation}

So for the interval between x=4 and x=6, that's:

\begin{equation}m = \frac{f(6) - f(4)}{6 - 4} \end{equation}

We can calculate and plot this using the following Python:

```python
%matplotlib inline

def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the a values
x1 = 4
x2 = 6

# Get the corresponding f(x) values
y1 = f(x1)
y2 = f(x2)

# Calculate the slope by dividing the deltas
a = (y2 - y1)/(x2 - x1)

# Create an array of x values for the secant line
sx = [x1,x2]

# Use the function to get the y values
sy = [f(i) for i in sx]

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the interval points
plt.scatter([x1,x2],[y1,y2], c='red')

# Plot the secant line
plt.plot(sx,sy, color='magenta')

# Display the calculated average rate of change
plt.annotate('Average change =' + str(a),(x2, (y2+y1)/2))

plt.show()
```

The average rate of change for the interval between x=4 and x=6 is 11/1 (or simply 11), meaning that for every **1** added to *x*, *f(x)* increases by **11**. Put another way, if x represents time in seconds and f(x) represents distance in meters, the average rate of change for distance over time (in other words, *velocity*) for the 4 to 6 second interval is 11 meters-per-second.

So far, this is just basic algebra; but what if instead of the average rate of change over an interval, we want to calculate the rate of change at a single point, say, where x = 4.5?

One approach we could take is to create a secant line between the point at which we want the slope and another point on the function line that is infintesimally close to it. So close in fact that the secant line is actually a tangent that goes through both points. We can then calculate the slope for the secant line as before. This would look something like the graph produced by the following code:

```python
%matplotlib inline

def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the x1 point, arbitrarily 4.5
x1 = 4.5
y1 = f(x1)

# Set the x2 point, very close to x1
x2 = 5.000000001
y2 = f(x2)

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the point
plt.scatter(x1,y1, c='red')
plt.annotate('x' + str(x1),(x1,y1), xytext=(x1-0.5, y1+3))

# Approximate the tangent slope and plot it
m = (y2-y1)/(x2-x1)
xMin = x1 - 3
yMin = y1 - (3*m)
xMax = x1 + 3
yMax = y1 + (3*m)
plt.plot([xMin,xMax],[yMin,yMax], color='magenta')

plt.show()
```

## Calculating a Derivative
In the Python code above, we created the (almost) tangential secant line by specifying a point that is very close to the point at which we want to calculate the rate of change. This is adequate to show the line conceptually in the graph, but it's not a particularly generalizable (or accurate) way to actually calculate the line so that we can get the rate of change at any given point.

If only we knew of a way to calculate a point on the line that is as close as possible to point with a given *x* value.

Oh wait, we do! It's a *limit*.

So how do we apply a limit in this scenario? Well, let's start by examining our general approach to calculating slope in a little more detail.Our tried and tested approach is to plot a secant line between two points at different values of x, so let's plot an arbitrary (*x,y*) point, and then add an arbitrary amount to *x*, which we'll call *h*. Then we know that we can plot a secant line between (*x,f(x)*) and (*x+h,f(x+h)*) and find its slope.

Run the cell below to see these points:

```python
%matplotlib inline

def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the x point
x1 = 3
y1 = f(x1)

# set the increment
h = 3

# set the x+h point
x2 = x1+h
y2 = f(x2)

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the x point
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

# Plot the x+h point
plt.scatter(x2,y2, c='red')
plt.annotate('(x+h, f(x+h))',(x2,y2), xytext=(x2+0.5, y2))

plt.show()
```

As we saw previously, our formula to calculate slope is:

\begin{equation}m = \frac{\Delta{f(x)}}{\Delta{x}} \end{equation}

The delta for *f(x)* is calculated by subtracting the *f(x + h)* and *f(x)* values of our points, and the delta for *x* is just the difference between *x* and *x + h*; in other words, *h*:

\begin{equation}m = \frac{f(x + h) - f(x)}{h} \end{equation}

What we actually need is the slope at the shortest possible distance between x and x+h, so we're looking for the smallest possible value of *h*. In other words, we need the limit as *h* approaches 0.

\begin{equation}\lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

This equation is generalizable, and we can use it as the definition of a function to help us find the slope at any given value of *x* on the line, and it's what we call the *derivative* of our original function (which in this case is called *f*). This is generally indicated in *Lagrange* notation like this:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

You'll also sometimes see derivatives written in *Leibniz's* notation like this:

\begin{equation}\frac{d}{dx}f(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

***Note:*** *Some textbooks use **h** to symbolize the difference between **x0** and **x1**, while others use **Δx**. It makes no diffrerence which symbolic value you use.*

#### Alternate Form for a Derivative
The formula above shows the generalized form for a derivative. You can use the derivative function to get the slope at any given point, for example to get the slope at point *a* you could just plug the value for *a* into the generalized derivative function:

\begin{equation}f'(\textbf{a}) = \lim_{h \to 0} \frac{f(\textbf{a} + h) - f(\textbf{a})}{h} \end{equation}

Or you could use the alternate form, which is specific to point *a*:

\begin{equation}f'(a) = \lim_{x \to a} \frac{f(x) - f(a)}{x - a} \end{equation}

These are mathematically equivalent.

### Finding the Derivative for a Specific Point
It's easier to understand differentiation by seeing it in action, so let's use it to find the derivitive for a specific point in the function ***f***.

Here's the definition of function ***f***:

\begin{equation}f(x) = x^{2} + x\end{equation}

Let's say we want to find ***f'(2)*** (the derivative for ***f*** when ***x*** is 2); so we're trying to find the slope at the point shown by the following code:

```python
%matplotlib inline

def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the point
x1 = 2
y1 = f(x1)

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the point
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

plt.show()
```

Here's our generalized formula for finding a derivative at a specific point (*a*):

\begin{equation}f'(a) = \lim_{h \to 0} \frac{f(a + h) - f(a)}{h} \end{equation}

So let's just start by plugging our *a* value in:

\begin{equation}f'(\textbf{2}) = \lim_{h \to 0} \frac{f(\textbf{2} + h) - f(\textbf{2})}{h} \end{equation}

We know that ***f(x)*** encapsulates the equation ***x2 + x***, so we can rewrite our derivative equation as:

\begin{equation}f'(2) = \lim_{h \to 0} \frac{((2+h)^{2} + 2 + h) - (2^{2} + 2)}{h} \end{equation}

We can apply the distribution property to ***(2 + h)2*** using the rule that *(a + b)2 = a2 + b2 + 2ab*:

\begin{equation}f'(2) = \lim_{h \to 0} \frac{(4 + h^{2} + 4h + 2 + h) - (2^{2} + 2)}{h} \end{equation}

Then we can simplify 22 + 2 (22 is 4, plus 2 gives is 6):

\begin{equation}f'(2) = \lim_{h \to 0} \frac{(4 + h^{2} + 4h + 2 + h) - 6}{h} \end{equation}

We can combine like terms on the left side of the numerator to make things a little clearer:

\begin{equation}f'(2) = \lim_{h \to 0} \frac{(h^{2} + 5h + 6) - 6}{h} \end{equation}

Which combines even further to get rid of the *6*:

\begin{equation}f'(2) = \lim_{h \to 0} \frac{h^{2} + 5h}{h} \end{equation}

And finally, we can simplify the fraction:

\begin{equation}f'(2) = \lim_{h \to 0} h + 5 \end{equation}

To get the limit when *h* is approaching 0, we can use direct substitution for h:

\begin{equation}f'(2) = 0 + 5 \end{equation}

so:

\begin{equation}f'(2) = 5 \end{equation}

Let's draw a tangent line with that slope on our graph to see if it looks right:

```python
%matplotlib inline

def f(x):
    return x**2 + x

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the point
x1 = 2
y1 = f(x1)

# Specify the derivative we calculated above
m = 5

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the point
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

# Plot the tangent line using the derivative we calculated
xMin = x1 - 3
yMin = y1 - (3*m)
xMax = x1 + 3
yMax = y1 + (3*m)
plt.plot([xMin,xMax],[yMin,yMax], color='magenta')

plt.show()
```

### Finding a Derivative for Any Point
Now let's put it all together and define a function that we can use to find the derivative for any point in the ***f*** function:

Here's our general derivative function again:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

We know that ***f(x)*** encapsulates the equation ***x2 + x***, so we can rewrite our derivative equation as:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{((x+h)^{2} + x + h) - (x^{2} + x)}{h} \end{equation}

We can apply the distribution property to ***(x + h)2*** using the rule that *(a + b)2 = a2 + b2 + 2ab*:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{(x^{2} + h^{2} + 2xh + x + h) - (x^{2} + x)}{h} \end{equation}

Then we can use the distributive property to expand ***- (x2 + x)***, which is the same thing as *-1(x2 + x)*, to ***- x2 - x***:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{x^{2} + h^{2} + 2xh + x + h - x^{2} - x}{h} \end{equation}

We can combine like terms on the numerator to make things a little clearer:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{h^{2} + 2xh + h}{h} \end{equation}

And finally, we can simplify the fraction:

\begin{equation}f'(x) = \lim_{h \to 0} 2x + h + 1 \end{equation}

To get the limit when *h* is approaching 0, we can use direct substitution for h:

\begin{equation}f'(x) = 2x + 0 + 1 \end{equation}

so:

\begin{equation}f'(x) = 2x + 1 \end{equation}

Now we have a function for the derivative of ***f***, which we can apply to any *x* value to find the slope of the function at ***f(x***).

For example, let's find the derivative of ***f*** with an *x* value of 5:

\begin{equation}f'(5) = 2\cdot5 + 1 = 10 + 1 = 11\end{equation}

Let's use Python to define the ***f(x)*** and ***f'(x)*** functions, plot ***f(5)*** and show the tangent line for ***f'(5)***:

```python
%matplotlib inline

# Create function f
def f(x):
    return x**2 + x

# Create derivative function for f
def fd(x):
    return (2 * x) + 1

from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [f(i) for i in x]

# Set the point
x1 = 5
y1 = f(x1)

# Calculate the derivative using the derivative function
m = fd(x1)

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the point
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

# Plot the tangent line using the derivative we calculated
xMin = x1 - 3
yMin = y1 - (3*m)
xMax = x1 + 3
yMax = y1 + (3*m)
plt.plot([xMin,xMax],[yMin,yMax], color='magenta')

plt.show()
```

## Differentiability
It's important to realize that a function may not be *differentiable* at every point; that is, you might not be able to calculate the derivative for every point on the function line.

To be differentiable at a given point:
- The function must be *continuous* at that point.
- The tangent line at that point cannot be vertical
- The line must be *smooth* at that point (that is, it cannot take on a sudden change of direction at the point)

For example, consider the following (somewhat bizarre) function:

\begin{equation}
q(x) = \begin{cases}
  \frac{40,000}{x^{2}}, & \text{if } x < -4, \\
  (x^{2} -2) \cdot (x - 1), & \text{if } x \ne 0 \text{ and } x \ge -4 \text{ and } x < 8, \\
  (x^{2} -2), & \text{if } x \ne 0 \text{ and } x \ge 8
\end{cases}
\end{equation}

```python
%matplotlib inline

# Define function q
def q(x):
    if x != 0:
        if x < -4:
            return 40000 / (x**2)
        elif x < 8:
            return (x**2 - 2) * x - 1
        else:
            return (x**2 - 2)

# Plot output from function g
from matplotlib import pyplot as plt

# Create an array of x values
x = list(range(-10, -5))
x.append(-4.01)
x2 = list(range(-4,8))
x2.append(7.9999)
x2 = x2 + list(range(8,11))

# Get the corresponding y values from the function
y = [q(i) for i in x]
y2 = [q(i) for i in x2]

# Set up the graph
plt.xlabel('x')
plt.ylabel('q(x)')
plt.grid()

# Plot x against q(x)
plt.plot(x,y, color='purple')
plt.plot(x2,y2, color='purple')

plt.scatter(-4,q(-4), c='red')
plt.annotate('A (x= -4)',(-5,q(-3.9)), xytext=(-7, q(-3.9)))

plt.scatter(0,0, c='red')
plt.annotate('B (x= 0)',(0,0), xytext=(-1, 40))

plt.scatter(8,q(8), c='red')
plt.annotate('C (x= 8)',(8,q(8)), xytext=(8, 100))

plt.show()
```

The points marked on this graph are non-differentiable:
- Point **A** is non-continuous - the limit from the negative side is infinity, but the limit from the positive side ≈ -57
- Point **B** is also non-continuous - the function is not defined at x = 0.
- Point **C** is defined and continuous, but the sharp change in direction makes it non-differentiable.

## Derivatives of Equations

We've been talking about derivatves of *functions*, but it's important to remember that functions are just named operations that return a value. We can apply what we know about calculating derivatives to any equation, for example:

\begin{equation}\frac{d}{dx}(2x + 6)\end{equation}

Note that we generally switch to *Leibniz's* notation when finding derivatives of equations that are not encapsulated as functions; but the approach for solving this example is exactly the same as if we had a hypothetical function with the definition *2x + 6*:

\begin{equation}\frac{d}{dx}(2x + 6) = \lim_{h \to 0} \frac{(2(x+h) + 6) - (2x + 6)}{h} \end{equation}

After factoring out the* 2(x+h)* on the left and the *-(2x - 6)* on the right, this is:

\begin{equation}\frac{d}{dx}(2x + 6) = \lim_{h \to 0} \frac{2x + 2h + 6 - 2x - 6}{h} \end{equation}

We can simplify this to:

\begin{equation}\frac{d}{dx}(2x + 6) = \lim_{h \to 0} \frac{2h}{h} \end{equation}

Now we can factor *h* out entirely, so at any point:

\begin{equation}\frac{d}{dx}(2x + 6) = 2 \end{equation}

If you run the Python code below to plot the line created by the equation, you'll see that it does indeed have a constant slope of 2:

```python
%matplotlib inline
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10 to plot
x = list(range(1, 11))

# Use the function to get the y values
y = [(2*i) + 6 for i in x]

# Set up the graph
plt.xlabel('x')
plt.xticks(range(1,11, 1))
plt.ylabel('y')
plt.yticks(range(8,27, 1))
plt.grid()

# Plot the function
plt.plot(x,y, color='purple')

plt.show()
```

## Derivative Rules and Operations
When working with derivatives, there are some rules, or shortcuts, that you can apply to make your life easier.

### Basic Derivative Rules
Let's start with some basic rules that it's useful to know.

- If *f(x)* = *C* (where *C* is a constant), then *f'(x)* = 0

    This makes sense if you think about it for a second. No matter what value you use for *x*, the function returns the same constant value; so the graph of the function will be a horizontal line. There's no rate of change in a horiziontal line, so its slope is 0 at all points. This is true of any constant, including symbolic constants like *π* (pi).

    So, for example:

\begin{equation}f(x) = 6 \;\; \therefore \;\; f'(x) = 0 \end{equation}

    Or:

\begin{equation}f(x) = \pi \;\; \therefore \;\; f'(x) = 0 \end{equation}

- If *f(x)* = *Cg(x)*, then *f'(x)* = *Cg'(x)*

    This rule tells us that if a function is equal to a second function multiplied by a constant, then the derivative of the first function will be equal to the derivative of the second function multiplied by the same constant. For example:

\begin{equation}f(x) = 2g(x) \;\; \therefore \;\; f'(x) = 2g'(x) \end{equation}

- If *f(x)* = *g(x)* + *h(x)*, then *f'(x)* = *g'(x)* + *h'(x)*

    In other words, if a function is the sum of two other functions, then the derivative of the first function is the sum of the derivatives of the other two functions. For example:

\begin{equation}f(x) = g(x) + h(x) \;\; \therefore \;\; f'(x) = g'(x) + h'(x) \end{equation}

    Of course, this also applies to subtraction:

\begin{equation}f(x) = k(x) - l(x) \;\; \therefore \;\; f'(x) = k'(x) - l'(x) \end{equation}

As discussed previously, functions are just equations encapsulated as a named entity that return a value; and the rules can be applied to any equation. For example:

\begin{equation}\frac{d}{dx}(2x + 6) = \frac{d}{dx} 2x +  \frac{d}{dx} 6\end{equation}

So we can take advantage of the rules to make the calculation a little easier:

\begin{equation}\frac{d}{dx}(2x) = \lim_{h \to 0} \frac{2(x+h) - 2x}{h} \end{equation}

After factoring out the* 2(x+h)* on the left, this is:

\begin{equation}\frac{d}{dx}(2x) = \lim_{h \to 0} \frac{2x + 2h - 2x}{h} \end{equation}

We can simplify this to:

\begin{equation}\frac{d}{dx}(2x) = \lim_{h \to 0} \frac{2h}{h} \end{equation}

Which gives us:

\begin{equation}\frac{d}{dx}(2x) = 2 \end{equation}

Now we can turn our attention to the derivative of the constant 6 with respect to *x*, and we know that the derivative of a constant is always 0, so:

\begin{equation}\frac{d}{dx}(6) = 0\end{equation}

We add the two derivatives we calculated:

\begin{equation}\frac{d}{dx}(2x + 6) = 2 + 0\end{equation}

Which gives us our result:

\begin{equation}\frac{d}{dx}(2x + 6) = 2\end{equation}

### The Power Rule
The *power rule* is one of the most useful shortcuts in the world of differential calculus. It can be stated like this:

\begin{equation}f(x) = x^{n} \;\; \therefore \;\; f'(x) = nx^{n-1}\end{equation}

So if our function for *x* returns *x* to the power of some constant (which we'll call *n*), then the derivative of the function for *x* is *n* times *x* to the power of *n* - 1.

It's probably helpful to look at a few examples to see how this works:

\begin{equation}f(x) = x^{3} \;\; \therefore \;\; f'(x) = 3x^{2}\end{equation}

\begin{equation}f(x) = x^{-2} \;\; \therefore \;\; f'(x) = -2x^{-3}\end{equation}

\begin{equation}f(x) = x^{2} \;\; \therefore \;\; f'(x) = 2x\end{equation}

In each of these examples, the exponential of *x* in the function definition becomes the coefficient for *x* in the derivative definition, with the exponential is decremented by 1.

Here's a worked example to find the derivative of the following function:

\begin{equation}f(x) = x^{2}\end{equation}

So we start with the general derivative function:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

We can plug in our definition for *f*:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{(x + h)^{2} - x^{2}}{h} \end{equation}

Now we can factor out the perfect square binomial on the left:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{x^{2} + h^{2} + 2xh - x^{2}}{h} \end{equation}

The x2 terms cancel each other out so we get to:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{h^{2} + 2xh}{h} \end{equation}

Which simplifies to:

\begin{equation}f'(x) = \lim_{h \to 0} h + 2x \end{equation}

With *h* approaching 0, this is:

\begin{equation}f'(x) = 0 + 2x \end{equation}

So our answer is:

\begin{equation}f'(x) = 2x \end{equation}

Note that we could have achieved the same result by simply applying the power rule and transforming x2 to 2x1 (which is the same as 2x).

### The Product Rule
The product rule can be stated as:

\begin{equation}\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x) \end{equation}

OK, let's break that down. What it's saying is that the derivative of *f(x)* multiplied by *g(x)* is equal to the derivative of *f(x)* multiplied by the value of *g(x)* added to the value of *f(x)* multiplied by the derivative of *g(x*).

Let's see an example based on the following two functions:

\begin{equation}f(x) = 2x^{2} \end{equation}

\begin{equation}g(x) = x + 1 \end{equation}

Let's start by calculating the derivative of *f(x)*:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{2(x + h)^{2} - 2x^{2}}{h} \end{equation}

This factors out to:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{2x^{2} + 2h^{2} + 4xh - 2x^{2}}{h} \end{equation}

Which when we cancel out the 2x2 and -2x2 is:

\begin{equation}f'(x) = \lim_{h \to 0} \frac{2h^{2} + 4xh}{h} \end{equation}

Which simplifies to:

\begin{equation}f'(x) = \lim_{h \to 0} 2h + 4x \end{equation}

With *h* approaching 0, this is:

\begin{equation}f'(x) = 4x \end{equation}

Now let's look at *g'(x)*:

\begin{equation}g'(x) = \lim_{h \to 0} \frac{(x + h) + 1 - (x + 1)}{h} \end{equation}

We can just remove the brackets on the left and factor out the *-(x + 1)* on the right:

\begin{equation}g'(x) = \lim_{h \to 0} \frac{x + h + 1 - x - 1}{h} \end{equation}

Which can be cleaned up to:

\begin{equation}g'(x) = \lim_{h \to 0} \frac{h}{h} \end{equation}

Enabling us to factor *h* out completely to give a constant derivative of *1*:

\begin{equation}g'(x) = 1 \end{equation}

So now we can calculate the derivative for the product of these functions by plugging the functions and the derivatives we've calculated for them into the product rule equation:

\begin{equation}\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x) \end{equation}

So:

\begin{equation}\frac{d}{dx}[f(x)g(x)] = (4x \cdot (x + 1)) + (2x^{2} \cdot 1) \end{equation}

Which can be simplified to:

\begin{equation}\frac{d}{dx}[f(x)g(x)] = (4x^{2} + 4x) + 2x^{2} \end{equation}

Which can be further simplified to:

\begin{equation}\frac{d}{dx}[f(x)g(x)] = 6x^{2} + 4x \end{equation}

### The Quotient Rule
The *quotient rule* applies to functions that are defined as a quotient of one expression divided by another; for example:

\begin{equation}r(x) = \frac{s(x)}{t(x)} \end{equation}

In this situation, you can apply the following quotient rule to find the derivative of *r(x)*:

\begin{equation}r'(x) = \frac{s'(x)t(x) - s(x)t'(x)}{[t(x)]^{2}} \end{equation}

Here are our definitions for *s(x)* and *t(x)*:

\begin{equation}s(x) = 3x^{2} \end{equation}

\begin{equation}t(x) = 2x\end{equation}

Let's start with *s'(x)*:

\begin{equation}s'(x) = \lim_{h \to 0} \frac{3(x + h)^{2} - 3x^{2}}{h} \end{equation}

This factors out to:

\begin{equation}s'(x) = \lim_{h \to 0} \frac{3x^{2} + 3h^{2} + 6xh - 3x^{2}}{h} \end{equation}

Which when we cancel out the 3x2 and -3x2 is:

\begin{equation}s'(x) = \lim_{h \to 0} \frac{3h^{2} + 6xh}{h} \end{equation}

Which simplifies to:

\begin{equation}s'(x) = \lim_{h \to 0} 3h + 6x \end{equation}

With *h* approaching 0, this is:

\begin{equation}s'(x) = 6x \end{equation}

Now let's look at *t'(x)*:

\begin{equation}t'(x) = \lim_{h \to 0} \frac{2(x + h) - 2x}{h} \end{equation}

We can just factor out the *2(x + h)* on the left:

\begin{equation}t'(x) = \lim_{h \to 0} \frac{2x + 2h - 2x}{h} \end{equation}

Which can be cleaned up to:

\begin{equation}t'(x) = \lim_{h \to 0} \frac{2h}{h} \end{equation}

Enabling us to factor *h* out completely to give a constant derivative of *2*:

\begin{equation}t'(x) = 2 \end{equation}

So now we can calculate the derivative for the quotient of these functions by plugging the function definitions and the derivatives we've calculated for them into the quotient rule equation:

\begin{equation}r'(x) = \frac{(6x \cdot 2x) - (3x^{2} \cdot 2)}{[2x]^{2}} \end{equation}

We can factor out the numerator terms like this:

\begin{equation}r'(x) = \frac{12x^{2} - 6x^{2}}{[2x]^{2}} \end{equation}

Which can then be combined:

\begin{equation}r'(x) = \frac{6x^{2}}{[2x]^{2}} \end{equation}

The denominator is [2x]2 (note that this is different from 2x2. [2x]2 is 2x • 2x, whereas  2x2 is 2 • x2):

\begin{equation}r'(x) = \frac{6x^{2}}{4x^{2}} \end{equation}

Which simplifies to:

\begin{equation}r'(x) = 1\frac{1}{2} \end{equation}

So the derivative of *r(x)* is 1.5.

### The Chain Rule

The *chain rule* takes advantage of the fact that equations can be encapsulated as functions, and since functions contain equations, it's possible to nest one function within another.

For example, consider the following function:

\begin{equation}u(x) = 2x^{2} \end{equation}

We could view the definition of *u(x)* as a composite of two functions,; an *inner* function that calculates x2, and an *outer* function that multiplies the result of the inner function by 2.

\begin{equation}u(x) = \widehat{\color{blue}2\color{blue}(\underline{\color{red}x^{\color{red}2}}\color{blue})} \end{equation}

To make things simpler, we can name these inner and outer functions:

\begin{equation}i(x) = x^{2} \end{equation}

\begin{equation}o(x) = 2x \end{equation}

Note that *x* indicates the input for each function. Function *i* takes its input and squares it, and function *o* takes its input and multiplies it by 2. When we use these as a composite function, the *x* value input into the outer function will be the output from the inner function.

Let's take a look at how we can apply these functions to get back to our original *u* function:

\begin{equation}u(x) = o(i(x)) \end{equation}

So first we need to find the output of the inner *i* function so we can use at as the input value for the outer *o* function. Well, that's easy, we know the definition of *i* (square the input), so we can just plug it in:

\begin{equation}u(x) = o(x^{2}) \end{equation}

We also know the definition for the outer *o* function (multiply the input by 2), so we can just apply that to the input:

\begin{equation}u(x) = 2x^{2} \end{equation}

OK, so now we know how to form a composite function. The *chain rule* can be stated like this:

\begin{equation}\frac{d}{dx}[o(i(x))] = o'(i(x)) \cdot i'(x)\end{equation}

Alright, let's start by plugging the output of the inner *i(x)* function in:

\begin{equation}\frac{d}{dx}[o(i(x))] = o'(x^{2}) \cdot i'(x)\end{equation}

Now let's use that to calculate the derivative of *o*, replacing each *x* in the equation with the output from the *i* function (*x2*):

\begin{equation}o'(x) = \lim_{h \to 0} \frac{2(x^{2} + h) - 2x^{2}}{h} \end{equation}

This factors out to:

\begin{equation}o'(x) = \lim_{h \to 0} \frac{2x^{2} + 2h - 2x^{2}}{h} \end{equation}

Which when we cancel out the 2x2 and -2x2 is:

\begin{equation}o'(x) = \lim_{h \to 0} \frac{2h}{h} \end{equation}

Which simplifies to:

\begin{equation}o'(x) = 2 \end{equation}

Now we can calculate *i'(x)*. We know that the definition of *i(x)* is x2, and we can use the power rule to determine that *i'(x)* is therefore 2x.

So our equation at this point is:

\begin{equation}\frac{d}{dx}[o(i(x))] = 2 \cdot 2x\end{equation}

Which is:

\begin{equation}\frac{d}{dx}[o(i(x))] = 4x\end{equation}

Commonly, the chain rule is stated using a slighly different notation that you may find easier to work with. In this case, we can take our equation:

\begin{equation}\frac{d}{dx}[o(i(x))] = o'(i(x)) \cdot i'(x)\end{equation}

and rewrite it as

\begin{equation}\frac{du}{dx} = \frac{do}{di}\frac{di}{dx}\end{equation}

We can then complete the calculations like this:

\begin{equation}\frac{du}{dx} = 2 \cdot 2x = 4x\end{equation}


# Example: Discrete Probability Distributions

```python
import numpy as np
import matplotlib.pyplot as plt
```

### Bernoulli Distribution
$P(X) = p$

```python
p = np.linspace(0, 1, 100)
var = p*(1-p)

plt.plot(p, var)
plt.title('variance vs p')
plt.show()
```

### Binomial Distribution
$p(X=k) = \binom{n}{k}p^kq^{n-k}$

```python
from utilities import binom_pmf
```

```python
# PMF

n = 10
k_range = np.arange(0, n+1)

for p in np.arange(.1, 1., .1):
    out = []
    for k in k_range:
        out.append(binom_pmf(k, n, p))

    plt.plot(k_range, out, label='p={}'.format(p.round(1)))

plt.xlabel('k')
plt.ylabel('p')

plt.legend()
plt.show()
```

```python
from utilities import binom_sampling
```

```python
# Sampling

for p in np.arange(.1, 1., .2):
    plt.hist(binom_sampling(10, p), bins=10, label='p={}'.format(p.round(1)))
plt.legend()
plt.show()
```

### Poisson Distribution
\begin{equation}
Y = \frac{\lambda^k}{k!}e^{-\lambda}
\end{equation}

```python
from utilities import poisson_pmf
```

```python
X = np.linspace(0, 10, 11)
```

```python
# PMF
for i, l in enumerate([1, 2, 3, 5, 7, 10]):
    Y = poisson_pmf(l, X)
    plt.plot(Y, label='lambda = {}'.format(l))
plt.legend()
plt.show()
```

```python
from utilities import poisson_sampling
```

```python
# sampling

n = 1000
for i, l in enumerate([.001, .002, .003, .005, .007, .01]):
    out = poisson_sampling(n, l)
    plt.hist(out, bins=20, label='lambda = {}'.format(round(l*n)))

plt.legend()
plt.show()
```

### Geometric Distribution

```python
from utilities import geometric_pmf, geometric_sampling
```

```python
# PMF

k = np.arange(1, 100)

for p in np.arange(.1, 1., .1):
    out = geometric_pmf(k, p)
    plt.plot(out[:5], label='p = {}'.format(round(p, 1)))

plt.legend()
plt.xlabel('k')
plt.show()
```

```python
# sampling

for p in np.arange(.1, 1., .1):
    plt.hist(geometric_sampling(p), label='p={}'.format(round(p, 1)))

plt.legend()
plt.xlabel('k')
plt.show()
```


For the Ronbrock method, we need to solve a linear system of the form
$$
M_{ij}x_{j}=b_{i} \;,
$$
with M a square matrix (repeated indecies imply summation).

Such systems are soved by (among  other methods) the so-called LU factorization (or decomposition),
where you decompose $M_{ij}=L_{ik}U_{kj}$ with $L_{i, j>i}=0$, $L_{i, j=i}=1$, $U_{i,j<i}=0$.

That is if $M$ is $N \times N$ matrix, L,U are defined as
\begin{align}
    &L=\left( \begin{matrix}
    1 & 0 & 0 & 0 & \dots &0 & 0 \\
    L_{2,1} & 1 & 0 & 0 & \dots &0 & 0\\
    L_{3,1} & L_{3,2} & 1 & 0 & \dots &0 & 0 \\
    \vdots & \vdots & \vdots & \ddots & \dots & \vdots & \vdots \\
    L_{N-1, 1} & L_{N-1, 2} & L_{N-1, 3} & L_{N-1, 4} & \dots & 1 & 0 \\
    L_{N, 1} & L_{N, 2} & L_{N, 3} & L_{N, 4} & \dots & L_{N, N-1} & 1 \\
    \end{matrix}\right) \;, \\
    %
    &U=\left( \begin{matrix}
    U_{1,1} & U_{1,2} & U_{1,3} & U_{1,4} & \dots & U_{1,N-1} & U_{1,N} \\
    0 & U_{2,2} & U_{2,3} & U_{2,4} & \dots & U_{2,N-1} & U_{2,N}\\
    0 & 0 & U_{3,3} & U_{3,4} & \dots & U_{3,N-1} & U_{3,N} \\
    \vdots & \vdots & \vdots & \ddots & \dots & \vdots & \vdots \\
    0 & 0 & 0 &0 & \dots & U_{N-1,N-1} & U_{N-1,N} \\
    0 & 0 & 0& 0 & \dots & 0 &U_{N,N} \\
    \end{matrix}\right)
    %
\end{align}

Then we have in general $M_{i, j} = \sum_{k=1}^{i}L_{i,k}U_{k,j}$. Since
$L_{i, k \geq i}=0$ and $U_{k>j,j}=0$, the sum runs up to $j$ if $i \geq j$ and
$i$ if $i \leq j$  (for $i=j$ then both are correct). That is

$$
M_{i, j \geq i} = \sum_{k=1}^{i-1}L_{i,k}U_{k,j}+ U_{i,j} \Rightarrow
U_{i,j }=M_{i,j } - \sum_{k=1}^{i-1}L_{i,k}U_{k,j }\; , \;\;\; j \geq i \\[0.5cm]
M_{i, j \leq i} = \sum_{k=1}^{j-1}L_{i,k}U_{k,j} +L_{i,j}U_{j,j} \Rightarrow
L_{i,j}=\left( M_{i,j} - \sum_{k=1}^{j-1}L_{i,k}U_{k,j} \right) U_{jj}^{-1} , \;\;\; j \leq i
$$

Since $U$ and $L$ are triangular matrices, we can solve these two systems sequentially
$$
L_{i,k}y_{k}=b_{k} \\
U_{k,j}x_{j}=y_{k},
$$

since
$$
y_1 = b_{1} \\
L_{2,1}y_{1}+y_{2}=b_{2} \Rightarrow y_{2}=b_{2}-L_{2,1}y_{1} \\
\vdots \\
y_{i}=b_{i} - \sum_{j=1}^{i-1}L_{i,j}y_{j}
$$

and

$$
U_{N,N}x_{N}=y_{N} \Rightarrow x_{N}=y_{N}/U_{N,N}\\
U_{N-1,N}x_{N}+U_{N-1,N-1}x_{N-1}=y_{N-1} \Rightarrow x_{N-1}=\left(y_{N-1} -U_{N-1,N}x_{N} \right)/U_{N-1,N-1} \\
\vdots \\
x_{i}=\dfrac{y_{i} -\displaystyle\sum_{j=i+1}^{N} U_{i,j}x_{j}  }{U_{i,i}}
$$

Since $U_{i,i}$  appears to denominator, if the diagonal terms of $U$ are small (or god forbid they vanish), we would have a problem.

To solve this problem we do $LUP$ decomposition, where $L \; U=P \; M$ with $P$ a permutation matrix so that the diagonal of $U$ has the dominant components in each row.

Then solving $M x =b$ is equavalent to solving $\left( P \; M \right) x =P \; b$ with LU decomposition of
$P \; M$. That is x solves both systems (no need to permute x).

There is a clever way to make the docomposition faster. This is by initializing
$L=1_{N \times N}$ and $U=M$. Then we have the follwing algorithm for LU decomposition without
pivoting:

```bash

Input: M, N
#M: matrix
#N: size of M

#initialize U
U=M
#initialize L
L=Unit(N,N)

    for k in [2,...,N] do
        for i in [k,...,N] do
            L[i][k-1]=U[i][k-1]/U[k-1][k-1]
            for j in [k-1,...,N] do
                U[i][j]=U[i][j]-L[i][k-1]*U[k-1][j]
            done
        done
    done
```

I will not write the algorithm including pivoting, as the code in python will not be different.

```python
import numpy as np
```

```python
def Abs(x):
    '''
    Input x  and you'll get the abs.
    '''
    #Not the best way, but it is an easy way to not use numpy
    return (x**2)**0.5

```

```python
def ind_max(row,N):
    '''
    Find the index of the maximum of a list (row) of lentgth N.
    '''
    _in=0
    _max=row[0]
    i=0
    while i<N:#the end of the row should be included (convension in how I use LUP..)
        if row[i]>_max:
            _max=row[i]
            _in=i
        i+=1

    return _in
```

```python
def Sum(List,N):
    '''
    Calculates the sum of a List of size N
    '''
    s=0
    for i in range(N):
        s+=List[i]
    return s
```

```python
def index_swap(A,index_1,index_2):
    '''
        index_swap takes an array and interchanges
         A[index_1] with A[index_2].

         Example:
             A=[0,1,2,3]
             index_swap(A,0,2)
             A
             >>[2, 1, 0, 3]
    '''

    tmp=A[index_1]
    A[index_1]=A[index_2]
    A[index_2]=tmp

def apply_permutations_vector(A,P,N):
    '''
    Applies the permutations given by P from LUP
    to a list A of length N, and returns the result.
    Example:
    A=[1,2,5,8,3]
    P=[2,4,0,3,1]

    apply_permutations_vector(A,P,5)
    >>[5, 3, 1, 8, 2]
    '''
    #that is, you make a list like this (P basically gives you the indices of A):)

    Ap=[A[ P[i] ] for i in range(N)]

    return Ap

def apply_permutations_matrix(M,P,N):
    '''
    Applies the permutations given by P from LUP
    to a N*N array M of length N, and returns the result.

    M=[
    [ 0,  2,  2 , 3 , 5],
    [-3, -1,  1 , 5 , 9],
    [ 1, -1,  1 , 4 , 7],
    [ 1, -1,  1 , 0 , 2],
    [ 1, -1,  1 , 0 , 3]
    ]

    P=[2,0,1,4,3]

    apply_permutations_matrix(M,P,5)
    >>[
      [ 1, -1, 1, 4, 7],
      [ 0,  2, 2, 3, 5],
      [-3, -1, 1, 5, 9],
      [ 1, -1, 1, 0, 3],
      [ 1, -1, 1, 0, 2]
      ]
    '''
    Mp=[ [M[ P[i] ][j]for j in range(N)] for i in range(N) ]

    return Mp
```

```python
def LUP(M,N,_tiny=1e-20):
    U=[  [ M[i][j] for j in range(N)] for i in range(N) ]
    L=[  [ 0 if i!=j else 1 for j in range(N)] for i in range(N) ]
    #this is the "permutation vector". if it is e.g. [2 1 0 3] it means you make 0<->2
    P=[  i for i in range(N) ]

    for k in range(1,N):
        for i in range(k,N):
            #find the index of the maximum in column
            _col=[Abs(U[_r][k-1]) for _r in range(k-1,N)]

            #find the index of the maximum of _col
            # notice that the length of _col is N-(k-1)
            len_col=N-(k-1)
            pivot=ind_max( _col ,len_col) + k - 1 #convert the index of _col (it has a length of len_col) to the index of  a row of U

            ##################################################
            #this was in LU_julia (instead of "<_tiny"  it had "== 0").
            #if you remove it, then you get a lot of infinities
            #it has to do with the fact that if U[pivot][k-1] <_tiny , then U[k-1][k-1] will be a zero,
            #L[i][k-1] explodes.
            #You are allowed to skip this i, then, because if U[pivot][k-1] <_tiny , then all U[i][k-1] are small!
            #Check that this is true by  uncommenting print(_col)
            if Abs(U[pivot][k-1]) < _tiny  :
                #print(_col)
                break
            ###################################################
            #if the maximum is not at k-1, swap!
            if pivot != k-1 :
                # Permute rows k-1 and pivot in U

                index_swap(P,k-1,pivot)

                tmpU=[U[k-1][_r] for _r in range(k-1,N)]

                #print(U)
                for _r in range(k-1,N):
                    U[k-1][_r]=U[pivot][_r]
                #print(U)
                for _r in range(k-1,N):
                    U[pivot][_r]=tmpU[_r-(k-1)]#again we have to convert the index of tmpU
                #print(U)
                #print("=========================")
                tmpL=[L[k-1][_r] for _r in range(k-1)]
                #print(L)
                for _r in range(k-1):
                    L[k-1][_r]=L[pivot][_r]
                #print(L)
                for _r in range(k-1):
                    L[pivot][_r]=tmpL[_r]

                #print(L)
                #print("========================")

            L[i][k-1]=U[i][k-1]/U[k-1][k-1]

            for j in range(k-1,N):
                U[i][j]=U[i][j]-L[i][k-1]*U[k-1][j]

    return L,U,P

```

```python
def Solve_LU(L,U,P,b,N):
    '''
    This solves P*M*x=P*b (x is also the solution to M*x=b)
    Input:
    L,U,P= LUP decomposition of M. with P*M=L*U
    b=the right hand side of the equation
    N=the number of equations
    '''
    #I know that this uses more memory than what it needs, but I want to keep b as is.
    #(in C++ I will make it a bit beter ;) )
    b=apply_permutations_vector(b,P,N)
    d=[0 for i in range(N) ]
    x=[0 for i in range(N) ]

    d[0]=b[0]
    for i in range(1,N):
        d[i]=b[i]-Sum(  [L[i][j]*d[j] for j in range(i)],i )

    x[N-1]  = d[N-1]/U[N-1][N-1]
    for i in range(N-2,-1,-1):
        x[i]=(d[i]-Sum( [U[i][j]*x[j] for j in  range(i+1,N)],N-(i+1) )  )/U[i][i]
        print( Sum( [U[i][j]*x[j] for j in  range(i+1,N)],N-(i+1) )  )

    b=apply_permutations_vector(b,P,N)#return b as it was
    return x
```

## tests

```python
#check if Solve_LU works

if True:

    NT=5000#NT tests
    N=12#N*N matrices
    testSol=[0 for i in range(NT)]

    for i in range(NT):

        #M=np.random.randint(-3,3,size=[N,N])
        b=np.random.rand(N)*13.-6.5
        M=np.random.rand(N,N)*4-2
        L,U,P=LUP(M,N)
        x=Solve_LU(L,U,P,b,N)
        testSol[i]=np.array( np.dot(M,x))-np.array(b)

    print(np.max(testSol))
```

    7.175948724125192e-11

```python
from scipy.linalg import lu_factor,lu_solve,lu

```

```python
#check LUP against numpy.
#in test I will have the maximum difference between my L,U with what np.lu returns,
#and the difference between my L*U-P*M. So, test should be an array with small numbers!

#even when I get difference with numpy it is not important, because the decomposition is still correct
#(no nan or inf)!!!!

#change to True to run tests
if True:

    NT=5000#NT tests
    N=12#N*N matrices
    testL=[0 for i in range(NT)]
    testU=[0 for i in range(NT)]
    testM=[0 for i in range(NT)]

    for i in range(NT):

        #M=np.random.randint(-3,3,size=[N,N])
        M=np.random.rand(N,N)*4-2
        L,U,P=LUP(M,N)
        Ps,Ls,Us=lu(M)

        testU[i]=np.max(np.array(U)-Us)
        testL[i]=np.max(np.array(L)-Ls)

        testM[i]=np.max(np.dot(L,U)- np.array(apply_permutations_matrix(M,P,N) ))
        if testL[i] > 1e-5:
            #print(np.array(L))
            #print(Ls)
            #print([U[_t][_t] for _t in range(N)])
            print(testM[i])
            pass

    print(np.max(testU) , np.max(testL) , np.max(testM))
```

    4.218847493575595e-14 3.7136960173711486e-14 2.220446049250313e-15

```python

```


```python
import sympy as sp
sp.init_printing()
from cymbol import Cymbol
```

```python
E_T = Cymbol(r'E_{\mathrm{T}}', codename='E_T_', real=True, nonnegative=True)
S_T = Cymbol(r'S_{\mathrm{T}}', codename='S_T_', real=True, nonnegative=True)
r_T = Cymbol(r'r_{\mathrm{T}}', codename='r_T_', real=True, nonnegative=True)
c_T = Cymbol(r'c_{\mathrm{T}}', codename='c_T_', real=True, nonnegative=True)
E_N = Cymbol(r'E_{\mathrm{N}}', codename='E_N_', real=True, nonnegative=True)
S_N = Cymbol(r'S_{\mathrm{N}}', codename='S_N_', real=True, nonnegative=True)
r_N = Cymbol(r'r_{\mathrm{N}}', codename='r_N_', real=True, nonnegative=True)
c_N = Cymbol(r'c_{\mathrm{N}}', codename='c_N_', real=True, nonnegative=True)
eta = Cymbol(r'\eta', codename='eta_', real=True, nonnegative=True)
r = Cymbol(r'r', codename='r_', real=True, nonnegative=True)
```

```python
omega_T = Cymbol(r'\omega_{\mathrm{T}}', codename='omega_T_', real=True, nonnegative=True)
omega_N = Cymbol(r'\omega_{\mathrm{N}}', codename='omega_N_', real=True, nonnegative=True)
```

```python
Y_T = Cymbol(r'Y_{\mathrm{T}}', codename='Y_T_', real=True, nonnegative=True)
Y_N = Cymbol(r'Y_{\mathrm{N}}', codename='Y_N_', real=True, nonnegative=True)
```

```python
S_NT = sp.sqrt(S_N*S_T)
c_NT = sp.sqrt(c_N*c_T)
```

```python
omega_NT = 1 - sp.sqrt((1-omega_N)*(1-omega_T))
```

```python
phi_N = (1 - omega_N)**c_N * S_N / (r+1) * (Y_N / S_N)**(r+1)
phi_N
```

```python
phi_T = (1 - omega_T)**c_T * S_T / (r+1) * (Y_T / S_T)**(r+1)
phi_T
```

```python
phi_NT = (1 - omega_NT)**c_NT * S_NT / (r+1) * ((Y_N + Y_T)/(S_NT))**(r+1)
phi_NT
```

```python
phi = (1 - eta)*(phi_N + phi_T) + eta*phi_NT
```

```python
dot_omega_N = phi.diff(Y_N)
dot_omega_T = phi.diff(Y_T)
```

```python
sp.simplify(dot_omega_N.subs(eta, 0))
```

```python
sp.simplify(dot_omega_T.subs(eta, 0))
```

```python
sp.simplify(dot_omega_N.subs(eta, 1))
```

```python
sp.simplify(dot_omega_T.subs(eta, 1))
```

```python

```


# Rate of Change
Functions are often visualized as a line on a graph, and this line shows how the value returned by the function changes based on changes in the input value.

## Linear Rate of Change

For example, imagine a function that returns the number of meters travelled by a cyclist based on the number of seconds that the cyclist has been cycling.

Here is such a function:

\begin{equation}q(x) = 2x + 1\end{equation}

We can plot the output for this function for a period of 10 seconds like this:

```python
%matplotlib inline

def q(x):
    return 2*x + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.xticks(range(0,11, 1))
plt.yticks(range(0, 22, 1))
plt.grid()

# Plot x against q(x)
plt.plot(x,q(x), color='green')

plt.show()
```

It's clear from the graph that ***q*** is a *linear* function that describes a slope in which distance increases at a constant rate over time. In other words, the cyclist is travelling at a constant speed.

But what speed?

Speed, or more technically, velocity is a measure of change - it measures how the distance travelled changes over time (which is why we typically express it as a unit of distance per a unit of time, like *miles-per-hour* or *meters-per-second*). So we're looking for a way to measure the change in the line created by the function.

The change in values along the line define its *slope*, which we know from a previous lesson is represented like this:

\begin{equation}m = \frac{\Delta{y}}{\Delta{x}} \end{equation}

We can calculate the slope of our function like this:

\begin{equation}m = \frac{q(x_{2}) - q(x_{1})}{x_{2} - x_{1}} \end{equation}

So we just need two ordered pairs of ***x*** and ***q(x)*** values from our line to apply this equation.

- After 1 second, ***x*** is 1 and ***q***(1) = **3**.
- After 10 seconds, ***x*** is 10 and ***q***(10) = 21.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{21 - 3}{10 - 1} \end{equation}

This is the same as:

\begin{equation}m = \frac{18}{9} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{2}{1} \end{equation}

So our rate of change is 2/1 or put another way, the cyclist is travelling at 2 meters-per-second.

## Average Rate of Change
OK, let's look at another function that calculates distance travelled for a given number of seconds:

\begin{equation}r(x) = x^{2} + x\end{equation}

Let's take a look at that using Python:

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

plt.show()
```

This time, the function is not linear. It's actually a quadratic function, and the line from 0 seconds to 10 seconds shows an exponential increase; in other words, the cyclist is *accelerating*.

Technically, acceleration itself is a measure of change in velocity over time; and velocity, as we've already discussed, is a measure of change in distance over time. So measuring accelleration is pretty complex, and requires *differential calculus*, which we're going to cover shortly. In fact, even just measuring the velocity at a single point in time requires differential calculus; but we can use algebraic methods to calculate an *average* rate of velocity for a given period shown in the graph.

First, we need to define a *secant* line that joins two points in our exponential arc to create a straight slope. For example, a secant line for the entire 10 second time span would join the following two points:

- 0, ***r***(0)
- 10, ***r***(10)

Run the following Python code to visualize this line:

```python
%matplotlib inline

def r(x):
    return (x)**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([0,10])

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.show()
```

Now, because the secant line is straight, we can apply the slope formula we used for a linear function to calculate the average velocity for the 10 second period:

- At 0 seconds, ***x*** is 0 and ***r***(0) = **0**.
- At 10 seconds, ***x*** is 10 and ***r***(10) = 110.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{110 - 0}{10 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{110}{10} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{11}{1} \end{equation}

So our rate of change is 11/1 or put another way, the cyclist is travelling at an average velocity of 11 meters-per-second over the 10-second period.

Of course, we can measure the average velocity between any two points on the exponential line. Use the following Python code to show the secant line for the period between 2 and 7 seconds, and calculate the average velocity for that period

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([2,7])

# Calculate rate of change
x1 = s[0]
x2 = s[-1]
y1 = r(x1)
y2 = r(x2)
a = (y2 - y1)/(x2 - x1)

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.annotate('Average Velocity =' + str(a) + ' m/s',((x2+x1)/2, (y2+y1)/2))

plt.show()

```

```python

```

MathsToML/Module02-Derivatives and Optimization/02-01-Rate of Change.ipynb


# Discrete Random Walk

Consider a discrete random walk.

Let $X_t = X_{t+1} + \epsilon_t$, with $p(\epsilon_t = 1) = 0.5$ and $p(\epsilon_t = -1) = 0.5$ be the stepwise random walk. Each possible arrival point $(t,X_{t})$ of the random walk is reached through a unique number of upsteps and downsteps, but the sequence of upsteps and downsteps does not matter, so that there may be many different paths. Since up-steps and down-steps are equally likely, the probability of any path of length $t$, corresponding to a specific sequence of innovations $\{\epsilon_t\}$, is $p(\{\epsilon_t\}) = \frac{1}{2^t}$. The point $(t,X_t)$ can be expressed in terms of the number of upsteps ($y$) and downsteps ($x$) (which amounts to a $45\deg$ coordinate rotation).

\begin{equation}
X_t = y-x\\
t = y+x
\end{equation}

From which follows:
\begin{equation}
x = \frac{t-X_t}{2}\\
y = \frac{t+X_t}{2}
\end{equation}

The total amount of paths that lead from $(0,0)$ to $(t,X_t)$ are given by all possible sequences of $x$ downsteps and $y$ upsteps.

\begin{equation}
n(t,X_t) = \left(\begin{array}{c}x+y\\x\end{array}\right) = \left(\begin{array}{c}x+y\\y\end{array}\right) =  \left(\begin{array}{c}t\\\frac{t+X_t}{2}\end{array}\right) = \left(\begin{array}{c}t\\\frac{t-X_t}{2}\end{array}\right)
\end{equation}

If $\frac{t\pm X_t}{2}$ are integer and 0 otherwise.

Given that $X_t = \sum^t_i \epsilon_t$, the probability $p(X_t = s)$ is:

\begin{equation}
p(X_t = s) = \sum^t_{\{\epsilon_t\}}\prod^t_ip(\epsilon_i)
\end{equation}

Where the sum is over all sequences of $\{\epsilon_t\}$ so that $\sum^t_i \epsilon_t = s$, and it follows:

\begin{equation}
p(s) = \left(\begin{array}{c}t\\\frac{t-X_t}{2} \end{array}\right)\frac{1}{2^t}
\end{equation}

```python
import numpy as np
from numpy import random as rnd
import matplotlib.pyplot as plt

def random_walk(t):
    """
    returns a single random walk
    """
    return np.cumsum([0]+[rnd.choice([-1,1]) for i in range(t)])

def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return int(res)

def binomial(n,k):
    if k >= 0 and n >= 0 and k <= n:
        return int(factorial(n)/(factorial(n-k)*factorial(k)))
    else:
        return 0

def p_s(s,t):

    """
    Probability of a path reaching t,s
    """

    x = (t - s)/2

    if x % 1 != 0:
        return 0
    else:
        return binomial(int(t),int(x))/(2**t)

tt = [int(10*i) for i in range(11)]
ss = [int(i) for i in range(-int(max(tt)/1.5),int(max(tt)/1.5)+1)]

plt.figure(figsize=(12,8))
for t in tt:

    ps = [p_s(s,t) for s in ss]

    plt.plot(ss,ps,'.-',alpha=0.7)

plt.legend(['t=%i' % i for i in tt])
plt.title("p(X_t=s)")
```

```python
for t in [50]:
    X_t = []
    for i in range(2000):
        walk = random_walk(50)
        X_t += [walk[-1]]

    bins = [i for i in range(-int(t/1.5),int(t/1.5))]
    plt.hist(X_t,density=True,bins=bins,align='left')
    ps = [p_s(s,t) for s in bins]
    plt.plot(bins,ps,'.',markersize=15)

    plt.legend(['simulated','analytical'])
```

# Counting Paths - Imbalanced Dyck Paths

As an example for calculating the probability of a certain type of path, I will look into the probability that a point $X_t,t$ is reached by a path that never dips below 0. This is similar to looking at Dyck Words, in that they are sequences where there are always at least as many "+1" steps as there are "-1" steps. Dyck Words are normally balanced, in that they contain the same number of "+1" and "-1", meaning that they correspond to paths that return to 0 without ever going negative. The number of Dyck paths with length 2n is given by the Catalan number $C_n$, so in that case the answer is known. Instead I will look at what I'll call "Imbalanced Dyck Paths", which are paths that never dip below zero but don't necessarily return to 0 at their end point.

Below is a plot of a few random walks, with those who qualify as Imbalanced Dyck Paths shown in red:

```python
def is_dyck(walk):
    """
    check whether the walk is an imbalanced dyck path, i.e. it never goes below the horizon
    """
    if any(walk < 0):
        return False
    else:
        return True

color_code = {True: 'red', False: 'royalblue'}

plt.figure(figsize=(16,8))

total = 0
dyck_words = 0
for i in range(1000):
    walk = random_walk(1000)

    cc = False
    total += 1
    if is_dyck(walk):
        dyck_words += 1
        cc = True

    plt.plot(walk,color = color_code[cc],alpha=0.2)

plt.title('%i paths, %i above horizon (%1.1f%%)' % (total,dyck_words,100*dyck_words/total))
```

In the ``Counting Paths - Catalan Numbers and Dyck Words`` notebook, I came up with the formula:

\begin{equation}
f(x,y) = \sum^y_{k=x}f(x-1,k)\\
\forall x>0,\ y\geq x
\end{equation}

With boundary condition:

\begin{equation}
f(y,0) = 1\\ \forall y\geq 0
\end{equation}

To describe the amount of imbalanced Dyke paths that reach a certain point $x$,$y$. (Proper Dyke Paths of length 2n correspond to $f(n,n)$). Here I exchanged x and y as compared to the other notebook, because in this problem I calculated so that $y\geq x$ and not the other way around.

Given the number of paths, and considering all paths in this example have probability $\frac{1}{2^t}$, the probability distribution for Dyck paths can be found as:

\begin{equation}
p_D(x,y) = \frac{1}{2^{x+y}}f(x,y) = \frac{1}{2^{x+y}}\sum^y_{k=x}f(x-1,k) = \frac{1}{2^{y-x+1}}\sum_{n=0}^{y-x}p_D(x-1,x+n)
\end{equation}

Changing variables:
\begin{equation}
p_D(X_t=s) = p_D(s,t) = \frac{1}{2^{s+1}}\sum_{n=0}^s 2^n
p_D(s'=n+1,t'=t-(s+1)+n)
\end{equation}
for $t>s>0$,
\begin{equation}
p_D(s,s) = \frac{1}{2^t} \\
\forall s\leq t\\
\end{equation}
whenever $s=t$, and

\begin{equation}
p_D(s,t) = 0\\
\forall s > t,\ \forall s < 0,\ \forall t < 0
\end{equation}

for the remaining cases.

```python
class Dyke_Path(object):
    """
    Class to calculate Dyke Path probabilities with Memoization. (Much faster)
    """

    def __init__(self):

        self.archive = {}

        pass

    def proba(self,s,t):
        """
        Probability of a point s,t being reached with a Dyck Path
        """

        if (s,t) in self.archive:
            return self.archive[(s,t)]

        elif s > t:
            return 0
        elif t < 0:
            return 0
        elif s < 0:
            return 0

        elif s == t:
            return 1/(2**s)
        else:

            p = 1/2**(s+1) * np.sum([2**n * self.proba(n+1,t-(s+1)+n) for n in range(s+1)])
            self.archive[(s,t)] = p

            return p

PD = Dyke_Path()

t_max = 22
tt = list(range(t_max))
ss = list(range(-2,t_max))[::-1]

P = np.zeros([len(ss),len(tt)])

for i in range(len(ss)):
    for j in range(len(tt)):
        P[i,j] = PD.proba(ss[i],tt[j])

print("Dyck Path Probabilities (log2, rounded)")
f = lambda s: str(int(int(1*np.log2(s))/1)).zfill(4) if s > 0 else ' -  '
for p,s in zip(P,ss):
    print('s=%i\t' % s,s,t_max*'%s  ' % tuple([f(i) for i in p]))
plt.figure(figsize=(12,8))

P[P == 0] = np.nan
plt.imshow(np.log2(P))
plt.colorbar()
```

### Probability of an Imbalanced Dyck Paths surviving until time T

There's probably a closed form solution, but I will just clumsily integrate the recursion equation I derived above.

```python
N = 1000
t_max = 50

dyck_prob_sim = []
dyck_prob = []
for t in range(t_max+1):
    d_count = 0
    for i in range(N):
        walk = random_walk(t)
        if is_dyck(walk):
            d_count += 1

    dyck_prob_sim += [d_count/N]

    d_prob = 0
    for s in range(t+1):
        d_prob += PD.proba(s,t)

    dyck_prob += [d_prob]

plt.figure(figsize=(12,8))
plt.plot(dyck_prob,'d-')
plt.plot(dyck_prob_sim,'o-')
plt.title('Probability of a Dyck Path')
plt.legend(['analytical','simulated'])
```

Combinatorics - Counting Paths, Discrete Random Walk.ipynb