 Boundary Value Problem

As opposed to initial value problems, [Boundary value problems](https://en.wikipedia.org/wiki/Boundary_value_problem) is a class of differential equation together with a set of additional constraints (boundary conditions). A BVP with second order D.E has the form :

$$
\frac{d^2y}{dx^2}\ =\ f\left(x, y(x), y^{\prime}(x)\right),\quad y(x_0) = a,\quad y(x_n) = b
$$

Unlike initial value problems, a BVP has no unique solution. Some of theqniques used for finding solution are described here:

 Shooting Method

In this approach, we convert the BVP to Initial Value Problem(IVP). So, if we take $u_\alpha$ to be the solution of the IVP, equivalent to the previous BVP, then it has the form :
$$
\frac{d^2u_\alpha}{dx^2}\ =\ f\left(x, u_\alpha(x), u_\alpha^{\prime}(x)\right),\quad u_\alpha(x_0) = a,\quad u^\prime_\alpha(x_0) = \alpha
$$

Here, $\alpha$ is choosen with an educated guess. Because of this guess, the value of this function $u_\alpha(x)$ at $x=x_n$ will not be same as $y(x_n)$. Let us take the difference as $g(\alpha)$, since the this depends on $\alpha$. Thus we have :

$$
g(\alpha)\ =\ u_\alpha(x_n) - y(x_n)\ =\ u_\alpha(x_n) - b \qquad\qquad \because y(x_n) = b
$$

So, inorder to solve this set of equations, one has to find zero of equation above. Here are some expamples using different methods:

### Bisection Shooting Method

**Question :** solve the D.E :

$$
y^{\prime\prime} + 4y^\prime\ =\ 5x, \quad y(0)=1, \quad y(1)=0
$$
**Solution :**
Lets convert this to a initial value problem.

$$
u_\alpha^{\prime\prime} + 4u_\alpha^\prime\ =\ 5x, \quad u_\alpha(0)=1, \quad u^\prime_\alpha(0)=\alpha
$$

lets pick initial values of $\alpha$ are 0 and 5, so that when we take it as 0 and solve, $u_\alpha(1)<0$ and for 5, $u_\alpha(1)>0$. Lets try implimenting a runge kutta method for finding D.E

### Runge Kutta

the equation :

$$
y^{\prime\prime} + 4y^\prime\ =\ 5x
$$

can be split into two differential equations by substituting for $y^\prime = u$, so that the equation becomes

\begin{align}
\frac{dy}{dx} &= u \\[0.5em]
\frac{du}{dx} &= 5x-4u
\end{align}

with constraints
$$
y(0) = 1, \quad u(0) = \alpha
$$

```python
# importing necessary modules
import numpy as np
import matplotlib.pyplot as plt
import scipy
```

```python
N = 200                      # Number of samples
x = np.linspace(0, 1, N)     # Creating the array x
dx = x[1] - x[0]             # Step size

# defining f(x, u) for derivatve du/dx = f(x, u)
def f(x, u):
    return 5*x - 4*u

# The following function takes a value of alpha and find the solution of the equation
# y'' + 4y' = 5x using Runge-kutta method

def desolve(alpha):
    y = np.zeros(N, dtype=float) # Creating array to store y
    u = np.zeros(N, dtype=float) # Creating array to store u

    # setting initial values
    y[0] = 1
    u[0] = alpha

    # Starting R-K loop
    for i in range(1, N):
        # k_x represents for y(x) and l_x for u(x)
        k1 = u[i-1]
        l1 = f(x[i-1], u[i-1])
        k2 = u[i-1] + (1/2)*l1*dx
        l2 = f(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l1*dx)
        k3 = u[i-1] + (1/2)*l2*dx
        l3 = f(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l2*dx)
        k4 = u[i-1] + l3*dx
        l4 = f(x[i-1] + dx, u[i-1] + l3*dx)

        # calculating the next y and u values
        y[i] = y[i-1] + (1/6)*dx*(k1 + 2*k2 + 2*k3 + k4)
        u[i] = u[i-1] + (1/6)*dx*(l1 + 2*l2 + 2*l3 + l4)
    # Now, the array y stores the result
    return y
```

Lets check the result with two initial guess of $\alpha$, $\alpha_1 = 1$ and $\alpha_2 = -10$

```python
desolve(1)[-1]    # last term of solution with alpha = 1
```

    1.6346151809563805

```python
desolve(-10)[-1]   # last term of solution with alpha = 5
```

    -1.0650168118209011

This makes a good guess, since both have opposite sign.

Now, using bisection method to find the value of alpha :

```python
a = 1
b = -10
# bisection
alpha = (a + b)/2
# final value of solution at alpha
fa = desolve(alpha)[-1]
while abs(fa)>1e-8:
    print("alpha = ", alpha)
    # If the last value of y is positive, then update a, else updtae b
    if fa > 0 :
        a = alpha
    else:
        b = alpha
    # update alpha and fa
    alpha = (a + b)/2
    fa = desolve(alpha)[-1]

```

    alpha =  -4.5
    alpha =  -7.25
    alpha =  -5.875
    alpha =  -5.1875
    alpha =  -5.53125
    alpha =  -5.703125
    alpha =  -5.6171875
    alpha =  -5.66015625
    alpha =  -5.681640625
    alpha =  -5.6708984375
    alpha =  -5.66552734375
    alpha =  -5.662841796875
    alpha =  -5.6614990234375
    alpha =  -5.66082763671875
    alpha =  -5.660491943359375
    alpha =  -5.6603240966796875
    alpha =  -5.660408020019531
    alpha =  -5.660449981689453
    alpha =  -5.660470962524414
    alpha =  -5.660460472106934
    alpha =  -5.660455226898193
    alpha =  -5.660452604293823
    alpha =  -5.660451292991638
    alpha =  -5.660450637340546
    alpha =  -5.660450965166092

```python
alpha
```

    -5.660451129078865

```python
#  Now we  have alpha, we can find the solution
y = desolve(alpha)
y[-1]
```

    3.284482948638645e-09

```python
# True solution find using a Computer Algebra System, for comparison
N = 200
x = np.linspace(0, 1, N)
ytrue = (5/8)*x**2 - (5/16)*x - (1/64)*(25*np.exp(4) + 59)/(np.exp(4) - 1) +\
        (21/16)*np.exp(-4*x + 4)/(np.exp(4) - 1) + (5/64)
ytrue[0], ytrue[-1]
```

    (1.0, -1.3877787807814457e-17)

```python
# visualizing the solutions
plt.figure(figsize=(10, 8))
plt.plot(x, y,"-", lw=5, label="calculated")
plt.plot(x, ytrue,"--",lw=5, label="exact")
plt.plot(x, desolve(1), "--", label="alpha=1")
plt.plot(x, desolve(-10), "--", label="alpha=-10")
plt.legend()
```

### Newton Raphson Shooting Method

Remember we have to solve IVP:
$$
\frac{d^2u_\alpha}{dx^2}\ =\ f\left(x, u_\alpha(x), u_\alpha^{\prime}(x)\right),\quad u_\alpha(x_0) = a,\quad u^\prime_\alpha(x_0) = \alpha
$$

with $g(\alpha)$ is given by:
$$
g(\alpha)\ =\ u_\alpha(x_n) - y(x_n)\ =\ u_\alpha(x_n) - b \qquad\qquad \because y(x_n) = b
$$

Here, to find root of equation $g(\alpha) = 0$, we use Newton-Raphson method:

$$
\alpha_{i+1} = \alpha_i - \frac{g(\alpha_i)}{g^\prime(\alpha_i)}
$$

But the problem here is, we don't have $g^\prime(\alpha)$. To calculate thats, we take derivative w.r.t $\alpha$ of the first equation:

\begin{align}
\frac{d}{d\alpha}\frac{d^2u_\alpha}{dx^2}\ &=\ \frac{d}{d\alpha}f\left(x, u_\alpha(x), u_\alpha^{\prime}(x)\right)\\[0.5em]
\implies \frac{d^2}{dx^2}\frac{du_\alpha}{d\alpha}\ &=\ \frac{\partial f}{\partial u_\alpha}\frac{du_\alpha}{d\alpha} + \frac{\partial f}{\partial u^\prime_\alpha}\frac{du^\prime_\alpha}{d\alpha}
\end{align}

putting $\frac{du_\alpha}{d\alpha}\ =\ v_\alpha(x)$, we have,
$$
v^{\prime\prime}_\alpha\ =\ \frac{\partial f}{\partial u_\alpha}v_\alpha + \frac{\partial f}{\partial u^\prime_\alpha}v^\prime_\alpha
$$

initial conditions of $v_\alpha(x)$ are:
$$
v_\alpha(x_0) = \frac{du_\alpha(x_0)}{d\alpha} = \frac{da}{d\alpha} = 0, \qquad v^\prime_\alpha(x_0) = \frac{du^\prime_\alpha(x_0)}{d\alpha} = \frac{d\alpha}{d\alpha} = 1
$$

Now we have another D.E to solve, as described above. As a result, the newton raphson ormula becomes:

$$
\alpha_{i+1} = \alpha_i - \frac{u_{\alpha_i}(x_n)-b}{v_{\alpha_i}(x_n)}
$$

So, inorder to solve the equation using newton raphson method, we have two IVP:

$$
\frac{d^2u_\alpha}{dx^2}\ =\ f\left(x, u_\alpha(x), u_\alpha^{\prime}(x)\right),\quad u_\alpha(x_0) = a,\quad u^\prime_\alpha(x_0) = \alpha
$$

$$
v^{\prime\prime}_\alpha\ =\ \frac{\partial f}{\partial u_\alpha}v_\alpha + \frac{\partial f}{\partial u^\prime_\alpha}v^\prime_\alpha, \quad v_\alpha(x_0) = 0, \quad v^\prime_\alpha(x_0) = 1
$$

and use this formula to better estimate $\alpha$ and repeat until convergence is achieved

$$
\alpha_{i+1} = \alpha_i - \frac{u_{\alpha_i}(x_n)-b}{v_{\alpha_i}(x_n)}
$$

**Question :** solve the D.E :

$$
y^{\prime\prime} + 4y^\prime\ =\ 5x, \quad y(0)=1, \quad y(1)=0
$$

**soluton :** This is same as the one we encounter in bisection method, so we can use the same desolve(alpha) to find solution for first IVP. for the second case, we will create desolve2(alpha)

we have The IVP:
$$
u_\alpha^{\prime\prime} + 4u_\alpha^\prime\ =\ 5x, \quad u_\alpha(0)=1, \quad u^\prime_\alpha(0)=\alpha
$$

$$
\therefore f = 5x - 4u^\prime_\alpha
$$
Hence
$$
\frac{\partial f}{\partial u_\alpha} = 0, \quad \frac{\partial f}{\partial u^\prime_\alpha} = -4
$$

so the 2nd IVP is:

$$
v^{\prime\prime}_\alpha\ = -4v^\prime_\alpha, \quad v_\alpha(x_0) = 0, \quad v^\prime_\alpha(x_0) = 1
$$

So, the DE to solve is,

$$
v^{\prime\prime} = -4v^\prime
$$

we make substitutio u = $v^\prime$, so that we got two coupled 1st order DE:

$$
\frac{dv}{dx} = u(x), \quad \texttt{and} \quad \frac{du}{dx} = -4u(x)
$$
with
$$
v(x_0) = 0 \quad \texttt{and} \quad u(x_0) = 1
$$

```python
N = 200                      # Number of samples
x = np.linspace(0, 1, N)     # Creating the array x
dx = x[1] - x[0]             # Step size

# defining f2(x, u) for derivatve du/dx = f2(x, u)
def f2(x, u):
    return -4*u

# The following function takes a value of alpha and find the solution of the equation
# y'' + 4y' = 5x using Runge-kutta method

def desolve2(alpha):
    v = np.zeros(N, dtype=float) # Creating array to store y
    u = np.zeros(N, dtype=float) # Creating array to store u

    # setting initial values
    v[0] = 0
    u[0] = 1

    # Starting R-K loop
    for i in range(1, N):
        # k_x represents for v(x) and l_x for u(x)
        k1 = u[i-1]
        l1 = f2(x[i-1], u[i-1])
        k2 = u[i-1] + (1/2)*l1*dx
        l2 = f2(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l1*dx)
        k3 = u[i-1] + (1/2)*l2*dx
        l3 = f2(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l2*dx)
        k4 = u[i-1] + l3*dx
        l4 = f2(x[i-1] + dx, u[i-1] + l3*dx)

        # calculating the next y and u values
        v[i] = v[i-1] + (1/6)*dx*(k1 + 2*k2 + 2*k3 + k4)
        u[i] = u[i-1] + (1/6)*dx*(l1 + 2*l2 + 2*l3 + l4)
    # Now, the array y stores the result
    return v
```

Now we have Newton-Raphson formula:

$$
\alpha_{i+1} = \alpha_i - \frac{u_{\alpha_i}(x_n)-b}{v_{\alpha_i}(x_n)} = \alpha_i - \frac{u_{\alpha_i}(x_n)}{v_{\alpha_i}(x_n)} \quad \because b=0
$$

```python
# starting guess for alpha
alpha = 1
# taking end point
fa =  desolve(alpha)[-1]
while abs(fa)>1e-8:
    print("alpha = ", alpha)
    alpha = alpha - desolve(alpha)[-1]/desolve2(alpha)[-1]
    fa = desolve(alpha)[-1]
```

    alpha =  1

```python
alpha
```

    -5.660451142461925

```python
# Now, we can calculate value of y for this alpha
ynr = desolve(alpha)
ynr[-1]
```

    -1.951563910473908e-15

```python
# visualizing the solutions
plt.figure(figsize=(10, 8))
plt.plot(x, ynr,"-", lw=5, label="newton-raphson")
plt.plot(x, ytrue,".",lw=5, label="exact")

plt.legend()
```

 Linear Shooting Method

A linear equation has the form :
$$
y^{\prime\prime}(x) + P(x)y^\prime(x) + Q(x)y(x) = R(x)
$$

If $y_1(x)$ and $y_2(x)$ are solutions of this equation, then so is $k_1y_1(x) + k_2y_2(x)$. This linearity makes it possible to not specifiying $\alpha$ and use root finding as did in earlier two methods.

cosider boundary value problem
$$
y^{\prime\prime} + P(x)y^\prime(x) + Q(x)y(x) = R(x), \quad y(x_0) = a, \quad y(x_n) = b
$$

we can construct two IVP:
$$
u^{\prime\prime} + P(x)u^\prime(x) + Q(x)u(x) = R(x), \quad u(x_0) = a, \quad u^\prime(x_0) = 0
$$

$$
v^{\prime\prime} + P(x)v^\prime(x) + Q(x)v(x) = 0, \quad v(x_0) = 0, \quad v^\prime(x_0) = s
$$

Here, s is a parameter. Now, if we take a linear combination of these two equations,
$$
\omega(x) = u(x) + \lambda v(x)
$$
This $\omega(x)$ satisfy left boundary condition:
$$
\omega(x_0) = u(x_0) + \lambda v(x_0) = a
$$

inorder to satisfy right boundary condition,
we have
$$
\omega(x_n) = u(x_n) + \lambda v(x_n) = b
$$

this gives,
$$
\lambda = \frac{b - u(x_n)}{v(x_n)}
$$

and hence the solution, is

$$
y(x) = u(x) + \frac{b - u(x_n)}{v(x_n)}v(x)
$$

So, as a summary, we need to setup two initial value problems and use the last equation to find the solution. This does not include any root finding steps, but if $v(x_n) = 0$, then the solution to BVP may not exist.

**Question :** solve the D.E :

$$
y^{\prime\prime} + 4y^\prime\ =\ 5x, \quad y(0)=1, \quad y(1)=0
$$

**solution :** This is a linear equation. so, we set up two IVPs (with s=1):

$$
u^{\prime\prime} + 4u^\prime = 5x, \quad u(0) = 1, \quad u^\prime(0) = 0
$$

$$
v^{\prime\prime} + 4v^\prime = 0, \quad v(0) = 0, \quad v^\prime(0) = 1
$$

solving both is same as we done above. so we just copy and paste it here, make simple changes

```python
# store x array
N = 200                      # Number of samples
x = np.linspace(0, 1, N)     # Creating the array x
dx = x[1] - x[0]             # Step size
```

```python
# Non-Homogeneous part

# defining f1(x, u) for derivatve du/dx = f1(x, u)
def f1(x, u):
    return 5*x - 4*u

# The following function takes a value of alpha and find the solution of the equation
# y'' + 4y' = 5x using Runge-kutta method

def desolve1():
    y = np.zeros(N, dtype=float) # Creating array to store y
    u = np.zeros(N, dtype=float) # Creating array to store u

    # setting initial values
    y[0] = 1
    u[0] = 0

    # Starting R-K loop
    for i in range(1, N):
        # k_x represents for y(x) and l_x for u(x)
        k1 = u[i-1]
        l1 = f1(x[i-1], u[i-1])
        k2 = u[i-1] + (1/2)*l1*dx
        l2 = f1(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l1*dx)
        k3 = u[i-1] + (1/2)*l2*dx
        l3 = f1(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l2*dx)
        k4 = u[i-1] + l3*dx
        l4 = f1(x[i-1] + dx, u[i-1] + l3*dx)

        # calculating the next y and u values
        y[i] = y[i-1] + (1/6)*dx*(k1 + 2*k2 + 2*k3 + k4)
        u[i] = u[i-1] + (1/6)*dx*(l1 + 2*l2 + 2*l3 + l4)
    # Now, the array y stores the result
    return y
```

```python
# Homogeneous part
# defining f2(x, u) for derivatve du/dx = f2(x, u)
def f2(x, u):
    return - 4*u

# The following function takes a value of alpha and find the solution of the equation
# y'' + 4y' = 5x using Runge-kutta method

def desolve2():
    y = np.zeros(N, dtype=float) # Creating array to store y
    u = np.zeros(N, dtype=float) # Creating array to store u

    # setting initial values
    y[0] = 0
    u[0] = 1

    # Starting R-K loop
    for i in range(1, N):
        # k_x represents for y(x) and l_x for u(x)
        k1 = u[i-1]
        l1 = f2(x[i-1], u[i-1])
        k2 = u[i-1] + (1/2)*l1*dx
        l2 = f2(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l1*dx)
        k3 = u[i-1] + (1/2)*l2*dx
        l3 = f2(x[i-1] + (1/2)*dx, u[i-1] + (1/2)*l2*dx)
        k4 = u[i-1] + l3*dx
        l4 = f2(x[i-1] + dx, u[i-1] + l3*dx)

        # calculating the next y and u values
        y[i] = y[i-1] + (1/6)*dx*(k1 + 2*k2 + 2*k3 + k4)
        u[i] = u[i-1] + (1/6)*dx*(l1 + 2*l2 + 2*l3 + l4)
    # Now, the array y stores the result
    return y
```

```python
# storing arrays
u = desolve1()
v = desolve2()
```

```python
# checking if v(x_n) = 0:
v[-1]
```

    0.24542109025247982

Now the solution is given by
$$
y(x) = u(x) + \frac{b - u(x_n)}{v(x_n)}v(x)
$$

```python
ylin = u + (-u[-1]/v[-1])*v
ylin[-1]
```

    0.0

```python
plt.figure(figsize=(10, 8))
plt.plot(x, ylin, lw=5, label="linear shooting")
plt.plot(x, ytrue, "--", lw=5, label="exact")
plt.legend()
```

```python

```

```python

```


## The Basis of Symbolic-Numeric Integration

In this section, we informally introduce the symbolic-numeric integration algorithm, as implemented by **SymbolicNumericIntegraion.jl**.

Let's start with a simple example, $f(x) = x \sin x$, and show how to integrate it using the *method of indeterminate coefficients*. The main idea is to write the solution, i.e., $S = \int x \sin x\,dx$, as a sum of multiple possible terms with unknown coefficients,

\begin{equation}
  S = \sum_i q_i \mathbb{T}_i(x)
  \,,
  \tag{1}
\end{equation}

where $q_i$ are constant coefficients and $\mathbb{T}_i(x)$ are *reasonable candidate* terms. For our first example, and considering that $(\sin x)' = \cos x$ and $(\cos x)' = -\sin x$, a reasonable set of terms is $\mathbb{T} = \{x, \sin x, \cos x, x\sin x, x\cos x\}$. Of course, we need a better method to find $\mathbb{T}$ than saying it should be a reasonable set! In fact, we will discuss this problem is details later, but for now assume that an oracle provides $\mathbb{T}$. We have

\begin{equation}
  S = q_1 x + q_2 \sin x  + q_3 \cos x + q_4 x \sin x + q_5 x \cos x
  \,.
  \tag{2}
\end{equation}

Differentiating with respect to $x$,

\begin{equation}
  S' = q_1 + (q_4 - q_3) \sin x + (q_2 + q_5) \cos x - q_5 x \sin x + q_4 x \cos x
  \,.
  \tag{3}
\end{equation}

By definition, $\int S\,dx = f$; therefore, $S' = f = x \sin x$ (note that, as it is customary in symbolic integration, we ignore the constant inegration term). We obtain the following linear system,

\begin{equation}
  \begin{array}{ll}
    q_1 = 0 \\
    q_4 - q_3 = 0 \\
    q_2 + q_5 = 0 \\
    -q_5 = 1 \\
    q_4 = 0
  \end{array}
  \tag{4}
\end{equation}

Solving the linear the system, we find $q_5 = -1$, $q_2 = 1$, and $q_1 = q_3 = q_4 = 0$. Therefore,

\begin{equation}
  S = \int x \sin x\,dx = \sin x - x \cos x
  \,.
  \tag{5}
\end{equation}

As it should be.

Note that the preceding calculations were all essentially symbolic and there was no need for numerical computation. However, this is not always the case. Let's look at another example. This time, let $f(x) = \sin^2 x$. We assume that the oracle, who knows the correct answer $\int \sin^2 x = (x - \sin x\cos x)/2$, gives us $\mathbb{T} = \{x, \sin x\cos x\}$ (in practice, the list will be longer, but we use the abbreviated one to reduce clutter). Following the same process as before,

\begin{equation}
  S = q_1 x + q_2 \sin x\cos x
  \,,
  \tag{6}
\end{equation}

and,

\begin{equation}
  S' = q_1 + q_2 \cos^2 x - q_2\sin^2 x
  \,.
  \tag{7}
\end{equation}

Equating $S'$ to $\sin^2 x$, we get

\begin{equation}
  \begin{array}{ll}
    q_1 = 0 \\
    q_2 = 0 \\
    q_2 = -1
  \end{array}
  \tag{8}
\end{equation}

which is a contradiction. We can resolve this problem by using $\sin^2 x + \cos^2 x = 1$ to write

\begin{equation}
  S' = q_1 + q_2 (1 - \sin^2 x) - q_2\sin^2 x =
       (q_1 + q_2) - 2q_2 \sin^2 x
  \,.
  \tag{9}
\end{equation}

Therefore,

\begin{equation}
  \begin{array}{ll}
    q_1 + q_2 = 0 \\
    -2q_2 = 1
  \end{array}
  \tag{10}
\end{equation}

Finally, we have the correct answer $q_1 = 1/2$ and $q_2 = -1/2$.

Numerical computation becomes necessary partly due to the limitations of **JuliaSymbolics** in converting expressions into unique *canonical* forms. Therefore, identities like $\sin^2 x + \cos^2 x = 1$ (and may more, some subtle and some complex) may not be correctly applied. In fact, the problem is more fundamental and according to the Richardson's theorem, the problem of finding canonical forms of transcendental expressions is undecided.

Another reason for using numerical computation is that the list of candidates may not be (and usually is not) linearly-independent. Finding a linearly-independent subset of a set of expressions using symbolical computation is a very difficult problem but can be done numerically.

The next example show cases the problems of linear dependence. Let $f(x) = \sinh x\cosh x$. Assume that the oracle returns the following candidate list (which is typical of such lists),

\begin{equation}
    \mathbb{T} = \{\cosh^2 x, \cosh x\sinh x, \sinh2 x, x\cosh^2 x, x\cosh x\sinh x, x\sinh^2 x\}
    \,.
    \tag{14}
\end{equation}

If we follow the same procedure described above, a singular matrix error occurs. The reason is the fact that $\cosh^2 x - \sinh^2 x = 1$; therefore, $x\cosh^2 x$ and $x\sinh^2 x$ are linearly dependent. The solution is to prune $\mathbb{T}$ to a linearly-independent subset,

\begin{equation}
    \mathbb{T} = \{\cosh^2 x, \cosh x\sinh x, x\cosh^2 x, x\cosh x\sinh x\}
    \,,
    \tag{15}
\end{equation}

Now, we can calculate the correct answer $\int \sinh x\cosh\,dx = \frac{1}{2}\cosh^2 x$.

```julia

```


# Machine Learning and Statistics for Physicists

## Homework 8

```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
import numpy as np
import pandas as pd
```

```python
from sklearn import model_selection, ensemble
```

```python
import autograd.numpy as anp
import autograd
```

```python
from mls import locate_data, cv_summary
```

### Problem 1

The default score function used by sklearn to evaluate how well a regression model predicts data is the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination) $R^2$. Implement the function below to calculate $R^2$:

```python
def calculate_R2(y_data, y_pred):
    """Calculate the coefficient of determination R2 for two arrays.

    Parameters
    ----------
    y_data : array
        Array of data values, must have the same shape as y_pred.
    y_pred : array
        Array of predicted values, must have the same shape as y_data.

    Returns
    -------
    float
        Calculated coefficient of determination R2.
    """
    assert y_data.shape == y_pred.shape
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# A correct solution should pass the tests below.
gen = np.random.RandomState(seed=123)
N = 100
x = gen.uniform(size=N)
y_pred = 2 * x - 1
y_data = y_pred + gen.normal(scale=0.1, size=N)
assert np.round(calculate_R2(y_data, y_pred), 3) == 0.961
assert np.round(calculate_R2(y_data, -y_pred), 3) == -2.935
assert np.round(calculate_R2(y_pred, y_pred), 3) == 1.000
assert np.round(calculate_R2(y_pred, -y_pred), 3) == -3.000
assert np.round(calculate_R2(y_data, np.full(N, np.mean(y_pred))), 3) == 0.000
```

### Problem 2

Implement the function below to perform a [grid-search cross validation](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) of a [random forest regression](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) over the following grid:
 - `min_samples_leaf` = 1, 10, 20
 - `n_estimators` = 5, 10, 15

Hint: you will need to ensure reproducible "random" behavior in order to pass all the tests.

```python
def cross_validate(X, Y, gen):
    """Perform cross validation of a random forest regression.

    Parameters
    ----------
    X : array
        Array with shape (N, DX) of N samples with DX features.
    Y : array
        Array with shape (N, DY) of N samples with DY features.
    gen : np.random.RandomState
        Random state to use for reproducible random numbers.

    Returns
    -------
    pandas.DataFrame
        Cross-validation summary table produced by cv_summary().
    """
    assert len(X) == len(Y)
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
X = pd.read_hdf(locate_data('spectra_data.hf5')).values
Y = pd.read_hdf(locate_data('spectra_targets.hf5')).values
```

```python
# A correct solution should pass the tests below.
gen = np.random.RandomState(seed=123)
cvs = cross_validate(X, Y, gen)
assert np.all(cvs.columns.values == [
    'mean_test', 'mean_train', 'min_samples_leaf', 'n_estimators', 'split0_test',
    'split0_train', 'split1_test', 'split1_train', 'std_test', 'std_train'])
assert np.all(cvs['min_samples_leaf'].values == [1, 1, 1, 10, 10, 10, 20, 20, 20])
assert np.all(cvs['n_estimators'].values == [15, 10, 5, 15, 10, 5, 15, 10, 5])
assert np.allclose(
    cvs['mean_test'].values,
    [0.962, 0.957, 0.944, 0.907, 0.904, 0.891, 0.610, 0.605, 0.593], atol=1e-3)
assert np.allclose(
    cvs['mean_train'].values,
    [0.994, 0.993, 0.991, 0.920, 0.920, 0.917, 0.621, 0.624, 0.615], atol=1e-3)
```

### Problem 3

In class, we hand-tuned a network to assign 2D points to the correct cluster, then repeated the exercise using tensorflow to automatically learn network parameters from the training data. In this problem, you will construct and optimize the same neural network by hand (without tensorflow) to get a deeper understanding of what is involved (and why tensorflow saves a lot of work).

First, load the training data:

```python
X = pd.read_hdf(locate_data('circles_data.hf5')).values
y = pd.read_hdf(locate_data('circles_targets.hf5')).values.astype(np.float)
```

```python
def plot_circles():
    cmap = sns.color_palette('colorblind', 2)
    colors = [cmap[int(c)] for c in y]
    plt.scatter(X[:, 0], X[:, 1], c=colors)
    plt.gca().set_aspect(1)

plot_circles()
```

Next, implement the function below to evalute the network prediction given its parameter values and a 2D input point $(x_1, x_2)$. Recall that the network consists of:
 - 2 input nodes
 - 4 hidden nodes with sigmoid activation
 - 1 output node with sigmoid activation

The corresponding mathematical function is:
$$
\begin{align}
y^{out}(\mathbf{x},\Theta)
&= \phi\left([u_1, u_2, u_3, u_4]\cdot \mathbf{G}(x_1, x_2) + a \right) \\
\mathbf{G}(x_1, x_2) &= \phi\left(
\begin{bmatrix}
v_{11} & v_{12} \\
v_{21} & v_{22} \\
v_{31} & v_{32} \\
v_{41} & v_{42}
\end{bmatrix} \cdot \begin{bmatrix}x_1 \\ x_2\end{bmatrix} +
\begin{bmatrix}b_1 \\ b_2 \\ b_3 \\ b_4\end{bmatrix}
\right)
\end{align} \\
\phi(s) = \frac{1}{1 + e^{-s}}
$$
In the function, we pack all the parameters into a single array to simplify the later steps.

```python
def network(x, params):
    """Evaluate the network output at (x1,x2) with the specified parameters.

    Parameters
    ----------
    x : array
        Array of length 2 giving the 2D coordinates of a point to classify.
    params : array
        Array of length 17 containing values for the parameters: u1, u2, u3, u4, a,
        v11, v12, v21, v22, v31, v32, v41, v42, b1, b2, b3, b4.

    Returns
    -------
    float
        Value of the network's single output node.
    """
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# A correct solution should pass the tests below.
initial = np.array([
    5,5,5,5,              # ui
    -18,                  # a
    8,0, -8,0, 0,8, 0,-8, # vij
    10,10,10,10           # bi
], dtype=float)
assert np.round(network([0, 0], initial), 3) == 0.881
assert np.round(network([0, 1], initial), 3) == 0.803
assert np.round(network([1, 0], initial), 3) == 0.803
assert np.round(network([1, -1], initial), 3) == 0.692
assert np.round(network([-1, 1], initial), 3) == 0.692
```

We can learn the parameters from the data by optimizing the [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) loss function:
$$
E(\Theta, \mathbf{X}^{train}, \mathbf{y}^{train}) =
-\sum_{i=1}^N\, \left[
y^{train}_i \log y^{out}(\mathbf{X}^{train}_i, \Theta) + (1 - y^{train}_i) \log (1 - y^{out}(\mathbf{X}^{train}_i, \Theta)) \right]
$$
(Refer to the Neural Network lecture notes for details on why this is a good loss function).

Implement the function below to evaluate this loss:

```python
def loss(params, X_train, y_train):
    """Evaluate the cross-entropy loss.

    Parameters
    ----------
    params : array
        Array of length 17 containing values for the parameters: u1, u2, u3, u4, a,
        v11, v12, v21, v22, v31, v32, v41, v42, b1, b2, b3, b4.
    X_train : array
        Array of shape (N, 2) with training values.
    y_train : array
        Array of N target values which are each either 0. or 1.

    Returns
    -------
    float
        The cross-entropy loss value.
    """
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# A correct solution should pass the tests below.
assert np.round(loss(initial, np.array([[0.,0.]]), np.array([0.])), 3) == 2.126
assert np.round(loss(initial, np.array([[0.,0.]]), np.array([1.])), 3) == 0.127
assert np.round(loss(initial, np.array([[2.,1.]]), np.array([0.])), 3) == 0.027
assert np.round(loss(initial, np.array([[2.,1.]]), np.array([1.])), 3) == 3.611
assert np.round(loss(initial, X, y), 1) == 41.4
```

We can check if the initial parameter values are at a local minimum of the loss by running 1D scans along each parameter axis. Scans are also useful for rough numerical estimates of the partial derivatives
$$
\frac{\partial}{\partial \Theta_k} E(\Theta, \mathbf{X}^{train}, \mathbf{y}^{train}) \; .
$$

```python
def plot_scans(params, X_train, y_train, step_size=0.5, n=3):
    names = 'u1,u2,u3,u4,a,v11,v12,v21,v22,v31,v32,v41,v42,b1,b2,b3,b4'.split(',')
    dp_grid = np.linspace(-step_size, +step_size, 2 * n + 1)
    L = np.empty_like(dp_grid)
    for i, p0 in enumerate(initial):
        pscan = params.copy()
        for j, dp in enumerate(dp_grid):
            pscan[i] = p0 + dp
            L[j] = loss(pscan, X_train, y_train)
        plt.plot(dp_grid, L, label=names[i])
        print('numerical grad[{}] = {:.3f}'.format(names[i], np.gradient(L, dp_grid)[n]))
    plt.legend(ncol=5)
    plt.ylim(35., 50.)
    plt.xlabel('Change in parameter value')
    plt.ylabel('Cross-entropy loss')

plot_scans(initial, X, y)
```

Finally, implement the function below to calculate the partial derivatives of the loss function,
$$
\frac{\partial}{\partial \Theta_k} E(\Theta, \mathbf{X}^{train}, \mathbf{y}^{train}) \; .
$$
These should be calculated to high accuracy, to allow stable optimization, so either using the autograd package for automatic differentiation (refer to the Optimization lecture) or else using derivative formulas that you calculate by hand.

```python
def loss_gradient(params, X_train, y_train):
    """Calculate the partial derivatives of the loss function.

    Parameters
    ----------
    params : array
        Array of length 17 containing values for the parameters: u1, u2, u3, u4, a,
        v11, v12, v21, v22, v31, v32, v41, v42, b1, b2, b3, b4.
    X_train : array
        Array of shape (N, 2) with training values.
    y_train : array
        Array of N target values which are each either 0. or 1.

    Returns
    -------
    array
        Array of length 17 containing the partial derivatives of the loss function
        with respect to each of the parameters.
    """
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# A correct solution should pass the tests below.
assert np.allclose(
    np.round(loss_gradient(initial, np.array([[0.,0.]]), np.array([0.])), 3),
    [ 0.881, 0.881, 0.881, 0.881, 0.881, 0., 0., 0.,
      0., 0., 0., 0., 0., 0., 0., 0., 0.], atol=1e-3)
assert np.allclose(
    np.round(loss_gradient(initial, np.array([[0.,0.]]), np.array([1.])), 3),
    [ -0.119, -0.119, -0.119, -0.119, -0.119, 0., 0.,
      0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], atol=1e-3)
assert np.allclose(
    np.round(loss_gradient(initial, np.array([[2.,1.]]), np.array([0.])), 3),
    [ 0.027, 0., 0.027, 0.024, 0.027, 0., 0., 0.001,
      0., 0., 0., 0.028, 0.014, 0., 0., 0., 0.014], atol=1e-3)
assert np.allclose(
    np.round(loss_gradient(initial, np.array([[2.,1.]]), np.array([1.])), 3),
    [ -0.973, -0.002, -0.973, -0.857, -0.973, -0., -0., -0.024,
      -0.012, 0., 0., -1.022, -0.511, 0., -0.012, 0., -0.511], atol=1e-3)
assert np.allclose(
    np.round(loss_gradient(initial, X, y), 1),
    [ -24.1, -24.1, -24.1, -24.1, -21.9, -0.5, 0.0, 0.5, 0.0,
      0.0, -0.5, 0.0, 0.4, 0.3, 0.3, 0.3, 0.3 ], atol=1e-1)
```

You have now implemented all of the pieces necessary to optimize the 17 network parameters using the gradient descent update rule,
$$
\Theta \rightarrow \Theta - \eta \nabla E(\Theta) \; ,
$$
where $\eta$ is the learning rate.  The large values of some of the derivatives we found above indicate that $\eta$ will need to be small in order for the optimization to converge.

```python
def optimize(initial, X_train, y_train, eta=0.001, n_steps=20):
    loss_history = [loss(initial, X_train, y_train)]
    params = initial.copy()
    for i in range(n_steps):
        params -= eta * loss_gradient(params, X_train, y_train)
        loss_history.append(loss(params, X_train, y_train))
        print('step {:02d}: loss={:.1f}'.format(i + 1, loss_history[-1]))
    plt.plot(loss_history, 'o-')
    plt.xlabel('Optimization step')
    plt.ylabel('Cross-entropy loss')

optimize(initial, X, y)
```


# Statistics Fundamentals
Statistics is primarily about analyzing data samples, and that starts with udnerstanding the distribution of data in a sample.

## Analyzing Data Distribution
A great deal of statistical analysis is based on the way that data values are distributed within the dataset. In this section, we'll explore some statistics that you can use to tell you about the values in a dataset.

### Measures of Central Tendency
The term *measures of central tendency* sounds a bit grand, but really it's just a fancy way of saying that we're interested in knowing where the middle value in our data is. For example, suppose decide to conduct a study into the comparative salaries of people who graduated from the same school. You might record the results like this:

| Name     | Salary      |
|----------|-------------|
| Dan      | 50,000      |
| Joann    | 54,000      |
| Pedro    | 50,000      |
| Rosie    | 189,000     |
| Ethan    | 55,000      |
| Vicky    | 40,000      |
| Frederic | 59,000      |

Now, some of the former-students may earn a lot, and others may earn less; but what's the salary in the middle of the range of all salaries?

#### Mean
A common way to define the central value is to use the *mean*, often called the *average*. This is calculated as the sum of the values in the dataset, divided by the number of observations in the dataset. When the dataset consists of the full population, the mean is represented by the Greek symbol ***μ*** (*mu*), and the formula is written like this:

\begin{equation}\mu = \frac{\displaystyle\sum_{i=1}^{N}X_{i}}{N}\end{equation}

More commonly, when working with a sample, the mean is represented by ***x̄*** (*x-bar*), and the formula is written like this (note the lower case letters used to indicate values from a sample):

\begin{equation}\bar{x} = \frac{\displaystyle\sum_{i=1}^{n}x_{i}}{n}\end{equation}

In the case of our list of heights, this can be calculated as:

\begin{equation}\bar{x} = \frac{50000+54000+50000+189000+55000+40000+59000}{7}\end{equation}

Which is **71,000**.

>In technical terminology, ***x̄*** is a *statistic* (an estimate based on a sample of data) and ***μ*** is a *parameter* (a true value based on the entire population). A lot of the time, the parameters for the full population will be impossible (or at the very least, impractical) to measure; so we use statistics obtained from a representative sample to approximate them. In this case, we can use the sample mean of salary for our selection of surveyed students to try to estimate the actual average salary of all students who graduate from our school.

In Python, when working with data in a *pandas.dataframe*, you can use the ***mean*** function, like this:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mean())
```

    71000.0

So, is **71,000** really the central value? Or put another way, would it be reasonable for a graduate of this school to expect to earn $71,000? After all, that's the average salary of a graduate from this school.

If you look closely at the salaries, you can see that out of the seven former students, six earn less than the mean salary. The data is *skewed* by the fact that Rosie has clearly managed to find a much higher-paid job than her classmates.

#### Median
OK, let's see if we can find another definition for the central value that more closely reflects the expected earning potential of students attending our school. Another measure of central tendancy we can use is the *median*. To calculate the median, we need to sort the values into ascending order and then find the middle-most value. When there are an odd number of observations, you can find the position of the median value using this formula (where *n* is the number of observations):

\begin{equation}\frac{n+1}{2}\end{equation}

Remember that this formula returns the *position* of the median value in the sorted list; not the value itself.

If the number of observations is even, then things are a little (but not much) more complicated. In this case you calculate the median as the average of the two middle-most values, which are found like this:

\begin{equation}\frac{n}{2} \;\;\;\;and \;\;\;\; \frac{n}{2} + 1\end{equation}

So, for our graduate salaries; first lets sort the dataset:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

There's an odd number of observation (7), so the median value is at position (7 + 1) ÷ 2; in other words, position 4:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
|***>54,000*** |
| 55,000      |
| 59,000      |
| 189,000     |

So the median salary is **54,000**.

The *pandas.dataframe* class in Python has a ***median*** function to find the median:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].median())
```

    54000.0

#### Mode
Another related statistic is the *mode*, which indicates the most frequently occurring value. If you think about it, this is potentially a good indicator of how much a student might expect to earn when they graduate from the school; out of all the salaries that are being earned by former students, the mode is earned by more than any other.

Looking at our list of salaries, there are two instances of former students earning **50,000**, but only one instance each for all other salaries:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

The mode is therefore **50,000**.

As you might expect, the *pandas.dataframe* class has a ***mode*** function to return the mode:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mode())
```

    0    50000
    dtype: int64

##### Multimodal Data
It's not uncommon for a set of data to have more than one value as the mode. For example, suppose Ethan receives a raise that takes his salary to **59,000**:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
|***>59,000***|
|***>59,000***|
| 189,000     |

Now there are two values with the highest frequency. This dataset is *bimodal*. More generally, when there is more than one mode value, the data is considered *multimodal*.

The *pandas.dataframe.**mode*** function returns all of the modes:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,59000,40000,59000]})

print (df['Salary'].mode())
```

    0    50000
    1    59000
    dtype: int64

### Distribution and Density
Now we know something about finding the center, we can start to explore how the data is distributed around it. What we're interested in here is understanding the general "shape" of the data distribution so that we can begin to get a feel for what a 'typical' value might be expected to be.

We can start by finding the extremes - the minimum and maximum. In the case of our salary data, the lowest paid graduate from our school is Vicky, with a salary of **40,000**; and the highest-paid graduate is Rosie, with **189,000**.

The *pandas.dataframe* class has ***min*** and ***max*** functions to return these values.

Run the following code to compare the minimum and maximum salaries to the central measures we calculated previously:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print ('Min: ' + str(df['Salary'].min()))
print ('Mode: ' + str(df['Salary'].mode()[0]))
print ('Median: ' + str(df['Salary'].median()))
print ('Mean: ' + str(df['Salary'].mean()))
print ('Max: ' + str(df['Salary'].max()))
```

    Min: 40000
    Mode: 50000
    Median: 54000.0
    Mean: 71000.0
    Max: 189000

We can examine these values, and get a sense for how the data is distributed - for example, we can see that the *mean* is closer to the max than the *median*, and that both are closer to the *min* than to the *max*.

However, it's generally easier to get a sense of the distribution by visualizing the data. Let's start by creating a histogram of the salaries, highlighting the *mean* and *median* salaries (the *min*, *max* are fairly self-evident, and the *mode* is wherever the highest bar is):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
salary.plot.hist(title='Salary Distribution', color='lightblue', bins=25)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

The ***mean*** and ***median*** are shown as dashed lines. Note the following:
- *Salary* is a continuous data value - graduates could potentially earn any value along the scale, even down to a fraction of cent.
- The number of bins in the histogram determines the size of each salary band for which we're counting frequencies. Fewer bins means merging more individual salaries together to be counted as a group.
- The majority of the data is on the left side of the histogram, reflecting the fact that most graduates earn between 40,000 and 55,000
- The mean is a higher value than the median and mode.
- There are gaps in the histogram for salary bands that nobody earns.

The histogram shows the relative frequency of each salary band, based on the number of bins. It also gives us a sense of the *density* of the data for each point on the salary scale. With enough data points, and small enough bins, we could view this density as a line that shows the shape of the data distribution.

Run the following cell to show the density of the salary data as a line on top of the histogram:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
density = stats.gaussian_kde(salary)
n, x, _ = plt.hist(salary, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*5)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

Note that the density line takes the form of an asymmetric curve that has a "peak" on the left and a long tail on the right. We describe this sort of data distribution as being *skewed*; that is, the data is not distributed symmetrically but "bunched together" on one side. In this case, the data is bunched together on the left, creating a long tail on the right; and is described as being *right-skewed* because some infrequently occurring high values are pulling the *mean* to the right.

Let's take a look at another set of data. We know how much money our graduates make, but how many hours per week do they need to work to earn their salaries? Here's the data:

| Name     | Hours |
|----------|-------|
| Dan      | 41    |
| Joann    | 40    |
| Pedro    | 36    |
| Rosie    | 30    |
| Ethan    | 35    |
| Vicky    | 39    |
| Frederic | 40    |

Run the following code to show the distribution of the hours worked:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Hours':[41,40,36,30,35,39,40]})

hours = df['Hours']
density = stats.gaussian_kde(hours)
n, x, _ = plt.hist(hours, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7)
plt.axvline(hours.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(hours.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

Once again, the distribution is skewed, but this time it's **left-skewed**. Note that the curve is asymmetric with the ***mean*** to the left of the ***median*** and the *mode*; and the average weekly working hours skewed to the lower end.

Once again, Rosie seems to be getting the better of the deal. She earns more than her former classmates for working fewer hours. Maybe a look at the test scores the students achieved on their final grade at school might help explain her success:

| Name     | Grade |
|----------|-------|
| Dan      | 50    |
| Joann    | 50    |
| Pedro    | 46    |
| Rosie    | 95    |
| Ethan    | 50    |
| Vicky    | 5     |
| Frederic | 57    |

Let's take a look at the distribution of these grades:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Grade':[50,50,46,95,50,5,57]})

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7.5)
plt.axvline(grade.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(grade.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

This time, the distribution is symmetric, forming a "bell-shaped" curve. The ***mean***, ***median***, and mode are at the same location, and the data tails off evenly on both sides from a central peak.

Statisticians call this a *normal* distribution (or sometimes a *Gaussian* distribution), and it occurs quite commonly in many scenarios due to something called the *Central Limit Theorem*, which reflects the way continuous probability works - more about that later.

#### Skewness and Kurtosis
You can measure *skewness* (in which direction the data is skewed and to what degree) and kurtosis (how "peaked" the data is) to get an idea of the shape of the data distribution. In Python, you can use the ***skew*** and ***kurt*** functions to find this:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' skewness: ' + str(df[col].skew()))
    print(df[col].name + ' kurtosis: ' + str(df[col].kurt()))
    density = stats.gaussian_kde(df[col])
    n, x, _ = plt.hist(df[col], histtype='step', normed=True, bins=25)
    plt.plot(x, density(x)*6)
    plt.show()
    print('\n')
```

Now let's look at the distribution of a real dataset - let's see how the heights of the father's measured in Galton's study of parent and child heights are distributed:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

import statsmodels.api as sm

df = sm.datasets.get_rdataset('GaltonFamilies', package='HistData').data

fathers = df['father']
density = stats.gaussian_kde(fathers)
n, x, _ = plt.hist(fathers, histtype='step', normed=True, bins=50)
plt.plot(x, density(x)*2.5)
plt.axvline(fathers.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(fathers.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

As you can see, the father's height measurements are approximately normally distributed - in other words, they form a more or less *normal* distribution that is symmetric around the mean.

### Measures of Variance
We can see from the distribution plots of our data that the values in our dataset can vary quite widely. We can use various measures to quantify this variance.

#### Range
A simple way to quantify the variance in a dataset is to identify the difference between the lowest and highest values. This is called the *range*, and is calculated by subtracting the minimim value from the maximum value.

The following Python code creates a single Pandas dataframe for our school graduate data, and calculates the *range* for each of the numeric features:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' range: ' + str(df[col].max() - df[col].min()))
```

    Salary range: 149000
    Hours range: 11
    Grade range: 90

#### Percentiles and Quartiles
The range is easy to calculate, but it's not a particularly useful statistic. For example, a range of 149,000 between the lowest and highest salary does not tell us which value within that range a graduate is most likely to earn -  it doesn't tell us nothing about how the salaries are distributed around the mean within that range.  The range tells us very little about the comparative position of an individual value within the distribution - for example, Frederic scored 57 in his final grade at school; which is a pretty good score (it's more than all but one of his classmates); but this isn't immediately apparent from a score of 57 and range of 90.

##### Percentiles
A percentile tells us where a given value is ranked in the overall distribution. For example, 25% of the data in a distribution has a value lower than the 25th percentile; 75% of the data has a value lower than the 75th percentile, and so on. Note that half of the data has a value lower than the 50th percentile - so the 50th percentile is also the median!

Let's examine Frederic's grade using this approach. We know he scored 57, but how does he rank compared to his fellow students?

Well, there are seven students in total, and five of them scored less than Frederic; so we can calculate the percentile for Frederic's grade like this:

\begin{equation}\frac{5}{7} \times 100 \approx 71.4\end{equation}

So Frederic's score puts him at the 71.4th percentile in his class.

In Python, you can use the ***percentileofscore*** function in the *scipy.stats* package to calculate the percentile for a given value in a set of values:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'strict'))
```

    71.42857142857143

We've used the strict definition of percentile; but sometimes it's calculated as being the percentage of values that are less than *or equal to* the value you're comparing. In this case, the calculation for Frederic's percentile would include his own score:

\begin{equation}\frac{6}{7} \times 100 \approx 85.7\end{equation}

You can calculate this way in Python by using the ***weak*** mode of the ***percentileofscore*** function:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'weak'))
```

    85.71428571428571

We've considered the percentile of Frederic's grade, and used it to rank him compared to his fellow students. So what about Dan, Joann, and Ethan? How do they compare to the rest of the class? They scored the same grade (50), so in a sense they share a percentile.

To deal with this *grouped* scenario, we can average the percentage rankings for the matching scores. We treat half of the scores matching the one we're ranking as if they are below it, and half as if they are above it. In this case, there were three matching scores of 50, and for each of these we calculate the percentile as if 1 was below and 1 was above. So the calculation for a percentile for Joann based on scores being less than or equal to 50 is:

\begin{equation}(\frac{4}{7}) \times 100 \approx 57.14\end{equation}

The value of **4** consists of the two scores that are below Joann's score of 50, Joann's own score, and half of the scores that are the same as Joann's (of which there are two, so we count one).

In Python, the ***percentileofscore*** function has a ***rank*** function that calculates grouped percentiles like this:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 50, 'rank'))
```

    57.142857142857146

##### Quartiles
Rather than using individual percentiles to compare data, we can consider the overall spread of the data by dividing those percentiles into four *quartiles*. The first quartile contains the values from the minimum to the 25th percentile, the second from the 25th percentile to the 50th percentile (which is the median), the third from the 50th percentile to the 75th percentile, and the fourth from the 75th percentile to the maximum.

In Python, you can use the ***quantile*** function of the *pandas.dataframe* class to find the threshold values at the 25th, 50th, and 75th percentiles (*quantile* is a generic term for a ranked position, such as a percentile or quartile).

Run the following code to find the quartile thresholds for the weekly hours worked by our former students:

```python
# Quartiles
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df['Hours'].quantile([0.25, 0.5, 0.75]))
```

    0.25    35.5
    0.50    39.0
    0.75    40.0
    Name: Hours, dtype: float64

Its usually easier to understand how data is distributed across the quartiles by visualizing it. You can use a histogram, but many data scientists use a kind of visualization called a *box plot* (or a *box and whiskers* plot).

Let's create a box plot for the weekly hours:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Hours'].plot(kind='box', title='Weekly Hours Distribution', figsize=(10,8))
plt.show()
```

The box plot consists of:
- A rectangular *box* that shows where the data between the 25th and 75th percentile (the second and third quartile) lie. This part of the distribution is often referred to as the *interquartile range* - it contains the middle 50 data values.
- *Whiskers* that extend from the box to the bottom of the first quartile and the top of the fourth quartile to show the full range of the data.
- A line in the box that shows that location of the median (the 50th percentile, which is also the threshold between the second and third quartile)

In this case, you can see that the interquartile range is between 35 and 40, with the median nearer the top of that range. The range of the first quartile is from around 30 to 35, and the fourth quartile is from 40 to 41.

#### Outliers
Let's take a look at another box plot - this time showing the distribution of the salaries earned by our former classmates:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8))
plt.show()
```

So what's going on here?

Well, as we've already noticed, Rosie earns significantly more than her former classmates. So much more in fact, that her salary has been identifed as an *outlier*. An outlier is a value that is so far from the center of the distribution compared to other values that it skews the distribution by affecting the mean. There are all sorts of reasons that you might have outliers in your data, including data entry errors, failures in sensors or data-generating equipment, or genuinely anomalous values.

So what should we do about it?

This really depends on the data, and what you're trying to use it for. In this case, let's assume we're trying to figure out what's a reasonable expectation of salary for a graduate of our school to earn. Ignoring for the moment that we have an extremly small dataset on which to base our judgement, it looks as if Rosie's salary could be either an error (maybe she mis-typed it in the form used to collect data) or a genuine anomaly (maybe she became a professional athelete or some other extremely highly paid job). Either way, it doesn't seem to represent a salary that a typical graduate might earn.

Let's see what the distribution of the data looks like without the outlier:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8), showfliers=False)
plt.show()
```

Now it looks like there's a more even distribution of salaries. It's still not quite symmetrical, but there's much less overall variance. There's potentially some cause here to disregard Rosie's salary data when we compare the salaries, as it is tending to skew the analysis.

So is that OK? Can we really just ignore a data value we don't like?

Again, it depends on what you're analyzing. Let's take a look at the distribution of final grades:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

Once again there are outliers, this time at both ends of the distribution. However, think about what this data represents. If we assume that the grade for the final test is based on a score out of 100, it seems reasonable to expect that some students will score very low (maybe even 0) and some will score very well (maybe even 100); but most will get a score somewhere in the middle.  The reason that the low and high scores here look like outliers might just be because we have so few data points. Let's see what happens if we include a few more students in our data:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

With more data, there are some more high and low scores; so we no longer consider the isolated cases to be outliers.

The key point to take away here is that you need to really understand the data and what you're trying to do with it, and you need to ensure that you have a reasonable sample size, before determining what to do with outlier values.

#### Variance and Standard Deviation
We've seen how to understand the *spread* of our data distribution using the range, percentiles, and quartiles; and we've seen the effect of outliers on the distribution. Now it's time to look at how to measure the amount of variance in the data.

##### Variance
Variance is measured as the average of the squared difference from the mean. For a full population, it's indicated by a squared Greek letter *sigma* (***σ2***) and calculated like this:

\begin{equation}\sigma^{2} = \frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}\end{equation}

For a sample, it's indicated as ***s2*** calculated like this:

\begin{equation}s^{2} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}\end{equation}

In both cases, we sum the difference between the individual data values and the mean and square the result. Then, for a full population we just divide by the number of data items to get the average. When using a sample, we divide by the total number of items **minus 1** to correct for sample bias.

Let's work this out for our student grades (assuming our data is a sample from the larger student population).

First, we need to calculate the mean grade:

\begin{equation}\bar{x} = \frac{50+50+46+95+50+5+57}{7}\approx 50.43\end{equation}

Then we can plug that into our formula for the variance:

\begin{equation}s^{2} = \frac{(50-50.43)^{2}+(50-50.43)^{2}+(46-50.43)^{2}+(95-50.43)^{2}+(50-50.43)^{2}+(5-50.43)^{2}+(57-50.43)^{2}}{7-1}\end{equation}

So:

\begin{equation}s^{2} = \frac{0.185+0.185+19.625+1986.485+0.185+2063.885+43.165}{6}\end{equation}

Which simplifies to:

\begin{equation}s^{2} = \frac{4113.715}{6}\end{equation}

Giving the result:

\begin{equation}s^{2} \approx 685.619\end{equation}

The higher the variance, the more spread your data is around the mean.

In Python, you can use the ***var*** function of the *pandas.dataframe* class to calculate the variance of a column in a dataframe:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].var())
```

    685.6190476190476

##### Standard Deviation
To calculate the variance, we squared the difference of each value from the mean. If we hadn't done this, the numerator of our fraction would always end up being zero (because the mean is at the center of our values). However, this means that the variance is not in the same unit of measurement as our data - in our case, since we're calculating the variance for grade points, it's in grade points squared; which is not very helpful.

To get the measure of variance back into the same unit of measurement, we need to find its square root:

\begin{equation}s = \sqrt{685.619} \approx 26.184\end{equation}

So what does this value represent?

It's the *standard deviation* for our grades data. More formally, it's calculated like this for a full population:

\begin{equation}\sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}}\end{equation}

Or like this for a sample:

\begin{equation}s = \sqrt{\frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}}\end{equation}

Note that in both cases, it's just the square root of the corresponding variance forumla!

In Python, you can calculate it using the ***std*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].std())
```

    26.184328282754315

#### Standard Deviation in a Normal Distribution

In statistics and data science, we spend a lot of time considering *normal* distributions; because they occur so frequently. The standard deviation has an important relationship to play in a normal distribution.

Run the following cell to show a histogram of a *standard normal* distribution (which is a distribution with a mean of 0 and a standard deviation of 1):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

# Create a random standard normal distribution
df = pd.DataFrame(np.random.randn(100000, 1), columns=['Grade'])

# Plot the distribution as a histogram with a density curve
grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, color='lightgrey', normed=True, bins=100)
plt.plot(x, density(x))

# Get the mean and standard deviation
s = df['Grade'].std()
m = df['Grade'].mean()

# Annotate 1 stdev
x1 = [m-s, m+s]
y1 = [0.25, 0.25]
plt.plot(x1,y1, color='magenta')
plt.annotate('1s (68.26%)', (x1[1],y1[1]))

# Annotate 2 stdevs
x2 = [m-(s*2), m+(s*2)]
y2 = [0.05, 0.05]
plt.plot(x2,y2, color='green')
plt.annotate('2s (95.45%)', (x2[1],y2[1]))

# Annotate 3 stdevs
x3 = [m-(s*3), m+(s*3)]
y3 = [0.005, 0.005]
plt.plot(x3,y3, color='orange')
plt.annotate('3s (99.73%)', (x3[1],y3[1]))

# Show the location of the mean
plt.axvline(grade.mean(), color='grey', linestyle='dashed', linewidth=1)

plt.show()
```

The horizontal colored lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

#### Z Score
So in a normal (or close to normal) distribution, standard deviation provides a way to evaluate how far from a mean a given range of values falls, allowing us to compare where a particular value lies within the distribution. For example, suppose Rosie tells you she was the highest scoring student among her friends - that doesn't really help us assess how well she scored. She may have scored only a fraction of a point above the second-highest scoring student. Even if we know she was in the top quartile; if we don't know how the rest of the grades are distributed it's still not clear how well she performed compared to her friends.

However, if she tells you how many standard deviations higher than the mean her score was, this will help you compare her score to that of her classmates.

So how do we know how many standard deviations above or below the mean a particular value is? We call this a *Z Score*, and it's calculated like this for a full population:

\begin{equation}Z = \frac{x - \mu}{\sigma}\end{equation}

or like this for a sample:

\begin{equation}Z = \frac{x - \bar{x}}{s}\end{equation}

So, let's examine Rosie's grade of 95. Now that we know the *mean* grade is 50.43 and the *standard deviation* is 26.184, we can calculate the Z Score for this grade like this:

\begin{equation}Z = \frac{95 - 50.43}{26.184} = 1.702\end{equation}.

So Rosie's grade is 1.702 standard deviations above the mean.

### Summarizing Data Distribution in Python
We've seen how to obtain individual statistics in Python, but you can also use the ***describe*** function to retrieve summary statistics for all numeric columns in a dataframe. These summary statistics include many of the statistics we've examined so far (though it's worth noting that the *median* is not included):

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df.describe())
```

                  Salary      Hours      Grade
    count       7.000000   7.000000   7.000000
    mean    71000.000000  35.428571  50.428571
    std     52370.475143   8.423324  26.184328
    min     40000.000000  17.000000   5.000000
    25%     50000.000000  35.500000  48.000000
    50%     54000.000000  39.000000  50.000000
    75%     57000.000000  40.000000  53.500000
    max    189000.000000  41.000000  95.000000

```python

```

Statistics and Probability by Hiren/04-02-Statistics Fundamentals.ipynb


#Linear Kalman Filter With Control Input Example

###Introduction

This notebook is designed to demonstrate how to use the StateSpace.jl package to execute the Kalman filter for a linear State Space model with control input. The example that has been used here closely follows the one given on "Greg Czerniak's Website". Namely the canonball example on [this page.](http://greg.czerniak.info/guides/kalman1/)

For those of you that do not need/want the explanation of the model and the code, you can skip right to the end of this notebook where the entire section of code required to run this example is given.

###The Problem

The problem considered here is that of firing a ball from a canon at a given angle and velocity from the canon muzzle. We will assume that measurements of the ball's position are recorded with a camera at a given (constant) interval. The camera has a significant error in its measurement. We also measure the ball's velocity with relatively precise detectors inside the ball.

#####Process Model

The kinematic equations for the system are:

$$
\begin{align}
x(t)   &= x_0 + V_0^x t, \\
V^x(t) &= V_0^x, \\
y(t)   &= y_0 + V_0^y t - \frac{1}{2}gt^2, \\
V^y(t) &= V_0^y - gt,
\end{align}
$$

where $x$ is the position of the ball in the x (horizontal) direction, $y$ is the position of the ball in the y (vertical) direction, $V^x$ is the velocity of the ball in the x (horizontal) direction, $V^y$ is the velocity of the ball in the y (vertical) direction, $t$ is time, $x_0$, $y_0$, $V_0^x$ and $V_0^y$ are the initial x and y postion and velocity of the ball. $g$ is the acceleration due to gravity, 9.81 m/s.

Since the filter is discrete we need to discretize our equations so that we get the value of the current state of the ball in terms of the previous. This leads to the following equations:

$$
\begin{align}
x_n   &= x_{n-1} + V_{n-1}^x \Delta t, \\
V^x_n &= V_{n-1}^x, \\
y_n   &= y_{n-1} + V_{n-1}^y \Delta t - \frac{1}{2}g\Delta t^2, \\
V^y_n &= V_{n-1}^y - g\Delta t.
\end{align}
$$

These equations. Can be written in matrix form as:

$$
\begin{bmatrix}
       x_n   \\[0.3em]
       V^x_n \\[0.3em]
       y_n   \\[0.3em]
       V^y_n
     \end{bmatrix}
     =
     \begin{bmatrix}
       1 & \Delta t & 0 & 0        \\[0.3em]
       0 & 1        & 0 & 0        \\[0.3em]
       0 & 0        & 1 & \Delta t \\[0.3em]
       0 & 0        & 0 & 1
     \end{bmatrix}
     \begin{bmatrix}
       x_{n-1}   \\[0.3em]
       V^x_{n-1} \\[0.3em]
       y_{n-1}   \\[0.3em]
       V^y_{n-1}
     \end{bmatrix}
     +
     \begin{bmatrix}
       0 & 0 & 0 & 0        \\[0.3em]
       0 & 0 & 0 & 0        \\[0.3em]
       0 & 0 & 1 & 0 \\[0.3em]
       0 & 0 & 0 & 1
     \end{bmatrix}
     \begin{bmatrix}
       0   \\[0.3em]
       0 \\[0.3em]
       -\frac{1}{2}g\Delta t^2   \\[0.3em]
       -g\Delta t
     \end{bmatrix},
$$

Which is in the form:
$$
\mathbf{x}_n = \mathbf{A}\mathbf{x}_{n-1} + \mathbf{B}\mathbf{u}_n,
$$

where $\mathbf{x}_n$ is the state vector at the current time step, $\mathbf{A}$ is the process matrix, $\mathbf{B}$ is the control matrix and $\mathbf{u}_n$ is the control input vector.

#####Observation Model

We assume that we measure the position and velocity of the canonball directly and hence the observation (emission) matrix is the identity matrix, namely:

$$
\begin{bmatrix}
       1 & 0 & 0 & 0        \\[0.3em]
       0 & 1 & 0 & 0        \\[0.3em]
       0 & 0 & 1 & 0 \\[0.3em]
       0 & 0 & 0 & 1
\end{bmatrix}
$$

###Setting up the problem
First we'll import the required modules

```julia
using StateSpace
using Distributions
using Gadfly
using Colors
```

#####Generate noisy observations
In this section we will generate the noisy observations using the kinematic equations defined above in their continuous form.

The first thing to do is to set the parameters of the model

```julia
elevation_angle = 45.0 #Angle above the (horizontal) ground
muzzle_speed = 100.0 #Speed at which the canonball leaves the muzzle
initial_velocity = [muzzle_speed*cos(deg2rad(elevation_angle)), muzzle_speed*sin(deg2rad(elevation_angle))] #initial x and y components of the velocity
gravAcc = 9.81 #gravitational acceleration
initial_location = [0.0, 0.0] # initial position of the canonball
Δt = 0.1 #time between each measurement
```

    0.1

Next we'll define the kinematic equations of the model as functions in Julia (we don't care about the horizontal velocity component because it's constant).

```julia
x_pos(x0::Float64, Vx::Float64, t::Float64) = x0 + Vx*t
y_pos(y0::Float64, Vy::Float64, t::Float64, g::Float64) = y0 + Vy*t - (g * t^2)/2
velocityY(Vy::Float64, t::Float64, g::Float64) = Vy - g * t
```

    velocityY (generic function with 1 method)

Let's now set the variances of the noise for the position and velocity observations. We'll make the positional noise quite big.

```julia
x_pos_var = 200.0
y_pos_var = 200.0
Vx_var = 1.0
Vy_var = 1.0
```

    1.0

Now we will preallocate the arrays to store the true values and the noisy measurements. Then we will create the measurements in a `for` loop

```julia
#Set the number of observations and preallocate vectors to store true and noisy measurement values
numObs = 145
x_pos_true = Vector{Float64}(numObs)
x_pos_obs = Vector{Float64}(numObs)
y_pos_true = Vector{Float64}(numObs)
y_pos_obs = Vector{Float64}(numObs)

Vx_true = Vector{Float64}(numObs)
Vx_obs = Vector{Float64}(numObs)
Vy_true = Vector{Float64}(numObs)
Vy_obs = Vector{Float64}(numObs)

#Generate the data (true values and noisy observations)
for i in 1:numObs
    x_pos_true[i] = x_pos(initial_location[1], initial_velocity[1], (i-1)*Δt)
    y_pos_true[i] = y_pos(initial_location[2], initial_velocity[2], (i-1)*Δt, gravAcc)
    Vx_true[i] = initial_velocity[1]
    Vy_true[i] = velocityY(initial_velocity[2], (i-1)*Δt, gravAcc)

    x_pos_obs[i] = x_pos_true[i] + randn() * sqrt(x_pos_var)
    y_pos_obs[i] = y_pos_true[i] + randn() * sqrt(y_pos_var)
    Vx_obs[i] = Vx_true[i] + randn() * sqrt(Vx_var)
    Vy_obs[i] = Vy_true[i] + randn() * sqrt(Vy_var)
end
#Create the observations vector for the Kalman filter
observations = [x_pos_obs Vx_obs y_pos_obs Vy_obs]'
```

    4x145 Array{Float64,2}:
      5.26679  28.426     26.8539  26.6246  …  1000.9     1008.6      1024.48
     70.0963   69.7485    70.6661  71.6596       69.3015    71.0265     72.1319
     18.6518    7.20288  -10.3564  28.2894       19.9131     1.48237    -9.53701
     71.1312   70.1222    69.3649  67.0026      -68.644    -68.4365    -70.083

The final step in the code block above just puts all of the observations in a single array. Notice that we transpose the array to make sure that the dimensions are consistent with the StateSpace.jl convention. (Each observation is represented by a single column).

#####Define Kalman Filter Parameters

Now we can set the parameters for the process and observation model as defined above. We also set values for the corresponding covariance matrices. Because we're very sure about the process model, we set the process covariance to be very small. The observations can be set to have a higher variance but you can play about with these parameters.
NOTE: Be carefull about setting the diagonal values to zero, these can result in calculation errors downstream in the matrix calculations - rather the values can be set to be very small.

```julia
process_matrix = [[1.0, Δt, 0.0, 0.0] [0.0, 1.0, 0.0, 0.0] [0.0, 0.0, 1.0, Δt] [0.0, 0.0, 0.0, 1.0]]'
process_covariance = 0.01*eye(4)
observation_matrix = eye(4)
observation_covariance = 0.2*eye(4)
control_matrix = [[0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 1.0, 0.0] [0.0, 0.0, 0.0, 1.0]]
control_input = [0.0, 0.0, -(gravAcc * Δt^2)/2, -(gravAcc * Δt)]

#Create an instance of the LKF with the control inputs
linCISMM = LinearGaussianCISSM(process_matrix, process_covariance, observation_matrix, observation_covariance, control_matrix, control_input)
```

    StateSpace.LinearGaussianCISSM{Float64}(4x4 Array{Float64,2}:
     1.0  0.1  0.0  0.0
     0.0  1.0  0.0  0.0
     0.0  0.0  1.0  0.1
     0.0  0.0  0.0  1.0,4x4 Array{Float64,2}:
     0.01  0.0   0.0   0.0
     0.0   0.01  0.0   0.0
     0.0   0.0   0.01  0.0
     0.0   0.0   0.0   0.01,4x4 Array{Float64,2}:
     1.0  0.0  0.0  0.0
     0.0  1.0  0.0  0.0
     0.0  0.0  1.0  0.0
     0.0  0.0  0.0  1.0,4x4 Array{Float64,2}:
     0.2  0.0  0.0  0.0
     0.0  0.2  0.0  0.0
     0.0  0.0  0.2  0.0
     0.0  0.0  0.0  0.2,4x4 Array{Float64,2}:
     0.0  0.0  0.0  0.0
     0.0  0.0  0.0  0.0
     0.0  0.0  1.0  0.0
     0.0  0.0  0.0  1.0,[0.0,0.0,-0.04905000000000001,-0.9810000000000001])

#####Initial Guess
Now we can make an initial guess for the state of the system (position and velocity). We've purposely set the y coordinate of the initial value way off. This is to show how quickly the Kalman filter can converge to the correct solution. This isn't generally the case. It really depends on how good you model is and also the covariance matrix that you assign to the process/observation models. In this case, if you increase the observation variance (i.e. `observation_covariance = 10*eye(4)` say) then you'll see that it takes the Kalman filter longer to converge to the correct value.

```julia
initial_guess_state = [0.0, initial_velocity[1], 500.0, initial_velocity[2]]
initial_guess_covariance = eye(4)
initial_guess = MvNormal(initial_guess_state, initial_guess_covariance)
```

    FullNormal(
    dim: 4
    μ: [0.0,70.71067811865476,500.0,70.71067811865474]
    Σ: 4x4 Array{Float64,2}:
     1.0  0.0  0.0  0.0
     0.0  1.0  0.0  0.0
     0.0  0.0  1.0  0.0
     0.0  0.0  0.0  1.0
    )

#####Perform Linear Kalman Filter Algorithm
Now we have all of the parameters:
1. noisy observations
2. process (transition) and observation (emission) model paramaters
3. initial guess of state

We can use the Kalman Filter to predict the true underlying state (The position - and velocity - of the canonball).

```julia
filtered_state = filter(linCISMM, observations, initial_guess)
```

    SmoothedState{Float64}

    145 estimates of 4-D process from 4-D observations
    Log-likelihood: -619371.3025340745

###Plot Results
Now we can plot the results to see how well the Kalman Filter predicts the true position of the ball.

```julia
x_filt = Vector{Float64}(numObs)
y_filt = Vector{Float64}(numObs)
for i in 1:numObs
    current_state = filtered_state.state[i]
    x_filt[i] = current_state.μ[1]
    y_filt[i] = current_state.μ[3]
end

n = 3
getColors = distinguishable_colors(n, Color[LCHab(70, 60, 240)],
                                   transform=c -> deuteranopic(c, 0.5),
                                   lchoices=Float64[65, 70, 75, 80],
                                   cchoices=Float64[0, 50, 60, 70],
                                   hchoices=linspace(0, 330, 24))

cannonball_plot = plot(
    layer(x=x_pos_true, y=y_pos_true, Geom.line, Theme(default_color=getColors[3])),
    layer(x=[initial_guess_state[1]; x_filt], y=[initial_guess_state[3]; y_filt], Geom.line, Theme(default_color=getColors[1])),
    layer(x=x_pos_obs, y=y_pos_obs, Geom.point, Theme(default_color=getColors[2])),
    Guide.xlabel("X position"), Guide.ylabel("Y position"),
    Guide.manual_color_key("Colour Key",["Filtered Estimate", "Measurements","True Value "],[getColors[1],getColors[2],getColors[3]]),
    Guide.title("Measurement of a Canonball in Flight")
    )
```

Looking for the full code without having to read through the entire document? We'll here you go :)

```julia
using StateSpace
using Distributions
using Gadfly
using Colors

#Set the Parameters
elevation_angle = 45.0
muzzle_speed = 100.0
initial_velocity = [muzzle_speed*cos(deg2rad(elevation_angle)), muzzle_speed*sin(deg2rad(elevation_angle))]
gravAcc = 9.81
initial_location = [0.0, 0.0]
Δt = 0.1

#Functions describing the position of canonball
x_pos(x0::Float64, Vx::Float64, t::Float64) = x0 + Vx*t
y_pos(y0::Float64, Vy::Float64, t::Float64, g::Float64) = y0 + Vy*t - (g * t^2)/2
#Function to describe the evolution of the velocity in the vertical direction
velocityY(Vy::Float64, t::Float64, g::Float64) = Vy - g * t

#Give variances of the observation noise for the position and velocity
x_pos_var = 200.0
y_pos_var = 200.0
Vx_var = 1.0
Vy_var = 1.0

#Set the number of observations and preallocate vectors to store true and noisy measurement values
numObs = 145
x_pos_true = Vector{Float64}(numObs)
x_pos_obs = Vector{Float64}(numObs)
y_pos_true = Vector{Float64}(numObs)
y_pos_obs = Vector{Float64}(numObs)

Vx_true = Vector{Float64}(numObs)
Vx_obs = Vector{Float64}(numObs)
Vy_true = Vector{Float64}(numObs)
Vy_obs = Vector{Float64}(numObs)

#Generate the data (true values and noisy observations)
for i in 1:numObs
    x_pos_true[i] = x_pos(initial_location[1], initial_velocity[1], (i-1)*Δt)
    y_pos_true[i] = y_pos(initial_location[2], initial_velocity[2], (i-1)*Δt, gravAcc)
    Vx_true[i] = initial_velocity[1]
    Vy_true[i] = velocityY(initial_velocity[2], (i-1)*Δt, gravAcc)

    x_pos_obs[i] = x_pos_true[i] + randn() * sqrt(x_pos_var)
    y_pos_obs[i] = y_pos_true[i] + randn() * sqrt(y_pos_var)
    Vx_obs[i] = Vx_true[i] + randn() * sqrt(Vx_var)
    Vy_obs[i] = Vy_true[i] + randn() * sqrt(Vy_var)
end
#Create the observations vector for the Kalman filter
observations = [x_pos_obs Vx_obs y_pos_obs Vy_obs]'

#Describe the system parameters
process_matrix = [[1.0, Δt, 0.0, 0.0] [0.0, 1.0, 0.0, 0.0] [0.0, 0.0, 1.0, Δt] [0.0, 0.0, 0.0, 1.0]]'
process_covariance = 0.01*eye(4)
observation_matrix = eye(4)
observation_covariance = 0.2*eye(4)
control_matrix = [[0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 1.0, 0.0] [0.0, 0.0, 0.0, 1.0]]
control_input = [0.0, 0.0, -(gravAcc * Δt^2)/2, -(gravAcc * Δt)]

#Create an instance of the LKF with the control inputs
linCISMM = LinearGaussianCISSM(process_matrix, process_covariance, observation_matrix, observation_covariance, control_matrix, control_input)

#Set Initial Guess
initial_guess_state = [0.0, initial_velocity[1], 500.0, initial_velocity[2]]
initial_guess_covariance = eye(4)
initial_guess = MvNormal(initial_guess_state, initial_guess_covariance)

#Execute Kalman Filter
filtered_state = filter(linCISMM, observations, initial_guess)

#Plot Filtered results
x_filt = Vector{Float64}(numObs)
y_filt = Vector{Float64}(numObs)
for i in 1:numObs
    current_state = filtered_state.state[i]
    x_filt[i] = current_state.μ[1]
    y_filt[i] = current_state.μ[3]
end

n = 3
getColors = distinguishable_colors(n, Color[LCHab(70, 60, 240)],
                                   transform=c -> deuteranopic(c, 0.5),
                                   lchoices=Float64[65, 70, 75, 80],
                                   cchoices=Float64[0, 50, 60, 70],
                                   hchoices=linspace(0, 330, 24))

cannonball_plot = plot(
    layer(x=x_pos_true, y=y_pos_true, Geom.line, Theme(default_color=getColors[3])),
    layer(x=[initial_guess_state[1]; x_filt], y=[initial_guess_state[3]; y_filt], Geom.line, Theme(default_color=getColors[1])),
    layer(x=x_pos_obs, y=y_pos_obs, Geom.point, Theme(default_color=getColors[2])),
    Guide.xlabel("X position"), Guide.ylabel("Y position"),
    Guide.manual_color_key("Colour Key",["Filtered Estimate", "Measurements","True Value "],[getColors[1],getColors[2],getColors[3]]),
    Guide.title("Measurement of a Canonball in Flight")
    )
```


# Introduction

Recently I read the very interesting [Compiling to Categories](http://conal.net/papers/compiling-to-categories/compiling-to-categories.pdf) paper by Conal Elliot.

He presents the idea to compile haskell programs into a expressions of closed cartesian categories by $\lambda$-elimination.

The process applied reminded me of the [bracket abstraction](https://crypto.stanford.edu/~blynn/lambda/sk.html) used when compiling $\lambda$-terms to $SKI$-Combinators.

In this notebook I'm studying this connection.

## bracket abraction in combinatory logic

$\lambda$-terms can be converted to variable free combinator terms with a process called bracket abstraction.
Bracket abstraction $[]$ is defined by the following equations:

\begin{align}
[\lambda x.x]   & = I \\
[\lambda x.y]   & = Ky \\
[\lambda x.P Q] & = S [\lambda x.P] [\lambda x.Q]
\end{align}

where the combinators $I$, $K$, and $S$ are defined as follows:

\begin{align}
I x & = x \\
K y x & = y \\
S p q x & = (p x) (q x)
\end{align}

In Haskell this combinators could be implemented as follows (we have to use lower case letters as Uppercase first letters are reserved for contructors in Haskell):

```haskell
i :: a -> a
i x = x

k :: a -> b -> a
k y _ = y

s :: (a -> b -> c) -> (a -> b) -> a -> c
s p q x = (p x) (q x)
```

Once the $\lambda$-terms are compiled to combinator terms, these terms can be interpreted quite efficiently as they don't contain any variables and so no environment-handling is needed.

Combinator terms also allow to apply several more advanced interpretation techniques like graph-reduction, node-sharing, parallel reduction, etc.

## Closed Cartesian Categories

Both $I$ and $K$ are part of the Haskell standard prelude under the names `id` and `const`:

```haskell
-- | Identity function.
id :: a -> a
id x =  x

-- | @const x@ is a unary function which evaluates to @x@ for all inputs.
const :: a -> b -> a
const x _ =  x
```

```haskell
i :: a -> a
i x = x

i 8

```

    8

```haskell

```

Closed Cartesian Categories and the SKI Combinators.ipynb


```python
import sympy as sp
import warnings
warnings.filterwarnings('ignore')
sp.init_printing()
eps = sp.Symbol("e")
sig = sp.Symbol("s")
rad = sp.Symbol("r")
energyRad = 4 * eps * ((sig/rad)**12 - (sig/rad)**6)
energyRad.diff(rad)
```

```python
import sympy as sp
import warnings
warnings.filterwarnings('ignore')
sp.init_printing()
deltaT = sp.Symbol("dt")
deltaR = sp.Symbol("drot")
theta = sp.Symbol("t")
eq = deltaT*sp.sin(theta+deltaR)
eq.diff(theta)

```

```python
import sympy as sp
import warnings
warnings.filterwarnings('ignore')
sp.init_printing()
x = sp.Symbol("x")
y = sp.Symbol("y")
lx = sp.Symbol("lx")
ly = sp.Symbol("ly")
eq = sp.sqrt((x-lx)**2+(y-ly)**2)
eq.diff(x)

```

```python

```


```python
from sympy import *

a = Symbol("a")
x1 = Symbol("x1")
x2 = Symbol("x2")
```

$$
\begin{cases}
x_1 + a x_2 = 1\\
a x_1 + x_2 = 0\\
\end{cases}
$$

```python
e = [
    x1 + a * x2 - 1,
    a * x1 + x2,
]
```

```python
r = solve(e, [x1, x2])

for n in [0.99, 0.991]:
    print(f"a={n}")
    for k in r:
        print(f"{k}: {r[k].subs(a, n)}")
    print("---")
```

    a=0.99
    x2: -49.7487437185929
    x1: 50.2512562814070
    ---
    a=0.991
    x2: -55.3044254701713
    x1: 55.8066856409397
    ---

$$
\begin{cases}
x_1 + a x_2 = 1\\
a x_1 + 4x_2 = 0\\
\end{cases}
$$

```python
e = [
    x1 + a * x2 - 1,
    a * x1 + 4 * x2,
]

r = solve(e, [x1, x2])

for n in [0.99, 0.991]:
    print(f"a={n}")
    for k in r:
        print(f"{k}: {r[k].subs(a, n)}")
    print("---")
```

    a=0.99
    x2: -0.327825424682937
    x1: 1.32454717043611
    ---
    a=0.991
    x2: -0.328371967571032
    x1: 1.32541661986289
    ---