$\newcommand{\mb}[1]{\mathbf{ #1 }}$
$\newcommand{\bs}[1]{\boldsymbol{ #1 }}$
$\newcommand{\bb}[1]{\mathbb{ #1 }}$

$\newcommand{\R}{\bb{R}}$

$\newcommand{\ip}[2]{\left\langle #1, #2 \right\rangle}$
$\newcommand{\norm}[1]{\left\Vert #1 \right\Vert}$

$\newcommand{\der}[2]{\frac{\mathrm{d} #1 }{\mathrm{d} #2 }}$
$\newcommand{\derp}[2]{\frac{\partial #1 }{\partial #2 }}$

# Cart Pole

Consider a cart on a frictionless track. Suppose a pendulum is attached to the cart by a frictionless joint. The cart is modeled as a point mass $m_c$ and the pendulum is modeled as a massless rigid link with point mass $m_p$ a distance $l$ away from the cart.

Let $\mathcal{I} = (\mb{i}^1, \mb{i}^2, \mb{i}^3)$ denote an inertial frame. Suppose the position of the cart is resolved in the inertial frame as $\mb{r}_{co}^{\mathcal{I}} = (x, 0, 0)$. Additionally, suppose the gravitational force acting on the pendulum is resolved in the inertial frame as $\mb{f}_g^{\mathcal{I}} = (0, 0, -m_p g)$.

Let $\mathcal{B} = (\mb{b}^1, \mb{b}^2, \mb{b}^3)$ denote a body reference frame, with $\mb{b}^2 = \mb{i}^2$. The position of the pendulum mass relative to the cart is resolved in the body frame as $\mb{r}_{pc}^\mathcal{B} = (0, 0, l)$.

The kinetic energy of the system is:

\begin{equation}
    \frac{1}{2} m_c \norm{\dot{\mb{r}}_{co}^\mathcal{I}}_2^2 + \frac{1}{2} m_p \norm{\dot{\mb{r}}_{po}^\mathcal{I}}_2^2
\end{equation}

First, note that $\dot{\mb{r}}_{co}^{\mathcal{I}} = (\dot{x}, 0, 0)$.

Next, note that $\mb{r}_{po}^\mathcal{I} = \mb{r}_{pc}^\mathcal{I} + \mb{r}_{co}^\mathcal{I} = \mb{C}_{\mathcal{I}\mathcal{B}}\mb{r}_{pc}^\mathcal{B} + \mb{r}_{co}^\mathcal{I}$, where $\mb{C}_{\mathcal{I}\mathcal{B}}$ is the direction cosine matrix (DCM) satisfying:

\begin{equation}
    \mb{C}_{\mathcal{I}\mathcal{B}} = \begin{bmatrix} \ip{\mb{i}_1}{\mb{b}_1} & \ip{\mb{i}_1}{\mb{b}_2} & \ip{\mb{i}_1}{\mb{b}_3} \\ \ip{\mb{i}_2}{\mb{b}_1} & \ip{\mb{i}_2}{\mb{b}_2} & \ip{\mb{i}_2}{\mb{b}_3} \\ \ip{\mb{i}_3}{\mb{b}_1} & \ip{\mb{i}_3}{\mb{b}_2} & \ip{\mb{i}_3}{\mb{b}_3} \end{bmatrix}.
\end{equation}

We parameterize the DCM using $\theta$, measuring the clockwise angle of the pendulum from upright in radians. In this case, the DCM is:

\begin{equation}
    \mb{C}_{\mathcal{I}\mathcal{B}} = \begin{bmatrix} \cos{\theta} & 0 & \sin{\theta} \\ 0 & 1 & 0 \\ -\sin{\theta} & 0 & \cos{\theta} \end{bmatrix},
\end{equation}

following from $\cos{\left( \frac{\pi}{2} - \theta \right)} = \sin{\theta}$. Therefore:

\begin{equation}
    \mb{r}_{po}^\mathcal{I} = \begin{bmatrix} x + l\sin{\theta} \\ 0 \\ l\cos{\theta} \end{bmatrix}
\end{equation}

We have $\dot{\mb{r}}_{po}^\mathcal{I} = \dot{\mb{C}}_{\mathcal{I}\mathcal{B}} \mb{r}_{pc}^\mathcal{B} + \dot{\mb{r}}_{co}^\mathcal{I}$, following from $\dot{\mb{r}}_{pc}^\mathcal{B} = \mb{0}_3$ since the pendulum is rigid. The derivative of the DCM is:

\begin{equation}
    \der{{\mb{C}}_{\mathcal{I}\mathcal{B}}}{\theta} = \begin{bmatrix} -\sin{\theta} & 0 & \cos{\theta} \\ 0 & 0 & 0 \\ -\cos{\theta} & 0 & -\sin{\theta} \end{bmatrix},
\end{equation}

finally yielding:

\begin{equation}
    \dot{\mb{r}}_{po}^\mathcal{I} = \dot{\theta}  \der{\mb{C}_{\mathcal{I}\mathcal{B}}}{\theta} \mb{r}^{\mathcal{B}}_{pc} + \dot{\mb{r}}_{co}^\mathcal{I} = \begin{bmatrix} l\dot{\theta}\cos{\theta} + \dot{x} \\ 0 \\ -l\dot{\theta}\sin{\theta} \end{bmatrix}
\end{equation}

Define generalized coordinates $\mb{q} = (x, \theta)$ with configuration space $\mathcal{Q} = \R \times \bb{S}^1$, where $\bb{S}^1$ denotes the $1$-sphere. The kinetic energy can then be expressed as:

\begin{align}
    T(\mb{q}, \dot{\mb{q}}) &= \frac{1}{2} m_c \begin{bmatrix} \dot{x} \\ \dot{\theta} \end{bmatrix}^\top \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} \dot{x} \\ \dot{\theta} \end{bmatrix} + \frac{1}{2} m_p \begin{bmatrix} \dot{x} \\ \dot{\theta} \end{bmatrix}^\top \begin{bmatrix} 1 & l \cos{\theta} \\ l \cos{\theta} & l^2 \end{bmatrix} \begin{bmatrix} \dot{x} \\ \dot{\theta} \end{bmatrix}\\
    &= \frac{1}{2} \dot{\mb{q}}^\top\mb{D}(\mb{q})\dot{\mb{q}},
\end{align}

where inertia matrix function $\mb{D}: \mathcal{Q} \to \bb{S}^2_{++}$ is defined as:

\begin{equation}
    \mb{D}(\mb{q}) = \begin{bmatrix} m_c + m_p & m_p l \cos{\theta} \\ m_p l \cos{\theta} & m_pl^2 \end{bmatrix}.
\end{equation}

Note that:

\begin{equation}
    \derp{\mb{D}}{x} = \mb{0}_{2 \times 2},
\end{equation}

and:

\begin{equation}
    \derp{\mb{D}}{\theta} = \begin{bmatrix} 0 & -m_p l \sin{\theta} \\ -m_p l \sin{\theta} & 0 \end{bmatrix},
\end{equation}

so we can express:

\begin{equation}
    \derp{\mb{D}}{\mb{q}} = -m_p l \sin{\theta} (\mb{e}_1 \otimes \mb{e}_2 \otimes \mb{e}_2 + \mb{e}_2 \otimes \mb{e}_1 \otimes \mb{e}_2).
\end{equation}

The potential energy of the system is $U: \mathcal{Q} \to \R$ defined as:

\begin{equation}
    U(\mb{q}) = -\ip{\mb{f}_g^\mathcal{I}}{\mb{r}^{\mathcal{I}}_{po}} = m_p g l \cos{\theta}.
\end{equation}

Define $\mb{G}: \mathcal{Q} \to \R^2$ as:

\begin{equation}
    \mb{G}(\mb{q}) = \left(\derp{U}{\mb{q}}\right)^\top = \begin{bmatrix} 0 \\ -m_p g l \sin{\theta} \end{bmatrix}.
\end{equation}

Assume a force $(F, 0, 0)$ (resolved in the inertial frame) can be applied to the cart. The Euler-Lagrange equation yields:

\begin{align}
    \der{}{t} \left( \derp{T}{\dot{\mb{q}}} \right)^\top - \left( \derp{T}{\mb{q}} - \derp{U}{\mb{q}} \right)^\top &= \der{}{t} \left( \mb{D}(\mb{q})\dot{\mb{q}} \right) - \frac{1}{2}\derp{\mb{D}}{\mb{q}}(\dot{\mb{q}}, \dot{\mb{q}}, \cdot) + \mb{G}(\mb{q})\\
    &= \mb{D}(\mb{q})\ddot{\mb{q}} + \derp{\mb{D}}{\mb{q}}(\cdot, \dot{\mb{q}}, \dot{\mb{q}}) - \frac{1}{2}\derp{\mb{D}}{\mb{q}}(\dot{\mb{q}}, \dot{\mb{q}}, \cdot) + \mb{G}(\mb{q})\\
    &= \mb{B} F,
\end{align}

with static actuation matrix:

\begin{equation}
    \mb{B} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}.
\end{equation}

Note that:

\begin{align}
    \derp{\mb{D}}{\mb{q}}(\cdot, \dot{\mb{q}}, \dot{\mb{q}}) - \frac{1}{2}\derp{\mb{D}}{\mb{q}}(\dot{\mb{q}}, \dot{\mb{q}}, \cdot) &= -m_p l \sin{\theta} (\mb{e}_1 \dot{\theta}\dot{\theta} + \mb{e}_2\dot{x}\dot{\theta}) + \frac{1}{2} m_p l \sin{\theta} (\dot{x}\dot{\theta} \mb{e}_2 + \dot{\theta}\dot{x} \mb{e}_2)\\
    &= \begin{bmatrix} -m_p l \dot{\theta}^2 \sin{\theta} \\ 0 \end{bmatrix}\\
    &= \mb{C}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}},
\end{align}

with Coriolis terms defined as:

\begin{equation}
    \mb{C}(\mb{q}, \dot{\mb{q}}) = \begin{bmatrix} 0 & -m_p l \sin{\theta} \\ 0 & 0 \end{bmatrix}.
\end{equation}

Finally, we have:

\begin{equation}
    \mb{D}(\mb{q})\ddot{\mb{q}} + \mb{C}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \mb{G}(\mb{q}) = \mb{B}F
\end{equation}

```python
from numpy import array, concatenate, cos, dot, reshape, sin, zeros

from core.dynamics import RoboticDynamics

class CartPole(RoboticDynamics):
    def __init__(self, m_c, m_p, l, g=9.81):
        RoboticDynamics.__init__(self, 2, 1)
        self.params = m_c, m_p, l, g

    def D(self, q):
        m_c, m_p, l, _ = self.params
        _, theta = q
        return array([[m_c + m_p, m_p * l * cos(theta)], [m_p * l * cos(theta), m_p * (l ** 2)]])

    def C(self, q, q_dot):
        _, m_p, l, _ = self.params
        _, theta = q
        _, theta_dot = q_dot
        return array([[0, -m_p * l * theta_dot * sin(theta)], [0, 0]])

    def U(self, q):
        _, m_p, l, g = self.params
        _, theta = q
        return m_p * g * l * cos(theta)

    def G(self, q):
        _, m_p, l, g = self.params
        _, theta = q
        return array([0, -m_p * g * l * sin(theta)])

    def B(self, q):
        return array([[1], [0]])

m_c = 0.5
m_p = 0.25
l = 0.5
cart_pole = CartPole(m_c, m_p, l)
```

We attempt to stabilize the pendulum upright, that is, drive $\theta$ to $0$. We'll use the normal form transformation:

\begin{equation}
    \bs{\Phi}(\mb{q}, \dot{\mb{q}}) = \begin{bmatrix} \bs{\eta}(\mb{q}, \dot{\mb{q}}) \\ \mb{z}(\mb{q}, \dot{\mb{q}}) \end{bmatrix} = \begin{bmatrix} \theta \\ \dot{\theta} \\ x \\ m_p l \dot{x} \cos{\theta} + m_p l^2 \dot{\theta} \end{bmatrix}.
\end{equation}

```python
from core.dynamics import ConfigurationDynamics

class CartPoleOutput(ConfigurationDynamics):
    def __init__(self, cart_pole):
        ConfigurationDynamics.__init__(self, cart_pole, 1)
        self.cart_pole = cart_pole

    def y(self, q):
        return q[1:]

    def dydq(self, q):
        return array([[0, 1]])

    def d2ydq2(self, q):
        return zeros((1, 2, 2))

output = CartPoleOutput(cart_pole)
```

```python
from numpy import identity

from core.controllers import FBLinController, LQRController

Q = 10 * identity(2)
R = identity(1)
lqr = LQRController.build(output, Q, R)
fb_lin = FBLinController(output, lqr)
```

```python
from numpy import linspace, pi

x_0 = array([0, pi / 4, 0, 0])
ts = linspace(0, 10, 1000 + 1)

xs, us = cart_pole.simulate(x_0, fb_lin, ts)
```

```python
from matplotlib.pyplot import subplots, show, tight_layout
```

```python
_, axs = subplots(2, 2, figsize=(8, 8))
ylabels = ['$x$ (m)', '$\\theta$ (rad)', '$\\dot{x}$ (m / sec)', '$\\dot{\\theta}$ (rad / sec)']

for ax, data, ylabel in zip(axs.flatten(), xs.T, ylabels):
    ax.plot(ts, data, linewidth=3)
    ax.set_ylabel(ylabel, fontsize=16)
    ax.grid()

for ax in axs[-1]:
    ax.set_xlabel('$t$ (sec)', fontsize=16)

tight_layout()
show()
```

```python
_, ax = subplots(figsize=(4, 4))

ax.plot(ts[:-1], us, linewidth=3)
ax.grid()
ax.set_xlabel('$t$ (sec)', fontsize=16)
ax.set_ylabel('$F$ (N)', fontsize=16)

show()
```


# Investigating propagation of errors

This notebook was made to investigate the propagation of errors formula.
We imagine that we have a function $q(x,y)$ and we want to propagate the
uncertainty on $x$ and $y$ (denoted $\sigma_x$ and $\sigma_y$, respectively) through to the quantity $q$.

The most straight forward way to do this is just randomly sample $x$ and $y$, evaluate $q$ and look at it's distribution. This is really the definition of what we mean by propagation of uncertianty. It's very easy to do with some simply python code.

The calculus formula for the propagation of errors is really an approximation. This is the formula for a general $q(x,y)$
\begin{equation}
\sigma_q^2 = \left( \frac{\partial q}{\partial x} \sigma_x \right)^2 + \left( \frac{\partial q}{\partial y}\sigma_y \right)^2
\end{equation}

In the special case of addition  $q(x,y) = x\pm y$ we have $\sigma_q^2 = \sigma_x^2 + \sigma_y^2$.

In the special case of multiplication $q(x,y) = x y$ and division $q(x,y) = x / y$ we have $(\sigma_q/q)^2 = (\sigma_x/x)^2 + (\sigma_y/y)^2$, which we can rewrite as $\sigma_q = (x/y) \sqrt{(\sigma_x/x)^2 + (\sigma_y/y)^2}$

Let's try out these formulas and compare the direct approach of making the distribution to the prediction from these formulas

```python
#%pylab inline --no-import-all
```

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as scs
```

```python
from ipywidgets import widgets
from ipywidgets import interact, interactive, fixed
```

## Setup repeated observations of two variables x, y

```python
mean_x = .8
std_x = .15
mean_y = 3.
std_y = .9
N = 1000
x = np.random.normal(mean_x, std_x, N)
y = np.random.normal(mean_y, std_y, N)
```

## Check propagation of errors under addition

```python
x = np.random.normal(mean_x, std_x, N)
y = np.random.normal(mean_y, std_y, N)

q_of_x_y = x+y

pred_mean_q = mean_x+mean_y
pred_std_q = np.sqrt(std_x**2+std_y**2)

counts, bins, patches = plt.hist(q_of_x_y,
                                 bins=np.linspace(pred_mean_q-3*pred_std_q,pred_mean_q+3*pred_std_q,30),
                                 density=True, alpha=0.3)

plt.plot(bins, scs.norm.pdf(bins, pred_mean_q, pred_std_q), c='r', lw=2)
plt.legend(('pred','hist'))
plt.xlabel('q(x)')
plt.ylabel('p(x)')
```

### same thing with an interactive widget

```python
def plot_x_plus_y(mean_x, std_x, mean_y, std_y, N):
    x = np.random.normal(mean_x, std_x, N)
    y = np.random.normal(mean_y, std_y, N)

    q_of_x_y = x+y

    pred_mean_q = mean_x+mean_y
    pred_std_q = np.sqrt(std_x**2+std_y**2)

    counts, bins, patches = plt.hist(q_of_x_y,
                                     bins=np.linspace(pred_mean_q-3*pred_std_q,pred_mean_q+3*pred_std_q,30),
                                     density=True, alpha=0.3)

    plt.plot(bins, scs.norm.pdf(bins, pred_mean_q, pred_std_q), c='r', lw=2)
    plt.legend(('pred','hist'))
    plt.xlabel('q(x)')
    plt.ylabel('p(x)')

# now make the interactive widget
interact(plot_x_plus_y,
         mean_x=(0.,3.,.1), std_x=(.0, 2., .1),
         mean_y=(0.,3.,.1), std_y=(.0, 2., .1),
         N=(0,10000,1000))
```

    interactive(children=(FloatSlider(value=1.5, description='mean_x', max=3.0), FloatSlider(value=1.0, descriptio…

    <function __main__.plot_x_plus_y(mean_x, std_x, mean_y, std_y, N)>

## Single variable example: division

As a warm up, let's consider $q(x) = 1/x$

```python
counts, bins, patches = plt.hist(x, bins=50, density=True, alpha=0.3)
gaus_x = scs.norm.pdf(bins, mean_x,std_x)

q_for_plot = 1./bins

plt.plot(bins, gaus_x, lw=2)
plt.plot(bins, q_for_plot, lw=2)

plt.xlabel('x')
plt.ylabel('q(x)')
```

```python
plt.xlabel('x')

q_of_x = 1./x

pred_mean_q = 1./mean_x
pred_std_q = np.sqrt((std_x/mean_x)**2)/mean_x

counts, bins, patches = plt.hist(q_of_x, bins=50, density=True, alpha=0.3)

plt.plot(bins, scs.norm.pdf(bins, pred_mean_q, pred_std_q), c='r', lw=2)
plt.legend(('pred','hist'))
plt.xlabel('x')
plt.ylabel('p(x)')
```

### Now let's do the same thing with an interactive widget!

```python
def plot_1_over_x(mean_x, std_x, N):
    x = np.random.normal(mean_x, std_x, N)

    q_of_x = 1./x

    pred_mean_q = 1./mean_x
    pred_std_q = np.sqrt((std_x/mean_x)**2)/mean_x

    counts, bins, patches = plt.hist(q_of_x,
                                     bins=np.linspace(pred_mean_q-3*pred_std_q,pred_mean_q+3*pred_std_q,30),
                                     density=True, alpha=0.3)

    plt.plot(bins, scs.norm.pdf(bins, pred_mean_q, pred_std_q), c='r', lw=2)
    plt.legend(('pred','hist'))
    plt.xlabel('q(x)')
    plt.ylabel('p(x)')
```

```python
# now make the interactive widget
interact(plot_1_over_x,mean_x=(0.,3.,.1), std_x=(.0, 2., .1), N=(0,10000,1000))
```

    interactive(children=(FloatSlider(value=1.5, description='mean_x', max=3.0), FloatSlider(value=1.0, descriptio…

    <function __main__.plot_1_over_x(mean_x, std_x, N)>

### Check propagation of errors under division

```python
def plot_x_over_y(mean_x, std_x, mean_y, std_y, N):
    x = np.random.normal(mean_x, std_x, N)
    y = np.random.normal(mean_y, std_y, N)

    q_of_x_y = x/y

    pred_mean_q = mean_x/mean_y
    pred_std_q = np.sqrt((std_x/mean_x)**2+(std_y/mean_y)**2)*mean_x/mean_y

    counts, bins, patches = plt.hist(q_of_x_y,
                                     bins=np.linspace(pred_mean_q-3*pred_std_q,pred_mean_q+3*pred_std_q,30),
                                     density=True, alpha=0.3)

    plt.plot(bins, scs.norm.pdf(bins, pred_mean_q, pred_std_q), c='r', lw=2)
    plt.legend(('pred','hist'))
    plt.xlabel('q(x)')
    plt.ylabel('p(x)')

interact(plot_x_over_y,mean_x=(0.,3.,.1), std_x=(.0, 2., .1), mean_y=(0.,3.,.1), std_y=(.0, 2., .1),N=(0,100000,1000))
```

    interactive(children=(FloatSlider(value=1.5, description='mean_x', max=3.0), FloatSlider(value=1.0, descriptio…

    <function __main__.plot_x_over_y(mean_x, std_x, mean_y, std_y, N)>

```python

```

```python

```

```python

```

```python

```


# Demo series: Controllers & Agents

---

## Active Inference for controlling a driven damped harmonic oscillator

Wouter Kouw, 02-08-2021

### System

This project considers a [harmonic oscillator](https://en.wikipedia.org/wiki/Harmonic_oscillator#Driven_harmonic_oscillators). Consider $y(t)$ as observed displacement, $x(t)$ as the unobserved state, $u(t)$ as the driving force and $v(t), w(t)$ as measurement and process noise, respectively. The continuous-time dynamics of the system are:

$$\begin{align}
m \frac{d^2 x(t)}{dt^2} =&\ - c \frac{d x(t)}{dt} - k x(t) + u(t) \\
y(t) =&\ x(t) + e(t)
\end{align}$$
where
$$
m     = \text{mass} \, , \quad
c     = \text{damping} \, , \quad
k     = \text{spring stiffness} \, ,
$$
constitute the physical parameters.

The process noise is a Wiener process, where the increment is Gaussian distributed $w(t) \sim \mathcal{N}(0, \zeta^{-1}dt)$ with $\zeta$ representing the precision of the process. The measurement noise is also a Wiener process, with $v(t) \sim \mathcal{N}(0, \xi^{-1}dt)$ and $\xi$ as precision parameter.

We cast this to the following discrete-time system. Using a forward finite difference method, i.e.,

$$\begin{align}
\frac{d^2 x(t)}{dt^2} = \frac{x_{t+1} - 2x_t + x_{t-1}}{(\Delta t)^2} \, , \quad \text{and} \quad
\frac{d x(t)}{dt} = \frac{x_{t+1} - x_t}{\Delta t} \, ,
\end{align}$$

we get:

$$\begin{align}
m \frac{x_{t+1} - 2x_t + x_{t-1}}{(\Delta t)^2}  =& - c \frac{x_{t+1} - x_t}{\Delta t} -  k x_t + u_t \\
x_{t+1} =&\ -\frac{-2m + k (\Delta t)^2}{m + c \Delta t} x_t - \frac{m - c \Delta t}{m + c \Delta t} = \frac{(\Delta t)^2}{m + c \Delta t} u_t \, .
\end{align}$$

If we substitute variables we get:

$$\begin{align}
x_{t+1} =&\ \theta_1 x_t + \theta_2 x_{t-1} + \eta u_t \\
y_t =&\ x_t + v_t \, .
\end{align}$$

```julia
using Pkg
Pkg.activate(".")
Pkg.instantiate();
```

      Activating environment at `C:\Users\kouww\Research\actinf-oscillator\Project.toml`

```julia
using Revise
using Plots
pyplot();
```

```julia
# Sampling step size
Δt = 1.0

# Dynamical parameters
m = 9.0
c = 0.3
k = 1.5

# Measurement noise precision
ξ_true = Inf

# Pack parameters
sys_params = (m, c, k, ξ_true);
```

```julia
# State transition coefficients
θ1 = -(-2*m + k*Δt^2)/(m + c*Δt)
θ2 = -(m - c*Δt)/(m + c*Δt)
θ_true = [θ1, θ2]

# Control coefficient
η_true = Δt^2/(m + c*Δt)

# Pack substituted variables
subs_params = (θ_true, η_true, ξ_true);
```

```julia
function sim_sys(input, state, params)
   "Simulate dynamic system"

    # Unpack state
    x_kmin1, x_kmin2 = state

    # Unpack parameters
    θ, η, ξ = params

    # State transition
    x_k = θ[1]*x_kmin1 + θ[2]*x_kmin2 + η*input

    # Generate observation
    y_k = x_k + sqrt(inv(ξ))*randn(1,)[1]

    return y_k, x_k
end;
```

```julia
# Time
T = 200
stop = T-100

# Input signal
input = [cos.((1:stop)./ (6*π)); zeros(length(stop+1:T),)]
# input = zeros(T,); input[1] = 1.0

# Preallocate arrays
states = zeros(T,)
output = zeros(T,)

for k = 4:T
    output[k], states[k] = sim_sys(input[k], states[[k-1, k-2]], subs_params)
end

p100 = plot(1:T, states, linewidth=1, color="blue", label="states", size=(900,300))
plot!(1:T, input, linewidth=1, color="red", label="input")
vline!([stop], color="green", label="")
scatter!(1:T, output, markersize=1, color="black", label="output")
```

```julia
savefig(p100, "figures/example-input-output_seq1.png")
```

## Model 1: Latent Auto-Regressive model with eXogenous inputs

We cast the above system into matrix form:

$$ \underbrace{\begin{bmatrix} x_{k} \\ x_{k-1} \end{bmatrix}}_{z_k} = \Big(\underbrace{\begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix}}_{S} + \underbrace{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}_{s} \begin{bmatrix} \theta_1 & \theta_2 \end{bmatrix} \Big) \underbrace{\begin{bmatrix} x_{k-1} \\ x_{k-2} \end{bmatrix}}_{z_{k-1}} + \begin{bmatrix} 1 \\ 0 \end{bmatrix} \eta u_t \, ,$$

Next, we add white noise with unknown precision to the first element in the model and white noise with a tiny amount of noise to the other elements:

$$w_k \sim \mathcal{N}\big(0, V(\zeta) \big) \, , \quad \text{with} \quad V(\zeta) = \begin{bmatrix} \zeta^-1 & 0 \\ 0 & \epsilon \end{bmatrix} \, .$$

The state transition can thus be written as:

$$ z_k = \underbrace{\big(S + s\theta^{\top} \big)}_{A(\theta)} z_{k-1} + \underbrace{s\eta}_{B(\eta)} u_k + w_k \, .$$

### Likelihood

Integrating out $w_k$ and $v_t$ lets us formulate the dynamics as a Gaussian state-space model:

$$\begin{align}
z_k \sim&\ \mathcal{N}(A(\theta) z_{k-1} + B(\eta) u_k , V(\zeta)) \\
y_k \sim&\ \mathcal{N}(s^{\top} z_k, \xi^{-1}) \, .
\end{align}$$

### Priors

We choose Gaussian priors for the unknown coefficients and Gamma priors for the unknown noise precisions:

$$\begin{align}
p(\theta) = \mathcal{N}(m^{0}_{\theta}, V^{0}_{\theta}) \, , \quad
p(\eta) = \mathcal{N}(m^{0}_{\eta}, v^{0}_{\eta}) \, , \quad
p(\zeta)= \Gamma(a^{0}_\zeta, b^{0}_\zeta) \, , \quad
p(\xi)= \Gamma(a^{0}_\xi, b^{0}_\xi) \, .
\end{align}$$

We also need a prior distribution for the initial state:

$$ p(z_0) = \mathcal{N}({\bf 0}, V^{z_0}) \, .$$

### Posteriors

We can apply the Bayesian filtering equations to obtain recursive posterior estimates:

$$\begin{align}\label{eq:filtering}
    \underbrace{p(z_k | x_{1:t})}_{\text{state posterior}} &= \frac{\overbrace{p(x_t | z_t)}^{\text{likelihood}}}{\underbrace{p(x_t | x_{1:t-1})}_{\text{evidence}} }  \overbrace{\iint \underbrace{p(z_t| z_{t-1},u_{t})}_{\text{state transition}} \underbrace{q(u_t)}_{\substack{\text{control} \\ \text{signal}}} \underbrace{p(z_{t-1} | x_{1:t-1})}_{\text{state prior}} \d z_{t-1} \d u_t}^{\text{prior predictive $p(z_t | x_{1:t-1})$}} \, ,
\end{align}$$

### Variational Bayesian inference

#### Recognition model

The recognition model will follow the generative model:

$$\begin{align}
q(\theta) = \mathcal{N}(m_{\theta}, V_{\theta}) \ , \quad
q(\eta) = \mathcal{N}(m_{\eta}, v_{\eta}) \ , \quad
q(\zeta)= \Gamma(a_\zeta, b_\zeta) \, , \quad
q(\xi)= \Gamma(a_\xi, b_\xi) \, .
\end{align}$$

#### Free Energy function

```julia
using LinearAlgebra
using LaTeXStrings
using ProgressMeter
using Optim
using ForneyLab
```

```julia
using LARX
```

```julia
include("util.jl");
```

```julia
graph1 = FactorGraph()

# Coefficients
@RV θ ~ GaussianMeanPrecision(placeholder(:m_θ, dims=(2,)), placeholder(:w_θ, dims=(2,2)))
@RV η ~ GaussianMeanPrecision(placeholder(:m_η), placeholder(:w_η))

# Noise precisions
@RV ζ ~ Gamma(placeholder(:a_ζ), placeholder(:b_ζ))
@RV ξ ~ Gamma(placeholder(:a_ξ), placeholder(:b_ξ))

# State prior
@RV x ~ GaussianMeanPrecision(placeholder(:m_x, dims=(2,)), placeholder(:w_x, dims=(2, 2)), id=:x_k)

# Autoregressive state transition
@RV z ~ LatentAutoregressiveX(θ, x, η, placeholder(:u), ζ, id=:z_k)

# Likelihood
@RV y ~ GaussianMeanPrecision(dot([1. , 0.], z), ξ, id=:y_k)
placeholder(y, :y);

# Specify recognition model
q = PosteriorFactorization(z, x, θ, η, ζ, ξ, ids=[:z, :x, :θ, :η, :ζ, :ξ])

# Specify and compile message passing algorithm
algo = messagePassingAlgorithm([z, x, θ, η, ζ, ξ], q)
eval(Meta.parse(algorithmSourceCode(algo)));
```

### Experiment 1: Online system Identification from known inputs

```julia
# Time horizon
T = 300

# Design input signal
inputs = cos.(range(1, stop=T) * π/6);
# inputs = [mean(sin.(t./ (0.1:.1:6 .*π))) for t in 1:T];
```

```julia
using LARX
```

```julia
# Preallocation
states = zeros(T,)
outputs = zeros(T,)

# Initialize constraints
constraints = Dict()

# Initialize marginals in factor graph
marginals = Dict()
marginals[:x] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.01*eye(2))
marginals[:z] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.01*eye(2))
marginals[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.01*eye(2))
marginals[:η] = ProbabilityDistribution(Univariate, GaussianMeanPrecision, m=0.0, w=.01)
marginals[:ζ] = ProbabilityDistribution(Univariate, Gamma, a=1e3, b=1)
marginals[:ξ] = ProbabilityDistribution(Univariate, Gamma, a=1e2, b=1e-2)

# Number of variational updates
num_iterations = 10

# Track estimated states
est_states = (zeros(T,), zeros(T,))
est_coeffs = (zeros(T,2), zeros(T,2,2))

@showprogress for k = 3:T

    # Execute input and observe output
    outputs[k], states[k] = sim_sys(inputs[k], states[[k-1,k-2]], sys_params)

    "Update model params"

    # Update constraints
    constraints = Dict(:y => outputs[k],
                       :u => inputs[k],
                       :m_x => mn(marginals[:z]),
                       :w_x => pc(marginals[:z]),
                       :m_θ => mn(marginals[:θ]),
                       :w_θ => pc(marginals[:θ]),
                       :m_η => mn(marginals[:η]),
                       :w_η => pc(marginals[:η]),
                       :a_ζ => marginals[:ζ].params[:a],
                       :b_ζ => marginals[:ζ].params[:b],
                       :a_ξ => marginals[:ξ].params[:a],
                       :b_ξ => marginals[:ξ].params[:b])

    # Iterate recognition factor updates
    for ii = 1:num_iterations

        # Update parameters
        stepθ!(constraints, marginals)
        stepη!(constraints, marginals)

        # Update states
        stepz!(constraints, marginals)
        stepx!(constraints, marginals)

        # Update noise precisions
        stepζ!(constraints, marginals)
        stepξ!(constraints, marginals)

    end

    # Store state estimates
    est_states[1][k] = mn(marginals[:z])[1]
    est_states[2][k] = sqrt(inv(pc(marginals[:z])[1,1]))
    est_coeffs[1][k,:] = mn(marginals[:θ])
    est_coeffs[2][k,:,:] = inv(pc(marginals[:θ]))
end
```

```julia
p1 = plot(3:T, inputs[3:T], color="red", label="inputs")
p2 = scatter(3:T, outputs[3:T], markersize=2, color="black", label="outputs")
plot!(3:T, est_states[1][3:T], color="purple", label="inferred")
plot!(3:T, est_states[1][3:T], ribbon=[est_states[2][3:T], est_states[2][3:T]], color="purple", label="")
plot(p1, p2, layout=(2,1), size=(900,400))
```

```julia
p1 = plot(3:T, est_coeffs[1][3:T,1], ribbon=[sqrt.(est_coeffs[2][3:T,1,1]), sqrt.(est_coeffs[2][3:T,1,1])], color="blue", label="θ1 inferred")
plot!(θ_true[1]*ones(T-2,), label="θ1 true", color="green")
p2 = plot(3:T, est_coeffs[1][3:T,2], ribbon=[sqrt.(est_coeffs[2][3:T,2,2]), sqrt.(est_coeffs[2][3:T,2,2])], color="darkblue", label="θ2 inferred")
plot!(θ_true[2]*ones(T-2,), label="θ2 true", color="darkgreen")
plot(p1,p2, layout=(2,1), size=(900,300))
```

```julia
mn(marginals[:θ])
```

#### Validate identified system

We validate the identified system by computing simulation error on the validation set.

```julia
# Time horizon
T_val = 1000

# Input signal
inputs_val = sin.(range(1, stop=T_val)*π/6);
# inputs_val = [mean(sin.(t./ (0.1:.1:6 .*π))) for t in 1:T_val];
```

```julia
# # Prediction graph
# graph = FactorGraph()

# # Autoregressive node
# @RV z ~ LatentAutoregressiveX(placeholder(:θ, dims=(2,)), placeholder(:x, dims=(2,)), placeholder(:η), placeholder(:u), placeholder(:ζ), id=:z_pred)

# # Inference algorithm
# q = PosteriorFactorization(z, ids=[:_pred])
# algo = messagePassingAlgorithm([z], q)
# eval(Meta.parse(algorithmSourceCode(algo)));
```

```julia
# Preallocate arrays
states_val = zeros(T_val,)
outputs_val = zeros(T_val,)

# # Initialize marginal
# marginals[:pred] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))

# Preallocate prediction arrays
preds = (zeros(T_val,), zeros(T_val,))

@showprogress for k = 3:T_val

    # Simulate forward
    outputs_val[k], states_val[k] = sim_sys(inputs_val[k], states_val[[k-1,k-2]], sys_params)

    # Inferred prior states
    z_kmin1 = preds[1][[k-1,k-2]]

    # Posterior predictive
    preds[1][k] = mn(marginals[:θ])'*z_kmin1 + mn(marginals[:η])*inputs_val[k]
    preds[2][k] = z_kmin1'*pc(marginals[:θ])*z_kmin1 + inputs_val[k]'*pc(marginals[:η])*inputs_val[k] + inv(mn(marginals[:ξ]))

end
```

```julia
zoom = 3:min(100,T_val)

# Plot predictions
p23 = scatter(zoom, outputs_val[zoom], label="outputs", xlabel="time (t)", color="black", size=(900,300), legend=:topleft)
plot!(zoom, inputs_val[zoom], label="inputs", color="red")
plot!(zoom, preds[1][zoom], label="simulation", color="blue")
# plot!(zoom, preds[1][zoom], ribbon=[sqrt.(inv.(preds[2][zoom])), sqrt.(inv.(preds[2][zoom]))], label="", color="blue")
```

```julia
Plots.savefig(p23, "figures/simulations.png")
```

```julia
# Compute prediction error
sq_pred_error = (preds[1] .- outputs_val).^2

# Simulation error
MSE_sim = mean(sq_pred_error)

# Scatter error over time
p24 = scatter(1:T_val, sq_pred_error, color="black", xlabel="time (t)", ylabel="Prediction error", label="")
title!("MSE = "*string(MSE_sim))
```

```julia
Plots.savefig(p24, "figures/sim-errors.png")
```

### Experiment 2: Online system identification with agent-regulated inputs

```julia
goal_state = (.8, 1e-3)
```

```julia
function EFE_k(action, prior_state, goal_state, model_params)

    # Unpack model parameters
    θ, η, ζ, ξ = model_params

    # Process noise
    Σ_z = inv(ζ) *[1. 0.; 0. 0.]

    # Helper matrices
    S = [0. 0.; 1. 0.]
    s = [1.; 0.]

    # Unpack prior state
    μ_kmin = prior_state[1]
    Σ_kmin = prior_state[2]

    # Predicted observation
    y_hat = θ'*μ_kmin + η*action[1]

    # Covariance matrix
    Σ_k = Σ_kmin + Σ_z
    Σ_11 = Σ_k
    Σ_21 = s'*Σ_k
    Σ_12 = Σ_k*s
    Σ_22 = s'*Σ_k*s .+ inv(ξ)

    # Calculate conditional entropy
    Σ_cond = Σ_22 - Σ_21 * inv(Σ_11) * Σ_12
    ambiguity = 0.5(log2π + log(Σ_cond[1]) + 1)

    # Risk as KL between marginal and goal prior
    risk = KLDivergence(y_hat, Σ_22[1], goal_state[1], goal_state[2])

    # Update loss.
    return risk + ambiguity
end
```

```julia
# Time horizon
T = 300

# Goal state (mean and std dev)
goal_state = (.8, 1e-4)

# Number of variational updates
num_iterations = 4

# Preallocation
inputs = zeros(T,)
states = zeros(T,)
outputs = zeros(T,)

# Initialize constraints
constraints = Dict()

# Initialize marginals in factor graph
marginals = Dict()
marginals[:x] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.1*eye(2))
marginals[:z] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.1*eye(2))
marginals[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=randn(2,), w=.1*eye(2))
marginals[:η] = ProbabilityDistribution(Univariate, GaussianMeanPrecision, m=randn(1,)[1], w=.1)
marginals[:ζ] = ProbabilityDistribution(Univariate, Gamma, a=1, b=1)
marginals[:ξ] = ProbabilityDistribution(Univariate, Gamma, a=1e2, b=1e-2)

# Track estimated states
est_states = (zeros(T,), zeros(T,))

@showprogress for k = 3:T

    "Find control"

    # Pack parameters estimated by model
#     model_params = (mn(marginals[:θ]), mn(marginals[:η]), mn(marginals[:ζ]), mn(marginals[:ξ]))
    model_params = (θ_true, η_true, 1., ξ_true)

    # State prior variable
    prior_state = (mn(marginals[:z]), inv(pc(marginals[:z])))
#     prior_state = (states[[k-1,k-2]], eye(2))

    # Objective function
    G(u_k) = EFE_k(u_k, prior_state, goal_state, model_params)

    # Miminimize EFE
#     results = optimize(G, -1, 1, rand(1,), Fminbox(LBFGS()), Optim.Options(iterations=100); autodiff=:forward)
    results = optimize(G, rand(1,), LBFGS(), Optim.Options(iterations=100); autodiff=:forward)

    # Select first action in policy
    inputs[k] = Optim.minimizer(results)[1]

    "Execute control"

    # Execute input and observe output
    outputs[k], states[k] = sim_sys(inputs[k], states[[k-1,k-2]], sys_params)

    "Update model params"

    # Update constraints
    constraints = Dict(:y => outputs[k],
                       :u => inputs[k],
                       :m_x => mn(marginals[:z]),
                       :w_x => pc(marginals[:z]),
                       :m_θ => mn(marginals[:θ]),
                       :w_θ => pc(marginals[:θ]),
                       :m_η => mn(marginals[:η]),
                       :w_η => pc(marginals[:η]),
                       :a_ζ => marginals[:ζ].params[:a],
                       :b_ζ => marginals[:ζ].params[:b],
                       :a_ξ => marginals[:ξ].params[:a],
                       :b_ξ => marginals[:ξ].params[:b])

    # Iterate recognition factor updates
    for ii = 1:num_iterations

        # Update parameters
        stepη!(constraints, marginals)
        stepθ!(constraints, marginals)

        # Update states
        stepz!(constraints, marginals)
        stepx!(constraints, marginals)

        # Update noise precisions
#         stepζ!(constraints, marginals)
#         stepξ!(constraints, marginals)

    end

    # Store state estimates
    est_states[1][k] = mn(marginals[:z])[1]
    est_states[2][k] = sqrt(inv(pc(marginals[:z])[1,1]))

end
```

```julia
p1 = plot(3:T, inputs[3:T], color="red", label="inputs")
p2 = plot(3:T, outputs[3:T], color="blue", label="outputs")
plot!(3:T, est_states[1][3:T], color="purple", label="inferred")
plot!(3:T, est_states[1][3:T], ribbon=[est_states[2][3:T], est_states[2][3:T]], color="purple", label="")
plot!(3:T, goal_state[1]*ones(T-2,), color="green", label="goal state")
plot(p1, p2, layout=(2,1), size=(900,400))
```

#### Validation

```julia
# Time horizon
T_val = 1000

# Input signal
inputs_val = cos.(range(1, stop=T_val)*π/6);
# inputs_val = [mean(sin.(t./ (0.1:.1:6 .*π))) for t in 1:T_val];
```

```julia
# Preallocate arrays
states_val = zeros(T_val,)
outputs_val = zeros(T_val,)

# Preallocate prediction arrays
preds = (zeros(T_val,), zeros(T_val,))

@showprogress for k = 3:T_val

    # Simulate forward
    outputs_val[k], states_val[k] = sim_sys(inputs_val[k], states_val[[k-1,k-2]], sys_params)

    # Inferred prior states
    z_kmin1 = preds[1][[k-1,k-2]]

    # Posterior predictive
    preds[1][k] = mn(marginals[:θ])'*z_kmin1 + mn(marginals[:η])*inputs_val[k]
    preds[2][k] = z_kmin1'*pc(marginals[:θ])*z_kmin1 + inputs_val[k]'*pc(marginals[:η])*inputs_val[k] + mn(marginals[:ξ])

end
```

```julia
zoom = 3:T_val

# Plot predictions
p23 = plot(zoom, outputs_val[zoom], label="outputs", xlabel="time (t)", color="black", size=(900,300))
plot!(zoom, inputs_val[zoom], label="inputs", color="red")
plot!(zoom, preds[1][zoom], label="simulation", color="blue")
# plot!(zoom, preds[1][zoom], ribbon=[sqrt.(inv.(preds[2][zoom])), sqrt.(inv.(preds[2][zoom]))], label="", color="blue")
```

```julia
# Compute prediction error
sq_pred_error = (preds[1] .- outputs_val).^2

# Simulation error
MSE_sim = mean(sq_pred_error)

# Scatter error over time
p24 = scatter(1:T_val, sq_pred_error, color="black", xlabel="time (t)", ylabel="Prediction error", label="")
title!("MSE = "*string(MSE_sim))
```

```julia

```

```julia

```

```julia

```


## Quadrature

```python
import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib import cm
import seaborn as sns
from scipy.integrate import quad, fixed_quad
sns.set_context("talk", font_scale=1.5, rc={"lines.linewidth": 2.5})
sns.set_style("whitegrid")
import sympy as sp
from IPython.display import HTML
from matplotlib import animation
%matplotlib inline

# Don't tinker, or do
#%matplotlib nbagg
# from matplotlib import rcParams
#rcParams['font.family']='sans-serif'
#rcParams('font', serif='Helvetica Neue')
# rcParams['text.usetex']= True
#rcParams.update({'font.size': 22})
```

## Numerical integration schemes
Integration is a fundamental operation in calculus, and involves finding the area under a certain curve (geometrically). As and when you were first introduced to integration, you realized that certain functions cannot be analytically integrated (the reverse is not true however---differentiation of functions was more natural!). But the definite integral of such functions (indefinite integrals are important, but not central in engineering) can at least be estimated numerically. We look at some schemes/algorithms to do such an estimation. The problem we deal with is then to approximate
$$  \int_{a}^{b} f(x) dx  \approx \sum_{i=1}^{n} \omega_i f(x_i) $$
given an integrable function $f : [a,b] \to \mathbb{R}$ with *nodes* ($x_i$) and *weights* ($\omega_i$). Schemes that we will be discussing involve careful choices of these nodes and weights.

To get started, let us define a class that helps us implement any quadrature rule easily.

```python
class SpatialIntegrator(object):
    """ Class for wrapping a quadrature scheme with other goodies
    """
    def __init__(self, i_a, i_b, i_h):
        """ Initialize the integrator
        """
        # What forcing function are we using?
        self.forcing = None

        # What integration algorithm are we using?
        self.integrator = None

        self.a = i_a
        self.b = i_b
        self.h = i_h
        self.nsteps = int((i_b - i_a)/i_h)

    def set_forcing_function(self, t_func):
        """ Set forcing function (or) integrand to be used
        """
        if type(t_func) is not str:
            try:
                # If not string, try and evaluate the function
                t_func(0.0)
                # If the function works, set this function as forcing
                self.forcing = t_func
            except:
                raise RuntimeError('Provided function cannot be evaluated')

    def analytic(self):
        """  For testing convergence, defined as a special function """
        if self.forcing.__name__ == "composite_function":
            analytical_integral = (self.forcing(self.b)[1] - self.forcing(self.a)[1])
        else:
            def temp_func(x):
                return self.forcing(x)[0]
            analytical_integral, _ = quad(temp_func, self.a, self.b)
        return analytical_integral

    def integrate_using(self, integrator, renderer):
        """  Integrates the function and spits out the relative error
        """
        self.integrator = integrator.__name__
        all_starts = np.linspace(self.a, self.b - self.h, self.nsteps)
        all_ends = all_starts + self.h
        integral, start_end_values = integrator(self.forcing, all_starts, all_ends)
        integral = np.sum(integral)

        analytical_integral = self.analytic()
        print("Integral is {:20.16f}, Analytical integral is {:20.16f}, Relative error is {:20.16f}".format(integral, analytical_integral, np.abs(integral/analytical_integral - 1)))
        # Pass as (2xn) arrays. While reshaping do T to get (nx2) arrays
        self.draw(renderer, start_end_values)
        return np.abs(integral - analytical_integral)/np.abs(analytical_integral)

    def draw(self, renderer, st_end_vals):
        """ Draw the matplotlib canvas with the portrait we want
        """
        # If there is a timestepper, then there is numerical data
        # Plot them
        fine_mesh = np.linspace(self.a, self.b, 1001)
        renderer.plot(fine_mesh, self.forcing(fine_mesh)[0], 'r-', label=r'$f(x)$')

        # This step interleaves data present in 0,2,4,6... by transposing
        # them and then reshaping
        all_x_values = st_end_vals[0::2].T
        all_x_values = all_x_values.reshape(-1,)

        all_y_values = st_end_vals[1::2].T
        all_y_values = all_y_values.reshape(-1,)

        renderer.plot(all_x_values, all_y_values, 'k--')
        renderer.fill_between(all_x_values, all_y_values, alpha=0.2)
        renderer.legend()
        renderer.set_xlabel(r'$x$')
        renderer.set_ylabel(r'$y$')
        renderer.set_title(r'${}$'.format(self.integrator))
#         renderer.set_aspect('equal')

```

## Quadrature

A variety of rules for quadratures exist, and we are going to look at three simple ones : (a) Midpoint (b) Trapezoidal and (c) Simpson rules, although others (Clenshaw--Curtis, Gaussian quadrature) will be discussed on the way. We will attempt to compare these methods in terms of their ease (in understanding/implementation), order of accuracy (in comparison to the discretization $h$) and function evaluations for each step $h$.

## What do these rules do?
They approximate the area under the curve in a smart way---using local polynomials (in the case of interpolatory quadrature), you approximate the curve and integrate these polynomials instead. This approximation is not arbitrary, it represents a local Taylor series expansion. We'll first look at some schemes and what local polynomials they represent.

```python
def trapezoidal(func, lhs, rhs):
    """Does quadrature for one panel using the trapezoidal
    rule

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact integral
        as a tuple
    lhs : float/array-like
        The lower evaluation limit of the definite integral
    rhs : float/array-like
        The upper evalutation limit of the definite integral

    Returns
    -------
    integral : float/array-like
        Approximation of the integral using quadrature
    pts : np.array
        Points and values at which the local polynomials
        are drawn, specified as (x_1,fx_1,x_2,fx_2,...)
    """
    tmp1, tmp2 = func(lhs)[0], func(rhs)[0]
    return 0.5*(rhs-lhs)*(tmp1 + tmp2), np.array([lhs, tmp1, rhs, tmp2])
```

```python
def midpoint(func, lhs, rhs):
    """Does quadrature for one panel using the midpoint
    rule

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact integral
        as a tuple
    lhs : float/array-like
        The lower evaluation limit of the definite integral
    rhs : float/array-like
        The upper evalutation limit of the definite integral

    Returns
    -------
    integral : float/array-like
        Approximation of the integral using quadrature
    pts : np.array
        Points and values at which the local polynomials
        are drawn, specified as (x_1,fx_1,x_2,fx_2,...)
    """
    tmp = func(0.5*(lhs+rhs))[0]
    return (rhs-lhs)*tmp, np.array([lhs, tmp, rhs, tmp])
```

```python
def simpson(func, lhs, rhs):
    """Does quadrature for one panel using the Simpson
    rule

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact integral
        as a tuple
    lhs : float/array-like
        The lower evaluation limit of the definite integral
    rhs : float/array-like
        The upper evalutation limit of the definite integral

    Returns
    -------
    integral : float/array-like
        Approximation of the integral using quadrature
    pts : np.array
        Points and values at which the local polynomials
        are drawn, specified as (x_1,fx_1,x_2,fx_2,...)
    """
    tmp1, tmp2, tmp3 = func(lhs)[0], func(rhs)[0], func(0.5*(lhs+rhs))[0]
    return (rhs-lhs)/6. * (tmp1+tmp2+4*tmp3), np.array([lhs, tmp1, 0.5*(lhs+rhs), tmp3, rhs, tmp2])
```

## Testing quadrature?
Let's test these rules out between $[a,b]$ for simple polynomial (constant, linear, quadratic and cubic) curves and draw inferences from them (What's the error and so on...)

```python
# Bounds of integration
a = 0
b = 1
h = (b-a)

# Define function to be integrated and
# its integral
def test_func(x):
#     # Constant
#     return 1. + 0.*x, 0. + 1.*x

#     # Linear
#     return 0. + 1.*x, 0. + 0.5*x**2

#     # Quadratic
#     return 0. + 1.*x**2, 0. + x**3/3.

#    # Cubic
#     return 0. + 1.*x**3, 0. + x**4/4.

   # Quartic
    return 0. + 1.*x**4, 0. + x**5/5.
```

```python
fig, ax = plt.subplots(1,1, figsize=(6, 6))
si = SpatialIntegrator(a, b, h)
si.set_forcing_function(test_func)

# Try out different rules below
# si.integrate_using(midpoint, ax)
# si.integrate_using(trapezoidal, ax)
si.integrate_using(simpson, ax)
```

### Things to observe
- What local polynomial orders do they fit?
- Is there a relation between the errors of the midpoint and trapezoidal quadrature rule?
- How do they perform for functions that are more complicated (say, a cubic)?

## Composite quadrature
We saw that if the function is not simple, quadrature struggles to give us approximations that are close. When faced with this situtation, we do what we do best---throw in more points and hope for the best. This has a name---"Composite quadrature", which fits the curve into many small piecewise polynomials and integrates them instead. We can also rigorously show the error bounds for composite rules too. Let's do that now.

```python
# Bounds of integration
a = 0
b = 1

# All the change comes in this part only.
# Instead of simple interval, we pass many
# such intervals to the quadrature rule
# and sum them up
h = (b-a)/10
# h = (b-a)/50
# h = (b-a)/100

# Define not so simple function to be integrated
def composite_function(x):
#     return np.exp(2*x), 0.5*np.exp(2*x)

#     return np.exp(x)*(1-np.cos(2.*np.pi*x)), np.exp(x)*(1. + 4.*np.pi**2 - np.cos(2.*np.pi*x) - 2.*np.pi*np.sin(2.*np.pi*x))/(1. + 4.*np.pi**2)

    return np.sqrt(1-x**2), 0.5*( x * np.sqrt(1-x**2) + np.arcsin(x))

#     return np.log(2. + np.cos(2.*np.pi*x)), 2*x

#     return 1. + np.cos(2.*np.pi*x), x + np.sin(2.*np.pi*x)/(2.*np.pi)
```

```python
fig, ax = plt.subplots(1,1, figsize=(5, 5))
si = SpatialIntegrator(a, b, h)
si.set_forcing_function(composite_function)

# Try out different rules below
# si.integrate_using(midpoint, ax)
# si.integrate_using(trapezoidal, ax)
si.integrate_using(simpson, ax)
```

## Order of accuracy of composite quadrature rules
What's **order** of accuracy? Order of accuracy quantifies the rate of convergence of a numerical approximation of a integral to the exact integral.
The numerical solution ${u}$ is said to be $n^{\text{th}}$-order accurate if the error, $e(h):=\lvert\tilde{{u}}-{u} \rvert$ is proportional to the discretization-size $ h $, to the $n^{\text{th}}$ power. That is

$$ e(h)=\lvert\tilde{{u}}-{u} \rvert\leq C(h)^{n} $$

Details of this are given in the slides. Here, we focus on integrating a simple function (which has a non-negligible contribution from H.O.T in the Taylor series ) and figure out the order of convergence. The model problem that we deal with is

$$ \int_{0}^{1} e^{2x} dx $$
which as we know has the analytical solution $ \tilde{y} = \frac{e^{2}}{2} $, and so error can be calculated.

### Gaussian quadrature
Before doing tests for convergence, let us also implement Gaussian quadrature using `scipy.integrate.fixed_quad`

```python
def gauss(func, lhs, rhs):
    """Does quadrature for one panel using the Gaussian rule

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact integral
        as a tuple
    lhs : float/array-like
        The lower evaluation limit of the definite integral
    rhs : float/array-like
        The upper evalutation limit of the definite integral

    Returns
    -------
    integral : float/array-like
        Approximation of the integral using quadrature
    pts : np.array
        Points and values at which the local polynomials
        are drawn, specified as (x_1,fx_1,x_2,fx_2,...)
    """
    def temp_func(x):
        return func(x)[0]

    quad_sum = 0.0
    for i in range(lhs.shape[0]):
        quad_sum += fixed_quad(temp_func, lhs[i], rhs[i], n=3)[0]
    return quad_sum, np.array([lhs, 0.0*lhs, rhs, 0.0*rhs])
```

```python
# Bounds of integration
a = 0
b = 1

# All functions that you coded up
impl_list = [midpoint, trapezoidal, simpson, gauss]

# Time steps and associated errors from 25*1 to 25*12
h_steps = np.arange(1, 12, dtype=np.int16)
errors_list = [[None for i in h_steps] for impl in impl_list]

# Run simulations and collect errors
for i_impl, impl in enumerate(impl_list):
    for i_step, step in enumerate(h_steps):
        new_h = (b-a)/(25*(step))
        si = SpatialIntegrator(a, b, new_h)
        si.set_forcing_function(composite_function)
        errors_list[i_impl][i_step] = si.integrate_using(impl, ax)
```

```python
# Draw error plots in a log-log plot
fig, ax = plt.subplots(1,1, figsize=(6, 6))

# x axis is time, y axis is error
for i_impl, impl in enumerate(impl_list):
    ax.plot((b-a)/(25.*h_steps), errors_list[i_impl], 'o-', label=impl.__name__)

# Draw helpful slope lines to compare
x_ax = (b-a)/(25.*h_steps)
ax.plot(x_ax, 1.2 * x_ax**2, 'k--')
ax.plot(x_ax, 0.5 * x_ax**2, 'k--')
ax.plot(x_ax, 0.015 * x_ax**4, 'k--')

# Make it readable
ax.set_xlabel(r'$h$')
ax.set_ylabel(r'$e(h)$')
ax.set_title('Order of accuracy')
ax.set_yscale('log')
ax.set_xscale('log')
ax.legend()
# fig.savefig('ooa.pdf')
```

### Things to observe
- What is the order of accuracy in the composite case?
- How does this compare to our error estimation (through Taylor series expansion) for non-composite and composite cases?
- Is the error function dependent? (change functions and see). If so, why?

## Analysis of the functions above
Some further analysis is needed to explain the results above.

```python
sp.x = sp.symbols('x')
# sp_expr = sp.exp(sp.x)*(1-sp.cos(2.*sp.pi*sp.x))
sp_expr = sp.sqrt(1-sp.x**2)
```

```python
expr_list = []
max_derivative = 3
for i in range(max_derivative):
    new_expr = sp.diff(sp_expr, sp.x, i)
    expr_list.append(sp.lambdify(sp.x, new_expr, 'numpy'))
```

```python
fig, ax = plt.subplots(1,1, figsize=(10, 10))
x = np.linspace(0.0, 1.0, 1001)
for i_expr, expr in enumerate(expr_list):
    ax.plot(x, expr(x), label=r'{} derivative'.format(i_expr))
ax.legend()
```


## 2.2 Two-Way Tables and Two Categorical Variables

Data science is all about relationships between variables. How do we summarize and visualize the relationship between two categorical variables?

For example, what can we say about the relationship between gender and survival on the Titanic?

```python
import pandas as pd
data_dir = "http://dlsun.github.io/pods/data/"
df_titanic = pd.read_csv(data_dir + "titanic.csv")
```

We can summarize each variable individually like we did in the previous lesson.

```python
df_titanic["gender"].value_counts()
```

```python
df_titanic["survived"].value_counts()
```

But this does not tell us how gender interacts with survival. To do that, we need to produce a _cross-tabulation_, or "cross-tab" for short. (Statisticians tend to call this a _contigency table_ or a _two-way table_.)

```python
pd.crosstab(df_titanic["survived"], df_titanic["gender"])
```

A cross-tabulation of two categorical variables is a two-dimensional array, with the levels of one variable along the rows and the levels of the other variable along the columns. Each cell in this array contains the number of observations that had a particular combination of levels. So in the Titanic data set, there were 359 females who survived and 1366 males who died. From the cross-tabulation, we can see that there were more females who survived than not, while there were more males who died than not. Clearly, gender had a strong influence on survival because of the Titanic's policy of "women and children first".

To get probabilities instead of counts, we specify `normalize=True`.

```python
joint_survived_gender = pd.crosstab(df_titanic["survived"], df_titanic["gender"],
                                    normalize=True)
joint_survived_gender
```

Notice that the four probabilities in this table add up to 1.0. Each of these probabilities is called a joint probability and can be notated, for example, as

$$ P(\text{female}, \text{died}) = 0.058903.$$

Collectively, these probabilities make up the _joint distribution_ of the variables **survived** and **gender**.

### 2.2.1 Marginal Distributions

Is it possible to recover the distribution of **gender** alone from the joint distribution of **survived** and **gender**?

Yes! We simply sum the probabilities for each **gender** over all the possible levels of **survived**.

$$\begin{align}
P(\text{female}) = P(\text{female}, \text{died}) + P(\text{female}, \text{survived}) &= 0.058903 + 0.162664 = 0.221567 \\
P(\text{male}) = P(\text{male}, \text{died}) + P(\text{male}, \text{survived}) &= 0.618940 + 0.159493 = 0.778433
\end{align}$$

In code, this can be achieved by summing the `DataFrame` _over_ one of the dimensions. We can specify which dimension to sum over, using the `axis=` argument to `.sum()`.

- `axis=0` refers to the rows. In the current example, **survived** is the variable along this axis.
- `axis=1` refers to the columns. In the current example, **gender** is the variable along this axis.

Since we want to sum _over_ the **survived** variable, we specify `.sum(axis=0)`.

```python
gender = joint_survived_gender.sum(axis=0)
gender
```

When calculated from a joint distribution, the distribution of one variable is called a _marginal distribution_. So the above is the marginal distribution of **gender**.

The name "marginal distribution" comes from the fact that it is customary to write these totals in the _margins_ of the table. In fact `pd.crosstab()` has an argument `margins=` that automatically adds these margins to the cross-tabulation.

```python
pd.crosstab(df_titanic["survived"], df_titanic["gender"],
            normalize=True, margins=True)
```

While the margins are useful for display purposes, they actually make computations more difficult, since it is easy to mix up which numbers correspond to joint probabilities and which ones correspond to marginal probabilities.

Likewise, to obtain the marginal distribution of **survived**, we sum over the possible levels of **gender** (which is the variable along `axis=1`).

```python
survived = joint_survived_gender.sum(axis=1)
survived
```

We can check this answer by calculating the distribution of **survived** directly from the original data, using the techniques from the previous lesson.

```python
df_titanic["survived"].value_counts(normalize=True)
```

### 2.2.2 Conditional Distributions

Let's take another look at the joint distribution of **survived** and **gender**.

```python
joint_survived_gender
```

From the joint distribution, it is tempting to conclude that females and males did not differ too much in their survival rates, since

$$ P(\text{female}, \text{survived}) = 0.162664 $$

is not too different from

$$ P(\text{male}, \text{survived}) = 0.159493. $$

This is because there were 359 women and 352 men who survived, out of 2207 passengers.

But this is the wrong comparison. The joint probabilities are affected by the baseline gender probabilities, and over three-quarters of the people aboard the Titanic were men. $P(\text{male}, \text{survived})$ and $ P(\text{female}, \text{survived})$ should not even be close if men were just as likely to survive as women, simply because of the sheer number of men aboard.

A better comparison is between the conditional probabilities. We ought to compare

$$ P(\text{survived} | \text{female}) $$

to

$$ P(\text{survived} | \text{male}). $$

To calculate each conditional probability, we simply divide the joint probability by the marginal probability. That is,

$$\begin{align}
P(\text{survived} | \text{female}) = \frac{P(\text{female}, \text{survived})}{P(\text{female})} &= \frac{0.162664}{0.221568} = .7341 \\
P(\text{survived} | \text{male}) = \frac{P(\text{male}, \text{survived})}{P(\text{male})} &= \frac{0.159493}{0.778432} = .2049
\end{align}$$

The conditional probabilities expose the stark difference in survival rates. One way to think about conditional probabilities is that they _adjust_ for the baseline gender probabilities. By dividing by $P(\text{male})$ and $P(\text{female})$, we adjust for the fact that there were more men and fewer women on the Titanic, thus enabling an apples-to-apples comparison.

In code, this can be achieved by dividing the joint distribution by the marginal distribution (of **gender**). However, we have to be careful:

- The joint distribution is a two-dimensional array. It is stored as a `DataFrame`.
- The marginal distribution (of **gender**) is a one-dimensional array. It is stored as a `Series`.

How is it possible to divide a two-dimensional object by a one-dimensional object? Only if we _broadcast_ the one-dimensional object over the other dimension. A toy example is illustrated below.

$$\begin{align}
\begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix} \Big/ \begin{bmatrix} 7 \\ 8 \end{bmatrix} &= \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix}  \Big/ \begin{bmatrix} 7 & 8 \\ 7 & 8 \\ 7 & 8 \end{bmatrix} \\
&= \begin{bmatrix} 1/7 & 2/8 \\ 3/7 & 4/8 \\ 5/7 & 6/8 \end{bmatrix}
\end{align}$$

To do this in `pandas`, we use the `.divide()` method, specifying the dimension on which to align the `Series` with the `DataFrame`. Since **gender** is on `axis=1` of `joint_survived_gender`, we align the `DataFrame` and `Series` along `axis=1`.

```python
cond_survived_gender = joint_survived_gender.divide(gender, axis=1)
# In this case, joint_survived_gender / gender would also haved worked,
# but better to play it safe and be explicit about the axis.

cond_survived_gender
```

Every probability in this table represents a conditional probability of gender given survival status. So from the table, we can read that

$$ P(\text{survived} | \text{female}) = 0.734151. $$

Notice that each row sums to $1.0$---as it must, since given the information that a person was female, there are only two possibilities: she either survived or died.

In other words, we have a distribution of **survived** for each level of **gender**. We might wish to compare these two distributions. When we call `.plot.bar()` on the `DataFrame`, it will plot the values in each column as a set of bars with its own color.

```python
cond_survived_gender.plot.bar()
```

A different way to visualize a conditional distribution is to use a stacked bar graph. Here, we want one bar for females and another for males, each one divided in proportion to the survival rates for that gender. First, let's take a look at the desired graph.

```python
cond_survived_gender.T.plot.bar(stacked=True)
```

Now, let's unpack the code that generated this graphic. Recall that `.plot.bar()` plots each column of a `DataFrame` in a different color. Here we want different colors for each level of **survived**, so we need to swap the rows and columns of `cond_survived_gender`. In other words, we need the _transpose_ of the `DataFrame`, which is accomplished using `.T`.

```python
cond_survived_gender.T
```

When we call `.plot.bar()` on this transposed `DataFrame`, with `stacked=True`, we obtain the stacked bar graph above.

### 2.2.3 Exercises

Exercises 1-4 ask you to continue working with the Titanic data set explored in this lesson.

1\. Filter the data to include passengers only. Calculate the joint distribution between a passenger's class and where they embarked.

2\. Using the joint distribution that you calculated in Exercise 1, calculate the following:

- the conditional distribution of their class given where they embarked
- the conditional distribution of where they embarked given their class

Use the conditional distributions that you calculate to answer the following questions:

- What proportion of 3rd class passengers embarked at Southampton?
- What proportion of Southampton passengers were in 3rd class?

3\. Make a visualization showing the distribution of a passenger's class, given where they embarked.

4\. Compare the survival rates of crew members versus passengers. Which group appears to survive at higher rates?

(_Hint:_ You will have to transform the **class** variable to a variable that indicates whether a person was a passenger or a crew member. Refer to the previous lesson.)

Exercises 5-6 ask you to work with the Florida Death Penalty data set, which is available at  `https://dlsun.github.io/pods/data/death_penalty.csv`. This data set contains information about the races of the defendant and the victim, as well as whether a death penalty verdict was rendered, in 674 homicide trials in Florida between 1976-1987.

5\. Use the joint distribution to summarize the relationship between the defendant's and the victim's races in Florida homicides.

6\. Does there appear to be a relationship between death penalty verdicts and the defendant's race? If so, in what direction?

TesterBook/02_Categorical_Data/.ipynb_checkpoints/2.2 Two Categorical Variables and Cross-Tabulations-checkpoint.ipynb

TesterBook/02_Categorical_Data/2.2 Two Categorical Variables and Cross-Tabulations.ipynb


# Simple Linear Regression with NumPy

In school, students are taught to draw lines like the following.

$$ y = 2 x + 1$$

They're taught to pick two values for $x$ and calculate the corresponding values for $y$ using the equation.
Then they draw a set of axes, plot the points, and then draw a line extending through the two dots on their axes.

```python
# Import matplotlib.
import matplotlib.pyplot as plt
```

```python
# Draw some axes.
plt.plot([-1, 10], [0, 0], 'k-')
plt.plot([0, 0], [-1, 10], 'k-')

# Plot the red, blue and green lines.
plt.plot([1, 1], [-1, 3], 'b:')
plt.plot([-1, 1], [3, 3], 'r:')

# Plot the two points (1,3) and (2,5).
plt.plot([1, 2], [3, 5], 'ko')
# Join them with an (extending) green lines.
plt.plot([-1, 10], [-1, 21], 'g-')

# Set some reasonable plot limits.
plt.xlim([-1, 10])
plt.ylim([-1, 10])

# Show the plot.
plt.show()
```

Simple linear regression is about the opposite problem - what if you have some points and are looking for the equation?
It's easy when the points are perfectly on a line already, but usually real-world data has some noise.
The data might still look roughly linear, but aren't exactly so.

***

## Example (contrived and simulated)

#### Scenario
Suppose you are trying to weigh your suitcase to avoid an airline's extra charges.
You don't have a weighing scales, but you do have a spring and some gym-style weights of masses 7KG, 14KG and 21KG.
You attach the spring to the wall hook, and mark where the bottom of it hangs.
You then hang the 7KG weight on the end and mark where the bottom of the spring is.
You repeat this with the 14KG weight and the 21KG weight.
Finally, you place your case hanging on the spring, and the spring hangs down halfway between the 7KG mark and the 14KG mark.
Is your case over the 10KG limit set by the airline?

#### Hypothesis
When you look at the marks on the wall, it seems that the 0KG, 7KG, 14KG and 21KG marks are evenly spaced.
You wonder if that means your case weighs 10.5KG.
That is, you wonder if there is a *linear* relationship between the distance the spring's hook is from its resting position, and the mass on the end of it.

#### Experiment
You decide to experiment.
You buy some new weights - a 1KG, a 2KG, a 3Kg, all the way up to 20KG.
You place them each in turn on the spring and measure the distance the spring moves from the resting position.
You tabulate the data and plot them.

#### Analysis
Here we'll import the Python libraries we need for or investigations below.

```python
# Make matplotlib show interactive plots in the notebook.
%matplotlib inline
```

```python
# numpy efficiently deals with numerical multi-dimensional arrays.
import numpy as np

# matplotlib is a plotting library, and pyplot is its easy-to-use module.
import matplotlib.pyplot as plt

# This just sets the default plot size to be bigger.
plt.rcParams['figure.figsize'] = (8, 6)
```

Ignore the next couple of lines where I fake up some data. I'll use the fact that I faked the data to explain some results later. Just pretend that w is an array containing the weight values and d are the corresponding distance measurements.

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

```python
# Let's have a look at w.
w
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

```python
# Let's have a look at d.
d
```

    array([  7.8941861 ,  14.4990969 ,  15.54494165,  34.10112763,
            28.09931362,  32.40950674,  47.95067306,  50.56800243,
            53.57449031,  57.26477658,  60.90854081,  66.42632473,
            65.84573316,  65.8210405 ,  73.13032259,  81.26976123,
            93.94740543,  98.1664509 , 102.50090033, 113.22448452,
           104.44046614])

Let's have a look at the data from our experiment.

```python
# Create the plot.

plt.plot(w, d, 'k.')

# Set some properties for the plot.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Model
It looks like the data might indeed be linear.
The points don't exactly fit on a straight line, but they are not far off it.
We might put that down to some other factors, such as the air density, or errors, such as in our tape measure.
Then we can go ahead and see what would be the best line to fit the data.

#### Straight lines
All straight lines can be expressed in the form $y = mx + c$.
The number $m$ is the slope of the line.
The slope is how much $y$ increases by when $x$ is increased by 1.0.
The number $c$ is the y-intercept of the line.
It's the value of $y$ when $x$ is 0.

#### Fitting the model
To fit a straight line to the data, we just must pick values for $m$ and $c$.
These are called the parameters of the model, and we want to pick the best values possible for the parameters.
That is, the best parameter values *given* the data observed.
Below we show various lines plotted over the data, with different values for $m$ and $c$.

```python
# Plot w versus d with black dots.
plt.plot(w, d, 'k.', label="Data")

# Overlay some lines on the plot.
x = np.arange(0.0, 21.0, 1.0)
plt.plot(x, 5.0 * x + 10.0, 'r-', label=r"$5x + 10$")
plt.plot(x, 6.0 * x +  5.0, 'g-', label=r"$6x +  5$")
plt.plot(x, 5.0 * x + 15.0, 'b-', label=r"$5x + 15$")

# Add a legend.
plt.legend()

# Add axis labels.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Calculating the cost
You can see that each of these lines roughly fits the data.
Which one is best, and is there another line that is better than all three?
Is there a "best" line?

It depends how you define the word best.
Luckily, everyone seems to have settled on what the best means.
The best line is the one that minimises the following calculated value.

$$ \sum_i (y_i - mx_i - c)^2 $$

Here $(x_i, y_i)$ is the $i^{th}$ point in the data set and $\sum_i$ means to sum over all points.
The values of $m$ and $c$ are to be determined.
We usually denote the above as $Cost(m, c)$.

Where does the above calculation come from?
It's easy to explain the part in the brackets $(y_i - mx_i - c)$.
The corresponding value to $x_i$ in the dataset is $y_i$.
These are the measured values.
The value $m x_i + c$ is what the model says $y_i$ should have been.
The difference between  the value that was observed ($y_i$) and the value that the model gives ($m x_i + c$), is $y_i - mx_i - c$.

Why square that value?
Well note that the value could be positive or negative, and you sum over all of these values.
If we allow the values to be positive or negative, then the positive could cancel the negatives.
So, the natural thing to do is to take the absolute value $\mid y_i - m x_i - c \mid$.
Well it turns out that absolute values are a pain to deal with, and instead it was decided to just square the quantity instead, as the square of a number is always positive.
There are pros and cons to using the square instead of the absolute value, but the square is used.
This is usually called *least squares* fitting.

```python
# Calculate the cost of the lines above for the data above.
cost = lambda m,c: np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)])

print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 10.0, cost(5.0, 10.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (6.0,  5.0, cost(6.0,  5.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 15.0, cost(5.0, 15.0)))
```

    Cost with m =  5.00 and c = 10.00:   525.70
    Cost with m =  6.00 and c =  5.00:  1809.49
    Cost with m =  5.00 and c = 15.00:   974.82

#### Minimising the cost
We want to calculate values of $m$ and $c$ that give the lowest value for the cost value above.
For our given data set we can plot the cost value/function.
Recall that the cost is:

$$ Cost(m, c) = \sum_i (y_i - mx_i - c)^2 $$

This is a function of two variables, $m$ and $c$, so a plot of it is three dimensional.
See the **Advanced** section below for the plot.

In the case of fitting a two-dimensional line to a few data points, we can easily calculate exactly the best values of $m$ and $c$.
Some of the details are discussed in the **Advanced** section, as they involve calculus, but the resulting code is straight-forward.
We first calculate the mean (average) values of our $x$ values and that of our $y$ values.
Then we subtract the mean of $x$ from each of the $x$ values, and the mean of $y$ from each of the $y$ values.
Then we take the *dot product* of the new $x$ values and the new $y$ values and divide it by the dot product of the new $x$ values with themselves.
That gives us $m$, and we use $m$ to calculate $c$.

Remember that in our dataset $x$ is called $w$ (for weight) and $y$ is called $d$ (for distance).
We calculate $m$ and $c$ below.

```python
# Calculate the best values for m and c.

# First calculate the means (a.k.a. averages) of w and d.
w_avg = np.mean(w)
d_avg = np.mean(d)

# Subtract means from w and d.
w_zero = w - w_avg
d_zero = d - d_avg

# The best m is found by the following calculation.
m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
# Use m from above to calculate the best c.
c = d_avg - m * w_avg

print("m is %8.6f and c is %6.6f." % (m, c))
```

    m is 4.958010 and c is 10.781211.

Note that numpy has a function that will perform this calculation for us, called polyfit.
It can be used to fit lines in many dimensions.

```python
np.polyfit(w, d, 1)
```

    array([ 4.95801012, 10.78121051])

#### Best fit line
So, the best values for $m$ and $c$ given our data and using least squares fitting are about $4.95$ for $m$ and about $11.13$ for $c$.
We plot this line on top of the data below.

```python
# Plot the best fit line.
plt.plot(w, d, 'k.', label='Original data')
plt.plot(w, m * w + c, 'b-', label='Best fit line')

# Add axis labels and a legend.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')
plt.legend()

# Show the plot.
plt.show()
```

Note that the $Cost$ of the best $m$ and best $c$ is not zero in this case.

```python
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (m, c, cost(m, c)))
```

    Cost with m =  4.96 and c = 10.78:   521.60

### Summary
In this notebook we:
1. Investigated the data.
2. Picked a model.
3. Picked a cost function.
4. Estimated the model parameter values that minimised our cost function.

### Advanced
In the following sections we cover some of the more advanced concepts involved in fitting the line.

#### Simulating data
Earlier in the notebook we glossed over something important: we didn't actually do the weighing and measuring - we faked the data.
A better term for this is *simulation*, which is an important tool in research, especially when testing methods such as simple linear regression.

We ran the following two commands to do this:

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

The first command creates a numpy array containing all values between 1.0 and 21.0 (including 1.0 but not including 21.0) in steps of 1.0.

```python
 np.arange(0.0, 21.0, 1.0)
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

The second command is more complex.
First it takes the values in the `w` array, multiplies each by 5.0 and then adds 10.0.

```python
5.0 * w + 10.0
```

    array([ 10.,  15.,  20.,  25.,  30.,  35.,  40.,  45.,  50.,  55.,  60.,
            65.,  70.,  75.,  80.,  85.,  90.,  95., 100., 105., 110.])

It then adds an array of the same length containing random values.
The values are taken from what is called the normal distribution with mean 0.0 and standard deviation 5.0.

```python
np.random.normal(0.0, 5.0, w.size)
```

    array([ -1.06010546,   6.11989391,   6.50528315,  -0.64401265,
             5.56020188,  -9.21568895,   1.73234209,  -1.60719357,
           -10.31804015,  -1.65038869,  -0.70955811,   4.51754854,
             4.30969704,  -4.40688318,   0.70951891,   1.63830816,
            -2.28882717,   4.62862745,   1.30286098,   5.19446648,
            -1.77633161])

The normal distribution follows a bell shaped curve.
The curve is centred on the mean (0.0 in this case) and its general width is determined by the standard deviation (5.0 in this case).

```python
# Plot the normal distrution.
normpdf = lambda mu, s, x: (1.0 / (2.0 * np.pi * s**2)) * np.exp(-((x - mu)**2)/(2 * s**2))

x = np.linspace(-20.0, 20.0, 100)
y = normpdf(0.0, 5.0, x)
plt.plot(x, y)

plt.show()
```

The idea here is to add a little bit of randomness to the measurements of the distance.
The random values are entered around 0.0, with a greater than 99% chance they're within the range -15.0 to 15.0.
The normal distribution is used because of the [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) which basically states that when a bunch of random effects happen together the outcome looks roughly like the normal distribution. (Don't quote me on that!)

#### Plotting the cost function
We can plot the cost function for a given set of data points.
Recall that the cost function involves two variables: $m$ and $c$, and that it looks like this:

$$ Cost(m,c) = \sum_i (y_i - mx_i - c)^2 $$

To plot a function of two variables we need a 3D plot.
It can be difficult to get the viewing angle right in 3D plots, but below you can just about make out that there is a low point on the graph around the $(m, c) = (\approx 5.0, \approx 10.0)$ point.

```python
# This code is a little bit involved - don't worry about it.
# Just look at the plot below.

from mpl_toolkits.mplot3d import Axes3D

# Ask pyplot a 3D set of axes.
ax = plt.figure().gca(projection='3d')

# Make data.
mvals = np.linspace(4.5, 5.5, 100)
cvals = np.linspace(0.0, 20.0, 100)

# Fill the grid.
mvals, cvals = np.meshgrid(mvals, cvals)

# Flatten the meshes for convenience.
mflat = np.ravel(mvals)
cflat = np.ravel(cvals)

# Calculate the cost of each point on the grid.
C = [np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)]) for m, c in zip(mflat, cflat)]
C = np.array(C).reshape(mvals.shape)

# Plot the surface.
surf = ax.plot_surface(mvals, cvals, C)

# Set the axis labels.
ax.set_xlabel('$m$', fontsize=16)
ax.set_ylabel('$c$', fontsize=16)
ax.set_zlabel('$Cost$', fontsize=16)

# Show the plot.
plt.show()
```

#### Coefficient of determination
Earlier we used a cost function to determine the best line to fit the data.
Usually the data do not perfectly fit on the best fit line, and so the cost is greater than 0.
A quantity closely related to the cost is the *coefficient of determination*, also known as the *R-squared* value.
The purpose of the R-squared value is to measure how much of the variance in $y$ is determined by $x$.

For instance, in our example the main thing that affects the distance the spring is hanging down is the weight on the end.
It's not the only thing that affects it though.
The room temperature and density of the air at the time of measurment probably affect it a little.
The age of the spring, and how many times it has been stretched previously probably also have a small affect.
There are probably lots of unknown factors affecting the measurment.

The R-squared value estimates how much of the changes in the $y$ value is due to the changes in the $x$ value compared to all of the other factors affecting the $y$ value.
It is calculated as follows:

$$ R^2 = 1 - \frac{\sum_i (y_i - m x_i - c)^2}{\sum_i (y_i - \bar{y})^2} $$

Note that sometimes the [*Pearson correlation coefficient*](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used instead of the R-squared value.
You can just square the Pearson coefficient to get the R-squred value.

```python
# Calculate the R-squared value for our data set.
rsq = 1.0 - (np.sum((d - m * w - c)**2)/np.sum((d - d_avg)**2))

print("The R-squared value is %6.4f" % rsq)
```

    The R-squared value is 0.9732

```python
# The same value using numpy.
np.corrcoef(w, d)[0][1]**2
```

    0.9731819489090523

#### The minimisation calculations
Earlier we used the following calculation to calculate $m$ and $c$ for the line of best fit.
The code was:

```python
w_zero = w - np.mean(w)
d_zero = d - np.mean(d)

m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
c = np.mean(d) - m * np.mean(w)
```

In mathematical notation we write this as:

$$ m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \qquad \textrm{and} \qquad c = \bar{y} - m \bar{x} $$

where $\bar{x}$ is the mean of $x$ and $\bar{y}$ that of $y$.

Where did these equations come from?
They were derived using calculus.
We'll give a brief overview of it here, but feel free to gloss over this section if it's not for you.
If you can understand the first part, where we calculate the partial derivatives, then great!

The calculations look complex, but if you know basic differentiation, including the chain rule, you can easily derive them.
First, we differentiate the cost function with respect to $m$ while treating $c$ as a constant, called a partial derivative.
We write this as $\frac{\partial m}{ \partial Cost}$, using $\delta$ as opposed to $d$ to signify that we are treating the other variable as a constant.
We then do the same with respect to $c$ while treating $m$ as a constant.
We set both equal to zero, and then solve them as two simultaneous equations in two variables.

###### Calculate the partial derivatives
$$
\begin{align}
Cost(m, c)                       &= \sum_i (y_i - mx_i - c)^2      \\[1cm]
\frac{\partial Cost}{\partial m} &= \sum 2(y_i - m x_i -c)(-x_i) \\
                                 &= -2 \sum x_i (y_i - m x_i -c) \\[0.5cm]
\frac{\partial Cost}{\partial c} & = \sum 2(y_i - m x_i -c)(-1) \\
                                 & = -2 \sum (y_i - m x_i -c)   \\
\end{align}
$$

###### Set to zero
$$
\begin{align}
& \frac{\partial Cost}{\partial m}                         = 0 \\[0.2cm]
& \Rightarrow -2 \sum x_i (y_i - m x_i -c)                 = 0 \\
& \Rightarrow \sum (x_i y_i - m x_i x_i - x_i c)           = 0 \\
& \Rightarrow \sum x_i y_i - \sum_i m x_i x_i - \sum x_i c = 0 \\
& \Rightarrow m \sum x_i x_i                               = \sum x_i y_i - c \sum x_i \\[0.2cm]
& \Rightarrow m                                            = \frac{\sum x_i y_i - c \sum x_i}{\sum x_i x_i} \\[0.5cm]
& \frac{\partial Cost}{\partial c}               = 0 \\[0.2cm]
& \Rightarrow -2 \sum (y_i - m x_i - c)          = 0 \\
& \Rightarrow \sum y_i - \sum_i m x_i - \sum c   = 0 \\
& \Rightarrow \sum y_i - m \sum_i x_i            = c \sum 1 \\
& \Rightarrow c = \frac{\sum y_i - m \sum x_i}{\sum 1} \\
& \Rightarrow c = \frac{\sum y_i}{\sum 1} - m \frac{\sum x_i}{\sum 1} \\[0.2cm]
& \Rightarrow c = \bar{y} - m \bar{x} \\
\end{align}
$$

###### Solve the simultaneous equations
Here we let $n$ be the length of $x$, which is also the length of $y$.

$$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum x_i y_i - (\bar{y} - m \bar{x}) \sum x_i}{\sum x_i x_i} \\
& \Rightarrow m \sum x_i x_i = \sum x_i y_i - \bar{y} \sum x_i + m \bar{x} \sum x_i \\
& \Rightarrow m \sum x_i x_i - m \bar{x} \sum x_i = \sum x_i y_i - \bar{y} \sum x_i \\[0.3cm]
& \Rightarrow m = \frac{\sum x_i y_i - \bar{y} \sum x_i}{\sum x_i x_i - \bar{x} \sum x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x} - n \bar{y} \bar{x} + n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x} - n \bar{x} \bar{x} + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - \sum y_i \bar{x} - \sum \bar{y} x_i + n \bar{y} \bar{x}}{\sum (x_i x_i) - \sum x_i \bar{x} - \sum \bar{x} x_i + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \\
\end{align}
$$

#### End


# Extracting Information from Audio Signals

## Measuring amplitude (Session 1.9) - Kadenze

### George Tzanetakis, University of Victoria

In this notebook we will explore different ways of measuring the amplitude of a sinusoidal signal. The use of the inner product to estimate the amplitude of a sinusoids in the presence of noise and other sinusoids will also be covered. As usual we start by defining a sinusoid generation function.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import IPython.display as ipd

# generate a discrete time sinusoidal signal with a specified frequency and duration
def sinusoid(freq=440.0, dur=1.0, srate=44100.0, amp=1.0, phase = 0.0):
    t = np.linspace(0,dur,int(srate*dur))
    data = amp * np.sin(2*np.pi*freq *t+phase)
    return data
```

One way of measuring the amplitude of an audio signal is by finding the maximum value. As long the array of samples contains a few cycles of a sinusoidal signal this estimation works well.

```python
def peak_amplitude(data):
    return np.max(data)
```

Let's check it out:

```python
freq = 550
data = sinusoid(freq, 0.5, amp =4.0)
print('Peak amplitude = %2.2f ' % peak_amplitude(data))
```

    Peak amplitude = 4.00

Now let's define a function that returns the Root of the Mean Squarred (RMS) amplitude. For an array containing a few cycles of a sinusoid signal we can estimate the RMS amplitude as follows:

```python
def rms_amplitude(data):
    rms_sum = np.sum(np.multiply(data,data))
    rms_sum /= len(data)
    return np.sqrt(rms_sum) * np.sqrt(2.0)
```

Let's check out that this method of estimation also works:

```python
freq = 550
data = sinusoid(freq, 0.5, amp =8.0)
print('Rms amplitude = %2.2f' %  rms_amplitude(data))
```

    Rms amplitude = 8.00

Now let's look at estimating the amplitude based on taking the dot product of two sinusoids.
Unlike the peak and RMS methods of estimating amplitude this method requires knowledge of the
frequency (and possibly phase) of the underlying sinusoid. However, it has the advantage that it is much more robust when there is interferring noise or other sinusoidal signals with other frequencies.

```python
def dot_amplitude(data1, data2):
    dot_product = np.dot(data1, data2)
    return 2 * (dot_product / len(data1))
```

First lets confirm that this amplitude estimation works for a single sinusoid

```python
data = sinusoid(300, 0.5, amp =4.0)
basis = sinusoid(300, 0.5, amp = 1)
print('Dot product amplitude = %2.2f' % dot_amplitude(data, basis))
plt.figure()
plt.plot(data[1:1000])
plt.plot(basis[1:1000])
```

Now lets add some noise to our signal. Notice that the dot-amplitude estimation works reliably, the RMS still does ok but the peak amplitude gets really affected by the added noise. Notice that the dot product amplitude estimation requires knowledge of the frequency to create the appropriate basis signal

```python
noise = np.random.normal(0, 1.0, len(data))
mix = data + noise
plt.figure()
plt.plot(data[1:1000])
plt.plot(noise[1:1000])
plt.plot(mix[1:1000])
plt.plot(basis[1:1000])
print('Dot product amplitude = %2.2f' % dot_amplitude(mix, basis))
print('Peak amplitude = %2.2f' % peak_amplitude(mix))
print('RMS amplitude = %2.2f' % rms_amplitude(mix))
```

```python
data_other = sinusoid(500, 0.5, amp = 3.0)
mix = data + data_other
plt.figure()
#plt.plot(data_other[1:1000])
#plt.plot(data[1:1000])
plt.plot(mix[1:1000])
plt.plot(basis[1:1000])
print('Dot product amplitude = %2.2f' % dot_amplitude(mix, basis))
print('Peak amplitude = %2.2f' % peak_amplitude(mix))
print('RMS amplitude = %2.2f' % rms_amplitude(mix))
```

To summarize, if we know the frequency of the sinusoid we are interested in, we can use the inner product with a sinusoid of the same frequency and phase as a robust way to estimate the amplitude in the presence of interferring noise and/or sinusoidal signals of different frequencies. If we don't know the phase we can use an iterative approach of trying every possible phase and selecting the one that gives the highest amplitude estimate - the brute force approach we talked about in a previous notebook.

However, there is a simpler approach to estimating both the amplitude and the phase of a sinusoidal signal of known frequency.

It is based on the following identity:
\begin{equation} a \sin(x) + b \cos(x) = R \sin (x + \theta)
\end{equation}
where
$ R = \sqrt{(a^2 + b^2)} \;\; \text{and} \;\; \theta = \tan^{-1} \frac{b}{a} $

So basically we can represent a sinusoidal signal of a particular amplitude and phase as a weighted sum (with appropriate weights $ a \;\; \text{and}\;\; b$ of a sine signal and a cosine signal. So to estimate the amplitude and phase of a sinusoid of known frequency we can take the inner product with a pair of sine and cosine signals of the same frequncy. Let's see how this would work. We will see later that these pairs of sines and cosine signals are what are called basis functions of the Discrete Fourier Transform.

```python
srate = 8000
amplitude = 3.0
k = 1000
phase = k * (2 * np.pi / srate)
print('Original amplitude  = %2.2f' % amplitude)
print('Original phase = %2.2f' % phase)

data = sinusoid(300, 0.5, amp =amplitude, phase = phase)
plt.plot(data[1:1000])
basis_sin = sinusoid(300, 0.5, amp = 1)
basis_cos = sinusoid(300, 0.5, amp = 1, phase = np.pi/2)

a = dot_amplitude(data, basis_sin)
b = dot_amplitude(data, basis_cos)
estimated_phase = np.arctan(b/a)
estimated_magnitude = np.sqrt(a*a+b*b)
print('Estimated Magnitude = %2.2f' % estimated_magnitude)
print('Estimated Phase = %2.2f' % estimated_phase)

```

```python

```

```python

```

```python

```


# Arbitrary precision
Typically numerical optimization is performed using binary 64 bit IEEE 754 floating point numbers (becuase modern processors offer good performance/precision characteristics for this data type). By using a math library which supports arbitrary precision we can achive higher precision (larger number of significant digits). By using ``pyneqsys`` we only need to write our equations once. We will be using [mpmath](http://www.mpmath.org) as our backend library for arbitrary precision.

```python
import sympy as sp
from pyneqsys.symbolic import SymbolicSys
sp.init_printing()
```

```python
def f(x):
    return [x[0]**2 + x[1],
            5*x[0]**2 - 3*x[0] + 2*x[1] - 3]

neqsys = SymbolicSys.from_callback(f, 2)
neqsys.exprs
```

Let us now find the roots for this system numerically, if we read the documentation of that library we will see that they recommend that the starting guess is obtained using a conventional solver.

So we will do just that: we will first solve the problem using SciPy's default solver and then we will perform a "refinement" by solving it using ``mpmath``'s solver

```python
x, sol = neqsys.solve([100, 100])
sol, f(sol['x'])
```

```python
mp_neqsys = SymbolicSys.from_callback(f, 2, module='mpmath')
mp_x, mp_sol = mp_neqsys.solve(x, solver='mpmath', dps=50)
mp_sol, f(mp_sol['x'])
```

So that gave us, as requested, close to 50 significant digits‒nifty!


# Introduction

In this section we learn to do the following:

* Import SymPy and set up pretty printing
* Use mathematical operations like `sqrt` and `sin`
* Make SymPy Symbols
* Take derivatives of expressions
* Simplify expressions

## Preamble

Just like NumPy and Pandas replace functions like `sin`, `cos`, `exp`, and `log` to powerful numeric implementations, SymPy replaces `sin`, `cos`, `exp` and `log` with powerful mathematical implementations.

```python
from sympy import *
init_printing()  # Set up fancy printing
```

```python
import math
math.sqrt(2)
```

```python
sqrt(2)  # This `sqrt` comes from SymPy
```

```python
cos(0)
```

### Exercise

Use the function `acos` on `-1` to find when cosine equals `-1`.  Try this same function with the math library.  Do you get the same result?

```python
# Call acos on -1 to find where on the circle the x coordinate equals -1

```

```python
# Call `math.acos` on -1 to find the same result using the builtin math module.
# Is the result the same?
# What does `numpy.arccos` give you?

```

## Symbols

Just like the NumPy `ndarray` or the Pandas `DataFrame`, SymPy has `Symbol`, which represents a mathematical variable.

We create symbols using the function `symbols`.  Operations on these symbols don't do numeric work like with NumPy or Pandas, instead they build up mathematical expressions.

```python
x, y, z = symbols('x,y,z')
alpha, beta, gamma = symbols('alpha,beta,gamma')
```

```python
x + 1
```

```python
log(alpha**beta) + gamma
```

```python
sin(x)**2 + cos(x)**2
```

### Exercise

Use `symbols` to create two variables, `mu` and `sigma`.

```python
?, ? = symbols('?')
```

### Exercise

Use `exp`, `sqrt` and Python's arithmetic operators like `+, -, *, **` to create the standard bell curve with SymPy objects

$$ e^{\frac{(x - \mu)^2}{ \sigma^2}} $$

```python
exp(?)
```

## Derivatives

One of the most commonly requested operations in SymPy is the derivative.  To take the derivative of an expression use the `diff` method

```python
(x**2).diff(x)
```

```python
sin(x).diff(x)
```

```python
(x**2 + x*y + y**2).diff(x)
```

```python
diff(x**2 + x*y + y**2, y) # diff is also available as a function
```

### Exercise

In the last section you made a normal distribution

```python
mu, sigma = symbols('mu,sigma')
```

```python
bell = exp((x - mu)**2 / sigma**2)
bell
```

Take the derivative of this expression with respect to $x$

```python
?.diff(?)
```

### Exercise

There are three symbols in that expression.  We normally are interested in the derivative with respect to `x`, but we could just as easily ask for the derivative with respect to `sigma`.  Try this now

```python
# Derivative of bell curve with respect to sigma

```

### Exercise

The second derivative of an expression is just the derivative of the derivative.  Chain `.diff( )` calls to find the second and third derivatives of your expression.

```python
#  Find the second and third derivative of `bell`

```

## Functions

SymPy has a number of useful routines to manipulate expressions.  The most commonly used function is `simplify`.

```python
expr = sin(x)**2 + cos(x)**2
expr
```

```python
simplify(expr)
```

### Exercise

In the last section you found the third derivative of the bell curve

```python
bell.diff(x).diff(x).diff(x)
```

You might notice that this expression has lots of shared structure.  We can factor out some terms to simplify this expression.

Call `simplify` on this expression and observe the result.

```python
# Call simplify on the third derivative of the bell expression

```

## Sympify

The `sympify` function transforms Python objects (ints, floats, strings) into SymPy objects (Integers, Reals, Symbols).

*note the difference between `sympify` and `simplify`.  These are not the same functions.*

```python
sympify('r * cos(theta)^2')
```

It's useful whenever you interact with the real world, or for quickly copy-paste an expression from an external source.


```python
from sympy import symbols, laplace_transform, \
                  DiracDelta, Heaviside, exp, sin, cos

s, t, a, n, b = symbols('s t a n b')

laplace_transform(DiracDelta(t), t, s)
```

    (1 - Heaviside(0), -oo, True)

```python
laplace_transform(Heaviside(t), t, s, noconds=True)
```

$\displaystyle \frac{1}{s}$

```python
laplace_transform(1, t, s, noconds=True)
```

$\displaystyle \frac{1}{s}$

```python
laplace_transform(t, t, s, noconds=True)
```

$\displaystyle \frac{1}{s^{2}}$

```python
laplace_transform(t**n, t, s, noconds=True)
```

$\displaystyle \frac{s^{- n} \Gamma\left(n + 1\right)}{s}$

```python
laplace_transform(exp(-a*t), t, s, noconds=True)
```

$\displaystyle \frac{1}{a + s}$

```python
laplace_transform(sin(b*t), t, s, noconds=True)
```

$\displaystyle \frac{b}{b^{2} + s^{2}}$

```python
laplace_transform(cos(b*t), t, s, noconds=True)
```

$\displaystyle \frac{s}{b^{2} + s^{2}}$

```python
laplace_transform(exp(-a*t)*sin(b*t), t, s, noconds=True)
```

$\displaystyle \frac{b}{b^{2} + \left(a + s\right)^{2}}$

```python
laplace_transform(exp(-a*t)*cos(b*t), t, s, noconds=True)
```

$\displaystyle \frac{a + s}{b^{2} + \left(a + s\right)^{2}}$


## Mathematics with Python

All notebooks can be found [https://github.com/drvinceknight/mwp](https://github.com/drvinceknight/mwp).

### Requirements for this workshop

- Python (version 3+)
- Sympy
- Numpy
- Matplotlib
- Jupyter notebooks (although if you're comfortable using Python in another way do not feel obliged to use notebooks)

### Arithmetic

It is possible to use Python to carry out arithmetic. For example

```python
2 + 2
```

    4

```python
538 * 612 / 24
```

    13719.0

### Exercises

- Calculate $42 ^ 2$
- Calculate $56 / 2 \times 5$

## Symbolic mathematics

Most of mathematics involves the use of symbolic variables. We can use a python library called `sympy` to do this. For example, let us compute:

$$x + x$$

```python
import sympy as sym
x = sym.Symbol("x")  # Creating a symbolic variable x
x + x
```

    2*x

### Exercises

- Compute $x - 2x$
- Compute $x + 2y - 3x + y$

## Nicer output

`sympy` can use $\LaTeX$ to display mathematics when using Jupyter notebooks:

```python
sym.init_printing()
x + x
```

## Substituting values in to expressions

If we need to compute a numerical value, it is possible to do so:

```python
expr = x + x
expr
```

```python
expr.subs({x: 4})
```

### Exercises

- Substitute $x=5, y=7$ in to the expression: $x + 2y$

## Expanding expressions

We can use `sympy` to verify expressions like the following:

$$(a + b) ^ 2 = a ^ 2 + 2 a b + b ^ 2$$

```python
a, b = sym.symbols("a, b")  # Short hand: note we're using `sym.symbols` and not `sym.Symbol`
expr = (a + b) ** 2
expr
```

```python
expr.expand()
```

A `sympy` expression not only retains mathematical information but also the form, indeed the following two expressions are not the same. They have equal values.

```python
expr == expr.expand()
```

    False

### Exercises

- Expand $(a + b + c)^2$
- Expand $(a + b) ^ 3$

## Factorising expressions

We can also factor expressions like:

$$a ^ 2 - b ^ 2 = (a - b)(a + b)$$

```python
sym.factor(a ** 2 - b ** 2)
```

### Exercises

- Factorise $a ^ 3 - b ^ 3$
- Factorise $4x + x ^ 2 - yx$

## Solving equations

We can use `sympy` to solve algebraic equations. Let us take a look at the quadratic equation:

$$ax^2 + b x + c = 0$$

```python
c = sym.Symbol('c')
eqn = a * x ** 2 + b * x + c
sym.solveset(eqn, x)
```

### Exercises:

- Obtain the solution to: $ax^3+bx^2+cx+d=0$
- Obtain the solution to $x^2 + 1 = 0$