# Introduction and overview

In this course, we will build on the techniques learnt in numerical analysis in CMIII and make some of the notions there more rigorous.

In addition to thinking about discretising in time (ODEs) we will also look at discretising differential equations in space (PDEs), and the interplay between them.

We'll start by recapitulating some of the terminology and results from CMIII, in particular stability and consistency of numerical methods for ODEs.

Some of this material is taken from [Jed Brown's](https://jedbrown.org) numerical computation course https://github.com/cucs-numcomp/numcomp-class/, licensed under BSD 2-clause.

This is a [Jupyter](https://jupyter.org) notebook, which you can execute. I recommend using something like the following:

```shell
$ python3 -m venv scicomp
$ . scicomp/bin/activate # for bash shells, other activation scripts are available
$ pip install matplotlib jupyter ipython numpy scipy
```

A readable textbook that covers this material (with more rigour than in this course, but still easy to follow) is Arieh Iserles' [*A first course in the numerical analysis of differential equations*](http://www.damtp.cam.ac.uk/user/ai/Arieh_Iserles/Textbook.html).

Our treatment will be somewhat more computational.

## Discretising an ODE

### Method of lines

When we eventually encounter spatial derivatives, along with time derivatives, we will first discretise in space, obtaining a system of ODEs and then in time. This is known as the _method of lines_. For now, we will just assume that we have already discretised in space to obtain the ODE system

$$
 \dot u = f(t, u), t \ge t_0, u(t_0) = u_0
$$

where $f : [t_0, \infty) \times \mathbb{R}^d \to \mathbb{R}^d$, $u \in \mathbb{R}^d$ and we require that $f$ is sufficiently "nice".

We will at the very least, require that $f$ is Lipschitz:

$$
\| f(t, x) - f(t, y) \| \le \lambda \| x - y \| \text{ for all } x, y \in \mathbb{R}^d, t \ge t_0,
$$

where $\lambda > 0$ is the _Lipschitz constant_. Informally, this says that we can bound the difference in the output of $f$ for two values by the difference of the values.

Sometimes we will want $f$ to be analytic (so that we can perform Taylor expansions).

This is a _first order_ ODE, since there is only one time derivative.

#### Uniqueness

If the Lipschitz condition is satisfied, one can prove that the ODE system has a unique solution.

We will call $u^*(t)$ a _solution_ to $\dot u = f(t, u)$, if, for all $t$

$$
\dot u^*(t) = f(t, u^*(t)).
$$

We note, and are not the first, that this solution usually contains a free parameter, so that for uniqueness we need to specify an _initial value_ $u(0) = u_0$. Such problems are therefore termed _initial value problems_ or IVPs.

## Discretising the $\dot u$ term

### Explicit Euler

We can formally write down the solution to our ODE problem by integrating the RHS:

$$
u(t) = u(t_0) + \int_{t_0}^t f(\tau, u(\tau)) \text{d}\tau.
$$

We can _approximate_ the integral using linear interpolation by making the approximation $f(t, u(t)) \approx f(t_0, u(t_0))$ for $t \in [t_0, t_0 + h]$ with $h > 0$ sufficiently small.

We then obtain

$$
u(t_0 + h) \approx u(t_0) + (t_0  + h - t_0)f(t_0, u(t_0))
$$

writing $u_n$ as the numerical estimate of the solution at $u(t_n)$ we obtain the explicit (or forward) Euler method:

$$
u_{n+1} = u_n + h f(t_n, u_n).
$$

Let's look at how this behaves for a sample problem (this problem is *stiff*, a property we will return to later)

$$
\dot u = -k(u - \cos{t})
$$

For an initial value $u(0) = u_0$, this problem has exact solution

$$
u(t) = \frac{k}{1+k^2}(\sin{t} + k \cos{t}) + \left(u_0 - \frac{k^2}{1+k^2}\right) \exp{(-kt)}.
$$

(hint: use an [integrating factor](https://en.wikipedia.org/wiki/Integrating_factor) and integrate by parts twice to compute the resulting integral). Note that most real problems will not have such easy to compute solutions.

Let's go ahead and integrate this equation numerically

```python
%matplotlib notebook
import numpy
from matplotlib import pyplot
pyplot.style.use("ggplot")

class f(object):
    def __init__(self, k):
        self.k = k

    def __call__(self, t, u):
        return -self.k * (u - numpy.cos(t))

    def __str__(self):
        return "f(k={})".format(self.k)

    def exact(self, t, u_0):
        k = self.k
        k2p1 = k/(1 + k**2)
        return (u_0 - k*k2p1)*numpy.exp(-k*t) + k2p1 * (numpy.sin(t) + k*numpy.cos(t))

def ode_euler(f, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    if isinstance(f, numpy.ndarray):
        f_ = f
        # Hack for linear problems
        f = lambda t_, u_: f_ @ u_
    while t < T:
        h = min(h, T - t)
        u = u + h * f(t, u)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)

u_0 = numpy.array(0.2)

pyplot.figure()

for k in [2, 10]:
    rhs = f(k)
    thist, uhist = ode_euler(rhs, u_0, h=.1, T=12)
    pyplot.plot(thist, uhist, "o", linestyle="solid", label=str(rhs)+' Forward Euler')
    pyplot.plot(thist, rhs.exact(thist, u_0), label=str(rhs)+' exact')
pyplot.legend(bbox_to_anchor=(0.5, 1), loc='center', ncol=2);
```

This looks pretty good in the eyeball norm.

#### Questions

1. What happens when you increase $k$ further?
2. What about if you try and increase $h$?

```python

```

## Convergence of a method

A timestepping method is _convergent_ if, as we take the timestep $h \to 0$, the numerical solution tends to the exact solution. Formally, we require for a time interval $[t_0, t_0 + t^*]$ that

$$
\lim_{h \to 0^+} \max_{n} \| u_{n, h} - u(t_n) \| = 0 \text{ for all } t^* > 0,
$$
where the maximum is taken over all timesteps, $h$ denotes the step size, and $u(t_n)$ denotes the exact solution at time $t_n$.

### Is explicit Euler convergent?

Let's check. Write $e_{n, h} := u_{n, h} - u(t_n)$ for the numerical error. Write $t_{n+1} = t_n + h$ and Taylor expand $u$ around $t_n$:

$$
u(t_{n+1}) = u(t_n) + h \dot u(t_n) + \mathcal{O}(h^2).
$$

Now substitute in the definition of $\dot u$ to obtain

$$
u(t_{n+1}) = u(t_n) + h f(t, u(t_n)) + \mathcal{O}(h^2)
$$

Subtract this from the numerical $u_{n+1}$ obtained from the Euler formula to obtain

$$
e_{n+1, h} = e_{n, h} + h[f(t, \underbrace{u(t_n) + e_{n, h}}_{u_n}) - f(t, u(t_n))] + \mathcal{O}(h^2)
$$

We're interested in bounding the size of this error, so let's take norms of both sides

$$
\|e_{n+1, h}\| = \|e_{n, h} + h[f(t, u(t_n) + e_{n, h}) - f(t, u(t_n))] + \mathcal{O}(h^2)\|
$$

Now use the triangle inequality

$$
\|a + b\| \le \|a\| + \|b\|
$$

to split the term on the right hand side, and use the Lipschitz condition

$$
\begin{align}
\|e_{n+1, h}\| &\le \|e_{n, h}\| + h\|[f(t, u(t_n) + e_{n, h}) - f(t, u(t_n))]\| + \|\mathcal{O}(h^2)\|\\
&\le (1 + h\lambda)\|e_{n,h}\| + \mathcal{O}(h^2) = (1 + h\lambda)\|e_{n,h}\| + c h^2
\end{align}
$$

By induction on $n$

$$
\|e_{n, h}\| \le \frac{c}{\lambda} h [(1 + h\lambda)^n - 1], n = 0, 1, \dots
$$

Finally we have $(1 + h\lambda) < e^{h\lambda}$ (hint: taylor expand exponential and note that both $h$ and $\lambda$ are positive) and so $(1 + h\lambda)^n < e^{nh\lambda}$. Since $n$ indexes the time interval, $\max_n = \lfloor t^* / h \rfloor$ and so $e^{nh\lambda} < e^{\lfloor t^*/h \rfloor h \lambda} \le e^{t^*\lambda}$. Hence

$$
\|e_{n, h}\| \le \frac{c}{\lambda} h (e^{t^* \lambda} -1),
$$

which converges to zero as $h \to 0$.

#### Observations

It is tempting to use this to provide an upper bound on the error of the method, however, the bound is not at all sharp (typically overestimating the error by many orders of magnitude).

#### Exercises

1. Prove the relationship $\|e_{n, h}\| \le \frac{c}{\lambda} h [(1 + h\lambda)^n - 1]$ by doing the induction on $n$. Hint: use $n = 0$ for a base case noting that $e_{0, h} = 0$.

2. For the scalar equation $\dot u = -100 u, u(0) = 1$, compute the bound on the error given by our convergence proof for $t^* = 1$. Hint: since $f$ is just linear scaling, $\lambda = 100$ and we can take $c = \lambda^2$. The exact solution is $e^{-100t}$.

3. For the same equation, derive an exact expression for the error. Hint: the value computed by forward Euler after $n$ timesteps can be computed in closed form for this case by repeatedly substituting into the definition of a single timestep. Compare the the error obtained from this expression from the bound in question 2 for various values of $h$.

```python

```

### The order of a method

Given some timestepping method

$$
y_{n+1} = \mathcal{Y}_n(f, h, y_0, \dots, y_n)
$$

we say it is of *order $p$* if

$$
y(t_{n+1}) - \mathcal{Y}_n(f, h, y_0, \dots, y_n) = \mathcal{O}(h^{p+1}).
$$

That is, for every timestep, we accumulate $\mathcal{O}(h^{p+1})$ error. Since we require $\mathcal{O}(h^{-1})$ timesteps, the global error will be $\mathcal{O}(h^p)$.

Note, this requires some further conditions (the method must at least be convergent), which we will come to.

### Implicit Euler

Rather than evaluating all the right hand side terms in our time discretisation *explicitly*, using already known values of $u$ at the beginning of the timestep, we can also evaluate them _implicitly_ using values at the end of the timestep. In that case, we end up with the discrete problem

$$
u(t + h) - h f(t + h, u(t + h)) = u(t).
$$

This is called _implicit_ (or backward) Euler. In the general case where $f$ is nonlinear, we must solve a nonlinear equation at this point. Let's skip over that for now and only consider the linear problem where we can write

$$
f(t, u) = A(t)u
$$

for some matrix $A(t)$ which may or may not depend on $t$.
Rearranging, we obtain

$$
(I - h A)u(t + h) = u(t)
$$

and so a single step now requires us to invert the matrix $(I - h A)$. This may be _significantly_ more expensive than just evaluating the right hand side (a matrix-vector product). So what does this buy us?

#### Aside: exact solution of linear problems

The solution to the problem
$$
\dot u = A u
$$
can be written using the [matrix exponential](https://en.wikipedia.org/wiki/Matrix_exponential) as
$$
u(t) = \exp{(A t)}u(0)
$$
where the exponential is formally defined using the Taylor series
$$
\exp{A} = \sum_{n=0}^{\infty} \frac{A^{n}}{n!},
$$
and there are many (both good and bad) [ways to compute it](https://doi.org/10.1137/S00361445024180).

If we can efficiently evaluate the matrix exponential, then we can directly compute the solution to our linear ODE at any time $t$.

### Back to Euler integration

Let's now look at the behaviour of explict and implicit Euler on a test problem with purely oscillatory solutions. Choosing

$$
A = \begin{bmatrix} 0 & 1\\ -1 & 0 \end{bmatrix}
$$
and
$$
u_0 = \begin{bmatrix} 0.75 \\ 0 \end{bmatrix}
$$
we expect
$$
u(t) = \begin{bmatrix} 0.75 \cos t\\-0.75 \sin t\end{bmatrix}.
$$

Rather than laboriously working this out each time, we can compute an "exact" solution by explicitly computing the matrix exponential. [scipy](https://scipy.org) provides a builtin function to do this, `scipy.linalg.expm`.

##### Question

If scipy didn't provide this function, can you think of a fast way of computing matrix exponentials if the matrix $A$ is diagonalisable? That is, we can write

$$
A = X \Lambda X^{-1}
$$

with

$$
\Lambda = \operatorname{diag}\lambda.
$$

Hint, substitute the expansion into the power series and note that $X X^{-1} = \mathbb{1}$, and that exponentiation of a diagonal matrix commutes. That is, if $\Lambda = \operatorname{diag}\lambda$, then $e^\Lambda = \operatorname{diag}e^{\lambda}$

```python
from scipy.linalg import expm

class linear(object):
    def __init__(self, A):
        self.A = A.copy()

    def __call__(self, t, u):
        return self.A @ u

    def exact(self, t, u_0):
        t = numpy.array(t, ndmin=1)
        return [numpy.real_if_close(expm(self.A*s) @ u_0) for s in t]

test = linear(numpy.array([[0, 1],
                           [-1, 0]]))
u_0 = numpy.array([.75, 0])
thist, uhist = ode_euler(test.A, u_0, h=.1, T=15)
pyplot.figure()
pyplot.plot(thist, uhist, '.', label='Euler')
pyplot.plot(thist, test.exact(thist, u_0), label='exact')
pyplot.legend(loc='upper right')
pyplot.title('Forward Euler');
```

Now let's look at backward Euler.

```python
def ode_beuler(A, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    while t < T:
        h = min(h, T - t)
        # u <- (I - h A)^{-1} u
        u = numpy.linalg.solve(numpy.eye(len(A)) - h*A, u)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)
```

```python
test = linear(numpy.array([[0, 1],
                           [-1, 0]]))
u_0 = numpy.array([.75, 0])
thist, uhist = ode_beuler(test.A, u_0, h=.01, T=15)
pyplot.figure()
pyplot.plot(thist, uhist, '.', label='Euler')
pyplot.plot(thist, test.exact(thist, u_0), label='exact')
pyplot.legend(loc='upper right')
pyplot.title('Backward Euler');
```

### The (implicit) midpoint method

Already for this simple problem neither forward nor backward Euler produce particularly accurate results unless we choose tiny timesteps. These are not the only choices we can make. We could instead evaluate the right hand side $f$ at the midpoint of the timestep

$$
u(t + h) = u(t) + h f\left(t + h/2, \frac{u(t) + u(t + h)}{2}\right).
$$

Which for linear problems reduces to

$$
\left(I - \frac{h}{2}A\right) u(t + h) = \left(I + \frac{h}{2}A\right)u(t)
$$
again requiring a solve.

### The trapezoidal rule

Another very similar scheme is obtained by making a trapezoidal approximation to the integral (rather than a linear approximation as we did for the Euler method).

$$
\begin{align}
u(t_0 + h) &= u(t_0) + \int_{t_0}^{t_0 + h} f(\tau, u(\tau)) \text{d}\tau\\
           &\approx u(t_0) + \frac{t_0 + h - t_0}{2}[f(t_0 + h, u(t_0 + h)) + f(t_0, u(t_0))]
\end{align}
$$

Resulting in the scheme:

$$
u_{n+1} = u_n + \frac{1}{2} h [f(t_n, u_n) + f(t_{n+1}, u_{n+1})]
$$

#### Questions

1. When do implicit midpoint and the trapezoidal rule differ? When are they the same?
2. Show that the trapezoidal method has order 2, i.e., that the error in a single step is $\mathcal{O}(h^3)$ (use the same mechanism as for the explicit Euler scheme, but you will need to Taylor expand one order higher)
3. Show that the trapezoidal method is convergent with convergence rate $\mathcal{O}(h^2)$.

```python
def ode_midpoint(A, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    I = numpy.eye(len(A))
    while t < T:
        h = min(h, T - t)
        # u <- (I - h/2 A)^{-1} (I + h/2 A) u
        u = numpy.linalg.solve(I - h/2*A, (I + h/2*A) @ u)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)
```

```python
test = linear(numpy.array([[0, 1],
                           [-1, 0]]))
u_0 = numpy.array([.75, 0])
thist, uhist = ode_midpoint(test.A, u_0, h=1, T=15)
pyplot.figure()
pyplot.plot(thist, uhist, '.', label='Midpoint')
pyplot.plot(thist, test.exact(thist, u_0), label='exact')
pyplot.legend(loc='upper right')
pyplot.title('Implicit Midpoint');
```

## Verifying the convergence of a method

An important task in scientific computing is always to verify that a numerical implementation of a scheme is correct, and as accurate as claimed. To do so, we usually compute the error as a function of step size and show that it (at least asymptotically) converges at the correct rate.

For simple test problems, we can usually construct an exact solution and explicitly measure the error. For real-world problems, this fails to be the case and we usually resort to comparing to some "ground-truth" numerical simulation (perhaps with tiny timesteps).

Construction of, and testing with, an exact solution is often called the [_method of manufactured solutions_](https://doi.org/10.1115/1.1436090).

Let's look at this for the problems we've seen so far.

```python
def mms(eqn, integrator, u_0, h=0.1, T=15):
    thist, uhist = integrator(eqn.A, u_0, h=h, T=T)
    return numpy.linalg.norm(uhist[-1] - eqn.exact(thist[-1], u_0), numpy.inf)

eqn = linear(numpy.array([[0, 1],
                          [-1, 0]]))
u_0 = numpy.array([.75, 0])

hs = numpy.array([1/2**i for i in range(10)])
pyplot.figure()
for name, method in [("Forward Euler", ode_euler),
                     ("Backward Euler", ode_beuler),
                     ("Midpoint", ode_midpoint)]:
    errors = []
    for h in hs:
        error = mms(eqn, method, u_0, h=h, T=4)
        errors.append(error)
    pyplot.loglog(hs, errors, ".", label=name)

pyplot.loglog(hs, 2*hs, label="$\mathcal{O}(h)$")
pyplot.loglog(hs, hs**2, label="$\mathcal{O}(h^2)$")
pyplot.legend()
pyplot.xlabel("h")
pyplot.ylabel("$e_{n,h}$");
```

## The importance of the Lipschitz condition

If our ODE does not obey the Lipschitz condition, we can expect that it will not perform as well as the convergence analysis suggests. As an example, consider the ODE (this example is taken from Iserles' book):

$$
\dot{u} = \left(u - \lfloor u \rfloor -\frac{3}{2}\right) \ln 3 , u(0) = 0.
$$

This has exact solution:

$$
u(t) = - \lfloor t \rfloor + \frac{1}{2}\left(1 - 3^{t - \lfloor t \rfloor}\right), t \ge 0.
$$

This equation fails the Lipschitz condition (hint: consider two points $x = m + \epsilon$ and $z = m - \epsilon$, $m \ge 1$ an integer and $\epsilon \in \left(0, \frac{1}{4}\right)$).

Let's look at what it does to the convergence of the methods we've tried.

```python
import scipy.optimize

## We'll come back to rootfinding later!
def ode_trapezoid_nonlinear(f, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    func = lambda un1, un: un1 - h/2*(f(0, un1) + f(0, un)) - un
    while t < T:
        h = min(h, T - t)
        # Solve u_{n+1} = u_n + h/2(f(t, u_{n+1}) + f(t, u_n))
        u = scipy.optimize.newton(func, u, args=(u, ), maxiter=200)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)

def ode_beuler_nonlinear(f, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    func = lambda un1, un: un1 - h*f(0, un1) - un
    while t < T:
        h = min(h, T - t)
        # Solve u_{n+1} = u_n + h f(t, u_{n+1}
        u = scipy.optimize.newton(func, u, args=(u, ), maxiter=100)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)
```

```python
class nonlipschitz(object):
    def __call__(self, t, u):
        return numpy.log(3) * (u - numpy.floor(u) - 3/2)
    def exact(self, t, u_0):
        return -numpy.floor(t) + 1/2 * (1 - 3**(t - numpy.floor(t)))
```

```python
pyplot.figure()
u_0 = numpy.array(0)
thist, uhist = ode_trapezoid_nonlinear(nonlipschitz(), u_0, h=0.01, T=8)
pyplot.plot(thist, uhist)
pyplot.plot(thist, nonlipschitz().exact(thist, u_0))
```

```python
def mms(eqn, integrator, u_0, h=0.1, T=15):
    thist, uhist = integrator(eqn, u_0, h=h, T=T)
    return numpy.abs(uhist[-1] - eqn.exact(thist[-1], u_0))

eqn = nonlipschitz()
u_0 = numpy.array(0)

hs = numpy.array([1/10**i for i in range(2, 5)])
pyplot.figure()
for name, method in [("Forward Euler", ode_euler),
                     ("Trapezoid", ode_trapezoid_nonlinear)]:
    errors = []
    for h in hs:
        error = mms(eqn, method, u_0, h=h, T=8)
        errors.append(error)
    pyplot.loglog(hs, errors, ".", label=name)

pyplot.loglog(hs, 2*hs, label="$\mathcal{O}(h)$")
pyplot.loglog(hs, 100*hs**2, label="$\mathcal{O}(h^2)$")
pyplot.legend()
pyplot.xlabel("h")
pyplot.ylabel("$e_{n,h}$");
```

Note how the trapezoid method, although a 2nd order method, only obtains first order convergence for this problem!


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

Note to lecturers:  This notebook is designed to work best as a classic Jupyter Notebook with nbextensions
* hide_input: to hide selected python cells particularly for just plotting
* RISE:  Interactive js slide presentations

## Why should I care?

*Because I have to be here for a requirement...*

*or I might actually need to solve something important...like can I ever retire?*

### An example: The Retirement Problem

$$A = \frac{P}{r} \left((1+r)^n - 1 \right)$$

$A$ is the total amount after n payments

$P$ is the incremental payment

$r$ is the interest rate per payment period

$n$ is the number of payments

Note that these can all be functions of $r$, $n$, and time

What is the minimum interest rate $r$ required to accrue \$1M over 20 years saving \\$1000 per month?

$$
A = \frac{P}{r} \left((1+r)^n - 1 \right)
$$

```python
def A(P, r, n):
    return P / r * ((1 + r)**n - 1)

P = 1000.
years = numpy.linspace(0,20,241)
n = 12*years
target = 1.e6
```

```python
# find the root using scipy's brentq method
from scipy.optimize import brentq
n_max = n[-1]
f = lambda r : target - A(P,r/12., n_max)
r_target = brentq(f,0.1,0.15)
```

```python
fig = plt.figure(figsize=(8,6))
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1)
for r in [0.02, 0.05, 0.08, 0.1, 0.12, 0.15]:
    axes.plot(years, A(P, r/12., n),label='r = {}'.format(r))
axes.plot(years, numpy.ones(years.shape) * target, 'k--',linewidth=2,label="Target")
axes.legend(loc='best')
axes.set_xlabel("Years",fontsize=18)
axes.set_ylabel("Annuity Value (Dollars)",fontsize=18)
axes.set_title("The Forward Problem $A(P,r,n)$",fontsize=20)
axes.grid()

axes = fig.add_subplot(1, 2, 2)
r = numpy.linspace(1.e-6,0.15,100)
target = 1.e6
axes.plot(A(P,r/12.,n_max),r,linewidth=2)
axes.plot(numpy.ones(r.shape) * target, r,'k--',linewidth=2)
axes.scatter(A(P,r_target/12,n_max),r_target,s=100,c='r')
axes.set_xlabel('Annuity Value (Dollars)',fontsize=18)
axes.set_title('The Inverse Problem $r(A,P,n)$, $r\geq$ {:.3f}'.format(r_target),fontsize=20)
axes.grid()
plt.show()
```

This is a rootfinding problem for a function of a single variable

### Another Example:  Boat race (numerical quadrature of arc length)
Given a river with coordinates $(x,f(x))$ find the total length actually rowed over a given interval $x\in [0,X]$

Example:  a sinusoidal river $$f(x) = A \sin x$$

```python
A=.5
fig = plt.figure(figsize=(8,6))
x = numpy.linspace(0,10,100)
plt.plot(x,A*numpy.sin(x),'r',linewidth=2,)
plt.xlabel('distance (km)')
plt.grid()
plt.title('The Boat Race: $f(x)=A\sin(x)')
plt.show()
```

We need to calculate the function $f(x)$'s arc-length from $[0, X]$ e.g.

\begin{align}
    L(X) &= \int_0^{X} \sqrt{1 + |f'(x)|^2} dx\\
         &= \int_0^{X} \sqrt{1 + A^2\cos^2(x)} dx\\
\end{align}

In general the value of this integral cannot be solved analytically (the answer is actually an elliptic integral). We usually need to approximate the integral using a numerical quadrature.

## Why is this exciting?

Computers have revolutionized our ability to explore...

[Top 10 Algorithms of the 20th Century](http://www.siam.org/pdf/news/637.pdf?t=1&cn=ZmxleGlibGVfcmVjcw%3D%3D&refsrc=email&iid=658bdab6af614c83a8df1f8e02035eae&uid=755271476&nid=244+285282312)

...and there is more to come!


PHYS341, F2019, Day #2

IPython Notebook to visualize a surface. We use the example:
\begin{equation}
z = x^2 + y^2
\end{equation}

```python
%matplotlib notebook
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
# Axes3D import adds feature, it enables using projection='3d' in add_subplot
import matplotlib.pyplot as plt
import random

def fun(x, y):
    return x**2 + y**2

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
x = y = np.arange(-3.0, 3.0, 0.05)
X, Y = np.meshgrid(x, y)
zs = np.array(fun(np.ravel(X), np.ravel(Y)))
Z = zs.reshape(X.shape)

ax.plot_surface(X, Y, Z)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

plt.show()
```

```python

```


# K-Nearest Neighbors Algorithm

* Last class, we introduced the probabilistic generative classifier.
* As discussed, the probabilistic generative classifier requires us to assume a parametric form for each class (e.g., each class is represented by a multi-variate Gaussian distribution, etc..).  Because of this, the probabilistic generative classifier is a *parametric* approach
* Parametric approaches have the drawback that the functional parametric form needs to be decided/assumed in advance and, if chosen poorly, might be a poor model of the distribution that generates the data resulting in poor performance.
* Non-parametric methods are those that do not assume a particular generating distribution for the data.  The $K$-nearest neighbors algorithm is one example of a non-parametric classifier.

* Nearest neighbor methods compare test point to the $k$ nearest training data points and then estimate an output value based on the desired/true output values of the $k$ nearest training points
* Essentially, there is no ``training'' other than storing the training data points and their desired outputs
* In test, you need to: (1) determine which $k$ training data points are closest to the test point; and (2) determine the output value for the test point
* In order to find the $k$ nearest neighbors in the training data, you need to define a *similarity measure* or a *dissimilarity measure*.  The most common dissimilarity measure is Euclidean distance.
    * Euclidean distance: $d_E = \sqrt{\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\left(\mathbf{x}_1-\mathbf{x}_2\right)}$
    * City block distance: $d_C = \sum_{i=1}^d \left| x_{1i} - x_{2i} \right|$
    * Mahalanobis distance: $\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\Sigma^{-1}\left(\mathbf{x}_1-\mathbf{x}_2\right)$
    * Geodesic distance
    * Cosine angle similarity: $\cos \theta = \frac{\mathbf{x}_1^T\mathbf{x}_2}{\left\|\mathbf{x}_1\right\|_2^2\left\|\mathbf{x}_2\right\|_2^2}$
    * and many more...

* If you are doing classification, once you find the $k$ nearest neighbors to your test point in the training data, then you can determine the class label of your test point using (most commonly) *majority vote*
* If there are ties, they can be broken randomly or using schemes like applying the label of the closest data point in the neighborhood
*  Of course, there are MANY modifications to you can make to this.  A common one is to weight the votes of each of the nearest neighbors by their distance/similarity measure value.  If they are closer, they get more weight.

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn import neighbors
%matplotlib inline

#figure params
h = .02  # step size in the mesh
figure = plt.figure(figsize=(17, 9))

#set up classifiers
n_neighbors = 3
classifiers = []
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='uniform'))
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='distance'))
names = ['K-NN_Uniform', 'K-NN_Weighted']

#Put together datasets
n_samples = 300
X, y = make_classification(n_samples, n_features=2, n_redundant=0, n_informative=2,
                           random_state=0, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(n_samples, noise=0.3, random_state=0),
            make_circles(n_samples,  noise=0.2, factor=0.5, random_state=1),
            linearly_separable]

i = 1
# iterate over datasets
for X, y in datasets:
    # preprocess dataset, split into training and test part
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.8)  #split into train/test folds

    #set up meshgrid for figure
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
    # and testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright, alpha=0.6)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
        clf.fit(X_train, y_train)
        score = clf.score(X_test, y_test)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)

        # Plot also the training points
        ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
        # and testing points
        ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright,
                   alpha=0.4)

        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1

figure.subplots_adjust(left=.02, right=.98)
plt.show()
```

# Error and Evaluation Metrics

* A key step in machine learning algorithm development and testing is determining a good error and evaluation metric.

* Evaluation metrics help us to estimate how well our model is trained and it is important to pick a metric that matches our overall goal for the system.

* Some common evaluation metrics include precision, recall, receiver operating curves, and confusion matrices.

### Classification Accuracy and Error

* Classification accuracy is defined as the number of correctly classified samples divided by all samples:

\begin{equation}
\text{accuracy} = \frac{N_{cor}}{N}
\end{equation}
where $N_{cor}$ is the number of correct classified samples and $N$ is the total number of samples.

* Classification error is defined as the number of incorrectly classified samples divided by all samples:

\begin{equation}
\text{error} = \frac{N_{mis}}{N}
\end{equation}
where $N_{mis}$ is the number of misclassified samples and $N$ is the total number of samples.

* Suppose there is a 3-class classification problem, in which we would like to classify each training sample (a fish) to one of the three classes (A = salmon or B = sea bass or C = cod).

* Let's assume there are 150 samples, including 50 salmon, 50 sea bass and 50 cod.  Suppose our model misclassifies 3 salmon, 2 sea bass and 4 cod.

* Prediction accuracy of our binary classification model is calculated as:

\begin{equation}
\text{accuracy} = \frac{47+48+46}{50+50+50} = \frac{47}{50}
\end{equation}

* Prediction error is calculated as:

\begin{equation}
\text{error} = \frac{N_{mis}}{N} = \frac{3+2+4}{50+50+50} = \frac{3}{50}
\end{equation}

### Confusion Matrices

* A confusion matrix summarizes the classification accuracy across several classes. It shows the ways in which our classification model is confused when it makes predictions, allowing visualization of the performance of our algorithm. Generally, each row represents the instances of a actual class while each column represents the instances in an predicted class.

* If our classifier is trained to distinguish between salmon, sea bass and cod. We can summarize the prediction result in the confusion matrix as follows:

| Actual/Predicted | Salmon | Sea bass | Cod  |
| --- | --- | --- | --- |
| Salmon | 47 | 2 | 1 |
| Sea Bass | 2 | 48 | 0 |
| Cod | 0 | 0 | 50 |

* In this confusion matrix, of the 50 actual salmon, the classifier predicted that 2 are sea bass, 1 is cod incorrectly and 47 are labeled salmon correctly. All correct predictions are located in the diagonal of the table. So it is easy to visually inspect the table for prediction errors, as they will be represented by values outside the diagonal.

### TP, FP, TN, and FN

* True positive (TP): correctly predicting event values
* False positive (FP): incorrectly calling non-events as an event
* True negative (TN): correctly predicting non-event values
* False negative (FN): incorrectly labeling events as non-event

* Precision is also called positive predictive value.

\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP}+\text{FP}}
\end{equation}

* Recall is also called true positive rate, probability of detection

\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP}+\text{FN}}
\end{equation}

* Fall-out is also called false positive rate, probability of false alarm.

\begin{equation}
\text{Fall-out} = \frac{\text{FP}}{\text{N}}= \frac{\text{FP}}{\text{FP}+\text{TN}}
\end{equation}

* *Consider the salmon/non-salmon classification problem, what are the TP, FP, TN, FN values?*

| Actual/Predicted | Salmon | Non-Salmon  |
| --- | --- | --- |
| Salmon | 47 | 3 |
| Non-Salmon | 2 | 98 |

### ROC curves

* The Receiver Operating Characteristic (ROC) curve is a plot between the true positive rate (TPR) and the false positive rate (FPR), where the TPR is defined on the $x$-axis and FPR is defined on the $y$-axis.

* $TPR = TP/(TP+FN)$ is defined as ratio between true positive prediction and all real positive samples. The defition used for $FPR$ in a ROC curve is often problem dependent.  For example, for detection of targets in an area, FPR may be defined as the ratio between the number of false alarms per unit area ($FA/m^2$).  In another example, if you have a set number of images and you are looking for targets in these collection of images, FPR may be defined as the number of false alarms per image.  In some cases, it may make the most sense to simply use the Fall-out or false positive rate.

* Given a binary classifier and its threshold, the (x,y) coordinates of ROC space can be calculated from all the prediction result.  You trace out a ROC curve by varying the threshold to get all of the points on the ROC.

* The diagonal between (0,0) and (1,1) separates the ROC space into two areas, which are left up area and right bottom area. The points above the diagonal represent good classification (better than random guess) which below the diagonal represent bad classification (worse than random guess).

* *What is the perfect prediction point in a ROC curve?*

### MSE and MAE

* *Mean Square Error* (MSE) is the average of the squared error between prediction and actual observation.

* For each sample $\mathbf{x}_i$, the prediction value is $y_i$ and the actual output is $d_i$. The MSE is

\begin{equation}
MSE = \sum_{i=1}^n \frac{(d_i - y_i)^2}{n}
\end{equation}

* *Root Mean Square Error* (RMSE) is simply the square root the MSE.

\begin{equation}
RMSE = \sqrt{MSE}
\end{equation}

* *Mean Absolute Error* (MAE) is the average of the absolute error.
\begin{equation}
MAE = \frac{1}{n} \sum_{i=1}^n \lvert d_i - y_i \rvert
\end{equation}

```python

```

Lecture07_KNN/Lecture 07 Non-parametric K-Nearest Neighbors Algorithm.ipynb


*Total work in progress; more than anything an early attempt at sorting it all out by myself. Code and derivations are my own and not necessarily bug-free (though about 80% of the equations here are standard results that are easy to find).*

# Combinatorics

*If there are m ways to do one thing, and n ways to do another, then there are mn ways of doing both.* - Deepak Chopra

I am using the language convention that 'collections' are unordered and 'sequences' are ordered.

# Principles

### Product Principle
if we have a partition of a set S into m blocks, each of size n, then S has size mn.

### Quotient Principle
If we partition a set P into q blocks, each of size r, then q = p/r.

### Sum Principle
if we have a partition of a set S, then the size of S is the sum of the sizes
of the blocks of the partition.

### Bijection Principle
Two sets have the same size if and only if there is a bijection between them.

# Combinatorics Code

- factorial
- binomial coefficent
- subsets of size n (choices of n out of S)
- subsets of size n+m where n and m are drawn from different, possibly overlappingsets
- multinomial
- permutations
- unordered multiplicities for all n-sized multisets with m classes (no permutations of multiplicities)
- multiplicities for all n-sized multisets with m classes

```python
import numpy as np

def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return int(res)

def binomial(n,k):
    """
    binomial coefficients. convention is that it's 0 if k>n
    """
    if k <= n:
        return int(factorial(n)/(factorial(k)*factorial(n-k)))
    else:
        return 0

def multinomial(n,k):
    """
    multinomial coefficient. k is multiindex. Returns error if sum(k) != n
    """
    assert np.sum(k) == n
    return int(factorial(n)/np.product([factorial(x) for x in k]))

def unordered_multiplicities(n,m,k0_max=None):
    """
    Generator that yields all unordered multiplicities for multisets with n objects of m classes.

    The convention is that they are sorted:
    n_1 >= n_2 >= n_3 >= ... >= n_m

    Example: (n=3,m=3) returns [3,0,0],[2,1,0],[1,1,1]

    I don't know what the proper math term for this is: If a set of n elements is partitioned into m subsets,
    then these are the possible relative sizes of that partitions can have.

    In a multiset of size n, so {1,1,2} = [2,0] is different from {1,2,2} = [0,2]. The multiplicities are ordered,
    and so [0,2] and [2,0] are counted twice.

    The "unordered multiplicites" here treat [2,0] and [0,2] as identical.

    As of now, I don't know how many there should be, but I think that she solution probably lies
    in creating a bijection to lattice paths. For the case m = n, I think the count might be the Catalan number C_n.
    I haven't checked.

    EDIT: These are called partitions and I am using the 'decreasing list' representation of partitions
    """

    if m == 0:
        yield []

    else:

        # define range of largest entry
        if not k0_max:
            k0_max = n

        k0_min = int(np.ceil(n/m))

        # iterate over largest entry, add combinations of remaining entries recursively
        for k0 in range(k0_min,k0_max+1):

            # the remaining entries are distributed as n-k0 objects over m-1 buckets, including constraint on maximum amount
            k_rem = unordered_multiplicities(n-k0,m-1,k0_max=np.min([n-k0,k0]))

            for k_n in k_rem:
                res = [k0] + k_n

                assert np.sum(res) == n

                yield res

def permutations(x):
    """
    generator that yields all permutations of an array x

    will be a total of factorial(len(x)) if all elements of x are distinguishable

    (you can just use itertools library)
    """

    pivots = set()

    if len(x) == 0 or len(x) == 1:
        yield x

    elif len(x) == 2:
        if x[0] == x[1]:
            yield x
        else:
            res = [x,x[::-1]]
            for i in range(2):
                yield res[i]

    else:
        for i in range(len(x)):
            p = x[i]
            if p in pivots:
                pass
            else:
                pivots.add(p)
                y = permutations(x[:i]+x[i+1:])

                for remainder in y:
                    yield [p] + remainder

def subsets(n,S):
    """
    Return all choices n out of S

    assumes all elements of the set are distinguishable
    """
    S = list(S)

    assert len(S) >= n

    if n == 0:
        yield []

    else:
        for i in range(len(S)-n+1):
            p = [S[i]]

            rem = subsets(n-1,S[i+1:])

            for r in rem:
                yield p+r

def choose(n,m,A,B):
    """
    Returns all unordered collections where n are chosen from A and m are chosen from B, where A & B may overlap.
    """

    C = [x for x in A if x in set(B)]
    A_ = [x for x in A if x not in B]
    B_ = [x for x in B if x not in A]

    a_ = len(A_)
    b_ = len(B_)
    c = len(C)

    for i in range(np.max([n-a_,0]),np.min([c,n])+1):
        for j in range(np.max([m-b_,0]),np.min([c-i,m])+1):

            c_C = subsets(i+j,C)
            for cc in c_C: # iterate over picks from intersection
                a_A = subsets(n-i,A_)
                for aa in a_A: # iterate over picks from A
                    b_B = subsets(m-j,B_)
                    for bb in b_B: # iterate over picks from B

                        yield aa+bb+cc

def multisets(n,m):
    """
    All multisets of n objects of m classes. All possible ways for n objects divided into m distinguishable classes.

    ex.: [1,0,0],[0,1,0],[0,0,1]

    Will be a total of (n multichoose m)

    This could be huge.
    """

    distributions = unordered_multiplicities(n,m)
    for distribution in distributions:
        mindexes = permutations(distribution)
        for mindex in mindexes:
            yield mindex
```

# Subsets and Permutations

### k-Element Permutations

A permutation is an ordered sequence of distinguishable elements drawn from some set without replacement.  Let $[S]$ be a set of $S$ distinguishable elements. A $k$-element permutation can be thought of as a injective function from $[k]=\{1,2,3,...,k\}$ to $[S]$. There are $S^{\underline{k}} = \frac{|S|!}{(|S|-k)!}$ different $k$-element permutations.

The notation $S^\underline{k} = S(S-1)(S-2)...(S-k+1)$ is the "falling factorial".

### Number of Subsets of Set with Size n

The number of subsets of a set $S$ with size $n$ is $2^n$. One way to calculate that:

\begin{equation}
n_{subsets} = \sum^n_{i=0} (\mathrm{ways\ of\ picking\ i\ out\ of\ n}) = \sum^n_{i=0}\left(\begin{array}{c}n\\i\end{array}\right) = 2^n
\end{equation}

There is a better way, though. A subset can be thought of as assigning a label to each element of $S$: it's in or out. That means that any possible subset cen be described by an $n$-element binary string and vice versa -- the subsets and the $n$-element strings can be linked by bijection. The number of subsets is therefore the number of $n$-element binary strings, which is:

\begin{equation}
n_{subsets} = 2^n
\end{equation}

### Number of ways to Divide a Set of size n into m Partitions

Equally well, one might ask: how many ways are there to partition $S$ into $k$ subsets? In that case there are $k$ labels, so there are $k^n$ ways. That means:

\begin{equation}
\sum_{\begin{array}{c}k_1,k_2,...,k_m\\ \sum_{k_i}=n\end{array}}\left(\begin{array}{c}n\\k_1 k_2 ... k_m \end{array}\right) = m^n
\end{equation}

Which is: (ways of taking $k_1$ out of $n$) x (ways of taking $k_2$ out of ($n-k_1$)) x (ways of ...). Progressively dividing the set into subsets, and adding up all ways of making the division.

# Sequences and Collections

### Sequences sampled with Replacement
Let [A] be a set of size $A$.

\begin{equation}
\begin{array}{c}
S = \left\{ (a_1,a_2,...,a_n) : a_1,a_2,...,a_n\in [A] \right\}\\
|S| = A^n
\end{array}
\end{equation}

The creation of n-tuples $(a_1,a_2,...,a_n)$ from elements of $A$ can be thought of in terms of functions that map each element of the tuple ($a_i$ for any choice of $i\in (1,n)$) to an element of $A$. For ordered tuples, these functions are always injective.

*Example: bit strings of length $3$: [0,0,0],[0,0,1],[0,1,1],[1,0,1],...*

### Collections sampled with Replacement (Multisets)
Let [A] a set of size A.
\begin{equation}
S = \left\{ \{a_1,a_2,...,a_n\} : a_1,a_2,...,a_n \in [A] \right\}
\end{equation}

This can be thought of as picking a total of $n$ objects from a choice of $k$ classes.

There are three entries, that can correspond to up to $\min(3,A)$ different elements of $[A]$.

For $n=3$, the options are: they are all the same, two are the same, or they're all different. There are $A$ ways for them to be all the same, $2*A*(A-1)$ ways for two of them to be the same (the factor 2 is because $(a,a,b) \neq (b,b,a)$) and $\left(\begin{array}{c}A\\3\end{array}\right)$ ways for them to all be different.

Instead of expressing the elements of S in terms of the unordered multiset $\{a,b,c\}$, we can write them in terms of an ordered sequence that contains the multiplicity of the elements of the multisets. That is, if $[A] = \{1,2,3,4\}$, we write $\{1,1,2\} \in S$ as $[2_1|1_2|0_3|0_4]$. This is a bijection between elements of $S$ and the elements of $N = \{[i_1,i_2,...,i_k]: i_1,i_2,...,i_k \in \mathbb{N}_0^+ , \sum_{i_j} i_j = n\}$ (all length $k$ sequences of integers $[i_1,i_2,...,i_k]$ so that $\sum_{i_j} i_j = n$). The bijection implies that $S$ and $K$ have the same size.

The amount of elements in $N$ is the Bose-Einstein Coefficient that is derived in the ``Bose Einstein Coefficients - Stars and Bars`` notebook. Therefore:

\begin{equation}
|S| = \left(\begin{array}{c}n+k-1\\n\end{array}\right)
\end{equation}

This is also known as *multichoose*. It is the number of $k$-element multisets on $n$ symbols.

*Example: You are 8 years old and you get to pick n pieces of candy from a candy store that has k different types of candy.*

*Example: Ways for n bosons to be distributed over k degenerate states.*

*Example: The number of ways to draw n elements from k equally likely classes, when the order does not matter.*

*Example: How many ways are there to partition a set with n elements (apart the empty set). (In this case k=n)*

*Example: In a universe of k stocks, how many portfolios are possible that consist of n stocks?*

### Sequences sampled without Replacement
Assume $a,b,c$ are all drawn from a single set $[A]$ without replacement. Then $|S| = A*(A-1)*(A-2)$, i.e. $\frac{n!}{(n-k)!} = n^\underline{k}$

*Example: Possible ways of assigning first, second and third place in a competition.*

### Collections sampled without Replacement

This means, selecting $k$ out of $n$:

\begin{equation}
|S| = \frac{n^\underline{k}}{k!} = \left(\begin{array}{c}n\\k\end{array}\right)
\end{equation}

Which is the *binomial coefficient*.

That's the same as taking the number of ordered $k$-tuples and dividing by the number of internal orderings $k!$.

*Example: Possible ways that k fermions might be distributed over n degenerate states*

*Example: Possible groups of k people that can be formed out of a pool of n.*

### Collections sampled from several Disjoint Sets with or without Replacement
Let [A], [B] and [C] be disjoint subsets of size $A$, $B$ and $C$, respectively.

\begin{equation}
\begin{array}{c}
S = \left\{ \{a,b,c\} : a\in [A], b\in [B], c\in[C] \right\}\\
|S| = ABC
\end{array}
\end{equation}

The mappings between the domain $S$ and co-domain are injective: each entry in the n-tuples is mapped to a disjoint subset of the co-domain. It is impossible for two tuples in $S$ to be permutations of each other, so that different permutations doesn't lead to overcounting.

*Example: Choose one drink $a$, one sandwich $b$ and one free T-shirt $c$. How many options are there?*

### Collections from several Overlapping Sets with Replacement
Let $[A]$ and $[B]$ be two sets of size $A$ and $B$ respectively, where $A$ and $B$ may or may not overlap.

Let:

\begin{equation}
S = \{\{a_1,a_2,...,a_n,b_1,b_2,...,b_m\}: a\in[A],b\in[B]\}
\end{equation}

Then, I *think*, but haven't checked:

\begin{equation}
|S| = \left( \begin{array}{c}n+a-1\\n \end{array}\right)\left( \begin{array}{c}n+b-1\\m \end{array}\right)
\end{equation}

### Collections from several Overlapping Sets without Replacement

Let $[A]$ and $[B]$ be two sets of size $A$ and $B$ respectively, so that $[A] \neq [B]$ but $[A]\cap[B]=[C]$, with the $C$ the size of $[C]$. The set $S$ consists of all collections of $n$ items from $[A]$ and $m$ items of $[B]$.

Let:

\begin{equation}
S = \{\{a_1,a_2,...,a_n,b_1,b_2,...,b_m\}: a\in[A],b\in[B]\}
\end{equation}

Then the size of $|S|$:

\begin{equation}
|S| = \sum_{\max(n-A+C,0)\leq i \leq \min(C,n)\\\max(m-B+C,0)\leq j \leq \min(C,m) \\i+j\leq C} \left(\begin{array}{c}C\\i+j\end{array}\right)\left(\begin{array}{c}A-C\\n-i\end{array}\right)\left(\begin{array}{c}B-C\\m-j\end{array}\right)
\end{equation}

This can be thought of choosing unordered collections from the 3 disjoint sets $[C]$,$[A]\setminus[C]$ and $[B]\setminus[C]$ and summing over the ways in which $[A]$ and $[B]$-assigned elements can be drawn from $[C]$.

TO DO: Generalize this to more than two sets. The key will be to reduce the problem to sampling from disjoint sets again.

*Example: You want to travel to 6 countries, of which at least 3 are Spanish speaking and at least 3 are in Europe.*

## Experiments

Testing almost everything.

```python
# Permutations

print("Permutations:")
for n in range(10):
    x = list(range(n))
    print('objects: %i\t permutations: %i %i' % (n,len(list(permutations(x))),factorial(n)))

#Subsets
nn = [2,5,10]
mm = [2,5,10]

print("\nTotal number of subsets of an n-sized set")
for n in nn:
    n_subsets = 0

    for i in range(n+1):
        n_subsets += binomial(n,i) # choose i out of n
    print("n: %i\t %i %i" % (n,n_subsets,2**n))

#Number of Partitionings into m Partitions
print("\nTotal number of partitionings of an n-sized set into m")
for n in nn:
    for m in mm:
        n_partitions = 0
        for mindex in multisets(n,m):
            n_partitions += multinomial(n,mindex)

        print("n: %i m: %i\t %i %i" % (n,m,n_partitions,m**n))

print("\nSubsets Generator:")
S = 'A,B,C,D,E'.split(',')
for n in range(6):
    print('choose %i out of %i\t size: %i %i' % (n,len(S),len(list(subsets(n,S))),binomial(len(S),n)))
    print([''.join(x) for x in subsets(n,S)])

# Class balances when choosing n from k classes
print("\nMultiplicities:")
print("Class balances when choosing n from k classes")
m = 4
for n in range(10):
    x = list(range(n))
    print('objects: %i classes: %i\t size: %i %i' % (n,m,len(list(unordered_multiplicities(n,m))),0))
    print(list(unordered_multiplicities(n,m)))

# Collections of n objects from k classes with replacement
print("\nMultisets:")
print("(Collections of n objects from k classes with replacement)")
m = 4
for n in range(7):
    x = list(range(n))
    print('objects: %i classes: %i\t size: %i %i' % (n,m,len(list(multisets(n,m))),
                                                      factorial(n+m-1)/(factorial(n)*factorial(m-1))))
    print(list(multisets(n,m)))

A = list('1234')
ordered_pairs = np.vstack(np.vstack(np.vstack([[[[(a,b,c) for a in A ] for b in A]] for c in A])))
print('\nOrdered Sequences with Replacement %i %i' % (len(ordered_pairs),len(A)**3))
print([''.join(x) for x in ordered_pairs])

unordered_pairs = {tuple(sorted(pair)) for pair in ordered_pairs}
print('\nUnordered Collections with Replacement %i %i' % (len(unordered_pairs),factorial(3+len(A)-1)/(factorial(3)*factorial(len(A)-1))))
print([''.join(x) for x in sorted(list(unordered_pairs))])

unordered_pairs_without_replacement = []
for i in range(len(A)):
    for j in range(i+1,len(A)):
        for k in range(j+1,len(A)):
            unordered_pairs_without_replacement+=[(A[i],A[j],A[k])]

print('\nUnordered Collections without Replacement %i %i' % (len(unordered_pairs_without_replacement),
                                                  factorial(len(A))/(factorial(3)*factorial(len(A)-3))))
print([''.join(x) for x in sorted(unordered_pairs_without_replacement)])

def count_choices(n,m,A,B):
    """
    Count choices for drawing n from A and m from B without replacement, where A and B may overlap.
    """
    C = [x for x in A if x in B]
    a = len(A)
    b = len(B)
    c = len(C)

    i_min = np.max([n-a+c,0])
    j_min = np.max([m-b+c,0])

    s = 0
    for i in range(i_min,np.min([c,n])+1):
        for j in range(j_min,np.min([c-i,m])+1):
            s += binomial(c,i+j)*binomial(a-c,n-i)*binomial(b-c,m-j)

    return s

A = 'Spain,Mexico,Cuba'.split(',') # spanish speaking countries
B = 'Spain,France,Germany'.split(',') # countries in Europe

print("\nChoose from Overlapping Sets")
for n in range(len(A)):
    for m in range(len(B)):
        S = list(choose(n,m,A,B))
        print("Spanish speaking: %i European: %i \t %i %i" % (n,m,len(S),count_choices(n,m,A,B)))
        print(['-'.join(x) for x in S])
```

    Permutations:
    objects: 0	 permutations: 1 1
    objects: 1	 permutations: 1 1
    objects: 2	 permutations: 2 2
    objects: 3	 permutations: 6 6
    objects: 4	 permutations: 24 24
    objects: 5	 permutations: 120 120
    objects: 6	 permutations: 720 720
    objects: 7	 permutations: 5040 5040
    objects: 8	 permutations: 40320 40320
    objects: 9	 permutations: 362880 362880

    Total number of subsets of an n-sized set
    n: 2	 4 4
    n: 5	 32 32
    n: 10	 1024 1024

    Total number of partitionings of an n-sized set into m
    n: 2 m: 2	 4 4
    n: 2 m: 5	 25 25
    n: 2 m: 10	 100 100
    n: 5 m: 2	 32 32
    n: 5 m: 5	 3125 3125
    n: 5 m: 10	 100000 100000
    n: 10 m: 2	 1024 1024
    n: 10 m: 5	 9765625 9765625
    n: 10 m: 10	 10000000000 10000000000

    Subsets Generator:
    choose 0 out of 5	 size: 1 1
    ['']
    choose 1 out of 5	 size: 5 5
    ['A', 'B', 'C', 'D', 'E']
    choose 2 out of 5	 size: 10 10
    ['AB', 'AC', 'AD', 'AE', 'BC', 'BD', 'BE', 'CD', 'CE', 'DE']
    choose 3 out of 5	 size: 10 10
    ['ABC', 'ABD', 'ABE', 'ACD', 'ACE', 'ADE', 'BCD', 'BCE', 'BDE', 'CDE']
    choose 4 out of 5	 size: 5 5
    ['ABCD', 'ABCE', 'ABDE', 'ACDE', 'BCDE']
    choose 5 out of 5	 size: 1 1
    ['ABCDE']

    Multiplicities:
    Class balances when choosing n from k classes
    objects: 0 classes: 4	 size: 1 0
    [[0, 0, 0, 0]]
    objects: 1 classes: 4	 size: 1 0
    [[1, 0, 0, 0]]
    objects: 2 classes: 4	 size: 2 0
    [[1, 1, 0, 0], [2, 0, 0, 0]]
    objects: 3 classes: 4	 size: 3 0
    [[1, 1, 1, 0], [2, 1, 0, 0], [3, 0, 0, 0]]
    objects: 4 classes: 4	 size: 5 0
    [[1, 1, 1, 1], [2, 1, 1, 0], [2, 2, 0, 0], [3, 1, 0, 0], [4, 0, 0, 0]]
    objects: 5 classes: 4	 size: 6 0
    [[2, 1, 1, 1], [2, 2, 1, 0], [3, 1, 1, 0], [3, 2, 0, 0], [4, 1, 0, 0], [5, 0, 0, 0]]
    objects: 6 classes: 4	 size: 9 0
    [[2, 2, 1, 1], [2, 2, 2, 0], [3, 1, 1, 1], [3, 2, 1, 0], [3, 3, 0, 0], [4, 1, 1, 0], [4, 2, 0, 0], [5, 1, 0, 0], [6, 0, 0, 0]]
    objects: 7 classes: 4	 size: 11 0
    [[2, 2, 2, 1], [3, 2, 1, 1], [3, 2, 2, 0], [3, 3, 1, 0], [4, 1, 1, 1], [4, 2, 1, 0], [4, 3, 0, 0], [5, 1, 1, 0], [5, 2, 0, 0], [6, 1, 0, 0], [7, 0, 0, 0]]
    objects: 8 classes: 4	 size: 15 0
    [[2, 2, 2, 2], [3, 2, 2, 1], [3, 3, 1, 1], [3, 3, 2, 0], [4, 2, 1, 1], [4, 2, 2, 0], [4, 3, 1, 0], [4, 4, 0, 0], [5, 1, 1, 1], [5, 2, 1, 0], [5, 3, 0, 0], [6, 1, 1, 0], [6, 2, 0, 0], [7, 1, 0, 0], [8, 0, 0, 0]]
    objects: 9 classes: 4	 size: 18 0
    [[3, 2, 2, 2], [3, 3, 2, 1], [3, 3, 3, 0], [4, 2, 2, 1], [4, 3, 1, 1], [4, 3, 2, 0], [4, 4, 1, 0], [5, 2, 1, 1], [5, 2, 2, 0], [5, 3, 1, 0], [5, 4, 0, 0], [6, 1, 1, 1], [6, 2, 1, 0], [6, 3, 0, 0], [7, 1, 1, 0], [7, 2, 0, 0], [8, 1, 0, 0], [9, 0, 0, 0]]

    Multisets:
    (Collections of n objects from k classes with replacement)
    objects: 0 classes: 4	 size: 1 1
    [[0, 0, 0, 0]]
    objects: 1 classes: 4	 size: 4 4
    [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]
    objects: 2 classes: 4	 size: 10 10
    [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1, 0], [0, 1, 0, 1], [0, 0, 1, 1], [2, 0, 0, 0], [0, 2, 0, 0], [0, 0, 2, 0], [0, 0, 0, 2]]
    objects: 3 classes: 4	 size: 20 20
    [[1, 1, 1, 0], [1, 1, 0, 1], [1, 0, 1, 1], [0, 1, 1, 1], [2, 1, 0, 0], [2, 0, 1, 0], [2, 0, 0, 1], [1, 2, 0, 0], [1, 0, 2, 0], [1, 0, 0, 2], [0, 2, 1, 0], [0, 2, 0, 1], [0, 1, 2, 0], [0, 1, 0, 2], [0, 0, 2, 1], [0, 0, 1, 2], [3, 0, 0, 0], [0, 3, 0, 0], [0, 0, 3, 0], [0, 0, 0, 3]]
    objects: 4 classes: 4	 size: 35 35
    [[1, 1, 1, 1], [2, 1, 1, 0], [2, 1, 0, 1], [2, 0, 1, 1], [1, 2, 1, 0], [1, 2, 0, 1], [1, 1, 2, 0], [1, 1, 0, 2], [1, 0, 2, 1], [1, 0, 1, 2], [0, 2, 1, 1], [0, 1, 2, 1], [0, 1, 1, 2], [2, 2, 0, 0], [2, 0, 2, 0], [2, 0, 0, 2], [0, 2, 2, 0], [0, 2, 0, 2], [0, 0, 2, 2], [3, 1, 0, 0], [3, 0, 1, 0], [3, 0, 0, 1], [1, 3, 0, 0], [1, 0, 3, 0], [1, 0, 0, 3], [0, 3, 1, 0], [0, 3, 0, 1], [0, 1, 3, 0], [0, 1, 0, 3], [0, 0, 3, 1], [0, 0, 1, 3], [4, 0, 0, 0], [0, 4, 0, 0], [0, 0, 4, 0], [0, 0, 0, 4]]
    objects: 5 classes: 4	 size: 56 56
    [[2, 1, 1, 1], [1, 2, 1, 1], [1, 1, 2, 1], [1, 1, 1, 2], [2, 2, 1, 0], [2, 2, 0, 1], [2, 1, 2, 0], [2, 1, 0, 2], [2, 0, 2, 1], [2, 0, 1, 2], [1, 2, 2, 0], [1, 2, 0, 2], [1, 0, 2, 2], [0, 2, 2, 1], [0, 2, 1, 2], [0, 1, 2, 2], [3, 1, 1, 0], [3, 1, 0, 1], [3, 0, 1, 1], [1, 3, 1, 0], [1, 3, 0, 1], [1, 1, 3, 0], [1, 1, 0, 3], [1, 0, 3, 1], [1, 0, 1, 3], [0, 3, 1, 1], [0, 1, 3, 1], [0, 1, 1, 3], [3, 2, 0, 0], [3, 0, 2, 0], [3, 0, 0, 2], [2, 3, 0, 0], [2, 0, 3, 0], [2, 0, 0, 3], [0, 3, 2, 0], [0, 3, 0, 2], [0, 2, 3, 0], [0, 2, 0, 3], [0, 0, 3, 2], [0, 0, 2, 3], [4, 1, 0, 0], [4, 0, 1, 0], [4, 0, 0, 1], [1, 4, 0, 0], [1, 0, 4, 0], [1, 0, 0, 4], [0, 4, 1, 0], [0, 4, 0, 1], [0, 1, 4, 0], [0, 1, 0, 4], [0, 0, 4, 1], [0, 0, 1, 4], [5, 0, 0, 0], [0, 5, 0, 0], [0, 0, 5, 0], [0, 0, 0, 5]]
    objects: 6 classes: 4	 size: 84 84
    [[2, 2, 1, 1], [2, 1, 2, 1], [2, 1, 1, 2], [1, 2, 2, 1], [1, 2, 1, 2], [1, 1, 2, 2], [2, 2, 2, 0], [2, 2, 0, 2], [2, 0, 2, 2], [0, 2, 2, 2], [3, 1, 1, 1], [1, 3, 1, 1], [1, 1, 3, 1], [1, 1, 1, 3], [3, 2, 1, 0], [3, 2, 0, 1], [3, 1, 2, 0], [3, 1, 0, 2], [3, 0, 2, 1], [3, 0, 1, 2], [2, 3, 1, 0], [2, 3, 0, 1], [2, 1, 3, 0], [2, 1, 0, 3], [2, 0, 3, 1], [2, 0, 1, 3], [1, 3, 2, 0], [1, 3, 0, 2], [1, 2, 3, 0], [1, 2, 0, 3], [1, 0, 3, 2], [1, 0, 2, 3], [0, 3, 2, 1], [0, 3, 1, 2], [0, 2, 3, 1], [0, 2, 1, 3], [0, 1, 3, 2], [0, 1, 2, 3], [3, 3, 0, 0], [3, 0, 3, 0], [3, 0, 0, 3], [0, 3, 3, 0], [0, 3, 0, 3], [0, 0, 3, 3], [4, 1, 1, 0], [4, 1, 0, 1], [4, 0, 1, 1], [1, 4, 1, 0], [1, 4, 0, 1], [1, 1, 4, 0], [1, 1, 0, 4], [1, 0, 4, 1], [1, 0, 1, 4], [0, 4, 1, 1], [0, 1, 4, 1], [0, 1, 1, 4], [4, 2, 0, 0], [4, 0, 2, 0], [4, 0, 0, 2], [2, 4, 0, 0], [2, 0, 4, 0], [2, 0, 0, 4], [0, 4, 2, 0], [0, 4, 0, 2], [0, 2, 4, 0], [0, 2, 0, 4], [0, 0, 4, 2], [0, 0, 2, 4], [5, 1, 0, 0], [5, 0, 1, 0], [5, 0, 0, 1], [1, 5, 0, 0], [1, 0, 5, 0], [1, 0, 0, 5], [0, 5, 1, 0], [0, 5, 0, 1], [0, 1, 5, 0], [0, 1, 0, 5], [0, 0, 5, 1], [0, 0, 1, 5], [6, 0, 0, 0], [0, 6, 0, 0], [0, 0, 6, 0], [0, 0, 0, 6]]

    Ordered Sequences with Replacement 64 64
    ['111', '211', '311', '411', '121', '221', '321', '421', '131', '231', '331', '431', '141', '241', '341', '441', '112', '212', '312', '412', '122', '222', '322', '422', '132', '232', '332', '432', '142', '242', '342', '442', '113', '213', '313', '413', '123', '223', '323', '423', '133', '233', '333', '433', '143', '243', '343', '443', '114', '214', '314', '414', '124', '224', '324', '424', '134', '234', '334', '434', '144', '244', '344', '444']

    Unordered Collections with Replacement 20 20
    ['111', '112', '113', '114', '122', '123', '124', '133', '134', '144', '222', '223', '224', '233', '234', '244', '333', '334', '344', '444']

    Unordered Collections without Replacement 4 4
    ['123', '124', '134', '234']

    Choose from Overlapping Sets
    Spanish speaking: 0 European: 0 	 1 1
    ['']
    Spanish speaking: 0 European: 1 	 3 3
    ['France', 'Germany', 'Spain']
    Spanish speaking: 0 European: 2 	 3 3
    ['France-Germany', 'France-Spain', 'Germany-Spain']
    Spanish speaking: 1 European: 0 	 3 3
    ['Mexico', 'Cuba', 'Spain']
    Spanish speaking: 1 European: 1 	 8 8
    ['Mexico-France', 'Mexico-Germany', 'Cuba-France', 'Cuba-Germany', 'Mexico-Spain', 'Cuba-Spain', 'France-Spain', 'Germany-Spain']
    Spanish speaking: 1 European: 2 	 7 7
    ['Mexico-France-Germany', 'Cuba-France-Germany', 'Mexico-France-Spain', 'Mexico-Germany-Spain', 'Cuba-France-Spain', 'Cuba-Germany-Spain', 'France-Germany-Spain']
    Spanish speaking: 2 European: 0 	 3 3
    ['Mexico-Cuba', 'Mexico-Spain', 'Cuba-Spain']
    Spanish speaking: 2 European: 1 	 7 7
    ['Mexico-Cuba-France', 'Mexico-Cuba-Germany', 'Mexico-Cuba-Spain', 'Mexico-France-Spain', 'Mexico-Germany-Spain', 'Cuba-France-Spain', 'Cuba-Germany-Spain']
    Spanish speaking: 2 European: 2 	 5 5
    ['Mexico-Cuba-France-Germany', 'Mexico-Cuba-France-Spain', 'Mexico-Cuba-Germany-Spain', 'Mexico-France-Germany-Spain', 'Cuba-France-Germany-Spain']


# Continous Controller Design

### PID with Low Pass Filter on Derivative Term

A PID controller with a low pass filter on the derivative has the following form.

\begin{align}
    G_c(s) &= k_p + k_i\frac{1}{s} + k_d \frac{s}{s/\omega_p + 1} \\
    &= \frac{(k_p/\omega_p + k_d)s^2 + (k_p + k_i/\omega_p)s + k_i}{s(s/\omega_p + 1)} \\
    &=k_{dc} \frac{(s/\omega_{z1} + 1)(s/\omega_{z2} + 1)}{s(s/\omega_p + 1)}
\end{align}

where,

\begin{align}
    k_p &= k_{dc} \bigg(\frac{\omega_{z1} + \omega_{z2}}{\omega_{z1}\omega_{z2}} - \frac{1}{\omega_p}\bigg) \\
    k_i &= k_{dc} \\
    k_d &= k_{dc} \bigg(\frac{1}{\omega_{z1} \omega_{z2}} - \frac{\omega_{z1} + \omega_{z2}}{\omega_{z1}\omega_{z2} \omega_p} + \frac{1}{\omega_p^2}\bigg)
\end{align}

We see that the controller can be written out as a DC gain with two zeros, a pole, and a pole at the origin which represents 4 design variables that we can manipulate. We will define the bandwidth of our position controller and current regulator to be $\omega_c$ and $\omega_{ci}$ respectively.

We can systematically place these poles and zeros to achieve the phase margin and bandwidth that we desire. Note that this is two input variables but we have four parameters to determine so some engineering judgement must be taken into account. The arbitrary choices that we will be as follows:

\begin{align}
    \omega_{z1} &= \frac{\omega_c}{100} \\
    \omega_p &= 4\omega_c
\end{align}

$\omega_{z1}$ is placed two decades below our controller bandwidth because the kernel of our loop gain starts out at -180 degrees and we need to gain phase. Zero's increase the phase of the system.

$\omega_p$ is placed 4 times higher than the bandwidth of our controller such that it can filter out noise but doesn't affect the overall system dynamics.

The second zero location is chosen to give the desired phase margin as:

\begin{align}
    \omega_{z2} = \frac{\omega_c}{\tan{\big(\phi_{\textrm{PM}} + \pi/2 + \arctan{\frac{\omega_{c}}{\omega_{ci}}} - \arctan{\frac{\omega_{c}}{\omega_{ci}}} + \arctan{\frac{\omega_{z1}}{\omega_{p}}} \big)}}
\end{align}

Finally, the DC gain is set such that the gain cross-over frequency occurs at the desired controller bandwidth:

\begin{align}
    k_{dc} = \frac{1}{\bigg|G_c(j\omega_c)\big|_{k_{dc} = 1}\bigg||I_{CL}(j\omega_c)|G_p(j\omega_c)|}
\end{align}

# Continuous to Discrete Time Controller Implementation

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

```python
s, z_inv, T, kp, ki, kd, wp, M, E, tau = sp.symbols('s z^-1 T k_p k_i k_d \omega_p M(z) E(z) tau')
```

We can use the tustin transformation (trapezoidal integration) to transform our continous time controller into an approximately equivalent discrete time controller:

By definition:

\begin{align}
z = e^{sT} = \frac{e^{sT/2}}{e^{-sT/2}}
\end{align}

Using the first order terms of a Taylor series expansion for the exponential we get:

\begin{align}
z \approx \frac{1 + sT/2}{1 - sT/2}
\end{align}

solving for s we get:

\begin{align}
s \approx \frac{2}{T}\frac{z-1}{z+1} = \frac{2}{T}\frac{1-z^{-1}}{1+z^{-1}}
\end{align}

We will use this above expression to develop difference equations that we can implement in code for each term of our controller:

```python
s_expr = (2/T)*(1-z_inv)/(1 + z_inv)
s_expr
```

$\displaystyle \frac{2 \left(1 - z^{-1}\right)}{T \left(z^{-1} + 1\right)}$

### Proportional Control:

The proportional controller is simple. The manipulation is as follows:

\begin{align}
m_{n,P} = k_p e_n
\end{align}

### Integral Control

Using the Tustin transform on the integral term:

```python
I = ki/s
Ieq = sp.Eq(M/E, I.subs({s:s_expr}))
Ieq
```

$\displaystyle \frac{M(z)}{E(z)} = \frac{T k_{i} \left(z^{-1} + 1\right)}{2 \left(1 - z^{-1}\right)}$

We get the following difference equation:

\begin{align}
m_{n,I} = m_{n-1,I} + k_i \frac{T}{2} (e_n + e_{n-1})
\end{align}

### Derivative Control with Low-Pass Filter

```python
D = kd*(s/(s/wp + 1))
D
```

$\displaystyle \frac{k_{d} s}{1 + \frac{s}{\omega_p}}$

```python
Deq = sp.Eq(M/E, D.subs(s,s_expr))
Deq.expand().simplify().factor(z_inv)
```

$\displaystyle \frac{M(z)}{E(z)} = - \frac{2 \omega_p k_{d} \left(z^{-1} - 1\right)}{T \omega_p + z^{-1} \left(T \omega_p - 2\right) + 2}$

We get the following difference equation:

\begin{align}
m_n (T\omega_p + 2) - m_{n-1} (2-T\omega_p) = 2\omega_p k_d (e_n - e_{n-1})
\end{align}

\begin{align}
m_n = \frac{2-T\omega_p}{T\omega_p + 2} m_{n-1} + \frac{2\omega_p k_d}{T\omega_p + 2} (e_n - e_{n-1})
\end{align}

which can be rewritten as:

\begin{align}
m_{n,D} = k_{md} m_{n-1,D} + k_{ed} (e_n - e_{n-q})
\end{align}

where:

\begin{align}
k_{md} = \frac{2-T\omega_p}{T\omega_p + 2}
\end{align}

\begin{align}
k_{ed} = \frac{2\omega_p k_d}{T\omega_p + 2}
\end{align}

### Total Controller:

The total controller manipulation will be the sum of each of the PID terms as:

\begin{align}
m_n = m_{n,P} + m_{n,I} + m_{n,D}
\end{align}

\begin{align}
m_{n,P} = k_p e_n
\end{align}

\begin{align}
m_{n,I} = m_{n-1,I} + k_i \frac{T}{2} (e_n + e_{n-1}) = m_{n-1,I} + k_{ei} (e_n + e_{n-1})
\end{align}

\begin{align}
m_{n,D} = k_{md} m_{n-1,D} + k_{ed} (e_n - e_{n-1})
\end{align}

At this point, we could write out one long expression for the controller output in terms of the previous the current error, previous error and previous manipulations. We will make the concious decision not to do this however as we would like to keep the integral term seperate in order to be able to apply an anti-windup scheme to it.

### Anti-Windup

We will use clamping for an anti-windup scheme as it is easy to implement and does not have any parameters that require tuning. Clamping works by stopping (or clamping) the integration on the integral control as soon as the following two criteria are met:

1) The controller output is saturating: `|m_n| > maximum allowable manipulation`

2) The integral control is trying to make things worse: `sign(e_n) == sign(m_n)`

If we define the following boolean variable:

`clamp = saturating && integral_makes_worse`

then we can modify our integral control expression to the following:

\begin{align}
m_{n,I} = m_{n-1,I} + k_{ei} (e_n + e_{n-1})(1 - \textrm{clamp})
\end{align}

### Low Pass Filter on Input Commands

To smooth the initial transients of the start up and create a trajectory that can actually be followed, we can pass our commands through a low pass filter. To do this, we can measure the initial position of the rotor and then command that position until a certain time at which we use a step command to the desired centered position. The step command will be smoothed into an exponential curve when passed through the filter:

A simple low pass filter can be designed as a single pole (first order system):

\begin{align}
G(s) = \frac{1}{\tau s + 1}
\end{align}

The transfer function has a gain of unity and will respond to a step input as an exponential with time constant of $\tau$

```python
lowpass = 1/(tau*s + 1)
lowpass
```

$\displaystyle \frac{1}{s \tau + 1}$

```python
lowpass.subs({s:s_expr}).expand().simplify().factor(z_inv)
```

$\displaystyle \frac{T \left(z^{-1} + 1\right)}{T + 2 \tau + z^{-1} \left(T - 2 \tau\right)}$

We get the following difference equation:

\begin{align}
y_{n} (T + 2\tau) - y_{n-1} (2\tau - T) = T(x_{n} + x_{x-1})
\end{align}

\begin{align}
y_{n} = \frac{2\tau - T}{T + 2\tau} y_{n-1}  + \frac{T}{T + 2\tau}(x_{n} + x_{n-1}) = k_{yLP} y_{n-1} + k_{xLP} ( x_{n} + x_{n-1} )
\end{align}

Let's take a look at the discrete step response to the filter we've created

```python
Ts = 1/10_000
tau_n = 0.25

T_settle = 8*tau_n
T_on = 1

total_samples = int((T_on + T_settle)/Ts)
samples_off = int(T_on/Ts)

t_arr = np.linspace(0, total_samples*Ts, total_samples)

kmlp = (2*tau_n - Ts)/(Ts + 2*tau_n)
kelp = Ts/(Ts + 2*tau_n)

x = np.ones(total_samples)
x[0:samples_off] = 0
y = np.zeros_like(x)

for i, val in enumerate(x):

        if i == 0:
            pass
        else:
            y[i] = y[i-1]*kmlp + kelp*(x[i] + x[i-1])

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(t_arr, x, lw = 2, label = 'Input Command')
ax.plot(t_arr, y, label = 'Filtered Command', lw = 2)
ax.set_xlabel('Time, $t$ [s]')
ax.set_ylabel('$y_n$')
ax.legend();
```


```python
import jax.numpy as jnp
from sympy import *
```

```python
x,y,z=symbols('x y z')
init_printing(use_unicode=True)
```

```python
print("\ncos(x) =",diff(cos(x),x),"\n")
diff(exp(x**2),x)
```

Podemos calcular derivadas complejas:
Tomelos la séptima derivada de exp(xyz) con respecto a sus variables, por ejemplo:

## $\frac{\partial^7}{\partial x \partial y^2 \partial z^4}\exp(xyz)$

```python
expr= exp(x*y*z)
diff(expr,x,y,y,z,z,z,z)
```

Para las derivadas numéricas:

## $f'(x_i)\approx \frac{\Delta f}{\Delta x}= \frac{f(x_{i+1})-f(x_i)}{x_{i+1}-x_i}$

```python
import numpy as np
import matplotlib.pyplot as plt

def derivative(x,y):
    xp=np.zeros(len(x))
    yp=np.zeros(len(x))
    for i in range(len(x)-1):
        yp[i]=y[i+1]-y[i]
        xp[i]=x[i+1]-x[i]
    return (yp/xp)
```

```python
x1=np.linspace(-10,10,100)
f1=x1**2*np.sin(x1)
der=derivative(x1,f1)
```

    /Users/diegobarbosa/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:10: RuntimeWarning: invalid value encountered in true_divide
      # Remove the CWD from sys.path while we load stuff.

```python
plt.figure(figsize=(15,8))
plt.plot(x1,f1,'r*',label='Function')
plt.plot(x1,der,'kd',label='Numerical derivative')
plt.legend(fontsize=15)
plt.xlabel("x",fontsize=15)
plt.ylabel("y",fontsize=15)
plt.legend(fontsize=15)
plt.grid()
```

```python
from jax import grad,jit,vmap
import jax.numpy as jnp

x=jnp.linspace(-5,5,100)
grad_f=jit(vmap(grad(jnp.tanh)))
g2=jit(vmap(grad(grad(jnp.tanh))))
g3=jit(vmap(grad(grad(grad(jnp.tanh)))))

plt.figure(figsize=(15,8))
#plt.plot(x,y,'o')
plt.plot(x,np.tanh(x))
plt.plot(x,grad_f(x))
plt.plot(x,g2(x))
plt.plot(x,g3(x))

plt.xlabel("Mediciones",fontsize=15)
plt.ylabel("Observaciones",fontsize=15)
plt.legend(["Datos"],fontsize=15)
plt.grid()
plt.show()
```

```python

```