```
%matplotlib inline
from __future__ import  division
from IPython.html.widgets import interact,fixed
from scipy import stats
import sympy
from sympy import S, init_printing, Sum, summation,expand
from sympy import stats as st
from sympy.abc import k
import re
```

## Maximum likelihood estimation

$$\hat{p} = \frac{1}{n} \sum_{i=1}^n X_i$$

$$ \mathbb{E}(\hat{p}) = p $$

$$ \mathbb{V}(\hat{p}) = \frac{p(1-p)}{n} $$

```
sympy.plot( k*(1-k),(k,0,1),ylabel='$n^2 \mathbb{V}$',xlabel='p',fontsize=28)
```

```
b=stats.bernoulli(p=.8)
samples=b.rvs(100)
print var(samples)
print mean(samples)
```

    0.1659
    0.79

```
def slide_figure(n=100,m=1000,p=0.8):
    fig,ax=subplots()
    ax.axis(ymax=25)
    b=stats.bernoulli(p=p)
    v=iter(b.rvs,2)
    samples=b.rvs(n*1000)
    tmp=(samples.reshape(n,-1).mean(axis=0))
    ax.hist(tmp,normed=1)
    ax1=ax.twinx()
    ax1.axis(ymax=25)
    ax1.plot(linspace(0,1,200),stats.norm(mean(tmp),std(tmp)).pdf(linspace(0.0,1,200)),lw=3,color='r')
```

```
interact(slide_figure, n=(100,500),p=(0.01,1,.05),m=fixed(500));
```

## Maximum A-Posteriori (MAP) Estimation

```
from sympy.abc import p,n,k

sympy.plot(st.density(st.Beta('p',3,3))(p),(p,0,1) ,xlabel='p')
```

### Maximize the MAP function to get form of estimator

```
obj=sympy.expand_log(sympy.log(p**k*(1-p)**(n-k) * st.density(st.Beta('p',6,6))(p)))
```

```
sol=sympy.solve(sympy.simplify(sympy.diff(obj,p)),p)[0]
print sol
```

    (k + 5)/(n + 10)

$ \hat{p}_{MAP} = \frac{(5+\sum_{i=1}^n X_i )}{(n + 10)}  $

with corresponding expectation

$ \mathbb{E}  = \frac{(5+n p )}{(n + 10)} $

which is a biased estimator. The variance of this estimator is the following:

$$ \mathbb{V}(\hat{p}_{MAP}) = \frac{n (1-p) p}{(n+10)^2} $$

compare this to the variance of the maximum likelihood estimator, which is reproduced here:

$$ \mathbb{V}(\hat{p}_{ML}) = \frac{p(1-p)}{n} $$

```
n=5
def show_bias(n=30):
    sympy.plot(p,(5+n*p)/(n+10),(p,0,1),aspect_ratio=1,title='more samples reduce bias')

interact(show_bias,n=(10,500,10));
```

Compute the variance of the MAP estimator

```
sum(sympy.var('x1:10'))
expr=((5+(sum(sympy.var('x1:10'))))/(n+10))
```

```
def apply_exp(expr):
    tmp=re.sub('x[\d]+\*\*2','p',str(expand(expr)))
    tmp=re.sub('x[\d]+\*x[\d]+','p**2',tmp)
    tmp=re.sub('x[\d]+','p',tmp)
    return sympy.sympify(tmp)
```

```
ex2 = apply_exp(expr**2)
print ex2
```

    8*p**2/25 + 11*p/25 + 1/9

```
tmp=sympy.simplify(ex2 - (apply_exp(expr))**2 )
sympy.plot(tmp,p*(1-p)/10,(p,0,1))
```

### General case

```
def generate_expr(num_samples=10,alpha=6):
    n = sympy.symbols('n')
    obj=sympy.expand_log(sympy.log(p**k*(1-p)**(n-k) * st.density(st.Beta('p',alpha,alpha))(p)))
    sol=sympy.solve(sympy.simplify(sympy.diff(obj,p)),p)[0]
    expr=sol.replace(k,(sum(sympy.var('x1:%d'%(num_samples)))))
    expr=expr.subs(n,num_samples)
    ex2 = apply_exp(expr**2)
    ex =  apply_exp(expr)
    return (ex,sympy.simplify(ex2-ex**2))
```

```
num_samples=10
X_bias,X_v = generate_expr(num_samples,alpha=2)
p1=sympy.plot(X_v,(p,0,1),show=False,line_color='b',ylim=(0,.03),xlabel='p')
X_bias,X_v = generate_expr(num_samples,alpha=6)
p2=sympy.plot(X_v,(p,0,1),show=False,line_color='r',xlabel='p')
p3=sympy.plot(p*(1-p)/num_samples,(p,0,1),show=False,line_color='g',xlabel='p')
p1.append(p2[0])
p1.append(p3[0])
p1.show()
```

```
p1=sympy.plot(n*(1-p)*p/(n+10)**2,(p,0,1),show=False,line_color='b',ylim=(0,.05),xlabel='p',ylabel='variance')
p2=sympy.plot((1-p)*p/n,(p,0,1),show=False,line_color='r',ylim=(0,.05),xlabel='p')
p1.append(p2[0])
p1.show()
```

```
def show_variance(n=5):
    p1=sympy.plot(n*(1-p)*p/(n+10)**2,(p,0,1),show=False,line_color='b',ylim=(0,.05),xlabel='p',ylabel='variance')
    p2=sympy.plot((1-p)*p/n,(p,0,1),show=False,line_color='r',ylim=(0,.05),xlabel='p')
    p1.append(p2[0])
    p1.show()
interact(show_variance,n=(10,120,2));

```

The obvious question is what is the value of  a biased estimator? The key fact is that the MAP estimator is biased, yes, but it is biased   according to the prior probability of $\theta$. Suppose that the true parameter $p=1/2$ which is exactly at the peak of the prior probability function. In this case, what is the bias?

$ \mathbb{E}  = \frac{(5+n p )}{(n + 10)} -p \rightarrow 0$

and the variance of the MAP estimator at this point is the following:

$$ \frac{n p (1-p)}{(n+10)^2} \rightarrow $$

```
pv=.60
nsamp=30
fig,ax=subplots()
rv = stats.bernoulli(pv)
map_est=(rv.rvs((nsamp,1000)).sum(axis=0)+5)/(nsamp+10);
ml_est=(rv.rvs((nsamp,1000)).sum(axis=0))/(nsamp);
_,bins,_=ax.hist(map_est,bins=20,alpha=.3,normed=True,label='MAP');
ax.hist(ml_est,bins=20,alpha=.3,normed=True,label='ML');
ax.vlines(map_est.mean(),0,12,lw=3,linestyle=':',color='b')
ax.vlines(pv,0,12,lw=3,color='r',linestyle=':')
ax.vlines(ml_est.mean(),0,12,lw=3,color='g',linestyle=':')
ax.legend()
```

```

```


# Stability of timestepping schemes

You already saw in CMIII that for some problems if we make the timestep too large in the forward Euler method, we don't get a valid solution. Let's remind ourselves with an example.

## Stiff problems

Let's consider a seemingly benign linear system of equations (this example is taken from Iserles' book, section 4.1)

$$
\begin{bmatrix}
\dot u_0\\
\dot u_1\\
\end{bmatrix}
= \overbrace{\begin{bmatrix}
-100 & 1\\
0 & -1/10\\
\end{bmatrix}}^{\Lambda}
\begin{bmatrix}
u_0\\
u_1\\
\end{bmatrix}.
$$

Factorising the matrix $\Lambda = V D V^{-1}$, it is easy to show that the exact solution at time $t$ obeys

$$
u(t) = e^{-100 t} x_1 + e^{-t/10} x_2
$$

where $x_1$ and $x_2$ depend on the initial conditions, but not $t$.

For large $t$, the behaviour of the system is controlled by the slowest decaying of the two exponentials. That is, for all practical purposes, we can write:

$$
u(t) \approx e^{-t/10} x_2, t > 0.
$$

Let's try out explicit Euler for this problem.

```python
%matplotlib widget
import numpy
from matplotlib import pyplot
pyplot.style.use("ggplot")

from scipy.linalg import expm

class linear(object):
    def __init__(self, A):
        self.A = numpy.asarray(A)
    def f(self, t, u):
        return self.A @ u
    __call__ = f
    def explicit(self, t, u):
        return numpy.zeros_like(u)
    def J(self, t, u):
        return self.A
    def exact(self, t, u_0):
        t = numpy.array(t, ndmin=1)
        return [numpy.real_if_close(expm(self.A*s) @ u_0) for s in t]

test = linear([[-100, 1], [0, -1/10]])

u_0 = numpy.asarray([1, 1])

def ode_euler(f, u_0, h=0.1, T=1):
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    while t < T:
        h = min(h, T - t)
        u = u + h * f(t, u)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)

thist, uhist = ode_euler(test, u_0, h=1/100, T=1)
pyplot.figure()

pyplot.semilogy(thist, uhist, "o", linestyle="solid", label='Forward Euler')
pyplot.semilogy(thist, test.exact(thist, u_0), label='exact');
pyplot.legend();
```

### Observations

When the timestep is very small, we get a reasonably accurate answer. However, when the timestep gets big, suddently the numerical solution blows up.

This is due to the small _stability domain_ of the explicit Euler method. As we will see below, the problem is only stable in a small region in the complex plane.

This constraint forces us to _artificially_ constrain the step size to avoid instabilities in the numerical solution.

We will informally call these problems, where the timestep is small for stability, rather than accuracy, reasons *stiff*

### Questions

1. Show that the explicit Euler solution after $n$ steps of size $h$ can be written as
$$
u_n = (1 - 100h)^n x_1 + (1 - \frac{1}{10}h)^n x_2
$$
2. Use this to determine the appropriate maximum step size for the integration.
3. Confirm your analysis with some experiments

## An implicit method

Now let's consider the same problem, only this time we'll use backward Euler, an implicit method.

```python
def ode_beuler(eqn, u_0, h=0.1, T=1):
    A = eqn.A
    u = numpy.array(u_0)
    t = 0
    thist = [t]
    uhist = [u_0]
    while t < T:
        h = min(h, T - t)
        # u <- (I - h A)^{-1} u
        u = numpy.linalg.solve(numpy.eye(len(A)) - h*A, u)
        t += h
        thist.append(t)
        uhist.append(u.copy())
    return numpy.asarray(thist), numpy.asarray(uhist)
```

```python
thist, uhist = ode_beuler(test, u_0, h=1, T=10)
pyplot.figure()

pyplot.semilogy(thist, uhist, "o", linestyle="solid", label='Backward Euler')
pyplot.semilogy(thist, test.exact(thist, u_0), label='exact');
pyplot.legend();
```

```python

```

Now we can take large timesteps, and only need to make $h$ small if we want higher accuracy.

## Linear stability

Why is this?

To answer this, we consider the behaviour of the integrator on a linear test problem, the *Dahlquist test equation*:

$$
\dot u = \lambda u \quad \lambda \in \mathbb{C}.
$$

When taking a step of length $h$, this equation has the exact solution

$$
u(h) = u_0 e^{\overbrace{\lambda h}^z} = u_0 e^{\mathfrak{R} z}(\cos \mathfrak{I} z + i\sin\mathfrak{I} z),
$$
where we wrote $\mathbb{C} \ni z = \lambda h$.

This problem is _physically_ stable whenever the real part of $z$ (and hence $\lambda$) is less than zero: $\mathfrak{R} z \le 0$.

### Aside

There is also theory surrounding _nonlinear_ stability, but we will not cover it in this course.

## Stability regions

Let us now consider applying the timestepping schemes we've already encountered to the test equation to see how they behave.

### Explicit Euler

$$
u(h) = \underbrace{(1 + \lambda h)}_{R(z)}u_0.
$$

Repeated application of the scheme results in:

$$
u(mh) = R(z)^m u_0.
$$

This scheme is convergent (producing a bounded $u(mh)$ in the limit $m \to \infty$) only if

$$
|R(z)| \le 1.
$$

### Implicit Euler

Performing a similar calculation we obtain

$$
R(z) = \frac{1}{1 - z}.
$$

### Trapezoidal rule/implicit midpoint

$$
R(z) = \frac{1 + z/2}{1 - z/2}.
$$

### Definitions
The function $R(z)$ is called the *stability function* of the method, the set

$$
S = \{ z \in \mathbb{C} \colon |R(z)| \le 1 \}
$$

is called the *stability domain*.

## Stability plots

Visualising the stability domains, by plotting $|R(z)|$ is an insightful way of comparing methods:

```python
def plot_stability(x, y, R, label):
    pyplot.figure()
    C = pyplot.contourf(x, y, numpy.abs(R), numpy.linspace(0, 1, 10), cmap=pyplot.cm.coolwarm)

    pyplot.colorbar(C, ticks=numpy.linspace(0, 1, 10))
    pyplot.contour(x, y, numpy.abs(Rz), numpy.linspace(0, 1,4), colors='k')
    pyplot.xlabel("Re (z)")
    pyplot.ylabel("Im (z)")
    pyplot.title(label)
```

```python
x = numpy.linspace(-3, 3)
x, y = numpy.meshgrid(x, x)
z = x + 1j*y

Rs = [("Forward Euler", 1 + z),
      ("Backward Euler", 1/(1 - z)),
      ("Implicit Midpoint", (1 + z/2)/(1 - z/2))]

Rs = [("Heun", 1 + z * (1 + z/2))]
for label, Rz in Rs:
    plot_stability(x, y, Rz, label)
```

```python
pyplot.figure()
z = numpy.linspace(-2, 1)
pyplot.plot(z, 1 + z*(1 + z/2))
```

### Observations

While the physical problem is stable whenever $\mathfrak{R} \lambda \le 0$, the same is not true of all our methods. The explicit Euler method is only stable in a small region of the left half plane, implicit Euler is stable in the entire left half plane, and more, implicit midpoint is stable in exactly the left half plane.

A question naturally arises? Why would one ever use an explicit method? Can you think of any reasons?

## Definitions

### A-stability

A method is _A-stable_ if the stability domain

$$
S = \{z \in \mathbb{C} \colon |R(z)| \le 1\}
$$

contains the _entire_ left half plane

$$
\mathfrak{R} z \le 0.
$$

This means that we can take arbitrarily large timesteps ($h \to \infty$) without the method becoming unstable (diverging) for any problem that is physically stable. Note that this says nothing about the _accuracy_ of the method.

No explicit method is A-stable.

### Questions

1. Show that the midpoint method and implicit Euler really do contain the entire left half plane. Hint: multiply through by an appropriate choice of $1$, write $z = a + bi$ and show that $|R(z)| \le 1$ whenever $a \le 0$.

```python

```

# Higher order methods

So far, we've seen the implicit and explicit Euler methods which are only first order accurate, and the implicit midpoint method which is second order accurate. We might, especially if we have an accurate spatial discretisation, want more accuracy in our time integrator. We will now look at some methods that offer this. Certainly if the timestep size is no longer limited by stability requirements (for example when using an implicit integrator), it makes sense to take big steps, but then we need a high order method for accuracy.

In this section we will look at single-step Runge-Kutta methods (covered briefly at the very end of CMIII).

Many of these methods offer a clever way of estimating the local error accumulation at each step, and we will look at how this can be used to implement an adaptive timestepping scheme.

High-order timestepping schemes also have a higher cost of evaluation (as we shall see), and we'll discuss how to compare different schemes in a fair way using _work-precision_ diagrams.

## [Runge-Kutta methods](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods)

These are a class of _single-step_ methods. That is, we advance a solution $u_0$ from $t_0$ to $t_1$ using $u_0$ as an initial value, producing $u_1$, then when advancing from $t_1$ to $t_2$ we forget about $u_0$ and use $u_1$ as an initial value.

Recall that the single-step methods we have looked at so far (Euler and midpoint schemes) are all obtained by formally solving the ODE with integration and then approximating the integral.

$$
\dot u = f(t, u), u(t_0) = u_0
$$

and hence

$$
u(t) = u(t_0) + \int_{t_0}^t f(\tau, u) \text{d}\tau.
$$

A single step is then obtained by setting $t = t_0 + h$ and writing

$$
\int_{t_0}^{t_0 + h} f(\tau, u) \text{d}\tau = h\int_0^1 f(t_0 + h\tau, y(t_0 + h\tau))\text{d}\tau.
$$

We now _approximate_ the integral with a sum (using a quadrature scheme, but let's sweep that under the carpet for now), by picking a set of points $\{\xi_i\}_{i=1}^{N} \in [0, 1]$ and weights $\{w_i\}_{i=1}^{N} \in \mathbb{R}$

$$
\int_0^1 f(t_0 + h\tau, y(t_0 + h\tau))\text{d}\tau \approx \sum_{i=1}^{N} w_i f(t_0 + \xi_i h, y(t_0 + \xi_i h)).
$$

Giving us our update formula:
$$
u_{n+1} = u_n + h \sum_{i=1}^{N} w_i f(t_n + \xi_i h, u(t_n + \xi_i h)).
$$

For example, explicit Euler was obtained with $N=1$, $\{\xi_i\} = \{0\}$, and $\{w_i\} = \{1\}$.

Our challenge, for more general schemes, is to figure out how to approximate the (unknown) $u$s, we only know the initial value $u(t_0)$.

### Explicit Runge-Kutta methods

Write $Y_i$ for our numerical approximation to $u(t_0 + \xi_i h)$. _Explicit_ Runga-Kutta methods compute $Y_i, i = 2,\dots,N$ as a linear combination of $Y_j, j < i$.

That is, we write

$$
\begin{align}
Y_1 &= u_n\\
Y_2 &= u_n + h a_{2, 1} f(t_n, Y_1)\\
Y_3 &= u_n + h(a_{3,1} f(t_n, Y_1) + a_{3,2} f(t_n + c_2 h, Y_2)\\
&\vdots\\
Y_N &= u_n + h \sum_{i=1}^{N-1} a_{N, i} f(t_n + c_i h, Y_i)\\
u_{n+1} &= u_n + h \sum_{i=1}^{N} b_{i} f(t_n + c_i h, Y_i).
\end{align}
$$

Each of these intermediate steps is termed a *stage*, and so we just wrote down an $N$-stage Runge-Kutta method.

For an explicit method, the _RK matrix_ (or *coefficients*)

$$
A = \begin{bmatrix}
a_{1, 1} & a_{1, 2} & \dots & a_{1, N}\\
a_{2, 1} & a_{2, 2} & \dots & a_{2, N}\\
\vdots & \vdots & \ddots & \vdots\\
a_{N, 1} & a_{N, 2} & \dots & a_{N, N}\\
\end{bmatrix}
$$

is strictly lower-triangular ($a_{i, j} = 0, j \ge i$).

If we further write the _RK weights_ (or *completion weights*)

$$
b = \begin{bmatrix}
b_1 \\
b_2 \\
\vdots \\
b_N
\end{bmatrix}
$$

and the _RK nodes_ (or *abscissa*)

$$
c = \begin{bmatrix}
c_1 \\
c_2 \\
\vdots \\
c_N
\end{bmatrix}
$$

the entire scheme can be compactly represented in a _Butcher tableau_

$$
\begin{array}{c|c}
c & A\\
\hline
 & b^T
\end{array}
$$

(named after [John Butcher](https://en.wikipedia.org/wiki/John_C._Butcher) who developed much of the theory underpinning the analysis and design of Runge-Kutta methods).

### Determining $A$, $b$, and $c$

For low order methods, our first thought is to Taylor expand everything around $(t_n, u_n)$ and then equate terms with the Taylor expansion of the exact solution.

This is doable for $N=2$, but already becomes tremendously labourious at $N=3$. Fortunately, other people (notably Butcher) have developed graph theoretical tools to construct, and analyse the properties of, Runge-Kutta methods. Here, we will only the resulting methods.

One particular point to note is that the conditions on the entries in $A$, $b$, and $c$ do not uniquely specify the coefficients, so it is possible to come up with multiple different methods with the same number of stages.

After all that, let's write some code to do explict RK integration. The Butcher tableau is not just a useful mathematical device, it allows us to write generic code for RK integration (rather than one method for explicit Euler, one for implicit Euler, one for midpoint, etc...).

```python
class ButcherTable(object):
    def __init__(self, A, b, order=None, btilde=None):
        self.A = numpy.asarray(A)
        self.b = numpy.asarray(b)
        self.order = order
        self.btilde = numpy.asarray(btilde) if btilde else None
        self.c = self.A.sum(axis=1)

    def _repr_latex_(self):
        rows, cols = self.A.shape
        strs = ["$$", r"\left[", r"\begin{array}{c|%s}" % ("c"*cols)]
        for r in range(rows):
            row = " & ".join(map(str, [self.c[r]] + list(self.A[r, :])))
            strs.append(row)
            strs.append(r"\\")
        strs.append(r"\hline")

        strs.append("&")
        strs.append(" & ".join(map(str, self.b)))
        strs.append(r"\\")
        if self.btilde is not None:
            strs.append("&" + " & ".join(map(str, self.btilde)))
        strs.extend([r"\end{array}", r"\right]", "$$"])
        return "\n".join(strs)
```

```python
euler = ButcherTable([[0]], [1], order=1); euler
```

```python
beuler = ButcherTable([[1]], [1], order=1); beuler
```

```python
midpoint = ButcherTable([[1/2]], [1], order=2); midpoint
```

```python
def ode_rk_explicit(f, u_0, table, h=0.1, T=1):
    A = table.A
    b = table.b
    c = A.sum(axis=1)
    N = len(c)
    for i in range(N):
        if any(A[i, i:] > 0):
            raise ValueError("Only for explicit methods")
    t = 0
    u_0 = numpy.asarray(u_0)
    u = u_0.copy()
    hist = [(t, u.copy())]
    fY = numpy.zeros(u_0.shape + (N, ))
    while t < T:
        h = min(h, T - t)
        for i in range(N):
            Yi = u.copy()
            for j in range(i):
                Yi += h * A[i, j] * fY[:, j]
            fY[:, i] = f(t + h*c[i], Yi)
        u += h * fY @ b
        t += h
        hist.append((t, u.copy()))
    ts, us = zip(*hist)
    return numpy.asarray(ts), numpy.asarray(us)
```

```python
test = linear([[0, 1.0], [-1.0, 0.0]])

u_0 = numpy.asarray([1.0, 0])

ts, us = ode_rk_explicit(test, u_0, euler, h=0.1, T=15)
pyplot.figure()
pyplot.plot(ts, us, label="Forward Euler");
pyplot.legend();
```

We can also combine multiple explicit Euler steps for a higher order method, [Heun's method](https://en.wikipedia.org/wiki/Heun%27s_method) (also called "modified Euler" or "explicit trapezoid"). This has Butcher tableau

$$
\left[
\begin{array}{c|cc}
0 & 0 & 0\\
1 & 1 & 0\\
\hline
  & 1/2 & 1/2
\end{array}
\right]
$$

```python
heun = ButcherTable(numpy.asarray([[0, 0], [1, 0]]), numpy.asarray([1/2, 1/2]), order=2); heun
```

```python
ts, us = ode_rk_explicit(test, u_0, heun, h=0.1, T=15)
pyplot.figure()
pyplot.plot(ts, us, label="Heun");
pyplot.legend();
```

Perhaps _the_ most famous RK method is the explicit 4-th order RK4

```python
rk4 = ButcherTable(numpy.asarray([[0, 0, 0, 0],
                                  [1/2, 0, 0, 0],
                                  [0, 1/2, 0, 0],
                                  [0, 0, 1, 0]]),
                   numpy.asarray([1/6, 1/3, 1/3, 1/6]), order=4); rk4
```

```python
ts, us = ode_rk_explicit(test, u_0, rk4, h=0.1, T=15)
pyplot.figure()
pyplot.plot(ts, us, label="RK4");
pyplot.legend();
```

## Questions

1. Derive an expression for the stability function $R(z)$ for RK methods by applying one step to the Dahlquist test equation
2. Extend the `ButcherTable` class with a method to plot the method's stability region.
3. What do you see when comparing the stability of Heun and RK4 to that of explicit Euler?

```python

```

## Comparing methods

All other things being equal, a higher-order method is likely a superior choice, since it will converge faster to the exact solution with bigger timesteps. For example, let's compute the convergence rate for the explicit Euler, BS3, RK4, and DP5 methods on our linear osciliatory test problem.

```python
from functools import partial

def mms(eqn, integrator, u_0, h=0.1, T=15):
    thist, uhist = integrator(eqn, u_0, h=h, T=T)
    return numpy.linalg.norm(uhist[-1] - eqn.exact(thist[-1], u_0), numpy.inf)

eqn = linear(numpy.array([[0, 1],
                          [-1, 0]]))
u_0 = numpy.array([.75, 0])

hs = numpy.array([1/2**i for i in range(2, 10)])
pyplot.figure()
method_errors = {}
for name, method in [("Forward Euler", partial(ode_rk_explicit, table=euler)),
                     ("Heun", partial(ode_rk_explicit, table=heun)),
                     ("RK4", partial(ode_rk_explicit, table=rk4))]:
    errors = []
    for h in hs:
        error = mms(eqn, method, u_0, h=h, T=8)
        errors.append(error)
    method_errors[name] = numpy.asarray(errors)
    pyplot.loglog(hs, errors, ".", label=name)

pyplot.loglog(hs, 1.5*hs, label="$\mathcal{O}(h)$")
pyplot.loglog(hs, 2*hs**2, label="$\mathcal{O}(h^2)$")
pyplot.loglog(hs, 0.1*hs**4, label="$\mathcal{O}(h^4)$")
pyplot.legend()
pyplot.xlabel("h")
pyplot.ylabel("$e_{n,h}$");
```

### Questions

1. What timestep would you need for Heun's method and forward Euler respectively to obtain an error of $10^{-10}$?

```python

```

### Work precision diagrams

This comparison is slightly unfair, since when we are only looking at the error, higher-order methods will clearly perform better for the same timestep. Especially on this oscillatory test problem that has a very smooth solution.

A fairer method of comparison is to use _work precision_ diagrams. In these, we plot some proxy for the total _cost_ of a method against the obtained error. For explicit methods, the proxy is usually the _number of function evaluations_ (since it avoids arguments about inefficiencies in implementation) though _time to solution_ is also valid.

With this in mind, we can replot our convergence data.

```python
def work(hs, table, T=15):
    A = table.A
    b = table.b
    stages, _ = A.shape
    if numpy.allclose(A[-1, :], b):
        stages -= 1
    steps = T/hs
    return steps * stages
```

```python
cost = partial(work, hs, T=15)
pyplot.figure()
for name, table in [("Forward Euler", euler),
                    ("Heun", heun),
                    ("RK4", rk4)]:
    pyplot.loglog(cost(table), method_errors[name], marker="o", label=name)
pyplot.legend()
pyplot.xlabel("Number of function evaluations")
pyplot.ylabel("Error")
pyplot.title("Error vs Cost");
```

### Questions

1. This looks like the high-order method still always wins. But can you actually increase the step size sufficiently that you can extrapolate to the low-error case?

```python

```

## Error control

In CMIII you saw a relatively simple method for adaptive timestepping. Given a single timestepping scheme with step length $h$, the error can be estimated by performing two steps with step length $h/2$ and computing the difference in the resulting guess.

This is rather wasteful, in that we need to perform three time integrations to take one step.

For higher-order RK schemes, a more sophisticated approach is possible. Since the constraints on the coefficients in the Butcher table can admit more than one possible solution it is often possible to use the same set of stages with two different sets of *completion weights* (the $b$ vectors) to produce two updates of different order effectively "for free". The error can then be estimated by computing the difference between the two steps.

### Error estimation with embedded methods

Let us suppose we have an RK method

$$
\left[
\begin{array}{c|c}
c & A\\
\hline
& b^T\\
& \tilde{b}^T
\end{array}
\right]
$$

Where the completion weights $b^T$ produce a method of order $p$, and the weights $\tilde{b}^T$ produce a mthod of order $\tilde{p} = p - 1$. We then have, from a starting point $u_n$, two candidate solutions:

$$
\begin{align}
u_{n+1} &= u_n + h b^T f(Y)\\
\tilde{u}_{n+1} &= u_n + h \tilde{b}^T f(Y)\\
\end{align}
$$

where $f(Y)$ is the vector of intermediate function evaluations.

We can now estimate the error in a single step by

$$
e(h) = \| u_{n+1} - \tilde{u}_{n+1} \| = \| h (b - \tilde{b})^T f(Y) \| \in \mathcal{O}(h^p)
$$

Given some desired tolerance $\epsilon$, we want to find $h_*$ such that

$$
e(h_*) < \epsilon
$$

Since $e(h) \in \mathcal{O}(h^p)$ we can write $e(h) = c h^p$, then our requirement on $h_*$ means we want

$$
c h_*^p < \epsilon
$$

and so

$$
h_* < \left(\frac{\epsilon}{c}\right)^{1/p}.
$$

We estimate the unknown $c$ by using our error estimate:

$$
e(h) = c h^p \Leftrightarrow c = \frac{e(h)}{h^p}
$$

and substituting in

$$
h^* < h \left(\frac{\epsilon}{e(h)}\right)^{1/p}
$$

### BS3

A low-order method which implements this embedded idea is the [Bogacki-Shampine](https://en.wikipedia.org/wiki/Bogacki–Shampine_method) method of order 3, which comes with an embedded 2nd order method.

```python
bs3 = ButcherTable([[0, 0, 0, 0],
                    [1/2, 0, 0, 0],
                    [0, 3/4, 0, 0],
                    [2/9, 1/3, 4/9, 0]],
                   [2/9, 1/3, 4/9, 0], # order 3
                   order=3,
                   btilde=[7/24, 1/4, 1/3, 1/8]) # order 2
```

Although this method has 4 stages, it only requires 3 function evaluations, since if we stare closely we can see that the last stage has the same coefficients as the order 3 completion formula (so we can reuse the last stage for the next timestep). This is termed the _First Same as Last_ or FSAL property.

```python
bs3
```

### ODE45

The default ODE solver in MATLAB (amongst others) is the 5th order [Dormand-Prince](https://en.wikipedia.org/wiki/Dormand–Prince_method) method (with a 4th order embedded method). One can vaguely imagine that this table was not computed by frantically Taylor expanding!

```python
dp45 = ButcherTable([[0, 0, 0, 0, 0, 0, 0],
                     [1/5, 0, 0, 0, 0, 0, 0],
                     [3/40, 9/40, 0, 0, 0, 0, 0],
                     [44/45, -56/15, 32/9, 0, 0, 0, 0],
                     [19372/6561, -25360/2187, 64448/6561, -212/729, 0, 0, 0],
                     [9017/3168, -355/33, 46732/5247, 49/176, -5103/18656, 0, 0],
                     [35/384, 0, 500/1113, 125/192, -2187/6784, 11/84, 0]],
                    [35/384, 0, 500/1113, 125/192, -2187/6784, 11/84, 0],
                    order=5,# 5th order
                    btilde=[5179/57600, 0, 7571/16695, 393/640, -92097/339200, 187/2100, 1/40]) # 4th order
dp45
```

Note how this method also has the FSAL property for the 5th order completion formula.

## Adaptive error control

Let us look at an example use of adaptive error control. The step-size acceptance here is rather simple, for a more thorough treatment, see [Hairer & Wanner (1993), pages 167-168](https://www.springer.com/gp/book/9783540566700).

We solve the coupled chemical reaction system known as the "Brusselator" [Lefever & Nicolis (1971)](https://doi.org/10.1016/0022-5193(71)90054-3).

$$
\begin{align}
\dot{u}_1 &= A + u_1^2 u_2 - (B+1)u_1\\
\dot{u}_2 &= Bu_1 - u_1^2 u_2\\
A &= 1\\
B &= 3\\
u_1(0) &= 1.5\\
u_2(0) &= 3.0\\
\end{align}
$$

```python
class brusselator(object):
    def __init__(self,  A, B):
        self.A = A
        self.B = B

    def __call__(self, t, u):
        u1, u2 = u
        A = self.A
        B = self.B
        return numpy.asarray([A + u1**2*u2 - (B + 1)*u1, B*u1 - u1**2*u2])
```

For the error control we will use the embedded method in our RK scheme to calculate an error $\|u - \tilde{u}\|$. If this is larger than some specified $\epsilon$ we should reject the step. Additionally, an optimal $h$ for the next step (irrespective of whether we accepted the step or not) is chosen by using the inequality above.

We set:

$$
h_\text{next} = 0.9 * h_\text{current} \left( \frac{\| u - \tilde{u}\|}{\epsilon}\right)^{-1/p}
$$

where $p$ is the order of the method. The factor of 0.9 is a safety factor to ensure we have some wiggle room in our optimal step size estimation.

```python
def ode_rk_explicit_adaptive(f, u_0, table, eps=0.1, h_0=0.1, T=1):
    """Integrate f using an explicit RK scheme and adaptive timestepping.

    :arg f: The RHS a callable f(t, u) -> udot
    :arg u_0: initial value for u
    :arg table: a :class:`ButcherTable`.
    :arg eps: target local error.
    :arg h_0: Initial step size.
    :arg T: Final time."""
    A = table.A
    b = table.b
    btilde = table.btilde
    if btilde is None:
        raise ValueError("Provided scheme %r does not have embedded scheme" % table)
    order = table.order
    c = A.sum(axis=1)
    N = len(c)
    t = 0
    u_0 = numpy.asarray(u_0)
    u = u_0.copy()
    hist = [(t, u.copy(), h_0)]
    fY = numpy.zeros(u_0.shape + (N, ))
    h = h_0
    while t < T:
        accept = True
        h = min(h, T - t)
        fY[:] = 0
        # Standard RK stages
        for i in range(N):
            Yi = u.copy()
            for j in range(i):
                Yi += h * A[i, j] * fY[:, j]
            fY[:, i] = f(t + h*c[i], Yi)
        # Error estimation
        ulo_up = h * fY @ btilde
        u_up = h * fY @ b
        err = numpy.linalg.norm(u_up - ulo_up)
        relerr = err / eps
        if relerr > 1: # error bigger than eps
            # The step was too big
            accept = False
        if relerr > 0:
            next_h = h * 0.9*relerr**(-1/order)
        else:
            # Some large number
            next_h = h*2
        if accept:
            # Step was fine (error was controlled), so use it.
            u += u_up
            t += h
            hist.append((t, u.copy(), h))
        # New step
        h = next_h
    ts, us, hs = zip(*hist)
    return numpy.asarray(ts), numpy.asarray(us), numpy.asarray(hs)
```

```python
u = numpy.asarray([1.5, 3.0])
eqn = brusselator(A=1, B=4)

thist, uhist, hhist = ode_rk_explicit_adaptive(eqn, u, bs3, eps=1e-3, h_0=0.1, T=20)
```

```python
pyplot.figure()
pyplot.plot(thist, uhist);
pyplot.figure()
pyplot.plot(thist, hhist);
pyplot.xlabel("t")
pyplot.ylabel("h");
```

## Questions

1. Use a small timestep, high-order integrator to compute an "exact" solution. Use this to produce MMS convergence plots for the BS3 and DP45 methods using this adaptive error control mechanism. How do you need to vary $\epsilon$ to get the convergence order you expect?

2. Using the same high-order "exact" solution, produce a work-precision diagram comparing fixed step-size methods against adaptive methods. Are the adaptive methods always superior in terms of work for a given accuracy? You may find it useful to augment the `brusselator` class with a count object that keeps track of the number of function evaluations (don't forget to create a fresh object for each integration though!).

3. Compare, using a work-precision diagram, the cost of this "embedded" adaptivity scheme with the "try with h/2 and compare" approach of CMIII.

```python

```

# Back to stiff problems

## Implicit Runge-Kutta methods

We have already seen that we will want an implicit, rather than explicit, method when taking large timesteps. The same formalism for expressing Runge-Kutta methods is also applicable to implicit ones. In this case, $A$ is no longer strictly lower-triangular. Popular implicit RK schemes are typically at least lower triangular ($a_{i,j} = 0, j > i$). This way, although we need to solve a system for each stage, we can still do forward-substitutions, and the stage equations become:

$$
Y_i - h a_{i,i} f(t_n + c_i h, Y_i) = u_n + h \sum_{j < i} a_{i, j} f(t_n + c_j h, Y_j).
$$

These are termed DIRK (_diagonally_ implicit Runge-Kutta) methods.

A subclass that is even better (if achievable) is SDIRK (_singly_ diagonally implicit Runge-Kutta) methods. This last implicit class has $a_{i,i} = \alpha$ for all $i$.

These methods are attractive because we can often reuse (part of) the setup cost for solving the systems at each stage.

Let's return to the _stiff_ equation we first saw

$$
\dot{u} = -k(u - \cos t)
$$

and try solving it when $k$ is very large (this increases the stiffness).

```python
class stiff(object):
    def __init__(self, k):
        self.k = k
    def f(self, t, u):
        return -self.k * (u - numpy.cos(t))

    def explicit(self, t, u):
        # This part of the forcing function will be moved to the rhs
        return self.k * numpy.cos(t)

    __call__ = f

    def J(self, t, u):
        # This is the linearly implicit part of the forcing function (solved for)
        # df/du
        return numpy.asarray(-self.k).reshape(1, 1)

    def exact(self, t, u_0):
        k = self.k
        k2p1 = k/(1 + k**2)
        return (u_0 - k*k2p1)*numpy.exp(-k*t) + k2p1 * (numpy.sin(t) + k*numpy.cos(t))
```

```python
def ode_dirk_linear(eqn, u_0, table, h=0.1, T=1):
    A = table.A
    b = table.b
    order = table.order
    c = A.sum(axis=1)
    N = len(c)
    for i in range(N):
        if any(A[i, i+1:] > 0):
            raise ValueError("Only for diagonally implicit RK schemes")
    t = 0
    u_0 = numpy.asarray(u_0).reshape(-1)
    u = u_0.copy()
    hist = [(t, u.copy())]
    fY = numpy.zeros(u_0.shape + (N, ))
    while t < T:
        h = min(h, T - t)
        fY[:] = 0
        # Standard RK stages
        for i in range(N):
            Yi = u.copy()
            tstage = t + h*c[i]
            for j in range(i):
                Yi += h * A[i, j] * fY[:, j]
            if A[i, i] != 0:
                # Solving
                # Yi - h a_ii f(tstage, Yi) = u_n + \sum_{i<j} h A[i, j] f(t + c[j] h, Y_j)
                # f(stage, Yi) splits into linearly implicit part and (non)linear explicit part (moved to rhs)
                J = eqn.J(tstage, Yi)
                rhs = Yi + h * A[i, i] * eqn.explicit(tstage, Yi)
                Yi = numpy.linalg.solve(numpy.eye(J.shape[0]) - h * A[i, i] * J, rhs)
            fY[:, i] = eqn(tstage, Yi)
        u += h * fY @ b
        t += h
        hist.append((t, u.copy()))
    ts, us = zip(*hist)
    return numpy.asarray(ts), numpy.asarray(us)
```

```python
beuler = ButcherTable([[1]], [1], order=1)

test = stiff(200)

u_0 = numpy.array(0.2)
thist, uhist = ode_dirk_linear(test, u_0, beuler, h=0.1, T=12)
pyplot.figure()
pyplot.plot(thist, uhist, "o", linestyle="solid", label='Backward Euler')
pyplot.plot(thist, test.exact(thist, u_0), label='exact');
pyplot.legend();
```

```python
# 4 stage, 3rd order scheme from Ascher et al. (1997) https://doi.org/10.1137/0732037
sdirk43 = ButcherTable([[1/2, 0, 0, 0],
                       [1/6, 1/2, 0, 0],
                       [-1/2, 1/2, 1/2, 0],
                       [3/2, -3/2, 1/2, 1/2]],
                      [3/2, -3/2, 1/2, 1/2], order=3)
sdirk43
```

```python
u_0 = numpy.array(0.2)

test = stiff(k=1000)
thist, uhist = ode_dirk_linear(test, u_0, sdirk43, h=0.1, T=20)
pyplot.figure()
pyplot.plot(thist, uhist, "o", linestyle="solid", label='DIRK43')
pyplot.plot(thist, test.exact(thist, u_0), label='exact');
pyplot.legend()
```

## Questions

1. Produce work-precision diagrams for the implicit methods on the stiff equation above. Compare them to using an explicit integrator instead.

```python

```


# Sum-of-Squares (SOS) optimization

## Background
Sum-of-Squares (SOS) optimization is a special type of convex optimization. This type of optimization is formulated as a Semidefinite Programming (SDP) problem, which can be solved through several solvers supported by Drake's MathematicalProgram.

A polynomial $p(x)$ is sum-of-squares if
\begin{align}
p(x) = \sum_i q_i(x)^2
\end{align}
where each $q_i(x)$ is also a polynomial of the indeterminates $x$. $p(x)$ being sum-of-squares is a *sufficient* condition for $p(x)$ being non-negative for all $x$. In some cases it is also a necessary and sufficient condition. For more details on the relationship between sum-of-squares and non-negativity of a polynomial, you could refer to Chapter 3 of [Semidefinite Optimization and Convex Algebraic Geometry](http://www.mit.edu/~parrilo/sdocag/).

A polynomial $p(x)$ being sum-of-squares is equivalent to the existence of a positive semidefinite (psd) matrix $Q$, such that
\begin{align}
p(x) = z(x)^T Q z(x), Q\succeq 0
\end{align}
where $z(x)$ contains all the monomials of $x$ up to a certain degree. If $x\in\mathbb{R}^n$ and $p(x)$ has degree $2d$, then $z(x)$ contains all the monomials of $x$ up to degree $d$, namely the $z(x)$ has size ${{n+d}\choose{d}}$. So searching for a sum-of-squares polynomial $p(x)$ is equivalent to searching for a positive semidefinite matrix $Q$, with the additional constraint that the polynomials $z(x)^TQz(x)$ and $p(x)$ are the same (they have the same coefficients).

For example, for the polynomial
\begin{align}
p(x) = 2x_1^4 + 5x_2^4 - 2x_1^2x_2^2 + 2x_1^3x_2 + 2x_1 + 2,
\end{align}
we have
\begin{align}
p(x) = \underbrace{\begin{bmatrix} 1 \\x_1 \\x_2 \\ x_1^2 \\ x_1x_2 \\ x_2^2\end{bmatrix}^T}_{z(x)^T} \underbrace{\frac{1}{3}\begin{bmatrix} 6 & 3 & 0 & -2 & 0 & -2\\ 3 & 4 & 0 & 0 & 0 & 0\\0 & 0 & 4 & 0 & 0 & 0\\-2 & 0 & 0 & 6 & 3 & -4\\ 0 & 0 & 0 & 3 & 5 & 0\\ -2 & 0 & 0 & -4 & 0 & 15\end{bmatrix}}_{Q} \underbrace{\begin{bmatrix} 1 \\ x_1 \\x_2 \\x_1^2 \\x_1x_2\\ x_2^2\end{bmatrix}}_{z(x)}
\end{align}
where this $Q$ matrix is positive semidefinite, hence $p(x)$ is a sum-of-squares polynomial.
In fact, $p(x)$ can be decomposed as sum-of-squares
\begin{align}
p(x) = \frac{4}{3}x_2^2 + \frac{1349}{705}x_2^4 + \frac{1}{12}(4x_1+3)^2+\frac{1}{15}(3x_1^2+5x_1x_2)^2+\frac{1}{315}(-21x_1^2+20x_2^2+10)^2+\frac{1}{59220}(328x_2^2-235)^2
\end{align}

## Solving sum-of-squares problem
In sum-of-squares problem, often we want to find a sum-of-squares polynomial that satisfies certain additional properties. Note that there are two different types of variables in sum-of-squares problems, the *indeterminates* and *decision variables*.
* Decision variables: the coefficients of the polynomial are affine functions of our decision variables. We are optimizing over the decision variables.
* Indeterminates: variables that can take any value. In the example above $x$ is an indeterminate -- the constraints that we write must hold $\forall x$, we are *not* optimizing over these indeterminates.

There are some functions commonly used in sum-of-squares optimization within Drake's MathematicalProgram. We list some of them here
* `MathematicalProgram.NewIndeterminates` Declares and returns new indeterminates.
* `MathematicalProgram.AddSosConstraint` Imposes sum-of-squares (SOS) constraint on a polynomial.
* `MathematicalProgram.NewSosPolynomial` Creates and returns a new polynomial, and constrain this polynomial to be sum-of-squares (SOS).
* `MathematicalProgram.NewFreePolynomial` Creates and returns a new polynomial. No constraint has been imposed on this polynomial yet.
* `MathematicalProgram.AddEqualityConstraintBetweenPolynomials` imposes the linear constraint that two polynomials are equal (they have the same coefficients for matching monomials).

We will describe the details of each function later. First let's look at some examples

### Imports

```python
import math

import numpy as np
import matplotlib.pyplot as plt

from pydrake.solvers.mathematicalprogram import MathematicalProgram, Solve
import pydrake.symbolic as sym
```

### Example 1
For example, if we want to solve the following optimization problem (Example 3.50 of [Semidefinite Optimization and Convex Algebraic Geometry](http://www.mit.edu/~parrilo/sdocag/))
\begin{align}
\min_{a}& -a_0 - a_1\\
\text{s.t }& x^4 + a_0 x + 2 + a_1 \text{ is SOS}\\
&(a_0 - a_1 + 1)x^2 + a_1x + 1\text{ is SOS}
\end{align}
in this case, $x$ is the indeterminate, and $a$ is the decision variables.

The code below shows how to model and solve this SOS problem in drake's MathematicalProgram.

```python
"""
Code for example 1.
"""

# Initialize an empty optimization program.
prog = MathematicalProgram()
# Declare "a" as decision variables.
a = prog.NewContinuousVariables(2)
# Declare "x" as indeterminates
x = prog.NewIndeterminates(1)

# Declare p1(x) = pow(x, 4) + a0 * x + 2 + a1. The second argument
# [x] indicates that the indeterminates in the polynomial is x.
p1 = sym.Polynomial(x[0]**4 + a[0] * x[0] + 2 + a[1], [x])
# Declare p2(x) = (a0 - a1 + 1) * pow(x, 2) + a1 * x + 1. The second argument
# [x] indicates that the indeterminates in the polynomial is x.
p2 = sym.Polynomial((a[0] - a[1] + 1) * x[0] * x[0] + a[1] * x[0] + 1, [x])
# Add the constraint that p1(x) and p2(x) are SOS
prog.AddSosConstraint(p1)
prog.AddSosConstraint(p2)

# Add the cost -a0 - a1
prog.AddCost(-a[0] - a[1])

# Solve the problem.
result = Solve(prog)

# Retrieve the solution.
print(result.get_solution_result())
print("a =", result.GetSolution(a))
```

### Example 2
\begin{align}
\min_{a, b, c}&\; a + b + c\\
\text{s.t }& x^4 + ax^3 + bx^2 + cx + 1 \ge 0 \quad\forall x\in[0, 1]
\end{align}

To solve this problem, we use a special case of Theorem 3.72 from
[Semidefinite Optimization and Convex Algebraic Geometry](http://www.mit.edu/~parrilo/sdocag/):
An even degree univariate polynomial $p(x)$ is nonnegative on interval $[0, 1]$ if and only if it can be written as
\begin{align}
p(x) = s(x) + x(1-x)t(x)
\end{align}
where $s(x), t(x)$ are both sum-of-squares.

Hence we can reformulate the problem as
\begin{align}
\min_{a, b, c, t}&\; a+b+c\\
\text{s.t }& x^4 + ax^3 + bx^2 + cx + 1 - x(1-x)t(x) \text{ is SOS}\\
& t(x) \text{ is SOS}
\end{align}
here $x$ is the indeterminate, $a, b, c$ and the coefficients of polynomial $t(x)$ are the decision variables. The code below shows how to model and solve this problem in drake's MathematicalProgram

```python
"""
Code for example 2.
"""

# Initialize an empty optimization program.
prog = MathematicalProgram()
# Declares indeterminates
x = prog.NewIndeterminates(1)

# Declares decision variable a, b, and c
a = prog.NewContinuousVariables(1)[0]
b = prog.NewContinuousVariables(1)[0]
c = prog.NewContinuousVariables(1)[0]

# Declare p(x), the second argument indicates this polynomial p(x) has x as the
# indeterminates.
poly_p = sym.Polynomial(x[0]**4 + a * x[0]**3 + b * x[0]** 2 + c * x[0] + 1, [x])
# Declares SOS polynomial t(x), such that s(x) = p(x) - x(1-x) * t(x) is SOS.
# The second return argument of NewSosPolynomial is the positive-semidefinite
# Gramian matrix of poly_s. We ignore this matrix here.
# t(x) should be a quadratic polynomial of x, hence the second input argument is
# 2, indicating the degree of the returned polynomial.
poly_t, _ = prog.NewSosPolynomial(sym.Variables(x), 2)
# Compute s(x) = p(x) - x(1-x)t(x) as a polynomial of indeterminate x.
poly_s = poly_p - sym.Polynomial(x[0] * (1 - x[0]), sym.Variables(x)) * poly_t
prog.AddSosConstraint(poly_s)

# Add the cost a + b + c
prog.AddCost(a + b + c)

result = Solve(prog)
print("Is success? ", result.is_success())
print(result.get_solution_result())
a_val = result.GetSolution(a)
b_val = result.GetSolution(b)
c_val = result.GetSolution(c)

# Now plot the polynomial p(x), that is always non-negative in the range [0, 1]
x_val = np.linspace(0, 1, 100)
y_val = np.power(x_val, 4) + a_val * np.power(x_val, 3) + b_val * np.power(x_val, 2) + c_val * x_val + 1
plt.xlabel("x")
plt.ylabel("p(x)")
plt.plot(x_val, y_val)
```

## Further readings

Sum-of-squares optimization has been widely used in numerous fields, including control theory, robotics, quantum information theory, etc. Here are some references

#### Sum-of-squares theory
* [Semidefinite Optimization and Convex Algebraic Geometry](http://www.mit.edu/~parrilo/sdocag/) by G Blekherman, P Parrilo and R Thomas
* [Sum of squares techniques and polynomial optimization](http://www.princeton.edu/~amirali/Public/Presentations/CDC_2016_Parrilo_1) Presentation slides by Pablo Parrilo

#### System control theory
* [Structured Semidefinite Programs and Semialgebraic Geometry Methods in Robustness and Optimization](http://www.mit.edu/~parrilo/pubs/files/thesis.pdf) Pablo Parrilo's PhD thesis
* [Robust Region-of-Attraction Estimation](https://ieeexplore.ieee.org/abstract/document/5337881) Ufuk Topcu, Andrew Packard, Peter Seiler, Gary Balas, IEEE Transactions on Automatic Control, 2009

#### Robotics
* [LQR-trees: Feedback motion planning via sums-of-squares verification](http://groups.csail.mit.edu/robotics-center/public_papers/Tedrake10.pdf) Russ Tedrake, Ian Manchester, Mark Tobenkin, John Roberts, International Journal of Robotics Research, 2010
* [Funnel libraries for real-time robust feedback motion planning](http://groups.csail.mit.edu/robotics-center/public_papers/Majumdar16.pdf) Anirudha Majumdar and Russ Tedrake, International Journal of Robotics Research, 2017


ECMM426 - Computer Vision / ECMM441 - Machine Vision (Professional)

Workshop 2

Corner Detection

## Imports

```
import urllib
import matplotlib, cv2
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (20.0, 10.0)
plt.rcParams['figure.dpi'] = 72

print('OpenCV version: {}'.format(cv2.__version__))
```

## Download Images
Download chessboard image

```
import os
if not os.path.exists('chessboard.zip'):
  !wget --no-check-certificate https://empslocal.ex.ac.uk/people/staff/ad735/ECMM426/chessboard.zip
  !unzip -q chessboard.zip
```

## Read and show the image

```
# read an image
img = cv2.imread('chessboard.png')

# convert image to grayscale
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY).astype(np.float32)

# show the image
plt.imshow(gray_img, cmap='gray')
```

## Add border to detect all the corners

```
gray_img = cv2.copyMakeBorder(gray_img.copy(), 10, 10, 10, 10, cv2.BORDER_CONSTANT, value=[0, 0, 0])
# show the image
plt.imshow(gray_img, cmap='gray')
```

## Harris corner detection
We know that corners are regions in the image with large variation in intensity in all the directions. One early attempt to find these corners was done by **Chris Harris & Mike Stephens** in their paper **A Combined Corner and Edge Detector** in 1988, so now it is called the **Harris Corner Detector**. He took this simple idea to a mathematical form. It basically finds the difference in intensity for a displacement of (u,v) in all directions. This is expressed as below:

$$E(u,v) = \sum_{x,y} \underbrace{w(x,y)}_\text{window function} \, [\underbrace{I(x+u,y+v)}_\text{shifted intensity}-\underbrace{I(x,y)}_\text{intensity}]^2$$

The window function is either a rectangular window or a Gaussian window which gives weights to pixels underneath. For sake of simplicity we can consider:

$$E(u,v) = \sum_{x,y} [\underbrace{I(x+u,y+v)}_\text{shifted intensity}-\underbrace{I(x,y)}_\text{intensity}]^2$$

We have to maximize this function $E(u,v)$ for corner detection.

Taylor's series expansion

$$f(x)=f(a+x-a)=\sum_{n=0}^{\infty} \frac{f^{(n)}(a)}{n!}(x-a)^n=f(a)+\frac{f^{(1)}(a)}{1!}(x-a)+\frac{f^{(2)}(a)}{2!}(x-a)^2+\ldots$$

Applying Taylor's expansion to the above equation

$$\begin{align}
I(x+u,y+v)&=I(x,y)+\frac{\partial I}{\partial x}u+\frac{\partial I}{\partial y}v+\frac{\partial^2 I}{\partial x^2}u^2+\frac{\partial^2 I}{\partial y^2}v^2+\ldots\\
&=I(x,y)+I_xu+I_yv+I_{xx}u^2+I_{yy}v^2+\ldots
\end{align}$$

Therefore,

$$E(u,v) \approx \sum_{x,y}(I_xu+I_yv)^2 \approx \sum_{x,y}I_xI_xu^2+2\sum_{x,y}I_xI_yuv+\sum_{x,y}I_yI_yv^2$$

 and using some mathematical steps (please refer to any standard text books you like for full derivation), we get the final equation as:

$$E(u,v) \approx \begin{bmatrix} u & v \end{bmatrix} M \begin{bmatrix} u \\ v \end{bmatrix}$$

where

$$M = \sum_{x,y} \begin{bmatrix}I_x I_x & I_x I_y \\ I_x I_y & I_y I_y \end{bmatrix}$$

Here, $I_x$ and $I_y$ are image derivatives in x and y directions respectively.

## Image gradient

```
# construct the Sobel x-axis kernel
sobelX = np.array(([-1, 0, 1], [-2, 0, 2], [-1, 0, 1]), dtype=np.float)
# construct the Sobel y-axis kernel
sobelY = np.array(([-1, -2, -1], [0, 0, 0], [1, 2, 1]), dtype=np.float)
# convolution
I_x = cv2.filter2D(gray_img, -1, sobelX)
I_y = cv2.filter2D(gray_img, -1, sobelY)
# plot
plt.subplot(1,3,1),plt.imshow(gray_img, cmap = 'gray')
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1,3,2),plt.imshow(I_x, cmap = 'gray')
plt.title('$I_x$'), plt.xticks([]), plt.yticks([])
plt.subplot(1,3,3),plt.imshow(I_y, cmap = 'gray')
plt.title('$I_y$'), plt.xticks([]), plt.yticks([])
plt.show()
```

## Gradient covariances

```
# Gradient covariance and light smoothing
I_x_I_x = cv2.GaussianBlur(I_x*I_x, (3, 3), sigmaX=1, sigmaY=1)
I_y_I_y = cv2.GaussianBlur(I_y*I_y, (3, 3), sigmaX=1, sigmaY=1)
I_x_I_y = cv2.GaussianBlur(I_x*I_y, (3, 3), sigmaX=1, sigmaY=1)

# plot
plt.subplot(1,4,1),plt.imshow(gray_img, cmap = 'gray')
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1,4,2),plt.imshow(I_x_I_x, cmap = 'gray')
plt.title('$I_xI_x$'), plt.xticks([]), plt.yticks([])
plt.subplot(1,4,3),plt.imshow(I_y_I_y, cmap = 'gray')
plt.title('$I_yI_y$'), plt.xticks([]), plt.yticks([])
plt.subplot(1,4,4),plt.imshow(I_x_I_y, cmap = 'gray')
plt.title('$I_xI_y$'), plt.xticks([]), plt.yticks([])
plt.show()
```

## Harris response calculation

Then comes the main part. After this, they created a score, basically an equation, which determines if a window can contain a corner or not.

$$R = \text{det}(M) - k(\text{trace}(M))^2$$

where
* $\text{det}(M) = \lambda_1 \lambda_2$
* $\text{trace}(M) = \lambda_1 + \lambda_2$
* $\lambda_1$ and $\lambda_2$ are the eigenvalues of $M$

```
k = 0.05
# determinant
detA = I_x_I_x * I_y_I_y - I_x_I_y ** 2
# trace
traceA = I_x_I_x + I_y_I_y
R = detA - k * traceA ** 2
```

## Find corners using R > 0

```
y, x = np.where(R > 0)
plt.figure()
plt.imshow(gray_img, cmap='gray')
plt.scatter(x, y, c='r', s=40)
plt.show()
```

##Find edges using R < 0

```
y, x = np.where(R < 0)
plt.figure()
plt.imshow(gray_img, cmap='gray')
plt.scatter(x, y, c='r', s=10)
plt.show()
```

## Find flat regions using R = 0

```
y, x = np.where(R == 0)
plt.figure()
plt.imshow(gray_img, cmap='gray')
plt.scatter(x, y, c='r', s=1)
plt.show()
```

## Non-maximum suppression

```
from skimage.feature import corner_peaks
# non-maximal suppression
corners = corner_peaks(R)
x, y = corners[:, 1], corners[:, 0]
plt.figure()
plt.imshow(gray_img, cmap='gray')
plt.scatter(x, y, c='r', s=40)
plt.show()
```

## `HarrisCorner` function

```
from skimage.feature import corner_peaks
def HarrisCorner(img, ksize=5, sigmaX=2, sigmaY=2, k=0.05, non_max_suppr=True):
  # convert image to grayscale
  gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY).astype(np.float32)
  # light Gaussian smoothing
  gray_img = cv2.GaussianBlur(gray_img, (ksize, ksize), sigmaX=sigmaX, sigmaY=sigmaY)
  # construct the Sobel x-axis kernel
  sobelX = np.array(([-1, 0, 1], [-2, 0, 2], [-1, 0, 1]), dtype=np.float)
  # construct the Sobel y-axis kernel
  sobelY = np.array(([-1, -2, -1], [0, 0, 0], [1, 2, 1]), dtype=np.float)
  # convolution
  I_x = cv2.filter2D(gray_img, -1, sobelX)
  I_y = cv2.filter2D(gray_img, -1, sobelY)
  # gradient covariances and light Gaussian smoothing
  I_x_I_x = cv2.GaussianBlur(I_x*I_x, (ksize, ksize), sigmaX=sigmaX, sigmaY=sigmaY)
  I_y_I_y = cv2.GaussianBlur(I_y*I_y, (ksize, ksize), sigmaX=sigmaX, sigmaY=sigmaY)
  I_x_I_y = cv2.GaussianBlur(I_x*I_y, (ksize, ksize), sigmaX=sigmaX, sigmaY=sigmaY)
  # determinant
  detA = I_x_I_x * I_y_I_y - I_x_I_y ** 2
  # trace
  traceA = I_x_I_x + I_y_I_y
  # Harris criteria
  R = detA - k * traceA ** 2
  if non_max_suppr:
    corners = corner_peaks(R)
    x, y = corners[:, 1], corners[:, 0]
  else:
    y, x = np.where(R > 0)
  return (x, y)
```

## Download some other images

```
import os
if not os.path.exists('images.zip'):
  !wget --no-check-certificate https://empslocal.ex.ac.uk/people/staff/ad735/ECMM426/images.zip
  !unzip -q images.zip
```

## Apply on the other images

```
img = cv2.imread('images/books.jpg')
x, y = HarrisCorner(img, sigmaX=3, sigmaY=3, k=0.1)
x_, y_ = HarrisCorner(img, sigmaX=3, sigmaY=3, k=0.1, non_max_suppr=False)
# plot
plt.subplot(1, 3, 1), plt.imshow(img)
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 2), plt.imshow(img)
plt.scatter(x, y, c='r', s=40)
plt.title('Corners with nonmax suppression'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 3), plt.imshow(img)
plt.scatter(x_, y_, c='r', s=40)
plt.title('Corners without nonmax suppression'), plt.xticks([]), plt.yticks([])
plt.show()
```

```
img = cv2.imread('images/piet.png')
x, y = HarrisCorner(img, sigmaX=3, sigmaY=3, k=0.1)
x_, y_ = HarrisCorner(img, sigmaX=3, sigmaY=3, k=0.1, non_max_suppr=False)
# plot
plt.subplot(1, 3, 1), plt.imshow(img)
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 2), plt.imshow(img)
plt.scatter(x, y, c='r', s=40)
plt.title('Corners with nonmax suppression'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 3), plt.imshow(img)
plt.scatter(x_, y_, c='r', s=40)
plt.title('Corners without nonmax suppression'), plt.xticks([]), plt.yticks([])
plt.show()
```

## Qualitative comparison with OpenCV implementation
Note the difference between the plotting of our implementation and OpenCV implementation

```
img = cv2.imread('images/books.jpg')
img_cv = img.copy()
x, y = HarrisCorner(img, ksize=3, sigmaX=3, sigmaY=3, k=0.01)
dst = cv2.cornerHarris(np.float32(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)), 3, 3, 0.1)
# Results are marked through the dilated corners
dst = cv2.dilate(dst, None, iterations = 3)
# Reverting back to the original image, with optimal threshold value
img_cv[dst > 0.01 * dst.max()] = [255, 0, 0]
# plot
plt.subplot(1, 3, 1), plt.imshow(img)
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 2), plt.imshow(img)
plt.scatter(x, y, c='r', s=40)
plt.title('Corners (our implementation)'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 3), plt.imshow(img_cv)
plt.title('Corners (opencv implementation)'), plt.xticks([]), plt.yticks([])
plt.show()
```

```
img = cv2.imread('images/piet.png')
img_cv = img.copy()
x, y = HarrisCorner(img, ksize=3, sigmaX=3, sigmaY=3, k=0.1)
dst = cv2.cornerHarris(np.float32(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)), 2, 3, 0.1)
# Results are marked through the dilated corners
dst = cv2.dilate(dst, None, iterations = 10)
# Reverting back to the original image, with optimal threshold value
img_cv[dst > 0.01 * dst.max()] = [255, 0, 0]
# plot
plt.subplot(1, 3, 1), plt.imshow(img)
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 2), plt.imshow(img)
plt.scatter(x, y, c='r', s=40)
plt.title('Corners (our implementation)'), plt.xticks([]), plt.yticks([])
plt.subplot(1, 3, 3), plt.imshow(img_cv)
plt.title('Corners (opencv implementation)'), plt.xticks([]), plt.yticks([])
plt.show()
```


Rechneranwendungen in der Physik - Übung N.4

Santiago.R (in Zusammenarbeit mit Diego Rubio Carrera)

```python
import numpy as np
import sympy as sy
from scipy import optimize
import matplotlib.pyplot as plt
```

Zuerst werden beide Suchfunktionen mit den jeweils wählbaren Parametern der Funktion und Startwerte als Variablen definiert.

# Bisektionsmethode

Die Bisektionsmethode beruht auf das sukzessive Prüfen eines Intervalls $[x1,x2]$ auf Nullstellen mit der Annahme, dass für $f(x1)*f(x2)<0$ eine Nullstelle im Intervall vorhanden sein muss. Dies ist auch wahr für Funktionen wo eine Nullstelle mit einem Vorzeichenwechsel vorliegt, was auch in den meisten Fällen zutrifft, aber bspw. bei $f(x)=x^2$ liegt eine Nullstelle bei $x_0=0$ im Intervall $[-1,1]$ vor, obwohl $f(-1)*f(1)=+1>0$ gilt. Diese Methode funktioniert somit auschließlich für Nullstellen mit Vorzeichenwechsel d.h. für $f(x_0)=0 \Rightarrow f'(x_0) \neq 0$. Ist dies der Fall wird die Nullstellensuche auf immer kleinere Intervalle $[x_1,x_3]$ geprüft mit $x_3=\frac{x2-x1}{2}$ bis der Bereich unterhalb eines Toleranzgebietes eingeschränkt und die Nullstellensuche abgebrochen wird.

```python
def Bisektionsuchfunktion(f, x1, x2, tol=1e-8):
    """Annähernde Lösung für f(x_0)=0 mithilfe der Bisektionsmethode.

    Parameter
    ----------
    f : (function)
        Die Funktion für die die Nullstelle x0 mit f(x0)=0 gesucht werden soll.
    x1,x2 : (floats)
        Startwerte für den Intervall, in dem nach der Funktion gesucht werden soll. x1 muss kleiner als x2 sein
        und eine Nullstelle mit Vorzeichenwechsel muss im gegeben Intervall vorliegen, ansonstens gibt die Funktion
        einen Fehler als Output an.
    tol : (positive floats)
        Toleranz des eingeschränkten Gebietes.

    Returns
    -------
    Nullstelle und Iterationsnummer: (floats)
        Die Nullstelle, die durch das sukzessive einschränken des Nullstellenintervalls angenähert wird sowie die Iterationsnummer n.
        Sind x1 und x2 gleich bis auf eine Differenz unterhalb des Toleranzgebietes, dann wird der Mittelpunkt
        x3 zwischen beiden als Nullstelle angenohmen und als Output der Bisektionssuchfunktion herausgegeben.
    """

    n = 0 # Iterationsnummer

    if f(x1)*f(x2) < 0 and x1<x2:

        f_Links = f(x1)

        while True:

            x_3 = (x1 + x2)/2 #Bilden des Mittelpunktes
            f_Rechts = f(x_3) #Funktionswert des Mittelpunktes
            n += 1 #aktualisieren der Iterationsnummer
            if f_Rechts == 0 or (x2 - x1)/np.abs(x2 + x1) < tol: #Nullstellentest
                Nullstelle = x_3
                break

            elif f_Links*f_Rechts < 0:
                x2 = x_3

            else:
                x1 = x_3
                f_Links = f_Rechts
        Iterationsnummer = n
        return Nullstelle, Iterationsnummer

    else:
        print("Fehler (Die Bisektion kann keine Nullstelle im ggb. Intervall finden)")
```

Test der Suchfunktion an einer Funktion $f(x)=x^3+x^2-10$ mit reeller Nullstelle und Vorzeichenwechsel an $x_0$

```python
f = lambda x: x**3+x**2-10
x = np.linspace(-5,5)
plt.plot(x,f(x))
plt.plot(x,np.zeros_like(x))
plt.plot()
Bisektionsuchfunktion( f , x1=0, x2=4) # test der Funktion ''
```

# Sekantenverfahren

Der hier angewandte Sekantenverfahren beruht auf einer Vereinfachung des Newton-Raphson Verfahrens, bei dem die Nullstellensuche mithilfe der sukzessiven Berechnung der Nullstellen $x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}$ einer Sekante mit $f'(x)=\frac{f(x_1+1e-4)-f(x_1)}{1e-4}$ angenähert wird. Somit werden in der Sekantensuchfunktion die Werte $x_{n+1}=x_n-\frac{f(x_n)*1e-4}{f(x_1+1e-4)-f(x_1)}$ sukzessiv berechnet, bis der gefundene Wert $x_{n+1}$ mit $f(x_{n+1})$ entweder $0$ ergibt oder unterhalb des Toleranzbereichs $tol=1e-8$ liegt. Ein Vorteil dieser Methode ist das auch Nullstellen ohne Vorzeichenwechsel ermittelt werden können, da die Sekanten auf die nähesten Funktionswerte zur X-Achse konvergieren, unabhängig vom Vorzeichen. Man braucht auch nicht zu wissen, im welchen Bereich die Nullstelle sein muss, denn bei genügend Iterationen wird das Sekantenverfahren trotzdem konvergieren. Ein Nachteil ist aber, dass bei mehreren Nullstellen es unintuitiv sein kann zu wissen, welche Nullstelle mit welchem Startparameter nun von der Sekantensuchfunktion berechnet wird. Außerdem funktioniert dieses Verfahren nicht, wenn $f'(x_0) \rightarrow 0$ geht, da nicht durch 0 dividiert werden kann.

```python
def Sekantensuchfunktion(f, x_0, tol=1e-8):

  """Annähernde Lösung f(x)=0 mit einem Startparameter x_0 mithilfe des Sekantenverfahrens.

    Parameter
    ----------
    f : (function)
        Die Funktion f(x) in der die Nullstelle gesucht wird.
    x_0 : (floats)
        Anfangswert der Sekantenmethode.
    tol : (positive floats)
        Toleranz.

    Returns
    -------
    Nullstelle und Iterationsnummer : (floats)
        X-Wert x_n+1 für den die Funktion unterhalb des Toleranzbereichs liegt und Iterationsnummer n.
    """

  n = 0 # Number of iteration
  if (f(x_0+0.0001)-f(x_0)!=0):
        while n<100:

            x_Nullstelle = x_0-(f(x_0)*0.0001/(f(x_0+0.0001)-f(x_0)))
            x_0 = x_Nullstelle
            n+=1 #aktualisieren der Iterationsnummer
            Iterationsnummer=n
            if f(x_Nullstelle) == 0 or np.abs(f(x_Nullstelle)) <= tol:
                Nullstelle = x_Nullstelle
                break

  else:
        Iterationsnummer = 0
        Nullstelle = 0
        print(f"Fehler: Steigung der Sekante ist gleich 0")

  return Nullstelle, Iterationsnummer
```

Test der Suchfunktion an einer Funktion $f(x)=x^3+x^2-10$ mit reeller Nullstelle und Vorzeichenwechsel an $x_0$

```python
g = lambda x: x**3+x**2-10
x = np.linspace(-5,5)

plt.plot(x,g(x))
plt.plot(x,np.zeros_like(x))

Sekantensuchfunktion(g, 0)
```

# Test der Nullstellensuchfunktionen für $f(x)=(x+\frac{1}{2})^n -2$ mit n=1,2,3,4

```python
x = np.linspace(-2,2,num=20000)
mylist1=[]
mylist2=[]
f = lambda x, i: (x+0.5)**i-2
for i in range(1,5,1):
    plt.plot(x, f(x,i),label=i)
    g = lambda x: (x+0.5)**i-2
    BNullstellen= Bisektionsuchfunktion( g , x1=0, x2=4)
    SNullstellen= Sekantensuchfunktion( g , 0)
    mylist1.append(BNullstellen)
    mylist2.append(SNullstellen)
plt.plot(x,np.zeros_like(x))
plt.legend(title="Potenz n",loc='upper right')
plt.xlabel('X-Achse')
plt.ylabel('Y-Achse')
startx, endx = -2, 2
starty, endy = -3, 3
plt.axis([startx, endx, starty, endy])
plt.show()
print("Die von der Bisektionssuchfunktion ermittelten Nullstellen und Iterationsnummer sind in aufsteigender Reihenfolge für n=1,2,3,4:", mylist1)
print("Die von der Sekantensuchfunktion ermittelten Nullstellen und Iterationsnummer sind in aufsteigender Reihenfolge für n=1,2,3,4:", mylist2)
```

# Test der Sekantensuchfunktion für $f(x)=(x+\frac{1}{2})^n -2$ mit n=1,2,3,4 und Startwerten $x_0 \in [-2,2]$ mit $\Delta x_0= 0.5$

```python
mylistN=[]
for i in range(1,5,1):
    g = lambda x: (x+0.5)**i-2
    for j in range(0,9,1):
        N=Sekantensuchfunktion( g , j/2-2)
        mylistN.append(N)
n = 1
Iterationszahlen=[x[n] for x in mylistN]
x=np.linspace(-2,2,num=9)
plt.plot(x,Iterationszahlen[:9],label=1)
plt.plot(x,Iterationszahlen[9:18],label=2)
plt.plot(x,Iterationszahlen[18:27],label=3)
plt.plot(x,Iterationszahlen[27:36],label=4)
plt.legend(title="Potenz n",loc='upper right')
plt.xlabel('Startwert x_0')
plt.ylabel('Iterationszahl')
```

Da bei eins der Startwerte die Sekantensuchfunktion auf eine Stelle traf mit $f'(x_{n+1})=0$ wurde die Suchfunktion in dieser Iterationsfolge unterbrochen mit dem Output ''Fehler: Steigung der Sekante ist gleich 0''


# Semi-Supervised Learning with Spectral Graph Convolutions
> [**How to do Deep Learning on Graphs with Graph Convolutional Networks** Part 2: Semi-Supervised Learning with Spectral Graph Convolutions](https://towardsdatascience.com/how-to-do-deep-learning-on-graphs-with-graph-convolutional-networks-62acf5b143d0)

## Spectral Graph Convolutions

### The Sum Rule

- The $i$th row in the aggregate (feature representation of the $i$th node) is computed:
$$
\begin{equation}
\begin{aligned}
aggregate(A, X)_i&=A_iX \\
&=\sum_{j=1}^{N}A_{ij}X_{j}
\end{aligned}
\end{equation}
$$
- We can compute the aggregate feature representation of the $i$th node as a **vector-matrix product** (e.g.,simple weighted sum)
- Corresponds to summing up the feature representations of the neighbors of the $i$th node

### The Mean Rule

- The $i$th row in the aggregate is computed (only consider on the raw adjacency matrix $A$ for simplicity):
$$
\begin{equation}
\begin{aligned}
aggregate(A, X)_i&=D^{-1}A_iX \\
&=\sum_{k=1}^{N}D_{ik}^{-1}\sum_{j=1}^{N}A_{ij}X_{j} \\
&=\sum_{j=1}^{N}D_{ii}^{-1}A_{ij}X_{j} \\
&=\sum_{j=1}^{N}\frac{A_{ij}}{D_{ii}}X_{j} \\
\end{aligned}
\end{equation}
$$
- The inverse degree matrix $D^{-1}$ is a diagonal matrix where the values along the diagonal are inverse node degrees s.t. the value at position $(i, i)$ is the inverse degree of the $i$th node
- The weights in the weighted sum are now guaranteed to sum to 1 by with the degree of the $i$th node
- Corresponds to a mean over the feature representations of the neighbors of the $i$th node

*Whereas the sum rule depends solely on the neighborhood defined by the adjacency matrix $A$, the mean rule also depends on node degrees.*

### The Spectral Rule

$$
\begin{equation}
\begin{aligned}
aggregate(A, X)_i&=D^{-0.5}A_iD^{-0.5}X \\
&=\sum_{k=1}^{N}D_{ik}^{-0.5}\sum_{j=1}^{N}A_{ij}\sum_{l=1}^{N}D_{jl}^{-0.5}X_{j} \\
&=\sum_{j=1}^{N}D_{ii}^{-0.5}A_{ij}D_{jj}^{-0.5}X_{j} \\
&=\sum_{j=1}^{N}\frac{A_{ij}}{D_{ii}^{0.5}D_{jj}^{0.5}}X_{j} \\
\end{aligned}
\end{equation}
$$
- Raise the degree matrix to the power of -0.5 and multiply it on each side of $A$
- Not only take into consideration the degree of the $i$th node, but also the degree of the $j$th node
- The spectral rule weighs neighbor in the weighted sum higher if they have a low-degree and lower if they have a high-degree (This may be useful when low-degree neighbors provide more useful information than high-degree neighbors)

## Semi-Supervised Classification with GCNs
- We have implicitly assumed that the entire graph is available, i.e., that we are in a transductive setting. In other words, we know all the nodes, but not all the node labels.
- In all the rules, we aggregate over node neighborhoods, and thus nodes that share neighbors tend to have similar feature representations. This property is very useful if the graph exhibits homophily, i.e., that connected nodes tend to be similar (e.g. have the same label).
- We can take this a step further by training the GCN on the labeled nodes, effectively propagating the node label information to unlabelled nodes by updating weight matrices that are shared across all nodes. This can be done as follows:
    1. Perform forward propagation through the GCN.
    2. Apply the sigmoid function row-wise on the last layer in the GCN.
    3. Compute the cross entropy loss on known node labels.
    4. Backpropagate the loss and update the weight matrices W in each layer.

## Load Zachary's Karate Club Network

```python
from networkx import read_edgelist, set_node_attributes
from pandas import read_csv, Series
import numpy as np
from keras.utils import to_categorical

# from github.com/tkipf/keras-gcn
def encode_onehot(labels):
    classes = set(labels)
    classes_dict = {c: np.identity(len(classes))[i, :] for i, c in enumerate(classes)}
    labels_onehot = np.array(list(map(classes_dict.get, labels)), dtype=np.int32)
    return labels_onehot

def loadZacharyKarateClubNetwork():
    network = read_edgelist('karate.edgelist', nodetype=int)
    attributes = read_csv('karate.attributes.csv', index_col=['node'])
    labels = encode_onehot(np.array(attributes['community']))
    for attr in attributes.columns.values:
        set_node_attributes(network,
                            values=Series(attributes[attr], index=attributes.index).to_dict(),
                            name=attr)

    idx_train = [node for node, data in network.nodes(data=True) if data['role'] in {'Administrator', 'Instructor'}]
    idx_test = [node for node, data in network.nodes(data=True) if data['role'] == 'Member']
    y_train = np.zeros(labels.shape, dtype=np.int32)
    y_test = np.zeros(labels.shape, dtype=np.int32)
    y_train[idx_train] = labels[idx_train]
    y_test[idx_test] = labels[idx_test]
    mask = np.zeros(labels.shape[0])
    mask[idx_train] = 1
    train_mask =  np.array(mask, dtype=np.bool)
    return network, labels, y_train, y_test, idx_train, idx_test, train_mask
```

```python
from networkx import spring_layout, draw
import matplotlib.pyplot as plt

zkc_net, labels, y_train, y_test, idx_train, idx_test, sample_mask = loadZacharyKarateClubNetwork()
pos = spring_layout(zkc_net)
node_map = {
    'Administrator' : 0,
    'Instructor' : 0.2
}
colors = [node_map.get(zkc_net.nodes[node]['community']) for node in zkc_net.nodes]
draw(zkc_net, cmap = plt.get_cmap('Accent'), node_color=colors, with_labels=True, pos=pos)
```

```python
from networkx import to_numpy_matrix

def normalize_adj(A):
    I = np.eye(zkc_net.number_of_nodes())
    A_s = A + I
    D_hat = np.diag(np.array(np.power(np.sum(A_s, axis=0), -0.5))[0])
    return D_hat.dot(A_s).dot(D_hat)

order = sorted(list(zkc_net.nodes()))
A = to_numpy_matrix(zkc_net, nodelist=order)
A_norm = normalize_adj(A)
```

### Representation 1

Use the sparse 34 × 34 identity matrix $I$, as the feature matrix $X$, i.e., a one-hot encoding of each node in the graph
- it can be used in any graphs
- but results in an input parameter for each node in the network which requires a substantial amount of memory and computional power for training on large networks and may result in overfitting

```python
def evaluate_preds(preds, labels, indices):
    return np.mean(np.equal(np.argmax(labels[indices], 1), np.argmax(preds[indices], 1)))
```

```python
from keras.models import Model
from keras.layers import Input, Dropout, Dense
from keras.optimizers import Adam
from GCN import GraphConv

X_1 = np.eye(zkc_net.number_of_nodes())

featureInput = Input(shape=(X_1.shape[1],))
adjInput = Input(shape=(None, None), batch_shape=(None, None))
H1 = GraphConv(4, activation='tanh', name='graph_conv_1')([featureInput, adjInput])
H2 = GraphConv(2, activation='tanh', name='graph_conv_2')([H1, adjInput])
H3 = Dense(2, activation='sigmoid')(H2)

model1 = Model(inputs=[featureInput, adjInput], outputs=H3)
model1.compile(optimizer=Adam(lr=0.01), loss='binary_crossentropy')
model1.summary()

history = model1.fit([X_1, A_norm], y_train, sample_weight=sample_mask,
          batch_size=X_1.shape[0], epochs=5000, shuffle=False, verbose=0)

preds = model1.predict([X_1, A_norm], batch_size=X_1.shape[0])
print("train_acc= {:.4f}".format(evaluate_preds(preds, labels, idx_train)),
      "test_acc= {:.4f}".format(evaluate_preds(preds, labels, idx_test)))

```

    Model: "model_35"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_71 (InputLayer)           (None, 34)           0
    __________________________________________________________________________________________________
    input_72 (InputLayer)           (None, None)         0
    __________________________________________________________________________________________________
    graph_conv_1 (GraphConv)        (None, 4)            140         input_71[0][0]
                                                                     input_72[0][0]
    __________________________________________________________________________________________________
    graph_conv_2 (GraphConv)        (None, 2)            10          graph_conv_1[0][0]
                                                                     input_72[0][0]
    __________________________________________________________________________________________________
    dense_3 (Dense)                 (None, 2)            6           graph_conv_2[0][0]
    ==================================================================================================
    Total params: 156
    Trainable params: 156
    Non-trainable params: 0
    __________________________________________________________________________________________________
    train_acc= 1.0000 test_acc= 0.9375

### Representation 2

Improve representation 1 by adding two features that are not specific to any node or area of the network, but measures the connectedness to the administrator and instructor.
Compute the shortest path distance from each node in the network to both the administrator and instructor and concatenate these two features to the previous representation
- the graph convolutional layers always have a local perspective and has a limited capacity to capture global information about the location of each node in the graph

```python
from networkx import shortest_path_length

X_2 = np.zeros((zkc_net.number_of_nodes(), 2))
node_distance_instructor = shortest_path_length(zkc_net, target=33)
node_distance_administrator = shortest_path_length(zkc_net, target=0)

for node in zkc_net.nodes():
    X_2[node][0] = node_distance_administrator[node]
    X_2[node][1] = node_distance_instructor[node]

X_2 = np.concatenate((X_1, X_2), axis=1)

featureInput = Input(shape=(X_2.shape[1],))
adjInput = Input(shape=(None, None), batch_shape=(None, None))
H1 = GraphConv(4, activation='tanh')([featureInput, adjInput])
H2 = GraphConv(2, activation='sigmoid')([H1, adjInput])

model2 = Model(inputs=[featureInput, adjInput], outputs=H2)
model2.compile(optimizer=Adam(lr=0.01), loss='binary_crossentropy', metrics=['accuracy'])

history = model2.fit([X_2, A_norm], y_train, sample_weight=sample_mask,
          batch_size=X_2.shape[0], epochs=250, shuffle=False, verbose=0)

preds = model2.predict([X_2, A_norm], batch_size=X_2.shape[0])
print("train_acc= {:.4f}".format(evaluate_preds(preds, labels, idx_train)),
      "test_acc= {:.4f}".format(evaluate_preds(preds, labels, idx_test)))
```

    train_acc= 1.0000 test_acc= 0.9688

```python

```


```python
import sympy as s
```

```python
phi,x,y = s.symbols('phi,x,y')
phi_d,x_d,y_d = s.symbols('phi_d,x_d,y_d')
x_i,y_i,beta_i,gamma_i,r_i = s.symbols('x_i,y_i,beta_i,gamma_i,r_i')
q_d = s.symbols('q_d')
```

```python
q_d = s.Matrix([[phi_d],[x_d],[y_d]])
q_d
```

$\displaystyle \left[\begin{matrix}\phi_{d}\\x_{d}\\y_{d}\end{matrix}\right]$

```python
first_trans = s.Matrix([[1,0,0],
                        [0,s.cos(phi),s.sin(phi)],
                        [0,-s.sin(phi),s.cos(phi)]])
first_trans
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & \cos{\left(\phi \right)} & \sin{\left(\phi \right)}\\0 & - \sin{\left(\phi \right)} & \cos{\left(\phi \right)}\end{matrix}\right]$

```python
second_trans = s.Matrix([[-y_i,1,0],
                         [x_i,0,1]])
second_trans
```

$\displaystyle \left[\begin{matrix}- y_{i} & 1 & 0\\x_{i} & 0 & 1\end{matrix}\right]$

```python
third_trans = s.Matrix([[s.cos(beta_i),s.sin(beta_i)],
                        [-s.sin(beta_i),s.cos(beta_i)]])
third_trans
```

$\displaystyle \left[\begin{matrix}\cos{\left(\beta_{i} \right)} & \sin{\left(\beta_{i} \right)}\\- \sin{\left(\beta_{i} \right)} & \cos{\left(\beta_{i} \right)}\end{matrix}\right]$

```python
fourth_trans = s.Matrix([[1/r_i,s.tan(gamma_i)/r_i]])
fourth_trans
```

$\displaystyle \left[\begin{matrix}\frac{1}{r_{i}} & \frac{\tan{\left(\gamma_{i} \right)}}{r_{i}}\end{matrix}\right]$

```python
trans = fourth_trans@third_trans@second_trans@first_trans
trans
```

$\displaystyle \left[\begin{matrix}x_{i} \left(\frac{\sin{\left(\beta_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}}\right) - y_{i} \left(- \frac{\sin{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)}}{r_{i}}\right) & - \left(\frac{\sin{\left(\beta_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}}\right) \sin{\left(\phi \right)} + \left(- \frac{\sin{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)}}{r_{i}}\right) \cos{\left(\phi \right)} & \left(\frac{\sin{\left(\beta_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}}\right) \cos{\left(\phi \right)} + \left(- \frac{\sin{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}} + \frac{\cos{\left(\beta_{i} \right)}}{r_{i}}\right) \sin{\left(\phi \right)}\end{matrix}\right]$

```python
s.simplify(trans)
```

$\displaystyle \left[\begin{matrix}\frac{x_{i} \left(\sin{\left(\beta_{i} \right)} + \cos{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)}\right) + y_{i} \left(\sin{\left(\beta_{i} \right)} \tan{\left(\gamma_{i} \right)} - \cos{\left(\beta_{i} \right)}\right)}{r_{i}} & \frac{- \sin{\left(\beta_{i} + \phi \right)} \tan{\left(\gamma_{i} \right)} + \cos{\left(\beta_{i} + \phi \right)}}{r_{i}} & \frac{\sin{\left(\beta_{i} + \phi \right)} + \cos{\left(\beta_{i} + \phi \right)} \tan{\left(\gamma_{i} \right)}}{r_{i}}\end{matrix}\right]$

```python
s.simplify(trans*(r_i*s.cos(gamma_i))).T
```

$\displaystyle \left[\begin{matrix}x_{i} \sin{\left(\beta_{i} + \gamma_{i} \right)} - y_{i} \cos{\left(\beta_{i} + \gamma_{i} \right)}\\\cos{\left(\beta_{i} + \gamma_{i} + \phi \right)}\\\sin{\left(\beta_{i} + \gamma_{i} + \phi \right)}\end{matrix}\right]$

```python

```


```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from sympy import *
from math import log, ceil
from random import randint
import functools
```

---

```python
class GeneticAlgo:

    def __init__(self, a,b, functionOptions,
                 populationOpts, operatorOpts, tournamentOpts,
                 freezeLimit = 6, precision = 10**(-6)):

        self.a, self.b = a, b
        self.func, self.funcVariables, self.funcFitness = functionOptions
        self.popNumber, self.popSize = populationOpts
        self.mutationChance, self.crossChance = operatorOpts
        self.tournamentMembersNumber = tournamentOpts
        self.freezeLimit = freezeLimit
        self.precision = precision

        self.length = self.__initLength()
        self.population = self.__initPopulation()
        self.bits = self.__initMaxBits()
        self.realPrecision = self.__initRealPrecision()

        self.fittest = []

        self.infoData = {
            "OffspringNUmber": 0,
            "CurrentFittest": []
        }

# Initializaton functions
#----------------------------------------------------------------------

    def __initLength(self): return self.b - self.a

    def __initPopulation(self): return np.random.randint(self.a, self.b + 1, self.popSize)

    def __initMaxBits(self): return ceil(log(self.length/self.precision, 2))

    def __initRealPrecision(self):

        maxBinaryinBaseTen = 2**self.bits / self.length

        return 1 / maxBinaryinBaseTen

# Main functions
#----------------------------------------------------------------------

    def solve(self, displayData = False):

        offspringNumber = 0
        freezeNumber = 0

        while (freezeNumber < self.freezeLimit):

            self.population = self.__offspring()

            if (not self.__newFitElemExist()):
                freezeNumber+=1

            offspringNumber+=1

            if displayData:
                self.__displayData(freezeNumber, offspringNumber)

        self.infoData["OffspringNUmber"] = offspringNumber

        return self.fittest

    def __offspring(self):

        children = []

        while len(children) < self.popSize:

            selected = self.__selectMultiple(2)
            mutated = self.__execute(self.__mutate, self.mutationChance, selected)
            crossed = self.__execute(self.__cross, self.crossChance, mutated)

            children.extend(self.__listToNumber(crossed))

        return np.array(children)

    def __select(self):

        indexes = np.random.choice(self.popSize, self.tournamentMembersNumber)

        competitors = self.population[indexes]

        winner = self.__tournament(competitors)

        return self.__toBinary(winner)

    def __mutate(self, elems):

        indexes = np.random.randint(1, self.bits, 2)

        for idx in range(len(elems)):
            index = indexes[idx]

            elems[idx][index] = self.__bitFlip(elems[idx][index])

        return elems

    def __cross(self, elems):

        index = randint(1, self.bits)

        return self.__merge(elems, index)

    def __newFitElemExist(self):

        sortedPopulation = sorted(self.population, key = lambda elem: self.func.subs({x:elem}))

        self.infoData["CurrentFittest"].append(sortedPopulation[0])

        if not self.fittest:
            self.fittest.append(sortedPopulation[0])
            return True

        for elem in sortedPopulation:
            for currFittest in self.fittest:

                if self.__isFittest(elem, currFittest):
                    self.__resetFittestList()
                    self.fittest.append(elem)
                    return True

        return False

    def __isEqualFit(self, a, b):
        print(f'{a}, {self.func.subs({x:a})}')
        print(f'{b}, {self.func.subs({x:b})}')

        return (
            abs(self.func.subs({x:a})) - abs(self.func.subs({x:b})) <= self.realPrecision * 2 and \
            abs(b - a) > self.precision * 10
        )

    def __isFittest(self, a, b): return True if self.funcFitness(self.func.subs({x:a}), self.func.subs({x:b})) else False

    def __getFittest(self, a, b): return a if self.__isFittest(a, b) else b

#----------------------------------------------------------------------

    def __tournament(self, competitors):

        winner = competitors[0]

        for idx in range(1, self.tournamentMembersNumber):
            winner = self.__getFittest(winner, competitors[idx])

        return winner

    def __merge(self, elems, index):
        elem1, elem2 = elems

        if index == 0: index+=1

        for idx in range(index, self.bits):
            elem1[idx], elem2[idx] = elem2[idx], \
                                     elem1[idx]
        return elem1, elem2

    def __evaluate(self): return [ (elem, self.func.subs({x:elem})) for elem in self.population ]

    def __resetFittestList(self): self.fittest = []

    def __displayData(self, freezeNumber, offspringNumber):
        print("\n -------- ")
        #print(f'population - {self.population}')
        print(f'fittest - {self.fittest}')
        print(f'freeze number - {freezeNumber}')
        print(f'offspring number - {offspringNumber}')
        #print(self.__evaluate())

# Helper functions
#----------------------------------------------------------------------

    def __selectMultiple(self, n):
        return [self.__select() for i in range(n)]

    def __mutateMultiple(self, elems):
        return [self.__mutate(elem) for elem in elems]

    def __execute(self, func, probability, value):
        return func(value) if (randint(1, 100) / 100) < probability else value

# Utility functions
#----------------------------------------------------------------------

    def __toBinary(self, number):

        baseTen = (number - self.a) * 2**self.bits / self.length

        return list("{0:b}".format(int(baseTen)).zfill(self.bits))

    def __toNumber(self, bitList):

        baseTwo = sum([ int(bitList[i]) * (2**(self.bits - 1 - i)) for i in range(self.bits)])

        return self.a + baseTwo * (self.length / 2**self.bits)

    def __listToNumber(self, list):
        return [ self.__toNumber(elem) for elem in list ]

    def __bitFlip(self, bit):
        return '1' if bit == '0' else '0'
```

---

### Initialization

```python
variables = x = Symbol("x")
```

```python
def fitnessFunc(a,b): return abs(a) < abs(b)
```

```python
a, b = -5, 5
func = x**2 + 5*x + 6 #x*sin(10*pi*x) + 1

functionOpts = func, variables, fitnessFunc
populationOpts = populationNumber, populationSize = 5, 100
operatorOpts = mutation, crossover = 0.2, 0.8
tournamentMembersOpts = 4
freezeLimit = 6
precision = 10**(-6)
```

### Calculation

```python
ga = GeneticAlgo(a,b, functionOpts, populationOpts, operatorOpts, tournamentMembersOpts, freezeLimit, precision)

ga.solve(True)
```

     --------
    fittest - [-2.1875]
    freeze number - 0
    offspring number - 1

     --------
    fittest - [-2.8749924898147583]
    freeze number - 0
    offspring number - 2

     --------
    fittest - [-2.0234376192092896]
    freeze number - 0
    offspring number - 3

     --------
    fittest - [-2.9902344942092896]
    freeze number - 0
    offspring number - 4

     --------
    fittest - [-2.9993897676467896]
    freeze number - 0
    offspring number - 5

     --------
    fittest - [-3.0000001192092896]
    freeze number - 0
    offspring number - 6

     --------
    fittest - [-3.0000001192092896]
    freeze number - 1
    offspring number - 7

     --------
    fittest - [-3.0000001192092896]
    freeze number - 2
    offspring number - 8

     --------
    fittest - [-3.0000001192092896]
    freeze number - 3
    offspring number - 9

     --------
    fittest - [-3.0000001192092896]
    freeze number - 4
    offspring number - 10

     --------
    fittest - [-3.0000001192092896]
    freeze number - 5
    offspring number - 11

     --------
    fittest - [-3.0000001192092896]
    freeze number - 6
    offspring number - 12

    [-3.0000001192092896]

fittest = []
population = []

ga = GeneticAlgo(a,b, func,variables, fitnessFunc, (5, 100), (0.2, 0.8), 2, freezeLimit, precision)
fitVal = ga.solve()

rangee = np.arange(0, ga.infoData["OffspringNUmber"])

fittest.extend(ga.infoData["CurrentFittest"])
solutionInOffspring = rangee

print(fitVal)
plt.plot(solutionInOffspring, fittest)
plt.xlabel('Offspring')
plt.ylabel('Current fittest')solutionInOffspring = []
population = []

for i in range(10, 1010, 100):
    ga = GeneticAlgo(a,b, func,variables, fitnessFunc, (5, i), (0.2, 0.8), 2, freezeLimit, precision)
    fitVal = ga.solve()

    population.append(i)
    solutionInOffspring.append(ga.infoData["OffspringNUmber"])

print(fitVal)
plt.plot(population, solutionInOffspring)
plt.xlabel('Population Size')
plt.ylabel('Offspring')solutionInOffspring = []
mutation = []

for i in np.arange(0.00, 1, 0.05):
    ga = GeneticAlgo(a,b, func,variables, fitnessFunc, (5, 100), (i, 0.8), 2, freezeLimit, precision)
    fitVal = ga.solve()

    mutation.append(i)
    solutionInOffspring.append(ga.infoData["OffspringNUmber"])

print(fitVal)
plt.plot(mutation, solutionInOffspring)
plt.xlabel('Mutation')
plt.ylabel('Offspring')solutionInOffspring = []
tournamentMembers = []

for i in np.arange(2, 20, 1):
    ga = GeneticAlgo(a,b, func,variables, fitnessFunc, (5, 500), (0.2, 0.8), i, freezeLimit, precision)
    fitVal = ga.solve()

    tournamentMembers.append(i)
    solutionInOffspring.append(ga.infoData["OffspringNUmber"])

print(fitVal)
plt.plot(tournamentMembers, solutionInOffspring)
plt.xlabel('Tournament members')
plt.ylabel('Offspring')

```python

```


# 2.1 Sensitivity Analysis

## Introduction

**This section corresponds to Section 4.1 of the original paper.**

The task of explaining a DNN's decision can be rephrased into the following question: given a DNN $f$, input image $x$ and the output of DNN $f(x)$, what is it of $x$ that made the DNN reach the particular decision $f(x)$?

To tackle this problem, we can define a relevance function $R$ that takes in an ith pixel $x_i$ of image $x$ as an argument and returns a scalar value $R(x_i)$ which indicates how much positive or negative contribution $x_i$ has had on the final decision $f(x)$. If the range of $R$ is limited to nonnegative real numbers, 0 indicates no contribution and any other positive scalar indicates positive contribution. Intuitively, the extent of positive/negative contribution must be proportional to the absolute value of $R(x_i)$.

In this section, we ask for a given an image x, which pixel(s) or group(s) of pixels makes it representative of a certain concept encoded in the output neuron of the deep neural network (DNN).

A first approach to identify the most important input features is sensitivity analysis. A common formulation of sensitivity analysis defines relevance scores as

\begin{equation}
R(x_i) = \left( \frac{\partial f}{\partial x_i} \right)^2
\end{equation}

where the gradient is evaluated at the data point $x_i$. Put simply, the relevance score indicates how influential a particular data point was on the DNN's decision. Therefore, the most relevant input features are those to which the output is most sensitive i.e., has high relevance scores. The relevance score can be calculated simply using backpropagation.

However, it is important to note that sensitivity analysis does not provide an explicit explanation of the function value $f(x)$, but rather suggestions. For instance, when applying sensitivity analysis to a DNN detecting cars, we answer the question *"what makes this image more/less a car"* rather than the more basic question *"what makes this image a car?"*. This may not make much sense to you. Let me offer an explanation with a simplified version of sensitivity analysis.

Suppose we have a binary classification function that separates 2-dimensional points into two classes:

\begin{equation}
f(x,y) =
\begin{cases}
x^2 + y^2 - 9 < 0 & \text{for class N(egative)} \\
x^2 + y^2 - 9 \geq 0 &  \text{for class P(ositive)}
\end{cases}
\end{equation}

For instance, $f(-1,1) = -7 < 0$ and therefore the point $(-1,1)$ is in class $N$. On the other hand, $f(-4,-4) = 7 > 0$ and therefore the point $(-4,-4)$ is in class $P$.

Let us now turn our attention to the gradient of $f$ at points $(-1,1)$ and $(-4,-4)$. The gradient of $f$ is:

\begin{equation}
\nabla f(x,y) =
\begin{bmatrix}
\partial f / \partial x & \partial f / \partial y
\end{bmatrix} =
\begin{bmatrix}
2x & 2y
\end{bmatrix}
\end{equation}

The gradient at $(-1,1)$ is given by $\begin{bmatrix}-2 & 2\end{bmatrix}$ and the sensitivity is $\begin{bmatrix}4 & 4\end{bmatrix}$. Note that the sensitivity does not tell *why* $(-1,1)$ is in class $N$. It merely suggests that increasing or decreasing $x$ and $y$ may produce a point that belongs to class $N$ more strongly.

Let's take a look at another example: $(-4,-4)$. The gradient at the point is given by $\begin{bmatrix}-8 & -8\end{bmatrix}$ and therefore the sensitivity is $\begin{bmatrix}64 & 64\end{bmatrix}$. Also in this case, the sensitivity only suggests that increasing or decreasing $x$ and $y$ may produce a point that DNN will classify into class $P$ with higher confidence (see the figure below for a visualization of the points with their normalized gradients indicated by arrows).

Likewise, the sensitivity scores for images do not tell us *why* the DNN made such a decision. It only tells us what changes in the image (e.g. emphasis or de-emphasis of a particular feature) may cause the DNN to classify the image with more certainty into the given class. Looking back, note how apt the name *"sensitivity analysis"* is. Sensitivity analysis tells us how *sensitive* the function value $f(x)$ is to changes in pixel values. The magnitude of sensitivity is quantified by the relevance score $R(x_i)$ given by the square of gradient of $f(x)$ at the pixel $x_i$.

## Tensorflow Walkthrough

### 1. Import Dependencies

```python
import os

from tensorflow.examples.tutorials.mnist import input_data
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np

from models.models_2_1 import MNIST_DNN, MNIST_CNN

%matplotlib inline

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

logdir = './tf_logs/2_1_SA/'
ckptdir = logdir + 'model'

if not os.path.exists(logdir):
    os.mkdir(logdir)
```

    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

### 2. Building Graph

In this step, a CNN classifier is initialized and necessary nodes for model training is added onto the graph. For sensitivity analysis, we only need three placeholders. `training` to indicate that we do not use dropout when during inference, `X` in order to pass test images into the computation graph and `logits` which is necessary to compute the gradient (we need to compute the gradient of `X` with respect to `logits`). Instead of using a fully connected neural network, I used a convolutional neural network because it provided better sensitivity analysis visualization.

```python
with tf.name_scope('Classifier'):

    # Initialize neural network
    CNN = MNIST_CNN('CNN')

    # Setup training process
    training = tf.placeholder_with_default(True, shape=[], name='training')
    X = tf.placeholder(tf.float32, [None, 784], name='X')
    Y = tf.placeholder(tf.float32, [None, 10], name='Y')

    logits = CNN(X, training)

    tf.add_to_collection('sensitivity_analysis', training)
    tf.add_to_collection('sensitivity_analysis', X)
    tf.add_to_collection('sensitivity_analysis', logits)

    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y))

    optimizer = tf.train.AdamOptimizer().minimize(cost, var_list=CNN.vars)

    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

cost_summary = tf.summary.scalar('Cost', cost)
accuray_summary = tf.summary.scalar('Accuracy', accuracy)
summary = tf.summary.merge_all()
```

### 3. Training Network

This is the step where the CNN is trained to classify the 10 digits of the MNIST images. Summaries are written into the logdir and you can visualize the statistics using tensorboard by typing this command: `tensorboard --lodir=./tf_logs`

```python
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

saver = tf.train.Saver()
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

# Hyper parameters
training_epochs = 15
batch_size = 100

for epoch in range(training_epochs):
    total_batch = int(mnist.train.num_examples / batch_size)
    avg_cost = 0
    avg_acc = 0

    for i in range(total_batch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        _, c, a, summary_str = sess.run([optimizer, cost, accuracy, summary], feed_dict={X: batch_xs, Y: batch_ys})
        avg_cost += c / total_batch
        avg_acc += a / total_batch

        file_writer.add_summary(summary_str, epoch * total_batch + i)

    print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.9f}'.format(avg_cost), 'accuracy =', '{:.9f}'.format(avg_acc))

    saver.save(sess, ckptdir)

print('Accuracy:', sess.run(accuracy, feed_dict={X: mnist.test.images, Y: mnist.test.labels, training: False}))

sess.close()
```

    Epoch: 0001 cost = 0.755203515 accuracy = 0.744127272
    Epoch: 0002 cost = 0.287379274 accuracy = 0.909418183
    Epoch: 0003 cost = 0.230599659 accuracy = 0.927818183
    Epoch: 0004 cost = 0.198606952 accuracy = 0.937472728
    Epoch: 0005 cost = 0.183995720 accuracy = 0.942200003
    Epoch: 0006 cost = 0.170081426 accuracy = 0.946763638
    Epoch: 0007 cost = 0.165785470 accuracy = 0.948218184
    Epoch: 0008 cost = 0.155800372 accuracy = 0.951472731
    Epoch: 0009 cost = 0.151487274 accuracy = 0.952563640
    Epoch: 0010 cost = 0.145716070 accuracy = 0.954636367
    Epoch: 0011 cost = 0.142153567 accuracy = 0.955272730
    Epoch: 0012 cost = 0.138724105 accuracy = 0.957436367
    Epoch: 0013 cost = 0.140947498 accuracy = 0.957090912
    Epoch: 0014 cost = 0.137629085 accuracy = 0.957836368
    Epoch: 0015 cost = 0.136273555 accuracy = 0.958454548
    Accuracy: 0.9892

### 4. Restoring Subgraph

Here we first rebuild the CNN graph from metagraph, restore CNN parameters from the checkpoint and then gather the necessary nodes for Sensitivity Analysis using the `tf.get_collection()` function. We can calculate the relevance scores using the `tf.square()` and the `tf.gradients()` functions.

```python
tf.reset_default_graph()

sess = tf.InteractiveSession()

new_saver = tf.train.import_meta_graph(ckptdir + '.meta')
new_saver.restore(sess, tf.train.latest_checkpoint(logdir))

SA = tf.get_collection('sensitivity_analysis')
training = SA[0]
X = SA[1]
logits = SA[2]

SA_scores = [tf.square(tf.gradients(logits[:,i], X)) for i in range(10)]
```

    INFO:tensorflow:Restoring parameters from ./tf_logs/2_1_SA/model

### 5. Calculating Relevance Scores $R_i(x)$

```python
images = mnist.train.images
labels = mnist.train.labels

sample_imgs = []
for i in range(10):
    sample_imgs.append(images[np.argmax(labels, axis=1) == i][3])

hmaps = np.reshape([sess.run(SA_scores[i], feed_dict={X: sample_imgs[i][None,:], training:False}) for i in range(10)], [10, 784])

sess.close()
```

### 6. Displaying Images

The relevance scores are visualized as heat maps over the original digit images. You can see which features/data points influenced the CNN most its decision making.

```python
plt.figure(figsize=(15,15))
for i in range(5):
    plt.subplot(5, 2, 2 * i + 1)
    plt.imshow(np.reshape(sample_imgs[2 * i], [28, 28]), cmap='gray')
    plt.imshow(np.reshape(hmaps[2 * i], [28, 28]), cmap='hot', alpha=0.5)
    plt.title('Digit: {}'.format(2 * i))
    plt.colorbar()

    plt.subplot(5, 2, 2 * i + 2)
    plt.imshow(np.reshape(sample_imgs[2 * i + 1], [28, 28]), cmap='gray')
    plt.imshow(np.reshape(hmaps[2 * i + 1], [28, 28]), cmap='hot', alpha=0.5)
    plt.title('Digit: {}'.format(2 * i + 1))
    plt.colorbar()

plt.tight_layout()
```

```python

```


# Proof template

This is a template for a generic proof of one ore more of the equations in the paper. Insert some description here about what we're trying to prove. We should try to use `sympy` as much as possible to actually prove things formally, but settle for a numerical verification if that's too difficult.

We begin with some generic imports:

```python
import numpy as np
import matplotlib.pyplot as pl
%matplotlib notebook
from IPython.display import display, Math
import sympy
from sympy import *
from sympy.functions.special.tensor_functions import KroneckerDelta

# Initialize the session
init_session(quiet=True)

# Let's report what version of sympy this is
print("Using sympy version", sympy.__version__)
```

    IPython console for SymPy 0.7.6.1 (Python 3.5.2-64-bit) (ground types: python)
    Using sympy version 0.7.6.1

As an example, let's prove some generic matrix identities.

```python
# Declare a few generic matrices
n, m = symbols('n m')
A = MatrixSymbol('A', n, n) # Square
B = MatrixSymbol('B', n, m) # Non-square
C = MatrixSymbol('C', m, m) # Square
```

Show that matrix multiplication is associative:

```python
(A * B) * C == A * (B * C)
```

    True

Show that the inverse of the inverse of a matrix is the matrix itself:

```python
(A.I).I
```

Show that a matrix times the identity is itself:

```python
A * Identity(n)
```

Show that matrix multiplication is not commutative

```python
try:
    A * B == B * A
except ShapeError:
    print("Matrix multiplication is not commutative!")
```

    Matrix multiplication is not commutative!

That's it for now!